mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
190 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
dependabot[bot]
|
c6dd823384
|
deps: bump md-5 from 0.10.6 to 0.11.0 (#3146)
Bumps [md-5](https://github.com/RustCrypto/hashes) from 0.10.6 to 0.11.0. <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
a307c11109
|
deps: bump tiktoken-rs from 0.11.0 to 0.12.0 (#3147)
Bumps [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs) from 0.11.0 to 0.12.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/zurawiki/tiktoken-rs/releases">tiktoken-rs's releases</a>.</em></p> <blockquote> <h2>v0.12.0</h2> <h2>Summary</h2> <p>This release backports OpenAI <code>tiktoken</code> 0.13.0 into <code>tiktoken-rs</code>. The main reason to upgrade is better alignment with upstream tokenization behavior, especially the upstream Rust core changes for large BPE pieces and error-aware encoding.</p> <p>For most users who call the high-level model/token counting helpers, this should behave the same aside from the new Rust compiler requirement. Users who call lower-level <code>CoreBPE</code> encoding methods directly should review the breaking changes below.</p> <h2>What Changed</h2> <ul> <li>Backported the vendored OpenAI <code>tiktoken</code> Rust core from 0.9.0 to 0.13.0.</li> <li>Added the upstream large-piece BPE merge path. Functionally, this improves behavior for very large or repetitive inputs that previously stressed the merge algorithm.</li> <li>Changed <code>CoreBPE::encode</code> to return <code>Result<(Vec<Rank>, usize), EncodeError></code>, matching upstream. Regex/tokenization failures can now be reported instead of being hidden behind infallible APIs.</li> <li>Updated <code>encode_as</code> and <code>count</code> to return <code>Result</code> because they call <code>encode</code>.</li> <li>Re-exported <code>EncodeError</code> so callers can handle encode failures directly.</li> <li>Aligned the vendored core with Rust 2024 and raised the crate MSRV to Rust 1.85.</li> <li>Synced model-to-tokenizer mappings with upstream <code>tiktoken</code> 0.13.0 while keeping local extra prefixes isolated.</li> <li>Hardened asset downloads with SHA-256 checks and a repo-root-aware asset path.</li> </ul> <h2>Breaking Changes</h2> <p>If your code calls <code>CoreBPE::encode</code>, unwrap or propagate the result before using the tokens:</p> <pre lang="rust"><code>let allowed = bpe.special_tokens(); let (tokens, last_piece_token_len) = bpe.encode("hello <|endoftext|>", &allowed)?; </code></pre> <p>The generic helpers changed similarly:</p> <pre lang="rust"><code>let (tokens, last_piece_token_len) = bpe.encode_as::<usize>(text, &allowed)?; let token_count = bpe.count(text, &allowed)?; </code></pre> <p><code>encode_ordinary</code>, <code>encode_ordinary_as</code>, <code>encode_with_special_tokens</code>, and <code>count_ordinary</code> remain infallible.</p> <p>Projects must now build with Rust 1.85 or newer.</p> <h2>Practical Impact</h2> <ul> <li>Applications processing long repeated text should see more robust tokenization behavior.</li> <li>Code that only uses helpers like <code>get_chat_completion_max_tokens</code>, <code>get_text_completion_max_tokens</code>, <code>bpe_for_model</code>, or singleton tokenizer constructors should not need call-site changes.</li> <li>Code using low-level <code>CoreBPE::encode</code>, <code>encode_as</code>, or <code>count</code> needs a small migration to handle <code>Result</code>.</li> </ul> <h2>Links</h2> <ul> <li>PR: <a href="https://redirect.github.com/zurawiki/tiktoken-rs/pull/164">zurawiki/tiktoken-rs#164</a></li> <li>Upstream <code>tiktoken</code> 0.13.0: <a href="https://github.com/openai/tiktoken/releases/tag/0.13.0">https://github.com/openai/tiktoken/releases/tag/0.13.0</a></li> <li>Full changelog: <a href="https://github.com/zurawiki/tiktoken-rs/compare/v0.11.0...v0.12.0">https://github.com/zurawiki/tiktoken-rs/compare/v0.11.0...v0.12.0</a></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
6e2e10f67a
|
deps: bump tokenizers from 0.22.2 to 0.23.1 (#3149)
Bumps [tokenizers](https://github.com/huggingface/tokenizers) from 0.22.2 to 0.23.1. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/huggingface/tokenizers/releases">tokenizers's releases</a>.</em></p> <blockquote> <h2>Release v0.23.1</h2> <h2>TL;DR</h2> <p><code>tokenizers 0.23.1</code> is the first proper stable release in the <code>0.23</code> line — <code>0.23.0</code> only ever shipped as <code>rc0</code> because the release pipeline itself was broken (Node side hadn't shipped multi-platform binaries since 2023, Python side was on <code>pyo3 0.27</code> without free-threaded support). <code>0.23.1</code> is the version where everything actually goes out the door together: full Node multi-platform wheels for the first time in years, Python 3.14 (regular <strong>and</strong> free-threaded <code>3.14t</code>), full type hints for every Python class, and a stack of measurable perf wins on the BPE / added-vocab hot paths.</p> <p>There is no functional <code>0.23.0</code> published — we tag <code>0.23.1</code> directly so users don't accidentally pull a never-shipped version.</p> <hr /> <h2>🚨 Breaking changes</h2> <ul> <li><strong>Drop Python 3.9</strong> (<a href="https://redirect.github.com/huggingface/tokenizers/issues/1952">#1952</a>) — <code>requires-python = ">=3.10"</code>; 3.9 users stay on <code>0.22.x</code>.</li> <li><strong><code>add_tokens</code> normalizes <code>content</code> at insertion</strong> (<a href="https://redirect.github.com/huggingface/tokenizers/issues/1995">#1995</a>) — re-saved <code>tokenizer.json</code> may differ in the <code>added_tokens</code> block. Existing files load unchanged.</li> <li><strong>Type stubs are precise</strong> (<a href="https://redirect.github.com/huggingface/tokenizers/issues/1928">#1928</a>, <a href="https://redirect.github.com/huggingface/tokenizers/issues/1997">#1997</a>) — methods that returned <code>Any</code> now return real types; <code>mypy --strict</code> may surface previously-hidden errors. Stub layout also moved from <code>tokenizers/<sub>/__init__.pyi</code> to <code>tokenizers/<sub>.pyi</code>. This breaks the surface of some of the processors like <code>RobertaProcessign</code>'s <code>__init__</code> .</li> <li><strong>3.14t-only</strong>: setters/getters return <code>PyResult<T></code> because of <code>Arc<RwLock<Tokenizer>></code>; a poisoned lock surfaces as <code>PyException</code> instead of a panic.</li> </ul> <hr /> <h2>⚡ Performance — measured locally on this Mac, not lifted from PRs</h2> <p>Run with <code>cargo bench --bench <name> -- --save-baseline v0_22_2</code> on <code>v0.22.2</code>, then <code>--baseline v0_22_2</code> on <code>v0.23.1</code>. Numbers are point-in-time wall clock on a single laptop; relative deltas are what matters, absolute numbers will differ on CI hardware.</p> <h3>Added-vocabulary deserialize — the headline win (<a href="https://redirect.github.com/huggingface/tokenizers/issues/1995">#1995</a>, <a href="https://redirect.github.com/huggingface/tokenizers/issues/1999">#1999</a>)</h3> <p><code>bench: improve added_vocab_deserialize to reflect real-world workloads</code> (<a href="https://redirect.github.com/huggingface/tokenizers/issues/2000">#2000</a>) is now representative of how transformers actually loads tokenizer.json files. The combined effect of <code>daachorse</code> for the matching automaton plus the normalize-on-insert refactor is enormous on this workload:</p> <table> <thead> <tr> <th>benchmark</th> <th align="right">v0.22.2</th> <th align="right">v0.23.1</th> <th align="right">change</th> </tr> </thead> <tbody> <tr> <td>100k tokens, special, no norm</td> <td align="right">~410 ms</td> <td align="right">248 ms</td> <td align="right"><strong>−40%</strong></td> </tr> <tr> <td>100k tokens, non-special, no norm</td> <td align="right">~7.1 s</td> <td align="right">273 ms</td> <td align="right"><strong>−96%</strong></td> </tr> <tr> <td>100k tokens, special, NFKC</td> <td align="right">~395 ms</td> <td align="right">235 ms</td> <td align="right"><strong>−40%</strong></td> </tr> <tr> <td>100k tokens, non-special, NFKC</td> <td align="right">~7.4 s</td> <td align="right">290 ms</td> <td align="right"><strong>−96%</strong></td> </tr> <tr> <td>400k tokens, special, no norm</td> <td align="right">~15 s</td> <td align="right">980 ms</td> <td align="right"><strong>−94%</strong></td> </tr> </tbody> </table> <p>Real-world impact: loading a Llama-3-style tokenizer with a large set of added tokens dropped from "noticeable pause" to "instant".</p> <h3>BPE encode</h3> <table> <thead> <tr> <th>benchmark</th> <th align="right">v0.22.2</th> <th align="right">v0.23.1</th> <th align="right">change</th> </tr> </thead> <tbody> <tr> <td><code>BPE GPT2 encode batch, no cache</code></td> <td align="right">530 ms</td> <td align="right">446 ms</td> <td align="right"><strong>−16%</strong></td> </tr> <tr> <td><code>BPE GPT2 encode batch</code> (cached)</td> <td align="right">690 ms</td> <td align="right">685 ms</td> <td align="right">noise</td> </tr> <tr> <td><code>BPE GPT2 encode</code> (single)</td> <td align="right">1.95 s</td> <td align="right">1.94 s</td> <td align="right">noise</td> </tr> <tr> <td><code>BPE Train (small)</code></td> <td align="right">32.6 ms</td> <td align="right">31.5 ms</td> <td align="right">−3%</td> </tr> <tr> <td><code>BPE Train (big)</code></td> <td align="right">1.01 s</td> <td align="right">988 ms</td> <td align="right">−2%</td> </tr> </tbody> </table> <p>The BPE per-thread cache PR (<a href="https://redirect.github.com/huggingface/tokenizers/issues/2028">#2028</a>) shows much larger wins on highly-parallel workloads (+47–62% at 88+ threads on a server box, per the PR's own measurements on Vera). Single-thread batch numbers above are flat or slightly improved because cache-hit overhead was already low without contention.</p> <h3>Llama-3 encode</h3> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
AxelRay
|
ef7e07e0f5
|
fix(policy): price net-cost mutations with the 1h cache-write tier (#2780)
## Description This fixes the net-cost mutation gate for requests using Anthropic's 1-hour prompt-cache TTL. The gate previously hardcoded the 5-minute cache-write multiplier of 1.25x. A 1-hour cache write costs 2.0x, so the old calculation understated the true write penalty and could incorrectly recommend mutation for 1-hour clients. Closes #2773 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added TTL-aware cache-write multiplier selection for 5-minute and 1-hour tiers. - Threaded the resolved TTL through the content router and compression policy helpers. - Preserved the existing 5-minute behavior as the default. - Added Python and Rust regression coverage for the 1-hour tier. - Retuned the netcost gate fixtures so the 1-hour write tier flips the decision in the full ContentRouter path. - Did not edit CHANGELOG.md. ## Testing - [x] Unit tests pass (pytest) - [x] Linting passes (ruff check .) - [ ] Type checking passes (mypy headroom) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text pytest tests/test_compression_policy.py -q 20 passed cargo test -p headroom-core --lib compression_policy -- --nocapture 14 passed pytest tests/test_netcost_gate.py -q 27 passed Ruff checks and formatting passed. git diff --check passed. ``` ## Real Behavior Proof - Environment: Linux x86_64 contributor checkout with Python and Rust test environments. - Exact command / steps: - Ran the Python compression policy test suite. - Ran the Rust compression policy unit tests. - Ran the netcost gate suite, including the 1-hour env and request-marker cases. - Exercised the new 1-hour TTL golden case alongside the existing 5-minute cases. - Observed result: The 1-hour case uses the 2.0x write multiplier and skips the same candidate that still mutates under 5-minute pricing. Existing 5-minute behavior remains covered and passing. - Not tested: A live Anthropic request through the proxy and production traffic. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit CHANGELOG.md - it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable for this backend policy fix. ## Additional Notes Ready for review. CI is green on the current tip. |
||
|
dependabot[bot]
|
322425c43b
|
deps: bump sha2 from 0.10.9 to 0.11.0 (#2288)
Bumps [sha2](https://github.com/RustCrypto/hashes) from 0.10.9 to 0.11.0. <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
5731be7e68
|
deps: bump axum from 0.7.9 to 0.8.9 (#2966)
Bumps [axum](https://github.com/tokio-rs/axum) from 0.7.9 to 0.8.9. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/tokio-rs/axum/releases">axum's releases</a>.</em></p> <blockquote> <h2>axum-v0.8.9</h2> <ul> <li><strong>added:</strong> <code>WebSocketUpgrade::{requested_protocols, set_selected_protocol}</code> for more flexible subprotocol selection (<a href="https://redirect.github.com/tokio-rs/axum/issues/3597">#3597</a>)</li> <li><strong>changed:</strong> Update minimum rust version to 1.80 (<a href="https://redirect.github.com/tokio-rs/axum/issues/3620">#3620</a>)</li> <li><strong>fixed:</strong> Set connect endpoint on correct field in MethodRouter (<a href="https://redirect.github.com/tokio-rs/axum/issues/3656">#3656</a>)</li> <li><strong>fixed:</strong> Return specific error message when multipart body limit is exceeded (<a href="https://redirect.github.com/tokio-rs/axum/issues/3611">#3611</a>)</li> </ul> <p><a href="https://redirect.github.com/tokio-rs/axum/issues/3597">#3597</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3597">tokio-rs/axum#3597</a> <a href="https://redirect.github.com/tokio-rs/axum/issues/3620">#3620</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3620">tokio-rs/axum#3620</a> <a href="https://redirect.github.com/tokio-rs/axum/issues/3656">#3656</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3656">tokio-rs/axum#3656</a> <a href="https://redirect.github.com/tokio-rs/axum/issues/3611">#3611</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3611">tokio-rs/axum#3611</a></p> <h2>axum v0.8.8</h2> <ul> <li>Clarify documentation for <code>Router::route_layer</code> (<a href="https://redirect.github.com/tokio-rs/axum/issues/3567">#3567</a>)</li> </ul> <p><a href="https://redirect.github.com/tokio-rs/axum/issues/3567">#3567</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3567">tokio-rs/axum#3567</a></p> <h2>axum v0.8.7</h2> <ul> <li>Relax implicit <code>Send</code> / <code>Sync</code> bounds on <code>RouterAsService</code>, <code>RouterIntoService</code> (<a href="https://redirect.github.com/tokio-rs/axum/issues/3555">#3555</a>)</li> <li>Make it easier to visually scan for default features (<a href="https://redirect.github.com/tokio-rs/axum/issues/3550">#3550</a>)</li> <li>Fix some documentation typos</li> </ul> <p><a href="https://redirect.github.com/tokio-rs/axum/issues/3550">#3550</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3550">tokio-rs/axum#3550</a> <a href="https://redirect.github.com/tokio-rs/axum/issues/3555">#3555</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3555">tokio-rs/axum#3555</a></p> <h2>axum v0.8.5</h2> <ul> <li><strong>fixed:</strong> Reject JSON request bodies with trailing characters after the JSON document (<a href="https://redirect.github.com/tokio-rs/axum/issues/3453">#3453</a>)</li> <li><strong>added:</strong> Implement <code>OptionalFromRequest</code> for <code>Multipart</code> (<a href="https://redirect.github.com/tokio-rs/axum/issues/3220">#3220</a>)</li> <li><strong>added:</strong> Getter methods <code>Location::{status_code, location}</code></li> <li><strong>added:</strong> Support for writing arbitrary binary data into server-sent events (<a href="https://redirect.github.com/tokio-rs/axum/issues/3425">#3425</a>)]</li> <li><strong>added:</strong> <code>middleware::ResponseAxumBodyLayer</code> for mapping response body to <code>axum::body::Body</code> (<a href="https://redirect.github.com/tokio-rs/axum/issues/3469">#3469</a>)</li> <li><strong>added:</strong> <code>impl FusedStream for WebSocket</code> (<a href="https://redirect.github.com/tokio-rs/axum/issues/3443">#3443</a>)</li> <li><strong>changed:</strong> The <code>sse</code> module and <code>Sse</code> type no longer depend on the <code>tokio</code> feature (<a href="https://redirect.github.com/tokio-rs/axum/issues/3154">#3154</a>)</li> <li><strong>changed:</strong> If the location given to one of <code>Redirect</code>s constructors is not a valid header value, instead of panicking on construction, the <code>IntoResponse</code> impl now returns an HTTP 500, just like <code>Json</code> does when serialization fails (<a href="https://redirect.github.com/tokio-rs/axum/issues/3377">#3377</a>)</li> <li><strong>changed:</strong> Update minimum rust version to 1.78 (<a href="https://redirect.github.com/tokio-rs/axum/issues/3412">#3412</a>)</li> </ul> <p><a href="https://redirect.github.com/tokio-rs/axum/issues/3154">#3154</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3154">tokio-rs/axum#3154</a> <a href="https://redirect.github.com/tokio-rs/axum/issues/3220">#3220</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3220">tokio-rs/axum#3220</a> <a href="https://redirect.github.com/tokio-rs/axum/issues/3377">#3377</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3377">tokio-rs/axum#3377</a> <a href="https://redirect.github.com/tokio-rs/axum/issues/3412">#3412</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3412">tokio-rs/axum#3412</a> <a href="https://redirect.github.com/tokio-rs/axum/issues/3425">#3425</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3425">tokio-rs/axum#3425</a> <a href="https://redirect.github.com/tokio-rs/axum/issues/3443">#3443</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3443">tokio-rs/axum#3443</a> <a href="https://redirect.github.com/tokio-rs/axum/issues/3453">#3453</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3453">tokio-rs/axum#3453</a> <a href="https://redirect.github.com/tokio-rs/axum/issues/3469">#3469</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3469">tokio-rs/axum#3469</a></p> <h2>axum v0.8.4</h2> <ul> <li><strong>added:</strong> <code>Router::reset_fallback</code> (<a href="https://redirect.github.com/tokio-rs/axum/issues/3320">#3320</a>)</li> <li><strong>added:</strong> <code>WebSocketUpgrade::selected_protocol</code> (<a href="https://redirect.github.com/tokio-rs/axum/issues/3248">#3248</a>)</li> <li><strong>fixed:</strong> Panic location for overlapping method routes (<a href="https://redirect.github.com/tokio-rs/axum/issues/3319">#3319</a>)</li> <li><strong>fixed:</strong> Don't leak a tokio task when using <code>serve</code> without graceful shutdown (<a href="https://redirect.github.com/tokio-rs/axum/issues/3129">#3129</a>)</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
bbe901319d
|
deps: bump tokio-tungstenite from 0.24.0 to 0.30.0 (#2967)
Bumps [tokio-tungstenite](https://github.com/snapview/tokio-tungstenite) from 0.24.0 to 0.30.0. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/snapview/tokio-tungstenite/blob/master/CHANGELOG.md">tokio-tungstenite's changelog</a>.</em></p> <blockquote> <h1>0.30.0</h1> <ul> <li>Update <code>tungstenite</code> to <code>0.30.0</code>. See <a href="https://github.com/snapview/tungstenite-rs/blob/master/CHANGELOG.md"><code>tungstenite</code> release</a>.</li> </ul> <h1>0.29.0</h1> <ul> <li>Update <code>tungstenite</code> to <code>0.29.0</code>. See <a href="https://github.com/snapview/tungstenite-rs/blob/master/CHANGELOG.md"><code>tungstenite</code> release</a>.</li> </ul> <h1>0.28.0</h1> <ul> <li>Update <code>tungstenite</code> to <code>0.28.0</code>. See <a href="https://github.com/snapview/tungstenite-rs/blob/master/CHANGELOG.md"><code>tungstenite</code> release</a>.</li> </ul> <h1>0.27.0</h1> <ul> <li>See <a href="https://github.com/snapview/tungstenite-rs/blob/master/CHANGELOG.md#0270">performance updates in <code>tungstenite-rs</code></a>.</li> </ul> <h1>0.26.2</h1> <ul> <li>Update <code>tungstenite</code>, see <a href="https://github.com/snapview/tungstenite-rs/blob/master/CHANGELOG.md#0262">changes here</a>.</li> </ul> <h1>0.26.1</h1> <ul> <li>Update <code>tungstenite</code> to address an issue that might cause UB in certain cases.</li> </ul> <h1>0.26.0</h1> <ul> <li>Update <code>tungstenite</code> to <code>0.26.0</code> (<a href="https://github.com/snapview/tungstenite-rs/blob/master/CHANGELOG.md#0260">breaking changes</a>).</li> </ul> <h1>0.25.0</h1> <ul> <li>Update <code>tungstenite</code> to <code>0.25.0</code> (<a href="https://github.com/snapview/tungstenite-rs/blob/master/CHANGELOG.md#0250">important updates!</a>).</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
b30f339d69
|
deps: bump criterion from 0.5.1 to 0.8.2 (#2965)
Bumps [criterion](https://github.com/criterion-rs/criterion.rs) from 0.5.1 to 0.8.2. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/criterion-rs/criterion.rs/releases">criterion's releases</a>.</em></p> <blockquote> <h2>criterion-plot-v0.8.2</h2> <h3>Other</h3> <ul> <li>Update Readme</li> </ul> <h2>criterion-v0.8.2</h2> <h3>Fixed</h3> <ul> <li>don't build alloca on unsupported targets</li> </ul> <h3>Other</h3> <ul> <li><em>(deps)</em> bump crate-ci/typos from 1.40.0 to 1.43.0</li> <li>Fix panic with uniform iteration durations in benchmarks</li> <li>Update Readme</li> <li>Exclude development scripts from published package</li> </ul> <h2>criterion-plot-v0.8.1</h2> <h3>Fixed</h3> <ul> <li>Typo</li> </ul> <h2>criterion-v0.8.1</h2> <h3>Fixed</h3> <ul> <li>Homepage link</li> </ul> <h3>Other</h3> <ul> <li><em>(deps)</em> bump crate-ci/typos from 1.23.5 to 1.40.0</li> <li><em>(deps)</em> bump jontze/action-mdbook from 3 to 4</li> <li><em>(deps)</em> bump actions/checkout from 4 to 6</li> </ul> <h2>criterion-plot-v0.8.0</h2> <p>No release notes provided.</p> <h2>criterion-v0.8.0</h2> <h3>BREAKING</h3> <ul> <li>Drop async-std support</li> </ul> <h3>Changed</h3> <ul> <li>Bump MSRV to 1.86, stable to 1.91.1</li> </ul> <h3>Added</h3> <ul> <li>Add ability to plot throughput on summary page.</li> <li>Add support for reporting throughput in elements and bytes - <code>Throughput::ElementsAndBytes</code> allows the text summary to report throughput in both units simultaneously.</li> <li>Add alloca-based memory layout randomisation to mitigate memory effects on measurements.</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/criterion-rs/criterion.rs/blob/master/CHANGELOG.md">criterion's changelog</a>.</em></p> <blockquote> <h2><a href="https://github.com/criterion-rs/criterion.rs/compare/criterion-v0.8.1...criterion-v0.8.2">0.8.2</a> - 2026-02-04</h2> <h3>Fixed</h3> <ul> <li>don't build alloca on unsupported targets</li> </ul> <h3>Other</h3> <ul> <li><em>(deps)</em> bump crate-ci/typos from 1.40.0 to 1.43.0</li> <li>Fix panic with uniform iteration durations in benchmarks</li> <li>Update Readme</li> <li>Exclude development scripts from published package</li> </ul> <h2><a href="https://github.com/criterion-rs/criterion.rs/compare/criterion-v0.8.0...criterion-v0.8.1">0.8.1</a> - 2025-12-07</h2> <h3>Fixed</h3> <ul> <li>Homepage link</li> </ul> <h3>Other</h3> <ul> <li><em>(deps)</em> bump crate-ci/typos from 1.23.5 to 1.40.0</li> <li><em>(deps)</em> bump jontze/action-mdbook from 3 to 4</li> <li><em>(deps)</em> bump actions/checkout from 4 to 6</li> </ul> <h2><a href="https://github.com/criterion-rs/criterion.rs/compare/criterion-v0.7.0...criterion-v0.8.0">0.8.0</a> - 2025-11-29</h2> <h3>BREAKING</h3> <ul> <li>Drop async-std support</li> </ul> <h3>Changed</h3> <ul> <li>Bump MSRV to 1.86, stable to 1.91.1</li> </ul> <h3>Added</h3> <ul> <li>Add ability to plot throughput on summary page.</li> <li>Add support for reporting throughput in elements and bytes - <code>Throughput::ElementsAndBytes</code> allows the text summary to report throughput in both units simultaneously.</li> <li>Add alloca-based memory layout randomisation to mitigate memory effects on measurements.</li> <li>Add doc comment to benchmark runner in criterion_group macro (removes linter warnings)</li> </ul> <h3>Fixed</h3> <ul> <li>Fix plotting NaN bug</li> </ul> <h3>Other</h3> <ul> <li>Remove Master API Docs links temporarily while we restore the docs publishing.</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
JD Davis
|
3077ac81e8
|
feat: add deterministic runtime rollout controls (#1490)
## Description Establish one centrally resolved, observable, deterministic, versioned runtime rollout-control mechanism for Headroom. Runtime rollout controls which behaviors an already-built artifact may expose; it does not select or qualify a Headroom release/version. ## Type of Change - [x] New feature (non-breaking change that adds functionality) - [x] Bug fix (non-breaking change that fixes rollout enforcement regressions) - [x] Documentation update - [x] Code refactoring (no functional changes) ## Changes Made - Added `RolloutChannel`, `HEADROOM_ROLLOUT_CHANNEL`, `--rollout-channel`, and a versioned immutable `RolloutSnapshot` shared by Python configuration boundaries. - Added schema/policy versions, canonical registry and snapshot SHA-256 identities, per-feature decision reasons, disable precedence, unsafe qualification poisoning, strict CLI validation, and fail-closed environment handling. - Added `headroom rollout status --json`, Python `/stats.rollout`, and Rust `/rollout/status` runtime provenance. - Added equivalent Rust snapshot semantics and shared Python/Rust policy vectors while retaining language-specific feature registries. - Enforced rollout policy at alternate Python server composition roots so `HEADROOM_READ_MATURATION=1` cannot bypass its beta gate. - Preserved typed rollout snapshots across multi-worker serialization with schema, policy, registry, snapshot-digest, type, and feature-name validation. - Made loopback runtime output-shaper updates replace the immutable snapshot atomically for request readers, retain explicit request/disable provenance, preserve channel and kill-switch precedence, invalidate cached stats, and return the effective rollout decision. - Made `headroom learn --verbosity --apply` report a channel-blocked update instead of claiming the shaper is live. - Made explicit CLI feature flags fail loudly when their current channel blocks them. - Made persistent interceptor installation select canary automatically, or reject an explicitly insufficient channel unless the break-glass override is set. - Updated architecture, proxy, rollout, learn, and output-shaper documentation with required channels and hot-reload semantics. ## Testing - [x] Unit tests pass - [x] Linting passes (`ruff check .` and `ruff format --check .`) - [x] Type checking passes (`mypy headroom --ignore-missing-imports`) - [x] New regression tests added for every corrected behavior - [x] Rust tests and production-target Clippy pass - [x] Documentation build passes ### Test Output ```text Focused rollout coverage suite 57 passed; headroom.rollout + rollout CLI: 98% coverage Affected proxy/rollout/transform/governance suites 222 passed; 0 failed Final changed regression suites 100 passed; 0 failed Cross-module hot-reload isolation regression 6 passed; 0 failed cargo test -p headroom-core -p headroom-proxy --quiet headroom-core: 924 passed; 1 ignored headroom-proxy and integration suites: all passed cargo clippy -p headroom-core -p headroom-proxy --lib --bins -- -D warnings cargo fmt --all -- --check ruff check . ruff format --check . mypy headroom --ignore-missing-imports git diff --check All passed cd docs && npm run build Compiled successfully; 164 static pages generated ``` The unsharded Windows-only CI selection exposed unrelated baseline failures, principally the existing `sqlite:///C:\\...` URL parser producing an invalid `\\C:\\...` path. At commit ` |
||
|
Andrei Boldyrev
|
f6398a6476
|
fix(proxy): port session-sticky beta headers to the Rust proxy (#2381)
## Description The Python proxy protects prompt caches with `SessionBetaTracker` (PR-A6, `headroom/proxy/helpers.py`): interactive clients (Claude Code, Codex CLI) may drop an `anthropic-beta` / `openai-beta` token between turn N and turn N+1 of the same conversation, and since beta headers are part of the bytes that determine the upstream prefix-cache key, the drop rotates the key and the provider re-writes the whole prefix at the customer's cost. The tracker unions the client's tokens with everything previously seen for that `(provider, session)` and forwards the union — a documented operator contract (`docs/configuration.mdx`, "Session Beta Header Tracking"). The Rust proxy has no equivalent, and Phase H (#2258) deletes the tracker together with `helpers.py` and its test file (`tests/test_anthropic_beta_session_sticky.py`). None of the Phase A–G plans port it (Phase F consumes beta headers for auth-mode classification only), so the protection would silently not survive the migration — and the Phase-H gate "Cache-hit-rate parity with direct upstream confirmed" can't catch the loss, because re-injection makes proxied traffic *beat* direct upstream on cache hits; when the mechanism disappears, proxied traffic degrades *to* direct-upstream levels, which that comparison reads as parity. This PR ports the tracker semantics into the Rust proxy so the protection lives in the codebase Phase H keeps. Closes #2380 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) (New Rust functionality, but a parity port of already-shipped, already-documented Python behavior — the PR title uses `fix:` per `REALIGNMENT/INDEX.md`: "Commit prefix: `fix:` for Rust-migration phase commits".) ## Changes Made - **`cache_stabilization/beta_sticky.rs`** — the tracker: bounded LRU (1000 sessions, same sizing rationale and `# Panics` contract as the drift detector's capacity) keyed by `(provider, session)`, storing the per-session ordered token list. Union preserves first-seen order; dedup is case-insensitive with first-seen casing winning; lookups touch recency; overflow evicts the oldest — mirroring the Python tracker. The header-plumbing lives in the module too (`apply_sticky_betas`), so the merge is unit-testable without booting a proxy. - **`proxy.rs` wiring** — on the intercepted POST routes (`/v1/messages`, `/v1/chat/completions`, `/v1/responses`), right after the drift-detector observation, reusing the drift detector's `derive_session_key` output so both cache-stability subsystems agree on conversation identity. - **`config.rs`** — `--beta-header-sticky` / `HEADROOM_PROXY_BETA_HEADER_STICKY` (`enabled` default; `disabled` forwards the client value verbatim and keeps no state), mirroring the `StripInternalHeaders` flag pattern and the existing `HEADROOM_*` → `HEADROOM_PROXY_*` Python→Rust env pairing. Since the merge runs inside the compression interceptor, startup logs a warning when the flag is `enabled` while `--compression` is off, and both the CLI doc and the docs row state the dependency. - **`tests/integration_beta_header_sticky.rs`** — 9 end-to-end tests against a wiremock upstream asserting the headers/bytes the upstream actually receives; 21 unit tests port the behavioral contract from `tests/test_anthropic_beta_session_sticky.py` and cover the header-map plumbing. - **`docs/content/docs/configuration.mdx`** — one row for `HEADROOM_PROXY_BETA_HEADER_STICKY` next to the existing Python/Rust flag pairs. ## Testing - [x] Unit tests pass (`cargo test -p headroom-proxy`; Python side via `make ci-precheck-python` — `pytest` subset, 174 passed) - [x] Linting passes (`cargo clippy --all-targets` — 0 warnings; `cargo fmt --check` clean; Rust-only change, so `ruff`/`mypy` are covered by the untouched-Python `ci-precheck-python` build) - [ ] Type checking passes (`mypy headroom`) — N/A, no Python files touched - [x] New tests added for new functionality - [x] Manual testing performed (RED/GREEN before-and-after runs below) ### Test Output ```text $ cargo test -p headroom-proxy --lib beta_sticky test result: ok. 21 passed; 0 failed; 0 ignored; 0 measured; 248 filtered out; finished in 0.03s $ cargo test -p headroom-proxy --test integration_beta_header_sticky test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.07s $ cargo test -p headroom-proxy # full crate: 37 suites, all ok $ cargo clippy -p headroom-proxy --all-targets # 0 warnings $ make ci-precheck-rust ci-precheck-python ci-precheck-commitlint # green ``` ## Real Behavior Proof - Environment: macOS arm64 (Darwin 24.6), `rustc 1.95.0`, real Rust proxy booted on an ephemeral port in front of a wiremock upstream (`tests/common::start_proxy_with`, `compression = true`). - Exact command / steps: two-turn conversation through the proxy — turn 1 `POST /v1/messages` with `anthropic-beta: context-management-2025-06-27,interleaved-thinking-2025-05-14`; turn 2, same conversation, client drops the second token. The wiremock responder captures the headers the upstream actually receives (`cargo test -p headroom-proxy --test integration_beta_header_sticky`). - Observed result: **before** the port (test written first, run against the unmodified proxy) the upstream sees the shrunken token set and the prefix-cache key rotates — ```text assertion `left == right` failed: turn 2 must re-inject the dropped token so the upstream prefix-cache key stays byte-stable left: Some("context-management-2025-06-27") right: Some("context-management-2025-06-27,interleaved-thinking-2025-05-14") ``` **After** the port the same scenario passes: the upstream receives the full union on turn 2, the internal `x-headroom-session-id` never crosses the upstream boundary, and the forwarded body is SHA-256-identical to what the client sent (asserted by `body_bytes_stay_byte_equal_while_header_is_rewritten`). - Not tested: live traffic against a real provider upstream (wiremock only); the WebSocket path and Bedrock/Vertex routes (out of scope — see Additional Notes). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A (proxy behavior; see Real Behavior Proof). ## Additional Notes Design decisions, and where I'd like reviewer judgment: 1. **Applies to all auth modes, like the Python handler.** The Phase-E module doctrine gates *body*-mutating normalizers on PAYG; this mechanism mutates headers only, and the Python source of truth applies it unconditionally — an auth-mode gate here would create a behavioral delta exactly where the PR's purpose is behavior preservation. It's also stealth-consistent by construction: the union only ever contains tokens this client itself sent (Headroom-added tokens are never recorded), `auth_mode.rs`'s own docs name "beta-header drift voids them" as the OAuth cache hazard (stickiness is the anti-drift), and F2's `CompressionPolicy` has no beta field — no gate is structurally expected. I've extended the `cache_stabilization/mod.rs` taxonomy with a third category ("re-echo client-sent state") to keep the module doctrine honest. Flagging explicitly since invariant #10 ("no beta drift") is subscription-critical: if you read it as "forward beta verbatim on Subscription", say so and I'll add the gate. 2. **One deliberate divergence from Python: sessions are keyed per conversation, not per `(model, system)` bucket.** The Python tracker keys on the store session id — explicit header, else a hash of model + leading system prompt — so a Claude Code session and every one of its subagents share one token union and cross-inherit tokens; two *different users* behind an org proxy with the same (model, system) do too. This port keys on the drift detector's conversation-aware key (#2301), so each conversation keeps its own union (pinned by `separate_conversations_do_not_leak_tokens`). That's the same conflation defect #2085/#2193/#2301 chased out of the other session-sticky subsystems, and it makes "the union only contains tokens this client sent" actually true — under the Python fallback key it isn't (cross-user union). Cost: Python's accidental cross-conversation repair is gone, and an OAuth access-token refresh mid-conversation re-keys the session (one turn forwards verbatim, then re-learns — fails safe). 3. **Repeated header lines are joined per RFC 9110 list semantics before recording.** A client sending two `anthropic-beta` lines gets both recorded; a later rewrite collapses to one line carrying the full set. (Reading only the first line — or Python's actual behavior, which keeps only the *last* line via its `dict(headers)` collapse — can shrink the upstream token set mid-conversation when a rewrite fires.) 4. **Scope: the three intercepted HTTP routes.** With the compression interceptor off the proxy is a strict byte-pipe (Phase-A invariant) — no header mutation, hence the startup warning. WebSocket keeps its behavior (Python's WS site keys on a per-connection UUID, so cross-turn accumulation is a near-no-op there; the Rust WS tunnel doesn't touch beta headers). Bedrock/Vertex are skipped by the same match that skips the drift detector (betas travel in the body as `anthropic_beta` on Bedrock). 5. **Log discipline**: `event=beta_header_merge` carries token *counts* only (beta tokens can carry experiment IDs; same privacy contract as Python's `log_beta_header_merge`, plus the drift detector's hashed session-key prefix instead of Python's raw session id). One deviation from Python's unconditional info: the no-op case logs at debug, matching the drift detector's silent-on-stable precedent — an info-level `beta_header_merge` always marks an actual cache-affecting rewrite. 6. **Capacity is a const (1000), not a flag** — following the drift-detector precedent rather than Python's `HEADROOM_BETA_TRACKER_MAX_SESSIONS` env var. Happy to make it configurable if you'd rather keep that operator knob. 7. **Fail-open everywhere**: non-ASCII client values are forwarded verbatim with nothing recorded; a poisoned tracker lock forwards the client value verbatim; an unencodable union (unreachable — every token came from a parsed header value) logs and forwards verbatim. The protection never delays or drops a request. |
||
|
Zhenjia ZHOU
|
6840153473
|
fix(tokenizer): price CJK in the Rust fixed-ratio estimator (Python parity) (#2260)
## Description The Rust `EstimatingCounter` priced every character at the Latin `chars_per_token` (default 4.0), but the Python `EstimatingTokenCounter` it explicitly mirrors already prices dense scripts (CJK / Kana / Hangul / full-width) at `CHARS_PER_TOKEN_CJK = 1.5` — so Rust under-counted CJK by ~2.5× and the two implementations diverged. #2080 fixed only the Python path; the Rust module doc still says "Mirrors …EstimatingTokenCounter" while it no longer did. This is the live count path for every provider-calibrated fixed-ratio counter (Anthropic 3.5, Google / Cohere 4.0, Moonshot 3.1), so CJK traffic was mis-budgeted (savings/estimates skewed). This counts dense-script codepoints — the same 8 `CJK_PATTERN` Unicode ranges Python uses — and prices them separately: `int(other / ratio + cjk / 1.5 + 0.5)`. Non-CJK output is byte-identical. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `crates/headroom-core/src/tokenizer/estimator.rs`: add `is_dense_script(c)` (8 ranges byte-mirroring Python `CJK_PATTERN`) and `CHARS_PER_TOKEN_CJK = 1.5`; `count_text` prices dense-script chars separately from Latin. - Reference tests for CJK / kana / full-width / mixed — values cross-checked against Python. ## Testing - [x] Unit tests pass (`cargo test`) - [x] Linting passes (`cargo clippy` / `cargo fmt`) - [x] New tests added - [x] Verified against Python (see Real Behavior Proof) ### Test Output ```text $ cargo test -p headroom-core --lib tokenizer test result: ok. 45 passed; 0 failed $ cargo clippy / fmt # clean ``` ## Real Behavior Proof - Environment: macOS (Darwin), Rust via cargo + Python in a uv venv, branch `feat/tokenizer-estimator-cjk` off `main`. - Exact command / steps: ran the same inputs through Python `EstimatingTokenCounter(4.0).count_text` and the Rust `EstimatingCounter::default().count_text`, comparing outputs. - Observed result: identical on every input — `数据库` → 2, `数据库连接失败` → 5 (was 2 under the old flat 7/4), `ひらが` → 2, full-width `ＡＰＩ` → 2 vs plain `API` → 1, mixed `api数据` → 2. The existing non-CJK reference tests (`a`×40 → 10, Claude-3.5 densities, `héllo`/emoji char-count) are unchanged, confirming no ASCII regression. - Not tested: nothing further — parity is verified directly against the Python reference (same values on both sides). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation — N/A (internal estimator) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — happy to add an entry if preferred. ## Additional Notes - Completes #2080 (which priced CJK in the Python fixed-ratio estimator) on the Rust side, restoring Rust↔Python parity for the density estimator. Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
dependabot[bot]
|
4925bf6a82
|
deps: bump hf-hub from 0.4.3 to 0.5.0 (#2285)
Bumps [hf-hub](https://github.com/huggingface/hf-hub) from 0.4.3 to 0.5.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/huggingface/hf-hub/releases">hf-hub's releases</a>.</em></p> <blockquote> <h2>v0.5.0</h2> <h2>What's Changed</h2> <ul> <li>Upgrade ureq by <a href="https://github.com/Narsil"><code>@Narsil</code></a> in <a href="https://redirect.github.com/huggingface/hf-hub/pull/114">huggingface/hf-hub#114</a></li> <li>Update indicatif to current version in Cargo.toml by <a href="https://github.com/gordonmessmer"><code>@gordonmessmer</code></a> in <a href="https://redirect.github.com/huggingface/hf-hub/pull/126">huggingface/hf-hub#126</a></li> <li>Fix failing API tests due to outdated model metadata expectations by <a href="https://github.com/bmqube"><code>@bmqube</code></a> in <a href="https://redirect.github.com/huggingface/hf-hub/pull/124">huggingface/hf-hub#124</a></li> <li>fix: fix typo by <a href="https://github.com/AndyDai-nv"><code>@AndyDai-nv</code></a> in <a href="https://redirect.github.com/huggingface/hf-hub/pull/121">huggingface/hf-hub#121</a></li> <li>Updating tests and dependencies. by <a href="https://github.com/Narsil"><code>@Narsil</code></a> in <a href="https://redirect.github.com/huggingface/hf-hub/pull/127">huggingface/hf-hub#127</a></li> <li>Remove markdown from Cargo.toml by <a href="https://github.com/gordonmessmer"><code>@gordonmessmer</code></a> in <a href="https://redirect.github.com/huggingface/hf-hub/pull/128">huggingface/hf-hub#128</a></li> <li>Fixup the docstrings for download function (which always downloads). by <a href="https://github.com/Narsil"><code>@Narsil</code></a> in <a href="https://redirect.github.com/huggingface/hf-hub/pull/130">huggingface/hf-hub#130</a></li> <li>Expose <code>metadata</code> and <code>pointer_path</code> methods by <a href="https://github.com/danieldk"><code>@danieldk</code></a> in <a href="https://redirect.github.com/huggingface/hf-hub/pull/136">huggingface/hf-hub#136</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/gordonmessmer"><code>@gordonmessmer</code></a> made their first contribution in <a href="https://redirect.github.com/huggingface/hf-hub/pull/126">huggingface/hf-hub#126</a></li> <li><a href="https://github.com/bmqube"><code>@bmqube</code></a> made their first contribution in <a href="https://redirect.github.com/huggingface/hf-hub/pull/124">huggingface/hf-hub#124</a></li> <li><a href="https://github.com/AndyDai-nv"><code>@AndyDai-nv</code></a> made their first contribution in <a href="https://redirect.github.com/huggingface/hf-hub/pull/121">huggingface/hf-hub#121</a></li> <li><a href="https://github.com/danieldk"><code>@danieldk</code></a> made their first contribution in <a href="https://redirect.github.com/huggingface/hf-hub/pull/136">huggingface/hf-hub#136</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/huggingface/hf-hub/compare/v0.4.3...v0.5.0">https://github.com/huggingface/hf-hub/compare/v0.4.3...v0.5.0</a></p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/huggingface/hf-hub/blob/main/RELEASE.md">hf-hub's changelog</a>.</em></p> <blockquote> <h1>Releasing hf-hub</h1> <p>This document covers the full release process for the <code>hf-hub</code> crate. If anything here is unclear or out of date, please open a PR.</p> <h2>What gets released</h2> <p>A single tag push releases one artifact:</p> <ul> <li><strong><code>hf-hub</code> Rust crate</strong> on <a href="https://crates.io/crates/hf-hub">crates.io</a>, via <code>.github/workflows/rust-release.yml</code>.</li> </ul> <p>The workflow triggers on tags matching <code>v*</code> (e.g., <code>v1.0.0</code>, <code>v1.0.0-rc.0</code>).</p> <p>There are no Python components in this repo. The other workspace members are not published:</p> <ul> <li><code>hfrs/</code> — CLI binary, distributed via <code>cargo install --git</code>.</li> <li><code>examples/</code>, <code>benches/</code>, <code>integration-tests/</code> — internal-only, version <code>0.0.0</code>, never published.</li> </ul> <h2>Pre-release checklist</h2> <ol> <li><strong>CI is green on <code>main</code>.</strong> The <code>Rust</code> workflow must be passing on every platform in the matrix (Ubuntu, Windows, macOS) with both feature configurations (<code>""</code> and <code>--all-features</code>).</li> <li><strong>Review the diff since the last release.</strong> <pre lang="bash"><code>git log --oneline v0.5.0..main git diff v0.5.0..main --stat -- hf-hub/ </code></pre> Pay particular attention to changes under <code>hf-hub/src/</code> — those are the only changes that actually ship to crates.io.</li> <li><strong>Identify breaking changes.</strong> Anything that changes the public Rust API (types, function signatures, removed re-exports, builder fields) needs to be reflected in the version bump per <a href="https://semver.org">semver</a> and called out in the release notes.</li> <li><strong>Run the full pre-release test sweep</strong> (see next section).</li> </ol> <h2>Pre-release test sweep</h2> <p>Run all of these from the repo root before tagging. They mirror what CI runs, plus a publish dry-run that CI does not currently do.</p> <h3>Format and lint</h3> <pre lang="bash"><code>cargo +nightly fmt --all -- --check cargo clippy --workspace --all-targets -- -D warnings cargo clippy --workspace --all-targets --all-features -- -D warnings </code></pre> <h3>Unit tests (<code>hf-hub</code>)</h3> <pre lang="bash"><code>cargo test -p hf-hub cargo test -p hf-hub --features blocking </code></pre> <h3>Integration tests (<code>integration-tests</code>)</h3> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
6448545a7f
|
deps: bump bytesize from 1.3.3 to 2.4.2 (#2286)
Bumps [bytesize](https://github.com/bytesize-rs/bytesize) from 1.3.3 to 2.4.2. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/bytesize-rs/bytesize/releases">bytesize's releases</a>.</em></p> <blockquote> <h2>bytesize: v2.4.2</h2> <ul> <li>Improve accuracy of parsing large non-decimal byte count strings.</li> </ul> <h2>bytesize: v2.4.1</h2> <ul> <li>Fix rounding error near power-of-unit boundaries.</li> </ul> <h2>bytesize: v2.4.0</h2> <ul> <li>Implement <code>Sum</code> for <code>ByteSize</code>.</li> <li>Minimum supported Rust version (MSRV) is now 1.85.</li> </ul> <h2>bytesize: v2.3.1</h2> <ul> <li>Fix unit truncation in error strings.</li> </ul> <h2>bytesize: v2.3.0</h2> <ul> <li>Add <code>Unit</code> enum.</li> <li>Add <code>UnitParseError</code> type.</li> </ul> <h2>bytesize: v2.2.0</h2> <ul> <li>Add <code>ByteSize::as_*()</code> methods to return equivalent sizes in KB, GiB, etc.</li> </ul> <h2>bytesize: v2.1.0</h2> <ul> <li>Support parsing and formatting exabytes (EB) & exbibytes (EiB).</li> <li>Migrate <code>serde</code> dependency to <code>serde_core</code>.</li> </ul> <h2>bytesize: v2.0.1</h2> <ul> <li>Add support for precision in <code>Display</code> implementations.</li> </ul> <h2>bytesize: v2.0.0</h2> <ul> <li>Add support for <code>no_std</code> targets.</li> <li>Use IEC (binary) format by default with <code>Display</code>.</li> <li>Use "kB" for SI unit.</li> <li>Add <code>Display</code> type for customizing printed format.</li> <li>Add <code>ByteSize::display()</code> method.</li> <li>Implement <code>Sub<ByteSize></code> for <code>ByteSize</code>.</li> <li>Implement <code>Sub<impl Into<u64>></code> for <code>ByteSize</code>.</li> <li>Implement <code>SubAssign<ByteSize></code> for <code>ByteSize</code>.</li> <li>Implement <code>SubAssign<impl Into<u64>></code> for <code>ByteSize</code>.</li> <li>Reject parsing non-unit characters after whitespace.</li> <li>Remove <code>ByteSize::to_string_as()</code> method.</li> <li>Remove top-level <code>to_string()</code> method.</li> <li>Remove top-level <code>B</code> constant.</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/bytesize-rs/bytesize/blob/master/CHANGELOG.md">bytesize's changelog</a>.</em></p> <blockquote> <h2>2.4.2</h2> <ul> <li>Improve accuracy of parsing large non-decimal byte count strings.</li> </ul> <h2>2.4.1</h2> <ul> <li>Fix rounding error near power-of-unit boundaries.</li> </ul> <h2>2.4.0</h2> <ul> <li>Implement <code>Sum</code> for <code>ByteSize</code>.</li> <li>Minimum supported Rust version (MSRV) is now 1.85.</li> </ul> <h2>2.3.1</h2> <ul> <li>Fix unit truncation in error strings.</li> </ul> <h2>2.3.0</h2> <ul> <li>Add <code>Unit</code> enum.</li> <li>Add <code>UnitParseError</code> type.</li> </ul> <h2>2.2.0</h2> <ul> <li>Add <code>ByteSize::as_*()</code> methods to return equivalent sizes in KB, GiB, etc.</li> </ul> <h2>2.1.0</h2> <ul> <li>Support parsing and formatting exabytes (EB) & exbibytes (EiB).</li> <li>Migrate <code>serde</code> dependency to <code>serde_core</code>.</li> </ul> <h2>2.0.1</h2> <ul> <li>Add support for precision in <code>Display</code> implementations.</li> </ul> <h2>v2.0.0</h2> <ul> <li>Add support for <code>no_std</code> targets.</li> <li>Use IEC (binary) format by default with <code>Display</code>.</li> <li>Use "kB" for SI unit.</li> <li>Add <code>Display</code> type for customizing printed format.</li> <li>Add <code>ByteSize::display()</code> method.</li> <li>Implement <code>Sub<ByteSize></code> for <code>ByteSize</code>.</li> <li>Implement <code>Sub<impl Into<u64>></code> for <code>ByteSize</code>.</li> <li>Implement <code>SubAssign<ByteSize></code> for <code>ByteSize</code>.</li> <li>Implement <code>SubAssign<impl Into<u64>></code> for <code>ByteSize</code>.</li> <li>Reject parsing non-unit characters after whitespace.</li> <li>Remove <code>ByteSize::to_string_as()</code> method.</li> <li>Remove top-level <code>to_string()</code> method.</li> <li>Remove top-level <code>B</code> constant.</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
522faa1a59
|
deps: bump rusqlite from 0.32.1 to 0.40.1 (#2287)
Bumps [rusqlite](https://github.com/rusqlite/rusqlite) from 0.32.1 to 0.40.1. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/rusqlite/rusqlite/releases">rusqlite's releases</a>.</em></p> <blockquote> <h2>0.40.1</h2> <h2>What's Changed</h2> <ul> <li>Fix clippy warnings <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1852">#1852</a></li> <li>Bump bundled SQLite version to 3.53.2 <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1853">#1853</a></li> <li>Bump hashlink version <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1855">#1855</a></li> <li>Fix SQL injection when SAVEPOINT name is tainted <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1854">#1854</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/rusqlite/rusqlite/compare/v0.40.0...v0.40.1">https://github.com/rusqlite/rusqlite/compare/v0.40.0...v0.40.1</a></p> <h2>0.40.0</h2> <h2>What's Changed</h2> <ul> <li>Breaking changes: Replace VTab macros by constructors <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1823">#1823</a></li> <li>Breaking changes: Fix VTab::best_index <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1824">#1824</a></li> <li>Asserts on VTab::connect aux and args <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1825">#1825</a></li> <li>Breaking changes: Fix VTab::connect / create <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1826">#1826</a></li> <li>Breaking changes: Allow opting out of using sqlite-wasm-rs on wasm32-unknown-unknown <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1828">#1828</a>, <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1829">#1829</a></li> <li>Derive Default for SeriesTabCursor/ArrayTabCursor <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1830">#1830</a></li> <li>Update link to pre-update hook <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1831">#1831</a></li> <li>Breaking changes: Fix VTab::connect <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1832">#1832</a></li> <li>impl From<!-- raw HTML omitted --> for FromSqlError <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1833">#1833</a></li> <li>Breaking changes: Fix vtab::dequote <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1835">#1835</a></li> <li>Bump bundled SQLCipher to version 4.14.0 <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1837">#1837</a></li> <li>sqlite3_set_errmsg <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1752">#1752</a></li> <li>Bump sqlite3-parser version <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1838">#1838</a></li> <li>Fix UB in ToSqlOutput::from_rc <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1839">#1839</a></li> <li>Ensure miri doesn't complain <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1840">#1840</a></li> <li>Bump to actions/checkout@v6 <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1842">#1842</a></li> <li>Add support to UtcDateTime <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1843">#1843</a>, <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1844">#1844</a></li> <li>Bump bundled SQLite version to 3.53.1 <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1848">#1848</a></li> <li>Replace some cfg(not by cfg_select <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1850">#1850</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/rusqlite/rusqlite/compare/v0.39.0...v0.40.0">https://github.com/rusqlite/rusqlite/compare/v0.39.0...v0.40.0</a></p> <h2>0.39.0</h2> <h2>What's Changed</h2> <ul> <li>Fix constraints on VTab Aux data <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1778">#1778</a>, <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1771">#1771</a></li> <li>Fix docs.rs generation <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1779">#1779</a></li> <li>Fix a small typo in <code>rollback_hook</code> docstring <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1780">#1780</a></li> <li>Fix some warnings from Intellij <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1781">#1781</a></li> <li>Minimal doc for features <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1783">#1783</a></li> <li>Clear hooks only for owning connections <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1785">#1785</a>, <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1784">#1784</a></li> <li>Fix link to SQLite C Interface, Prepare Flags <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1787">#1787</a></li> <li>Comment functions which are not usable from a loadable extension <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1789">#1789</a></li> <li>Factorize code <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1792">#1792</a></li> <li>Update getrandom to 0.4 <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1798">#1798</a></li> <li>Update Cargo.toml <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1800">#1800</a></li> <li>Fix appveyor <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1807">#1807</a></li> <li>Add support to unix timestamp for chrono, jiff and time <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1808">#1808</a>, <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1803">#1803</a></li> <li>fix(trace): check that the sql string pointer is not NULL <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1805">#1805</a></li> <li>Bump bundled SQLite version to 3.51.3 <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1818">#1818</a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
JD Davis
|
f236ef2e31
|
test(ccr): cross SQLite max lifetime boundary (#2794)
## Description
Fixes the failing Rust test on `main` after #2669 made SQLite CCR
entries valid at the exact TTL boundary. The integration test waited
only 3.3 seconds for a three-second ceiling; unix-second truncation can
represent that as exactly three seconds, so the entry is correctly still
valid. The test now crosses a guaranteed four-second elapsed boundary.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Extend the max-lifetime test's access loop from four to five 700 ms
gaps.
- Document why four gaps can land on the valid equality boundary and why
five are deterministic.
- Leave production SQLite TTL behavior and defaults unchanged.
## Testing
- [x] Unit tests pass (`cargo test -p headroom-core --test
ccr_backends`)
- [x] Formatting passes (`cargo fmt --all -- --check`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
cargo test -p headroom-core --test ccr_backends
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
5 consecutive repetitions of the previously failing test:
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 11 filtered out
```
## Real Behavior Proof
- Environment: macOS, Rust workspace at
`
|
||
|
Agistaris
|
d0a86d409f
|
fix(ccr): preserve exact SQLite TTL boundary (#2669)
## Description SQLite CCR timestamps have whole-second resolution. Expiring a row when `last_accessed + ttl == now` or `created_at + max_lifetime == now` can shorten the configured lifetime by almost one second. This change keeps entries valid at the exact boundary and expires them one second later. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Use strict expiration predicates for idle TTL and maximum lifetime. - Keep lookup predicates valid at the exact boundary. - Add deterministic fixed-time tests for both boundaries. ## Testing - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text running 3 tests test ccr::backends::sqlite::tests::exact_max_lifetime_boundary_is_still_valid ... ok test ccr::backends::sqlite::tests::exact_idle_ttl_boundary_is_still_valid ... ok test transforms::code_compressor::tests::english_exact_token_match_unchanged ... ok test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 911 filtered out ``` ## Real Behavior Proof - Environment: Linux x86_64, repository Rust toolchain. - Exact command / steps: `cargo test -p headroom-core exact_` - Observed result: Both SQLite boundary tests returned the stored payload at the exact configured boundary and removed it one second later. The focused command passed 3/3 selected tests, including one unrelated existing exact-token test. - Not tested: Live provider or model traffic; the change is isolated to the deterministic Rust SQLite backend. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project style guidelines - [x] I have performed a self-review of my code - [x] I have commented the boundary behavior - [ ] I have made corresponding documentation changes (not applicable; behavior and tests are local to the backend) - [x] My changes generate no new warnings - [x] I have added tests that prove the fix is effective - [x] New and existing relevant tests pass locally with my changes - [x] I did not edit `CHANGELOG.md` ## Screenshots (if applicable) Not applicable. ## Additional Notes Rust formatting and `headroom-core` Clippy pass. A broader package run passed 994 tests with three ignored; two unrelated ONNX parity tests were excluded after reproducing their pre-existing futex stall. |
||
|
Tejas Chopra
|
3e348f327f
|
fix(ccr): stop persisting retrieval markers as original content (#2694) (#2703)
## Description CCR entries could end up holding a `<<ccr:...>>` marker — or nothing at all — where the original bytes belonged, so `headroom_retrieve(hash)` answered with the very placeholder the caller was trying to resolve. For a base64/credential field that is permanent, silent data loss: the inner marker's hash is the only handle on the real payload, and it disappears from anywhere the model can see. Four sites, one root cause — **a compressed intermediate (or nothing) was stored in place of the source**, the same defect class as #1209 (tag placeholders persisted as originals). Closes #2694 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **`compaction/walker.rs`** — `walk_array` compacted through the store-LESS `compact()`. Opaque cells inside a compacted table got a marker whose payload was **never written**, so retrieval 404'd forever. Now uses `compact_with_store` so the emitted hash resolves. - **`compaction/classifier.rs`** — nothing stopped an already-marked string from being offloaded a second time, which stashed the MARKER as the new entry's "original". Marker-bearing text is our own output, not source content, so it is never classified opaque. One guard at the choke point both the walker and the table compactor share. - **`smart_crusher/crusher.rs`** — on the prose-hook path the row-drop marker hashed and stored rows whose leaves were **already** rewritten (prose compressed, blobs marker-substituted), so retrieving dropped rows returned compressed output. Now hashes and stashes the pre-processing array via `crush_array_with_source`. - **`content_router.py`** — compression pinning matched only `Retrieve more: hash=` / `Retrieve original: hash=`, **not** `<<ccr:`, so opaque-blob output was readmitted to the compressor on a later turn — the path that feeds the corruption above. Consolidated into `_is_already_compressed()` and applied at all three pinning sites. - **`cache/compression_store.py`** — store-level guard: refuse to persist a *bare* marker as `original_content` and log at ERROR, so a future producer regression surfaces loudly instead of silently converting "retrievable" into "gone". Deliberately narrow — originals may legally *contain* markers (nested offloads); only a bare marker is rejected. - **Regression tests** — `test_nested_table_markers_resolve_to_source_bytes` (asserts payloads are verbatim-retrievable, not merely that a marker was emitted) and `test_already_marked_content_is_not_re_offloaded`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ cargo build -p headroom-core Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 37s $ cargo test -p headroom-core --lib smart_crusher test result: ok. 329 passed; 0 failed; 0 ignored; 0 measured; 583 filtered out; finished in 0.19s $ python -m pytest tests/test_transforms/test_smart_crusher_ccr_roundtrip.py -q 16 passed in 0.68s $ python -m pytest tests/test_ccr_row_drop_store_bridge.py tests/test_ccr_tool_injection.py -q 50 passed in 12.52s $ python -m pytest tests/test_compression_store.py tests/test_lossless_mode.py -q 100 passed in 13.14s $ ruff check headroom/transforms/content_router.py headroom/cache/compression_store.py \ tests/test_transforms/test_smart_crusher_ccr_roundtrip.py All checks passed! $ mypy headroom/transforms/content_router.py headroom/cache/compression_store.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - **Environment:** macOS (Darwin 25.4.0), Python 3.12.6, headroom-ai 0.33.0 editable, `HEADROOM_CCR_BACKEND=memory`, Rust extension rebuilt via `maturin develop --release`. - **Exact command / steps:** compact a nested document — 5 rows whose `detail` field is a stringified sub-array of 6 base64 blobs (1600 B each) — then, for every `<<ccr:HASH>>` marker in the output, call `ccr_get(HASH)` and check the payload is the verbatim source rather than a marker. ```python inner = [{"k": f"key{i}", "v": i, "tok": blob(1200)} for i in range(6)] doc = {"rows": [{"id": i, "detail": json.dumps(inner), "note": "x"} for i in range(5)]} out = SmartCrusher().compact_document_json(json.dumps(doc)) for h in re.findall(r"<<ccr:([0-9a-f]+)", out): payload = crusher.ccr_get(h) # must be real bytes, not a marker ``` - **Observed result — BEFORE (on `main`):** all six payloads collapsed into a single dead marker. The rendered sub-table was re-classified opaque (`html`, because `<<` reads as a tag), offloaded again, and its payload never stored — so the six inner hashes were erased from the visible text *and* the outer hash resolved to nothing. ```text {"rows":"[5]{detail:string,id:int,note:string} <<ccr:3fb1d44933da,html,289B>>,0,x <<ccr:3fb1d44933da,html,289B>>,1,x ..."} 3fb1d44933da -> RUST MISS # unrecoverable — 6 × 1600 B gone ``` - **Observed result — AFTER (this branch):** the sub-table stays inline, each blob keeps its own marker, and every marker resolves to verbatim source. ```text {"rows":"[5]{detail:string,id:int,note:string} \"[6]{k:string,tok:string,v:int} key0,\"\"<<ccr:955b1fed2ef7,base64,1.6KB>>\"\",0 ..."} 6ad5846997f4: resolves, len=1600, is-verbatim-source=True 78a0bd9364a7: resolves, len=1600, is-verbatim-source=True 955b1fed2ef7: resolves, len=1600, is-verbatim-source=True a0cef69da7f0: resolves, len=1600, is-verbatim-source=True dfcde5e940c0: resolves, len=1600, is-verbatim-source=True e57c4e0a3ce8: resolves, len=1600, is-verbatim-source=True RESULT: PASS — every marker resolves to real source bytes ``` ## Notes for reviewers - The issue also reports **function words dropped from retained prose** (`is`, `a`, `the`) and **interleaved log output corrupting `headroom doctor`'s table borders**. Those are separate defects on different paths (extractive prose compression and log-handler buffering respectively) and are **not** addressed here — this PR is scoped to the CCR store/retrieve corruption. They should be tracked separately; the prose one overlaps #2586. - The `crusher.rs` prose-hook fix is on the Rust pipeline (`json_offload`) rather than the Python proxy path, but it is the same store-the-intermediate bug and was cheap to close while in the file. |
||
|
Tejas Chopra
|
e0ce4b1d48
|
fix: remove rtk and lean-ctx CLI context tools (#2677)
## Description Removes both third-party CLI context tools — **rtk** and **lean-ctx** — and with them the context-tool selector itself. Headroom no longer downloads, installs or configures either one, and there is no replacement. The previous pass (#2344) gated only three entry points inside `headroom/cli/wrap.py`. That left the feature reachable in practice: | Gap | Effect | |---|---| | `scripts/install.sh:1544`, `install.ps1:1681` | Ran `rtk init --global --auto-patch` from bash/PowerShell, **bypassing the Python gate entirely** — `curl \| sh` still wrote a Claude Code `PreToolUse` hook regardless of `HEADROOM_RTK` | | `wrap.py` `_setup_context_tool_for_agent` | **`wrap openhands` was broken by default**: `rtk_required=True` met a gate returning `None` → `SystemExit(1)`. Invisible because all 8 openhands tests patched `_ensure_rtk_binary` to a fake path | | `proxy/helpers.py`, `subscription/tracker.py` | Proxy shelled out to `rtk gain` from `/stats`, the dashboard and `headroom perf`; the tracker polled it per contribution (`_RTK_WIRING_DEFAULT = "enabled"`) | | No cleanup path | Nothing removed artifacts an earlier default had installed, so a machine that once ran the old default kept rtk in the loop forever (#1669, #1955) | Also worth noting: the rtk binary download had **no SHA or signature verification** — only `rtk --version` as a smoke test. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made **Removed** — `headroom/rtk/` and `headroom/lean_ctx/` packages, `headroom/cli/wrap_rtk_metrics.py`, `_selected_context_tool` / `_setup_context_tool_for_agent` / `_VALID_CONTEXT_TOOLS`, the `--rtk` / `--no-rtk` / `--no-project-rtk` / `--keep-rtk` flags across all 18 wrap subcommands, `HEADROOM_RTK*`, the proxy-side `rtk gain` polling, the dashboard CLI-filtering panel (rows + all 8 `cliFiltering*` Alpine getters), `paths.rtk_path()` / `lean_ctx_path()`, the SDK path helpers, `benchmarks/rtk_loop_learn_eval.py`, and the `headroom/rtk/**` CI path filters. **Fails loudly, not silently** — `--context-tool` / `--no-context-tool` / `HEADROOM_CONTEXT_TOOL` are kept solely to error out. They live in shell profiles, aliases and CI jobs, and accepting them as a no-op would read as Headroom having quietly stopped working. The installers reject them too, which matters more than it looks: their arg parsers forward the first unknown flag **and everything after it** to the wrapped tool, so a leftover `--no-rtk` would have silently swallowed a following `--port` and then been ignored downstream. **New `headroom/context_tool_cleanup.py`** — deleting the code cannot help a machine that already ran the old default, since the hooks, binaries and injected guidance are durable on disk. `purge_context_tool_artifacts()` runs once per `wrap`/`unwrap` and removes the registered hook entries, the generated hook scripts, the Headroom-managed `~/.local/bin` symlinks, the vendored `~/.headroom/bin/{rtk,lean-ctx}` binaries, the `lean-ctx` MCP server entry and the marker-fenced instruction blocks. Deliberately conservative: idempotent, **skips** a malformed config rather than overwriting it, and only unlinks a symlink resolving inside Headroom's own bin dir so a user's own build is untouched. It reports on **stderr**, because `wrap/unwrap openclaw --prepare-only` emit machine-readable JSON on stdout as their entire contract. Skipped for `wrap selfheal` (runs from a SessionStart hook; must not race Claude Code's writer for `~/.claude.json`) and for `--help`, which must stay read-only. **Client-config hardening** (discovered while investigating a "corrupted Serena settings file" report) — `wrap.py` reset a settings file to `{}` when an existing file would not parse, then wrote that back. One hand-edited typo or a transient `EACCES`/`EINTR` on a valid file destroyed the user's `permissions`, `env` and `hooks`, on **every `headroom wrap claude`**. It now refuses to write. Separately, `fsutil.write_text` is now atomic (temp file + `fsync` + `os.replace`), fixing all 14 non-atomic client-config writes at once; it follows symlinks rather than replacing them (dotfile managers) and preserves an existing file's mode. **Deliberately kept** — `rtk` stays in the wrapper-peel list in `transforms/content_router.py`. It sits beside `sudo`/`env`/`timeout` as shell-command grammar, so `rtk cat f` is still classified as a file read for anyone running their own rtk install, which the purge intentionally leaves alone. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check headroom/ tests/ e2e/ --exclude headroom/dashboard/templates All checks passed! $ ruff format --check headroom/ tests/ e2e/ --exclude headroom/dashboard/templates 1255 files already formatted $ mypy headroom/ Success: no issues found in 508 source files $ pytest tests/test_context_tool_cleanup.py -q 11 passed $ pytest tests/test_fsutil.py -q 12 passed $ pytest tests/test_cli/test_wrap_codex.py -q # 89 tests 89 passed in 431.68s $ pytest tests/test_cli/test_wrap_opencode.py -q 39 passed in 257.46s $ pytest tests/test_cli/test_wrap_helpers.py -q 45 passed $ pytest tests/test_paths.py -q 75 passed $ pytest tests/test_cli/test_unwrap_claude.py -q 14 passed $ pytest tests/test_proxy_savings_history.py -q 39 passed $ pytest tests/test_cli/test_wrap_copilot.py -q 27 passed $ pytest tests/test_cli/test_wrap_zcode.py -q 20 passed $ pytest tests/test_subscription_tracker.py -q 9 passed $ pytest tests/test_proxy_dashboard_stats_cache.py -q 5 passed, 1 skipped ``` Repo-wide grep for 14 removed symbols (`headroom.rtk`, `headroom.lean_ctx`, `_ensure_rtk_binary`, `_selected_context_tool`, `_get_context_tool_stats`, `rtk_path`, `lean_ctx_path`, `wrap_rtk_metrics`, `HEADROOM_RTK`, `cli_tokens_avoided`, `tokens_saved_rtk`, …) across `*.py`, `*.ts`, `*.sh`, `*.ps1`, `*.yml`, `*.html`: **zero hits**. Notable test changes: `test_wrap_openhands.py` no longer patches `_ensure_rtk_binary` and asserts `wrap openhands --prepare-only` exits 0 unpatched — the regression that was previously masked. `test_wrap_continue.py` and `test_wrap_hintfile_agents.py` were removed (every test drove RTK instruction injection). A new `test_subscription_tracker.py::test_load_state_written_before_cli_context_tools_were_removed` proves a pre-removal `subscription_state.json` still loads. ## Real Behavior Proof - **Environment:** macOS 15.4 (darwin 25.4.0), Python 3.12.6, Headroom @ this branch, real `~/.headroom` and `~/.claude` on the dev machine. - **Exact command / steps and observed result:** ```text # 1. Retired flag fails loudly instead of silently no-op'ing $ headroom wrap codex --prepare-only --context-tool rtk Error: CLI context tools (rtk, lean-ctx) have been removed from Headroom: they rewrote shell commands through a third-party binary Headroom no longer manages. Drop --context-tool / --no-context-tool and unset HEADROOM_CONTEXT_TOOL; `headroom wrap` uninstalls what they left behind on first run. $ HEADROOM_CONTEXT_TOOL=lean-ctx headroom wrap codex --prepare-only Error: CLI context tools (rtk, lean-ctx) have been removed from Headroom: ... # 2. install.sh rejects the retired flags (extracted parse_wrap_args harness) ['--no-rtk', '--port', '9999'] rc=1 ERROR: CLI context tools ... Drop --no-rtk ['--context-tool=rtk'] rc=1 ERROR: CLI context tools ... Drop --context-tool $ bash -n scripts/install.sh # syntax OK # 3. Purge ran against the real machine, which had all the orphaned artifacts $ python -c "from headroom.context_tool_cleanup import purge_context_tool_artifacts; ..." removed ~/.headroom/bin/lean-ctx (51 MB) removed ~/.headroom/bin/rtk (7.7 MB) removed ~/.local/bin/rtk (symlink into ~/.headroom/bin) removed ~/.claude/hooks/rtk-rewrite.sh removed 8 lean-ctx-* hook scripts # ~/.claude.json afterwards: 90 top-level keys, 19 projects, mcpServers unchanged # → ~59 MB reclaimed, no unrelated key touched # 4. stdout stays machine-readable while the purge reports (planted a fake artifact) $ headroom wrap openclaw --prepare-only --gateway-provider-id codex >out 2>err $ cat out {"enabled":true,"config":{"proxyPort":8787,...}} # parses as JSON $ cat err Retired CLI context tool cleanup: removed /Users/tcms/.headroom/bin/rtk # 5. --help is inert (planted artifact survives), a real run purges $ headroom wrap codex --help → artifact survived: CORRECT $ headroom wrap openclaw --prepare-only → purged: CORRECT # 6. MCP purge dry-run against a copy of the real 82 KB ~/.claude.json top-level keys 90 -> 90; projects 19 -> 19; LOST keys: none all content outside mcpServers byte-identical: True ``` Dashboard rendered via the Playwright test after the panel removal: "Token Savings" shows only `Proxy 0 (0.0%)` / `Of total wire: 36.86%`, and "Token Usage" reads Before Compression → Proxy Removed → After Compression with no "Filtered (this session)" row. Nothing below the removed panel broke. - **Not tested:** Windows and Linux (macOS only) — `install.ps1` is verified by brace-balance and inspection, not executed, since no `pwsh` is available locally. The wrap e2e suite (`e2e/wrap/run.py`) was updated but not run; it needs the Docker e2e image. `serena project index` interaction is exercised in the stacked base PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes **Stacked on #2676** (`tejas/serena-config-bootstrap`) — please merge that first; this PR's base should then be retargeted to `main`, or it will read as containing that fix too. **Breaking-change migration for users:** - Drop `--rtk`, `--no-rtk`, `--no-project-rtk`, `--keep-rtk`, `--context-tool`, `--no-context-tool` from any alias, script or CI job, and unset `HEADROOM_RTK*` / `HEADROOM_CONTEXT_TOOL`. They now error rather than being ignored, so the failure is immediate and self-explaining. - Previously-installed artifacts are purged automatically on the next `wrap`/`unwrap`; no manual cleanup needed. - `headroom perf --json` no longer carries a `cli_filtering` key, and `/stats` no longer returns a `context_tool` section. **Docs:** `docs/rtk-architecture.md` deleted; RTK/lean-ctx removed from `README.md`, `docs/content/docs/{configuration,opencode,grok-build,docker-install,filesystem-contract}.mdx`, `docs/observability.md` and the matching `wiki/` pages. `REALIGNMENT/09-phase-G-rtk-observability.md` is marked SUPERSEDED rather than deleted, to keep the planning record. **Follow-ups not in scope:** `_emit_wrap_interrupted` was deleted as dead code — its only caller was the `except KeyboardInterrupt` guarding the binary download, so with no download there is nothing slow left to interrupt. |
||
|
Zhenjia ZHOU
|
e825588bfb
|
fix(ccr): sliding idle-window TTL with max-lifetime ceiling in the Rust core backends (#2604) (#2631)
## Description
Rust-core counterpart of the CCR mid-session expiry fix. #2604 (and its
duplicate #2616) report that the 30-minute wall-clock TTL kills entries
in the middle of a normal multi-agent burst: the clock starts at
compression time and never refreshes, so an entry the session keeps
touching still dies.
#2607 fixes this on the Python side by turning the TTL into an idle
window that restarts on every successful retrieval, bounded by an
absolute max lifetime (8x the idle TTL) — but it explicitly notes the
caveat that the Rust core still measures TTL from insertion. This PR
closes that gap: all three Rust CCR backends (`InMemoryCcrStore`,
`SqliteCcrStore`, `RedisCcrStore`) now use the same sliding idle-window
+ max-lifetime-ceiling semantics as the Python `CompressionStore`.
Scoped to the Rust core only; it deliberately does not touch
`DEFAULT_TTL`'s value (1800), which #2607 bumps to 3600 — happy to
rebase in lockstep whichever lands first.
Refs #2604, #2616. Complements #2607.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `crates/headroom-core/src/ccr/mod.rs`:
`DEFAULT_MAX_LIFETIME_MULTIPLIER = 8` + `max_lifetime_for()` helper;
documents the idle-window semantics.
- `in_memory.rs`: entries track `last_accessed`; a hit refreshes it
under the shard write lock (`get_mut`), expiry checks idle window OR max
lifetime, and the existing `remove_if` TOCTOU protection now uses the
same predicate. New `with_capacity_and_ttls` constructor for independent
control of window and ceiling.
- `sqlite.rs`: new `last_accessed` column (legacy DBs migrated in place
via `ALTER TABLE`, backfilled from `created_at` so old rows keep their
original expiry baseline); lazy purge and the lookup honour both bounds;
a hit touches the row under the same connection mutex as the read. New
`open_with_ttls` constructor.
- `redis.rs`: a hit re-arms the key's expiry, capped by a companion
`{prefix}:{hash}:born` key whose remaining TTL marks the absolute
ceiling; entries written by pre-sliding builds (no born key) are
backfilled rather than dropped.
- `tests/ccr_backends.rs`: 6 new tests — sliding-window survival and
max-lifetime cap for in-memory and SQLite, legacy-schema migration, and
a gated Redis sliding test.
No public API is broken: existing constructors keep their signatures and
derive the ceiling as 8x the idle TTL.
## Testing
- [x] Unit tests pass (`cargo test -p headroom-core`)
- [x] Linting passes (`cargo clippy -p headroom-core --all-features`,
`cargo fmt --check`)
- [ ] Type checking passes (`mypy headroom`) — N/A, no Python files
touched
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ cargo test -p headroom-core --test ccr_backends
test result: ok. 12 passed; 0 failed; 0 ignored (8.12s)
$ cargo test -p headroom-core ccr # all ccr-named tests across suites
38 passed, 949 filtered out (13 suites)
$ cargo test -p headroom-core --test ccr_roundtrip --test live_zone_ccr
18 passed (2 suites)
$ cargo check -p headroom-core --features redis # cfg-gated backend compiles
Finished `dev` profile in 25.09s
$ cargo clippy -p headroom-core --all-features
No issues found
```
## Real Behavior Proof
- Environment: macOS (Darwin 25.5), local checkout at upstream `main`
(
|
||
|
Zhenjia ZHOU
|
e86c6390ce
|
fix(rust): port CJK-aware relevance-query matching to CodeCompressor (#2634)
## Description The Rust port of `CodeCompressor` (#1154, parity-only) did not carry over the CJK-aware relevance-query matching from `headroom/transforms/code_compressor.py` (`_CONTEXT_DELIMS` / `_CJK_CHARS` / `_query_context_tokens()` / `_symbol_in_context()`, lines 2353-2387, called from lines 987/1009): - Rust tokenized the context with an ASCII-only delimiter class `[\s,;:.()\[\]{}"']+`, so a CJK query (no spaces, CJK punctuation) collapses into a single blob and never isolates an ASCII symbol name. - The substring-fallback guard `chars().count() > 3` had no CJK relaxation, so a short ASCII name glued to CJK text (e.g. `run` in `修复run函数的报错`, `db` in `请保留db相关的逻辑`) could never receive the +3.0 context boost — while Python does boost it. Same `(code, context)` input, different `symbol_scores`. This PR ports the two Python helpers with identical semantics: - `query_context_tokens()` — delimiter class extended with the CJK/full-width punctuation and ideographic space from Python's `_CONTEXT_DELIMS`; returns `(words, lowered, has_cjk)` with CJK detection over U+3000-U+9FFF, U+AC00-U+D7AF, U+FF00-U+FFEF (Python's `_CJK_CHARS`). - `symbol_in_context()` — exact token match, plus the substring fallback gated by `> 3` **characters** (Python `len()`, not bytes), relaxed when the query contains CJK. The call site in `analyze_symbols` now uses these helpers; no other behavior changed. Pure-ASCII query behavior is identical to before (exact token match, `>3`-gated substring fallback), which the tests pin down. Closes #2630 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `crates/headroom-core/src/transforms/code_compressor.rs`: extract `query_context_tokens()` / `symbol_in_context()` free functions mirroring the Python helpers (CJK/full-width delimiter class, CJK detection, CJK-relaxed `>3`-character guard); replace the inline ASCII-only tokenization + guard in `analyze_symbols` with calls to them. - Unit tests mirroring `tests/test_transforms/test_code_compressor_cjk.py` case-for-case, plus a character-vs-byte guard test and an end-to-end `compress_with` test asserting `symbol_scores`. ## Testing - [x] Unit tests pass (`cargo test -p headroom-core` — Rust-only change; Python untouched) - [x] Linting passes (`cargo fmt --check`, `cargo clippy -p headroom-core --all-targets` — no new warnings) - [ ] Type checking passes (`mypy headroom`) — N/A, no Python changes - [x] New tests added for new functionality - [x] Manual testing performed New tests (all in `code_compressor.rs` `mod tests`): - `cjk_query_isolates_wrapped_ascii_symbol` — full-width parens isolate `parse_config` - `cjk_query_matches_short_ascii_name_glued_to_cjk` — `db` (len 2) glued to CJK matches via the relaxed guard - `english_short_name_substring_still_gated` — `db` vs "keep the database helper" must NOT match (ASCII guard unchanged) - `english_exact_token_match_unchanged`, `english_long_name_substring_fallback_unchanged`, `empty_context_matches_nothing` - `guard_counts_chars_not_bytes` — the guard is a character count, matching Python `len()` - `cjk_context_boosts_named_symbol_end_to_end` — full `compress_with` run asserting `symbol_scores` (red on main, green here — see proof) ### Test Output ```text $ cargo test -p headroom-core --lib -- code_compressor::tests test transforms::code_compressor::tests::empty_and_short_passthrough ... ok test transforms::code_compressor::tests::empty_context_matches_nothing ... ok test transforms::code_compressor::tests::estimate_tokens_uses_chars_div_4_min_1 ... ok test transforms::code_compressor::tests::py_round3_matches_cpython ... ok test transforms::code_compressor::tests::py_round_int_is_half_to_even ... ok test transforms::code_compressor::tests::cjk_query_isolates_wrapped_ascii_symbol ... ok test transforms::code_compressor::tests::english_short_name_substring_still_gated ... ok test transforms::code_compressor::tests::cjk_query_matches_short_ascii_name_glued_to_cjk ... ok test transforms::code_compressor::tests::english_exact_token_match_unchanged ... ok test transforms::code_compressor::tests::english_long_name_substring_fallback_unchanged ... ok test transforms::code_compressor::tests::guard_counts_chars_not_bytes ... ok test transforms::code_compressor::tests::cjk_context_boosts_named_symbol_end_to_end ... ok test transforms::code_compressor::tests::detect_language_basic ... ok test result: ok. 13 passed; 0 failed; 0 ignored; 0 measured; 899 filtered out; finished in 0.05s $ cargo test -p headroom-core # per-binary summaries lib .......................... ok. 911 passed; 0 failed; 1 ignored auth_mode .................... ok. 16 passed; 0 failed cache_control ................ ok. 14 passed; 0 failed ccr_backends ................. ok. 7 passed; 0 failed ccr_roundtrip ................ ok. 15 passed; 0 failed code_compressor_parity ....... ok. 1 passed; 0 failed (recorded byte-parity fixtures) live_zone_ccr ................ ok. 3 passed; 0 failed live_zone_dispatch ........... ok. 6 passed; 0 failed live_zone_thresholds ......... ok. 2 passed; 0 failed live_zone_token_validation ... ok. 3 passed; 0 failed recommendations_loader ....... ok. 4 passed; 0 failed tokenizer_proptest ........... ok. 5 passed; 0 failed doc-tests .................... ok. 1 passed; 0 failed; 2 ignored $ cargo fmt --check # clean $ cargo clippy -p headroom-core --all-targets # no new warnings ``` ## Real Behavior Proof - Environment: macOS (Darwin 25.5.0), repo-pinned Rust toolchain (`rust-toolchain.toml`), branch based on current `main`. - Exact command / steps: the end-to-end test was written first and run against unmodified `main` (red), then after the fix (green). Input: Python source with two signal-symmetric functions `run` and `keep`; context `修复run函数的报错`. The Python reference gives `run` the boost (`_symbol_in_context('run', ...) == True`, `_symbol_in_context('keep', ...) == False`, verified against the live Python implementation), so expected normalized scores are `run = 1.0`, `keep = 0.0`. - Observed result: on unmodified main the end-to-end test fails (`left: 0.5, right: 1.0` — the CJK query `修复run函数的报错` gives `run` no boost, both symbols collapse to 0.5, while Python scores `run=1.0, keep=0.0`); on this branch all 8 new tests pass and the same query boosts `run` to 1.0, matching Python. Full output: Before (unmodified `main` + new test only — Rust gives no boost, both symbols collapse to 0.5): ```text ---- transforms::code_compressor::tests::cjk_context_boosts_named_symbol_end_to_end stdout ---- thread '...cjk_context_boosts_named_symbol_end_to_end' panicked at crates/headroom-core/src/transforms/code_compressor.rs:1890:9: assertion `left == right` failed: run must get the context boost left: 0.5 right: 1.0 test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 904 filtered out ``` After (this branch): the same test passes, including its ASCII control case (`fix the runner` must NOT boost `run` — scores stay 0.5/0.5), proving pure-ASCII behavior is unchanged. The recorded byte-parity fixture suite (`code_compressor_parity`) also still passes. - Not tested: real proxy traffic end-to-end (change is confined to the symbol-scoring context boost inside the Rust compressor; the Python implementation is the behavioral reference and is untouched). `kompress_parity` was not run locally — it is model-gated and my sandbox blocks the model fetch; it is unrelated to this change and CI covers its skip path. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation — N/A (internal behavior fix) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes For background, the Python-side CJK handling comes from the merged CJK sweep work (#2220 and follow-ups); #1154 predates part of it, which is likely how the port missed it. Longer names wrapped in full-width punctuation happened to still match in Rust via the substring fallback, but the token set itself was wrong; this PR restores exact-token semantics for those too. |
||
|
Ruben A.
|
e530de5ad2
|
feat(rust): port CodeCompressor AST compressor to Rust (parity-only) (#1154)
Adds crates/headroom-core/src/transforms/code_compressor.rs (1,882 lines): the AST-aware CodeCompressor ported to Rust on tree-sitter, with grammars for Python, JavaScript, TypeScript, Go, Rust, Java, C and C++. Parity-only, like #1153. Nothing calls it: the only references outside the module are the pub mod / pub use declarations in transforms/mod.rs, and live_zone.rs still routes SourceCode to a no-op. The pyo3 bridge is untouched and no Python source changes, so the engine is unreachable from the shipped package. #1155 wires it into live-zone dispatch. Every grammar is pinned with '=' to the exact version of the corresponding Python tree-sitter-<lang> PyPI wheel. Same version on crates.io and PyPI means the same grammar.js, hence the same generated parser.c, hence node-for-node identical ASTs — the precondition for byte-parity. A canary over 9 samples x 8 languages confirmed identical node-type and line-span trees at these pins; bumping any pin requires re-running it and re-recording the fixtures. Ships 30 recorded parity fixtures, a CodeCompressorComparator in headroom-parity, and scripts/record_code_compressor_fixtures.py. Verified byte-identical to the recorded Python output: [code_aware_compressor] total=30 matched=30 skipped=0 diffed=0 Full harness on the merge result: 227 fixtures, 182 matched, 45 skipped (cache_aligner + ccr stubs), 0 diffed, exit 0 — with kompress at 21/21 under ONNX Runtime 1.24.4 (see #2591). Also verified cargo check -p headroom-core --no-default-features passes, so the static-musl path stays intact. |
||
|
Ruben A.
|
83e27e5036
|
feat(rust): port Kompress ML prose compressor to Rust (parity-only) (#1153)
Adds crates/headroom-core/src/transforms/kompress.rs (672 lines): the Kompress ML prose compressor ported to Rust, running the ModernBERT tokenizer plus the kompress-v2-base ONNX model through ort with a cache-only loader that never touches the network. Parity-only. Nothing calls it: the only references outside the module are the pub mod / pub use declarations in transforms/mod.rs. live_zone.rs still carries TODO(PR-B4), so PlainText dispatch remains a no-op and Python continues to serve prose compression. The pyo3 bridge is untouched and no Python source changes, so the new engine is unreachable from the shipped package. #1155 wires it up. Ships 21 recorded parity fixtures, a KompressComparator in headroom-parity, and scripts/record_kompress_fixtures.py. Verified byte-identical to the recorded Python output: [kompress] total=21 matched=21 skipped=0 diffed=0 That required ONNX Runtime >= 1.24 (see #2591) — below it ort deadlocks instead of erroring, which is why these fixtures had never been run. In CI the model is absent from the HF cache, so the comparator errors and the fixtures report Skipped rather than hanging. Also gates the module behind the ml feature, matching magika_detector: kompress.rs uses ort, which is optional = true, so an unconditional pub mod broke cargo check --no-default-features (the static-musl path). CI does not catch that class of break because cargo test --workspace only builds default features. |
||
|
Tejas Chopra
|
fd6abac87f
|
parity: promote log_compressor from stub to a real comparator (#2568)
Un-blinds the 20 recorded log_compressor fixtures, which reported Skipped since Phase 0. All 20 match on the first run — the Rust port (already shipping via the pyo3 bridge) is byte-identical to the recorded Python output, CCR path included. Two non-obvious details in the adapter: - bias. Python's signature is compress(content, context="", bias=1.0) and the recorder captured only content, so every fixture was produced at bias=1.0. - CCR store. Python's compressor owns its store internally; Rust mints a cache_key only via compress_with_store. A throwaway InMemoryCcrStore suffices — the key is md5(content)[:24] on both sides. Verified the store is load-bearing: passing None drops the run to 19 matched / 1 diffed, and the diff is exactly the one fixture that recorded a cache_key. Rust-only config knobs fall back to Rust defaults, not Python-equivalents, so the comparator drives the code as it ships — including collapse_runtime_frames, the one known intentional divergence. Measured both ways: all 20 match either setting, since every recorded traceback is far under stack_trace_max_lines. Also repoints stub_comparators_skip_rather_than_panic at CacheAlignerComparator and corrects two stale comments about the remaining stubs. Harness: total=176 matched=131 skipped=45 diffed=0. |
||
|
Tejas Chopra
|
c15e557da1
|
ci(parity): make the parity harness a real per-PR gate (#2567)
Three hardening steps on the Rust-vs-Python parity harness: - Drop the dead maturin/venv step. headroom-parity has no pyo3 dependency, so the venv requirement, the `maturin develop` rebuild, and the CI job's Python toolchain were all overhead. Verified no-op: identical report, exit 0. - Register a text_crusher comparator. 6 recorded fixtures were invisible because the transform was missing from builtin_comparators(); parity-run only walks directories it has a comparator for. All 6 match on the first run. - Promote parity to a blocking per-PR gate. Safe to harden now because parity-run exits non-zero only on a Diff, so the 65 still-stubbed fixtures report Skipped and cannot turn it red. Harness: total=176 matched=111 skipped=65 diffed=0, exit 0. Deliberately not widening the path filter to Python paths: the fixtures are frozen recordings of Python output and the harness never invokes Python, so it measures Rust-vs-snapshot and a Python edit cannot move the result. |
||
|
Rod Boev
|
9e0778553f
|
feat(rust): add structured prose offload plumbing (#334) (#2378)
## Description Structured payloads still leave long prose leaves without a dedicated prose compressor. The Rust pipeline already handles top-level log, diff, search, and JSON-array shapes, and the existing structured recursion rewrites stringified JSON and opaque blobs, but a plain prose string leaf inside structured content still falls back to generic opaque long-string handling instead of query-aware extractive compression. That wastes prompt budget on fields like `summary`, `description`, and `analysis` even though `headroom-core` already ships the deterministic, query-aware `TextCrusher`. This PR adds a bounded prose-field path for structured leaves. It introduces a reusable `ProseFieldOffload` backed by `TextCrusher`, then wires that offload into `JsonOffload`'s structured recursion with conservative byte and segment thresholds. Only detector-confirmed `PlainText` leaves are eligible. When a leaf clears those gates and the marker-inclusive output still saves bytes, the exact original leaf is written to CCR and the inline output carries a prose marker keyed to that store entry. Short prose, low-segment prose, diff-shaped strings, stringified JSON, and opaque base64 or HTML keep their existing behavior. This stays inside the Rust transform stack. It does not add a PyO3 shim, ONNX runtime, live-zone prose handling, or any new Python dependency. It also keeps the existing wrapper-level `JsonOffload` CCR entry, so the full structured payload remains recoverable as before. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `ProseFieldOffload` as a `ContentType::PlainText` pipeline offload backed by `TextCrusher`, with conservative byte, segment, and target-ratio thresholds. - Thread the prose offload into the structured `JsonOffload` recursion so nested prose leaves can compress and recover through the orchestrator store. - Add a pipeline-aware `JsonOffload::from_pipeline` constructor so `offload.prose_field` overrides actually reach the live prose hook instead of falling back to embedded defaults. - Preserve current behavior for short prose, low-segment prose, diff-shaped leaves, stringified JSON containers, and opaque base64 or HTML leaves. - Add focused config, routing, determinism, and CCR roundtrip coverage for the new prose path. - Leave changelog generation to the repo's conventional-commit release flow rather than editing `CHANGELOG.md` directly. ## Testing - [x] Unit tests pass (`cargo test -p headroom-core --lib transforms::pipeline::offloads::prose_field::tests`) - [x] Linting passes (`cargo clippy -p headroom-core -- -D warnings`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text cargo fmt --all -- --check cargo clippy -p headroom-core -- -D warnings cargo test -p headroom-core --lib transforms::pipeline::offloads::prose_field::tests test result: ok. 6 passed; 0 failed cargo test -p headroom-core --lib transforms::pipeline::offloads::json_offload::tests test result: ok. 17 passed; 0 failed cargo test -p headroom-core --lib transforms::smart_crusher::crusher::tests::default_crush_ignores_opt_in_prose_hook -- --exact test result: ok. 1 passed; 0 failed cargo test -p headroom-core --lib transforms::smart_crusher::crusher::tests::prose_hook_preserves_html_opaque_routing -- --exact test result: ok. 1 passed; 0 failed cargo test -p headroom-core --lib transforms::smart_crusher::crusher::tests::prose_hook_runs_for_dict_array_rows -- --exact test result: ok. 1 passed; 0 failed cargo test -p headroom-core --lib transforms::smart_crusher::crusher::tests::unchanged_stringified_json_container_skips_prose_hook -- --exact test result: ok. 1 passed; 0 failed cargo test -p headroom-core --test ccr_roundtrip nested_structured_prose_leaf_uses_ccr -- --exact test result: ok. 1 passed; 0 failed git diff --check ``` ## Real Behavior Proof - Environment: Windows 11, stable Rust toolchain, in-memory CCR store, no live provider - Exact command / steps: run the focused nested CCR roundtrip test through `CompressionPipeline::run` on a five-row structured payload containing a long prose leaf, then resolve the emitted prose marker key from the same orchestrator store - Observed result: the generic `CompressionPipeline` plus `JsonOffload` path applies, the nested prose leaf becomes shorter on the wire, and that prose key retrieves the byte-identical original leaf from the orchestrator store while HTML-shaped and diff-shaped leaves stay on their opaque marker routes - Not tested: live provider run ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - The upstream issue body originally parked PR3b behind a PyO3 shim or a later ONNX port. This PR takes the narrower Rust-native path instead by reusing the existing `TextCrusher` already in `headroom-core`. - This PR advances the pipeline-side PR3b slice from #334. It does not close #334, and it does not wire live-zone or PyO3 SmartCrusher callers to this path. - `CHANGELOG.md` is intentionally untouched because this repo's release pipeline generates changelog entries from conventional commits, and `repos/headroom/config.md` marks manual changelog edits as out of policy. - Python lint, type checking, and pytest are not part of the focused local proof for this slice because the change stays inside `crates/headroom-core`. |
||
|
Andrei Boldyrev
|
6744833afe
|
fix(proxy): key drift detector on conversations, not credentials; canonicalize drift hashes (#2301)
## Description
The Rust proxy's cache-bust drift detector
(`crates/headroom-proxy/src/cache_stabilization/drift_detector.rs`,
PR-E6) cannot currently tell drift from normal operation on interactive
agentic traffic, so it warns on nearly every turn and a real bust drowns
in the noise. Three compounding defects, all verified against live
Claude Code traffic:
1. `derive_session_key` stops at the credential hash — Claude Code sends
one OAuth bearer for every conversation, so all concurrent conversations
share one LRU slot and every conversation switch logs a false
`cache_drift_observed` (with `drift_dims` computed against the wrong
conversation's baseline).
2. The `early_messages` axis hashes the raw first-3-messages window, so
a lone conversation's normal growth (1 → 3 messages) and the client
relocating its `cache_control` breakpoint to the newest block both fire
a false `early_messages` drift at turn 2–3 of essentially every session.
3. `x-headroom-session-id` — the explicit session identity the Python
proxy honors everywhere session-sticky state exists — is ignored on the
Rust path.
This PR makes the detector's session identity conversation-scoped and
its comparison canonical, the same shape as the merged Python-side fix
for #2085 (`SessionTrackerStore.resolve_tracker` lineage resolution +
`_canonicalize_for_prefix_compare`):
- **`derive_session_key`**: honors `x-headroom-session-id` first
(hashed, like every other key input), then folds a conversation
discriminator into the credential/network arms: a 16-hex-char SHA-256
fingerprint of `(model, canonicalized first message)`. Provider prompt
caches are per-model, so a small-model sidecar call (title generation)
that reuses a conversation's opener stays a separate session instead of
false-drifting on `system`.
- **`canonicalize_for_hash`** on all axes and the discriminator: objects
rebuilt with sorted keys (this workspace enables serde_json
`preserve_order`, so a plain re-serialize would keep client wire order
and leave the hashes key-order sensitive) and `cache_control` stripped
outside opaque tool payloads (`input`/`arguments`/`json`/`input_schema`
— mirroring the Python canonicalizer's `_OPAQUE_PAYLOAD_KEYS`, so a user
field that happens to be *named* `cache_control` still counts as drift).
- **`early_messages`** becomes per-message hashes (`[Option<[u8; 32]>;
3]`) with a prefix-aware comparison: growing into the window is benign;
a settled message changing or disappearing under a stable session key is
still drift. `observe_drift` now gates the warning on drifted dimensions
rather than raw hash inequality.
True positives are preserved (`system`/`tools` changes, in-place history
rewrites under a pinned identity), and the detector remains a pure
observer — no forwarded byte changes, `does_not_mutate_input` still pins
that.
**Documented trade-off** (module doc + `conversation_discriminator`
doc): without the explicit header, a client that rewrites its first
message (history compaction, rolling-window truncation, Responses
chained mode) re-keys to a fresh session — the rewrite surfaces as
`cache_drift_first_request` rather than `cache_drift_observed` against
the old baseline. That is deliberate: the credential-keyed alternative
false-warned on every conversation switch, which buried those same
events anyway. `x-headroom-session-id` pins the identity and reports
rewrites as drift. Byte-identical openers on the same model under one
credential still conflate (rare; documented).
Closes #2300
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `derive_session_key`: `x-headroom-session-id` (hashed) wins;
credential/IP arms fold in `conversation_discriminator` — `(model,
canonicalized first message)`, 16 hex chars
- New `canonicalize_for_hash`: sorted-key object rebuild +
`cache_control` stripped outside `OPAQUE_PAYLOAD_KEYS`; applied to the
`system`/`tools`/`early_messages` axes and the discriminator
- `StructuralHash.early_messages`: `[u8; 32]` → `[Option<[u8; 32]>;
EARLY_MESSAGES_WINDOW]` per-message hashes;
`drift_dims`/`early_window_drifted` implement the prefix-aware rule;
`observe_drift` warns on non-empty dims instead of `!=`
- `conversation_messages` shape guard: bare-string message containers
only count for the Responses `input` sugar
- Docs: module header (canonicalization, trade-off, honest cost),
`conversation_discriminator` rationale + blind spots,
`DRIFT_DETECTOR_CAPACITY` cardinality note (per-conversation keys,
163-byte entry), `structural_hash_log_prefix` hex-length fix
- Tests: 13 new unit tests (conversation separation, turn-growth key
stability, explicit header priority, marker relocation + growth not
drift, rewrite/shrink still drift, per-model separation, key-order
neutrality, opaque-payload fields still count, Responses/Chat
discriminator shapes, string-container gating)
- `CHANGELOG.md`: Unreleased → Fixed entry
## Testing
- [x] Unit tests pass (`cargo test -p headroom-proxy` — full crate: lib
+ integration suites)
- [x] Linting passes (`cargo clippy -p headroom-proxy --all-targets` —
zero warnings; `cargo fmt --check` clean)
- [ ] Type checking passes (`mypy headroom`) — n/a, no Python files
touched
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ cargo test -p headroom-proxy --lib drift_detector
test result: ok. 27 passed; 0 failed; 0 ignored; 0 measured; 221 filtered out
$ cargo test -p headroom-proxy
(all suites) test result: ok. 248 passed (lib) + integration suites, 0 failed
$ cargo clippy -p headroom-proxy --all-targets
(no warnings)
```
## Real Behavior Proof
- Environment: macOS 15 (arm64), rustc 1.95.0, repo @
|
||
|
Zhenjia ZHOU
|
844d9caaa1
|
feat(text-crusher): fold full-width ASCII to half-width in CJK token keys (#2259)
## Description Real CJK content routinely mixes full-width and half-width forms (`ＡＰＩ` vs `API`, `０` vs `0`, the ideographic space ` ` vs a normal space). After #1504's ICU tokenization, these width variants produced *different* token keys, so `ＡＰＩ` and `API` didn't dedup or match as the same term on the CJK relevance and near-duplicate paths. This folds full-width ASCII (U+FF01–U+FF5E, via the U+FEE0 offset) and the ideographic space (U+3000 → space) to their half-width forms **when building the internal token key** inside `tokens_icu`. Only the token key is normalized — the kept output stays byte-verbatim, so TextCrusher's extractive / byte-faithful contract is preserved. CJK-gated (`tokens_icu` is the CJK path); the ASCII path is untouched. ## Type of Change - [x] Bug fix / enhancement (non-breaking) ## Changes Made - `crates/headroom-core/src/transforms/text_crusher/crusher.rs`: a `width_fold(c)` helper applied when building token keys in `tokens_icu`. - Rust unit tests: full-width ASCII folds to half-width in token keys; CJK segments split on full-width terminators. ## Testing - [x] Unit tests pass (`cargo test`) - [x] Linting passes (`cargo clippy` / `cargo fmt`) - [x] New tests added ### Test Output ```text $ cargo test -p headroom-core --lib text_crusher test result: ok. 13 passed; 0 failed $ cargo clippy / fmt # clean ``` ## Real Behavior Proof - Environment: macOS (Darwin), Rust via cargo, branch `feat/text-crusher-fullwidth-fold` off `main` (rebased after #1504 merged). - Exact command / steps: `cargo test -p headroom-core --lib text_crusher`. - Observed result: `fullwidth_ascii_folds_to_halfwidth` confirms `ＡＰＩ` and `API` now produce the same token key (so they dedup/relevance-match as one term); `cjk_splits_on_full_width_terminators` confirms full-width `！`/`？` terminate segments. All 13 text_crusher tests pass; the kept output is byte-verbatim (only the internal key is folded). - Not tested: no Python side — TextCrusher is Rust-only, and output stays byte-verbatim, so no parity fixtures change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation — N/A - [x] My changes generate no new warnings - [x] I have added tests that prove my change is effective - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — happy to add an entry if preferred. ## Additional Notes - Follow-up to #1504 (CJK-aware TextCrusher); it only touches the CJK-gated `tokens_icu` path, so English tokenization is unchanged. |
||
|
Zhenjia ZHOU
|
4035c04187
|
feat(text-crusher): CJK-aware segmentation + relevance via ICU (#1504)
## Description `TextCrusher` (the native extractive prose compressor added in #1171) only handled ASCII: `split_segments` split on `.!?`+whitespace and `tokens` split on whitespace/alphanumeric runs. CJK (Chinese/Japanese/Korean) has neither spaces nor ASCII terminators, so a whole CJK paragraph collapsed into **one segment / one token** — it passed through at ~0% compression, and BM25 relevance + salience scored zero terms. This makes `TextCrusher` CJK-aware. CJK-bearing content takes an ICU (`icu_segmenter`, UAX#29 sentence + dictionary word) segmentation path, with a length fallback for terminator-sparse runs, a local BM25 over the ICU word tokens, and ICU-token salience. Dispatch is on **content only**, so pure-ASCII text is byte-identical to before — the shared `BM25Scorer` and the ASCII path are untouched. It also adds a committed, reproducible answer-retention eval (`benchmarks/i18n_compression_eval.py`) with a deterministic zh/ja/ko CI regression gate, so the improvement below is permanently verifiable rather than a one-off measurement. Extends #1171. ## Type of Change - [x] Bug fix (CJK passed through near-uncompressed) - [x] New feature (CJK segmentation / relevance support) - [x] Performance improvement (CJK now compresses; ICU segmenters cached, not rebuilt per call) ## Changes Made - `is_cjk` predicate gates a CJK path (ideographs, kana, Hangul, CJK punctuation, full/half-width forms). - `split_segments` → ICU `SentenceSegmenter` for CJK + a mandatory length fallback (whitespace / CJK punctuation / hard cap) for terminator-sparse runs; ASCII path unchanged. - `tokens` → ICU `WordSegmenter` (dictionary) for CJK; ASCII path unchanged. - `relevance_cjk`: a local BM25 over ICU word tokens — the shared ASCII `BM25Scorer` scores zero terms for CJK and is parity-locked, so this is an intentional separate scorer (documented in code). - CJK salience uses ICU tokens (whitespace-split gave one giant "word" → zero salience). - `count_tokens`: CJK-aware so `compression_ratio` isn't nonsense for space-free text. - ICU segmenters resolved once in `static LazyLock` (compiled_data is static) instead of rebuilt per call. - New dep `icu_segmenter` 2.2, `compiled_data` only (see Dependency below). - `benchmarks/i18n_compression_eval.py` + `tests/test_transforms/test_text_crusher_cjk_eval.py`: a zh/ja/ko answer-retention eval — a deterministic needle CI gate (always-runs, no external data), real-transcript fidelity with CJK-aware salient, and optional `multi-wiki-qa` natural-data retention (loaded via the `[evals]` `datasets` extra, skipped if absent; data never vendored — CC-BY-NC-SA). ## Testing - [x] Unit tests pass (`pytest` + `cargo test`) - [x] Linting passes (`ruff check`/`format` on the new eval + test — clean) - [ ] Type checking passes (`mypy headroom`) — N/A, the only Python added is a benchmark + test, not `headroom/` source - [x] New tests added for new functionality - [x] Manual testing performed (see Real Behavior Proof) ### Test Output ```text $ cargo test -p headroom-core --lib text_crusher running 12 tests test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 841 filtered out $ .venv/bin/python -m pytest tests/test_transforms/test_text_crusher*.py 15 passed $ .venv/bin/python -m pytest tests/test_transforms/test_text_crusher_cjk_eval.py 6 passed # deterministic zh/ja/ko needle CI gate $ cargo clippy -p headroom-core && ruff check benchmarks/i18n_compression_eval.py # both clean ``` ## Real Behavior Proof - Environment: macOS (Darwin 25.3.0), Python in a uv venv, `headroom-core` built via `uv pip install -e .` (maturin), branch `feat/cjk-text-compression`. - Exact command / steps: built `_core`, then ran a mixed Chinese+Japanese doc (no spaces, `。` terminators) through `TextCrusher().compress(doc, "认证令牌缓存策略", 0.3)`; separately evaluated answer-retention on the public CMRC2018 Chinese QA dev set (bury the gold-answer paragraph among 25 distractors, query = the question, compress to 30%, check the gold answer survives), and end-to-end through `ContentRouter`. - Observed result: a mixed Chinese+Japanese doc compressed 189 → 78 tokens (ratio 0.41, kept 3/8 segments) with the query-relevant sentence surviving — before this change the same doc was a single segment → 100% passthrough. On the public CMRC2018 Chinese QA dev set, answer-retention under 30% compression rose 34% → 93% (multiple seeds). End-to-end through `ContentRouter` on real CJK content, aggregate savings rose 16% → 40%. Pure-ASCII (English) output stayed byte-identical (the English parity fixtures did not move). Demo terminal output: ```text ORIGINAL tokens= 189 chars=189 COMPRESS tokens= 78 ratio=0.41 segments kept 3/8 QUERY-RELEVANT sentence survived: True --- compressed output (verbatim kept CJK sentences) --- 认证令牌的缓存策略采用最近最少使用淘汰算法来管理过期条目。 请求重试使用指数退避并设置最大次数上限。 数据备份每天凌晨执行并保留最近三十天的快照。 ``` The committed eval now demonstrates this across all three CJK languages. The deterministic needle gate (in CI via `tests/test_transforms/test_text_crusher_cjk_eval.py`, 6 passed) has TextCrusher keep the query-relevant needle while truncate/random drop it in zh, ja, and ko. On real `multi-wiki-qa` natural data (n=80/lang), query-aware answer-retention is **zh 74% / ja 70% / ko 50%** vs **25–41%** for the truncate/random baselines: ```text === Part A: multi-wiki-qa answer-retention (n=80/lang, target_ratio=0.3) === lang text_crusher truncate random zh-cn 74% 25% 38% ja 70% 31% 39% ko 50% 26% 41% ``` Korean is measurably weaker (ICU has no Korean dictionary and falls back to UAX#29 word-breaking) — still well above baselines, and scoped as a follow-up. - Not tested: the live proxy HTTP path (validated at the `ContentRouter` / `TextCrusher` layer, not via a running proxy); no-space Korean (standard Korean is space-delimited and is covered); non-CJK SE-Asian scripts (out of scope). ## Dependency (per CONTRIBUTING supply-chain policy) `icu_segmenter` 2.2 (ICU4X), `features = ["compiled_data"]`: - **Why this package (vs. ourselves / existing deps):** CJK needs dictionary/UAX#29 segmentation. A hand-rolled char-bigram scored slightly worse on real data (CMRC2018 answer-retention: 92.5% ICU vs 91% bigram, 4 seeds); jieba/lindera are ZH-only or 13–207 MB dicts. ICU4X covers zh/ja/ko in one crate. The existing `unicode-segmentation` does UAX#29 only (no CJK dictionary), so it can't word-segment space-free CJK. - **Who maintains it:** the official `unicode-org` ICU4X project; active release cadence (2.2 in 2025); no known CVEs. - **Install surface:** ~13 new pure-Rust crates, no build scripts, no native code, no build/runtime network. `compiled_data` bundles locale data at compile time (hermetic). `auto`/`lstm` deliberately NOT enabled — LSTM covers SE-Asian scripts (Thai/Lao), not CJK, and would pull in `libm` for nothing. - **Why this version:** 2.x is the stabilized ICU4X API (1.x used a different data-provider model); floored at 2.2 (Cargo.lock pins the patch) since segmenter boundaries are observable in output and bumps should be deliberate. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (CHANGELOG) - [x] My changes generate no new warnings (clippy + fmt clean) - [x] I have added tests that prove my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md ## Additional Notes - **Parity:** the shared `BM25Scorer` (byte-exact parity-locked with `headroom/relevance/bm25.py`) is untouched. `relevance_cjk` is a separate local scorer because the shared one's tokenizer is ASCII-only. The whole CJK path lives in Rust (`text_crusher.py` is a thin wrapper over `_core`), so there is no Python mirror to keep in sync; the parity fixtures stay green (only the CJK `unicode` fixture was re-recorded, intentionally; English fixtures unchanged). - **Known by-design gap (not a bug):** CJK content + a pure-ASCII query yields no token overlap, so relevance falls back to recency + salience (cross-script query matching is unsupported). - The Python added is a benchmark (`benchmarks/i18n_compression_eval.py`) plus its test, not `headroom/` runtime source — both are `ruff`-clean; `mypy headroom` is unaffected. - **License:** the optional Part A pulls `alexandrainst/multi-wiki-qa` (CC-BY-NC-SA-4.0) at run time via the `[evals]` extra and is skipped if absent — the dataset is never vendored into the repo, and the always-run CI gate (Part C) uses only our own deterministic data. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
mstattma
|
3757a7cef3
|
fix(core): avoid unidiff panic on bash xtrace (#1506)
## Summary
- preflight unified diff inputs before calling the Rust unidiff parser
- catch parser panics so malformed-but-diff-looking input falls back to
non-diff
- add regressions for Bash xtrace lines like `+++ test.sh` and `+++
dirname test.sh`
## Repro
`detect_content_type("+++ test.sh")` could panic through the Rust
detector because unidiff treated the lone `+++` line as a target header
without a preceding source header.
## Tests
- `cargo fmt --all --check`
- `cargo test -p headroom-core --lib
transforms::unidiff_detector::tests`
- `cargo test -p headroom-core --lib`
Co-authored-by: Michael Stattmann <mstattma@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
Ashish
|
6469fcd018
|
feat(transforms): language-aware stack-trace collapse for Go/Rust/.NET/Java/Node (#1791)
## Description
Stack-trace handling covered only Python tracebacks and a generic ` at
symbol(` pattern. Go panics and Rust panics flowed to prose compression;
.NET traces were unrecognized; Java chained exceptions split into
separate traces at every `Caused by:` (so later chain heads fell off the
`max_stack_traces` cliff); and oversized traces were blindly
head-truncated — keeping runtime scheduler noise while dropping the app
frames and chain heads an agent actually needs.
This PR adds language-aware trace flavors (Go, Rust, .NET, Java chains,
Node async) to the Rust core and both Python mirrors, and replaces blind
truncation with a runtime-frame collapse: message lines, chain heads,
the trace head, and app-code frames survive; contiguous runtime/stdlib
frames fold into `[... N frames collapsed]` markers. A 147-line Go panic
dump compresses to 19 lines with the panic message, signal line, and app
frame intact.
Note one intentional behavior change: now that Go/Rust panics are
*detected*, panics ≤8KB in tool outputs gain the existing error-output
protection (`protect_error_outputs`) they previously missed — small
panics stay verbatim, exactly like small Python tracebacks already do.
## Type of Change
- [x] New feature (non-breaking change which adds functionality)
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring
## Changes Made
- `crates/headroom-core/src/transforms/log_compressor.rs`:
- New `TraceFlavor::GoPanic` (`panic:` / `fatal error:` / `goroutine N
[state]:` openers, tab-indented `.go:` frame lines, `created by` /
call-line continuation, blank-separated goroutine blocks) and
`TraceFlavor::DotNet` (`Unhandled exception.`, `at … ) in file:line N`
frames — checked before Java since those frames also satisfy the Java
shape — plus `--->` inner-exception heads and `--- End of` separators).
- Renamed the misnamed `Go` flavor to `RustBacktrace` (its `is_go_frame`
matched `N: 0x<hex>` — the Rust backtrace shape) and gave it real
openers (`thread '…' panicked at`, `stack backtrace:`); the free-text
panic-message line after the opener stays in the trace (`terminates` now
receives `lines_so_far`).
- Java: continues across `Caused by:` / `Suppressed:` / `... N more`,
and `is_java_at_frame` admits `/` so JPMS module frames (`at
java.base/…`) pass the opener re-check — without this, modern JDK traces
fragmented at the parse cap into ≤20-line groups.
- Frame collapse (`collapse_trace_frames`): for traces over
`stack_trace_max_lines`, keeps message/chain-head lines, first
`trace_head_frames` frames, up to `trace_app_frames` app frames; runtime
frames (prefix + path marker tables per language) fold into `[... N
frames collapsed]` markers that occupy the run's first line slot and
carry score 0.8 so the global cap doesn't drop them first. Collapsed
frame indices are excluded from the context-line pass (otherwise ±3
context re-added them), and the parse cap re-opens on continuation lines
so selection sees one contiguous trace. New config:
`collapse_runtime_frames=true`, `trace_head_frames=3`,
`trace_app_frames=5`; new sidecar stat `runtime_frames_collapsed`.
- `crates/headroom-py/src/lib.rs`: the three new knobs on the
`LogCompressorConfig` PyO3 signature.
- `headroom/transforms/log_compressor.py`: dataclass fields +
constructor pass-through; `_parse_lines` opener patterns mirrored per
the documented contract.
- `headroom/transforms/content_detector.py`: `_LOG_PATTERNS` additions
(Go panic/goroutine/frame lines, Rust panic/backtrace/numbered frames,
.NET, Java chain heads, Node `at async`); JS/Java `at` pattern admits
JPMS module paths.
- `tests/test_transforms_stack_traces.py` (new, 10 tests) + 7 new Rust
unit tests (flavor open/continue/terminate, chain grouping, collapse
keeps chain heads/app frames, collapse-off comparison, small traces
untouched).
## Testing
- [x] Added new tests for the changes
- [x] All existing tests pass
### Test Output
```
$ cargo test -p headroom-core
928 passed; 3 ignored
$ python -m pytest tests/test_transforms_stack_traces.py tests/test_log_compressor.py \
tests/test_transforms_log_compressor.py tests/test_transforms_content_detection.py \
tests/test_transforms_content_router.py tests/test_transforms/test_content_router.py \
tests/test_lossless_mode.py tests/test_compression_fidelity_regression.py -q
191 passed
```
## Real Behavior Proof
- Environment: macOS arm64, Python 3.13, repo main @
|
||
|
Matthew Jackson
|
cdba2eccdd
|
feat(core): gate ONNX transforms behind a default-on ml feature (static/lexical builds) (#2165)
## Description `TextCrusher` and the BM25 relevance path can run without the ONNX-backed ML stack, but `headroom-core` previously compiled `ort`, `fastembed`, and `magika` unconditionally. This made lexical-only downstream consumers carry the ONNX Runtime dependency even when they never used embedding relevance or Magika detection. This PR makes those ML crates optional behind a new default-on `ml` Cargo feature. Default builds keep the existing ML-backed behavior. Consumers that only need lexical compression can opt out with `default-features = false`; in that mode the ML modules are compiled out and the relevance path falls back to BM25. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `crates/headroom-core/Cargo.toml`: marks `ort`, `fastembed`, and `magika` optional; adds default-on `ml = ["dep:ort", "dep:fastembed", "dep:magika"]`. - `crates/headroom-core/src/lib.rs`: gates the shared ONNX CPU helper behind `ml`. - `crates/headroom-core/src/relevance/embedding.rs`: gates the fastembed implementation behind `ml` and provides a no-ml stub with the same scorer surface so `HybridScorer` naturally falls back to BM25. - `crates/headroom-core/src/transforms/detection.rs`: gates the Magika tier behind `ml`; no-ml builds start at the existing unidiff/plain-text fallback tiers. - `crates/headroom-core/src/transforms/mod.rs`: gates the Magika module and re-exports behind `ml`. ## Testing - [x] Default build compiles (`cargo build -p headroom-core`) - [x] Lexical-only build compiles (`cargo build -p headroom-core --no-default-features`) - [x] Default tests pass (`cargo test -p headroom-core`) - [x] Lexical-only tests pass (`cargo test -p headroom-core --no-default-features`) - [x] Dependency tree checked for no-ml build (`cargo tree -p headroom-core --no-default-features` contains no `fastembed`, `magika`, or `ort` packages) - [ ] Manual testing performed ## Real Behavior Proof - Environment: Windows 11 review worktree, Rust/Cargo workspace. - Exact command / steps: - `cargo build -p headroom-core` - `cargo build -p headroom-core --no-default-features` - `cargo test -p headroom-core` - `cargo test -p headroom-core --no-default-features` - `cargo tree -p headroom-core --no-default-features` - Observed result: both feature configurations build and test cleanly. The no-default dependency tree does not include `fastembed`, `magika`, or `ort`, while the default build still compiles the ML path. - Not tested: model-backed `RUN_FASTEMBED_TESTS=1` cases that require downloading the embedding model; those remain env-gated as before. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes The no-ml build intentionally degrades embedding relevance to the existing unavailable-model behavior, so `HybridScorer` takes its BM25 fallback path. Magika detection is skipped when `ml` is disabled; detection then proceeds through unidiff and plain-text fallback tiers. --------- Co-authored-by: Matthew Jackson <mattjackson86@gmail.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Zhenjia ZHOU
|
528517cff8
|
fix(diff-compressor): CJK-aware relevance scoring for hunk selection (#2220)
## Description `score_hunks` boosts diff hunks whose content overlaps the query/context (+`SCORE_CONTEXT_WORD_WEIGHT` per match); the resulting score decides which hunks survive when `max_hunks_per_file` fires. It split the context on whitespace, so a spaceless CJK query became one blob that only matched a hunk containing the whole query verbatim — relevant hunks weren't boosted and got dropped. This adds CJK character bigrams to the query match set so a CJK query boosts the hunks it overlaps. Rust-only (`diff_compressor.py` is a thin shim over Rust; hunk scoring lives only in Rust). CJK-gated: for a pure-ASCII query `cjk_bigrams` returns an empty set and the new loop is a no-op, so non-CJK scoring is byte-identical and the 20 diff parity fixtures stay green. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `crates/headroom-core/src/transforms/diff_compressor.rs`: add `is_cjk_char` + `cjk_bigrams`, and a separate loop in `score_hunks` that boosts hunks containing each CJK query bigram. The existing ASCII word loop is untouched. - Rust unit test (`cjk_bigrams` extraction) + an end-to-end test (a CJK query promotes the overlapping hunk into the kept set; the no-query baseline drops it). ## Testing - [x] Unit tests pass (`cargo test`) - [x] Linting passes (`cargo clippy` / `cargo fmt`) - [x] New tests added for new functionality - [x] Manual testing performed (see Real Behavior Proof) ### Test Output ```text $ cargo test -p headroom-core --lib diff_compressor test result: ok. 23 passed; 0 failed $ cargo clippy -p headroom-core # clean ``` ## Real Behavior Proof - Environment: macOS (Darwin), Rust via cargo, branch `feat/diff-compressor-cjk` off `main`. - Exact command / steps: `cargo test -p headroom-core --lib diff_compressor` — the `cjk_query_boosts_matching_hunk_into_kept_set` test builds a diff with 4 hunks (first / plain / cjk / last), `max_hunks_per_file = 3` (one contested middle slot between the plain hunk at change-density `0.12` and the CJK hunk at `0.06`), and compresses it once with the CJK context `数据库连接超时排查` and once with no query. - Observed result: with no query the higher-density plain hunk takes the slot (the CJK hunk `数据库连接失败重试` is dropped); with the CJK context its bigrams (`数据` / `据库` / `库连` / `连接`) match → score `0.06 + 4×0.2 = 0.86` beats the plain hunk's `0.12` → the CJK hunk survives. Both directions are asserted; before this change the spaceless CJK query matched neither hunk and the CJK hunk was always dropped. - Not tested: the Python side — `diff_compressor.py` is a thin shim that delegates `compress()` straight to Rust, so hunk scoring has no Python twin; and no new parity fixtures were recorded, since the 20 existing diff fixtures contain no CJK and therefore stay byte-identical. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation — N/A (internal scoring) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A: internal relevance-scoring fix, no user-facing surface change ## Additional Notes - Completes the relevance-scorer CJK sweep across the compressors (search, adaptive sizer, the shared BM25 tokenizer, and now diff). Rust-only — no Python parity mirror is needed because diff hunk scoring has no Python twin (the shim delegates `compress()` straight to Rust). |
||
|
Zhenjia ZHOU
|
5de12f75e3
|
docs(ccr): correct stale 5-minute TTL hints to 30 minutes (#2224)
## Description The CCR store default TTL is `DEFAULT_TTL = 1800s` (30 minutes — see `crates/headroom-core/src/ccr/mod.rs` and `config.py store_ttl_seconds=1800`), but several user-facing hints and docstrings still said "5 minutes", the old default. The opencode/openclaw retrieve tools surfaced `(default TTL: 5 minutes)` in their expiry hint — exactly the misleading message reported in #1023. (The CCR cache itself works; the row-drop store bridge that populates the retrieve store landed for #389.) This corrects the two plugin hints, the `InMemoryCcrStore` docstrings, the SQLite/backend default TTL comments, and the `smart_crusher` mirror comment. The `mod.rs` comment that references "the *old* 5-minute default" is intentionally left unchanged — it correctly describes history. ## Type of Change - [x] Documentation update ## Changes Made - `plugins/openclaw/src/tools/headroom-retrieve.ts` + `plugins/opencode/src/retrieve.ts`: retrieve-failure hint `5 minutes` → `30 minutes`. - `crates/headroom-core/src/ccr/backends/in_memory.rs`: two docstrings (`5 minutes by default`, `5-minute TTL`) → `30 minutes` / `30-minute`. - `crates/headroom-core/src/ccr/backends/mod.rs` + `sqlite.rs`: SQLite/default backend TTL comments `5-minute` → `30-minute`. - `headroom/transforms/smart_crusher.py`: mirror comment `defaults to 5 minutes` → `30 minutes`. ## Testing - [x] Linting passes (`ruff` / `cargo check`) - [x] Manual verification (see Real Behavior Proof) ### Test Output ```text $ ruff format --check headroom/transforms/smart_crusher.py # clean $ cargo check -p headroom-core # Finished, no errors ``` ## Real Behavior Proof - Environment: macOS (Darwin), branch `feat/ccr-ttl-hint-fix` off `main`. - Exact command / steps: grepped every `5 minutes` / `5-minute` TTL reference across the repo; confirmed the real default is `DEFAULT_TTL = Duration::from_secs(1800)` (`ccr/mod.rs:66`), that `InMemoryCcrStore::new()` uses `DEFAULT_TTL` (not a local 300s), and that `config.py` sets `store_ttl_seconds = 1800 # 30 minutes`. - Observed result: all stale CCR default-TTL "5 minutes" references now read "30 minutes"; the one historical reference (`mod.rs`: "the old 5-minute default") is left as-is because it is accurate. - Not tested: nothing runtime changed — these are docstring/comment/hint string edits only, so there is no behavior to exercise. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code — N/A (this PR is comments/strings) - [x] I have made corresponding changes to the documentation (this *is* the doc change) - [x] My changes generate no new warnings - [ ] I have added tests — N/A (no behavior change) - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A: user-facing hint/docstring correction, no functional change ## Additional Notes - Surfaced while root-causing #1023: the "cache permanently empty / TTL: 5 minutes" report is resolved on `main` (the store-bridge for #389 populates the retrieve store), but the stale "5 minutes" strings the reporter actually saw were still in the tree. This PR fixes those. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Doyeon Baek
|
36577d9547
|
fix(search_compressor): don't let a date in a path hijack the line-number parse (#2084)
## Description
`SearchCompressor::parse_match_line` splits a grep/ripgrep line into
`(file, line_number, content)` by finding the **leftmost**
`<sep><digits><sep>` triplet, where `<sep>` is `:` or `-`. A path
segment that itself contains such a triplet hijacks the parse — and that
shape is everyday, not exotic:
| real ripgrep line | parsed as |
|---|---|
| `logs/2026-05-03/app.log:12:ERROR boom` | `("logs/2026", 5,
"03/app.log:12:ERROR boom")` |
| `advisories/CVE-2021-44228.md:8:Log4Shell` | `("advisories/CVE", 2021,
"44228.md:8:Log4Shell")` |
| `src/v1-2-beta/mod.rs:3:fn x()` | `("src/v1", 2, "beta/mod.rs:3:fn
x()")` |
| `migrations/20240101-002-add_users.sql-9-…` | `("migrations/20240101",
2, "add_users.sql-9-…")` |
**This is silent corruption, not a drop.** The parse *succeeds*, so the
line is never counted in `stats.lines_unparsed` and never falls back to
passthrough. The bogus path becomes the **grouping key** in
`parse_search_results`, so unrelated files collapse into one bucket, and
the bogus path + line number + mangled body are what get scored, capped,
and rendered into the compressed output handed to the model. **The LLM
is shown a file and a line that do not exist.**
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
One file, one function:
`crates/headroom-core/src/transforms/search_compressor.rs`.
`parse_match_line` becomes a 3-tier scan:
- **Colon tier** — leftmost `:\d+:` whose path part contains no
whitespace. `:` is grep's *match* separator and a path practically never
contains one (the Windows drive colon is already skipped by the existing
`scan_start` logic), so leftmost is right. The whitespace bound stops a
`foo.rs:12:` reference *inside the body* of a `-` context line from
hijacking the parse.
- **Dash tier** — **last** `-\d+-` whose path part contains no
whitespace. `-` is grep's *context* separator, and unlike `:` it
genuinely appears inside real paths (`2026-05-03`, `CVE-2021-44228`,
`20240101-002-…`), so the marker is the *last* triplet in the path
token, not the first.
- **Permissive tier** — the original leftmost-any rule, byte-for-byte
unchanged. Only reached when neither typed tier matched (e.g. a path
containing a space), so those lines behave exactly as before.
- Also tightened in the typed tiers: the closing separator must equal
the opening one — grep emits `file:12:body` or `file-12-body`, never a
mix.
- Added 4 tests: 2 reproducing the bug, 2 regression guards against the
naive fixes.
**Safety argument (verified by execution):** with `parse_match_line`
temporarily forced to the Permissive tier alone, all 18 pre-existing
`search_compressor` tests still pass — i.e. the fallback is a faithful
reproduction of today's rule, so the change can only *add* correct
parses on lines a typed tier claims, never remove one.
This is the next bug in a family the module already tracks: the doc has
a "Bug fixes vs Python" section and three `fixed_in_3e2_*` tests
hardening this same parser against Windows drive colons and dashes in
filenames. `pre-commit-config.yaml-42-…` (dash before a *non*-digit) is
covered; `2026-05-03` (dash before a digit run followed by another dash)
was not.
## Testing
- [x] Unit tests pass
- [x] Linting passes (`cargo clippy -p headroom-core --all-targets` → 0
warnings)
- [x] Formatting passes (`cargo fmt --all -- --check`)
- [x] New tests added (4: 2 reproducing the bug, 2 regression guards
against naive fixes)
- [x] Manual testing performed (see Real Behavior Proof)
### Test Output
**Before the fix** (new tests run against the unmodified scan rule):
```text
$ cargo test -p headroom-core --lib search_compressor
---- transforms::search_compressor::tests::date_stamped_path_is_not_misread_as_line_number_marker stdout ----
assertion `left == right` failed
left: Some(("logs/2026", 5, "03/app.log:12:ERROR boom"))
right: Some(("logs/2026-05-03/app.log", 12, "ERROR boom"))
---- transforms::search_compressor::tests::date_stamped_paths_are_not_collapsed_into_one_bogus_file stdout ----
assertion `left == right` failed
left: ["logs/2026"]
right: ["logs/2026-05-03/app.log", "logs/2026-05-04/app.log"]
test result: FAILED. 18 passed; 2 failed; 0 ignored
```
**After the fix:**
```text
$ cargo test -p headroom-core --lib search_compressor
test result: ok. 20 passed; 0 failed; 0 ignored; 835 filtered out
$ cargo test -p headroom-core --lib # whole crate — no regressions
test result: ok. 854 passed; 0 failed; 1 ignored
$ cargo test -p headroom-parity
test result: ok. 4 passed; 0 failed
$ cargo fmt --all -- --check -> OK
$ cargo clippy -p headroom-core --all-targets -> 0 warnings, 0 errors
```
Regression guards added for the two ways a naive fix breaks:
- `digit_terminated_path_still_parses_ripgrep_context_line` —
`logs/app.log.1-42-rotated line` (path ends in a digit, so the context
separator is digit-preceded).
- `body_line_reference_does_not_hijack_a_context_line` —
`src/main.py-44-see foo.rs:12:bar` (body quotes a `file:line:`
reference).
## Real Behavior Proof
Per CONTRIBUTING — unit tests alone don't prove user-visible behavior,
so this was reproduced against the **released build** (`headroom-ai`
0.26.0 from PyPI, the compiled `_core.abi3.so`), driving the **public
`SearchCompressor.compress()` API** on **real `rg` output over real
files on disk** — not fixtures or mocks.
- Environment: macOS (Darwin 25.5.0, arm64), Python 3.13, released
`headroom-ai` 0.26.0 (`site-packages/headroom/_core.abi3.so`); patched
build = this branch compiled with `cargo build --release -p
headroom-py`, rustc 1.96.0.
- Exact command / steps: created 20 real log files at
`logs/2026-05-01/app.log` … `logs/2026-05-20/app.log` (12 real `ERROR`
lines each); ran `rg -n ERROR logs > rg_big.txt` (240 real match lines);
then called
`SearchCompressor(SearchCompressorConfig()).compress(open("rg_big.txt").read())`
on the shipped 0.26.0 build and on the patched build, comparing
`files_affected`, the rendered output, and whether each referenced path
exists on disk.
- Observed result: on shipped 0.26.0, the 20 distinct real files
collapse into **1 bogus bucket** `logs/2026` (a path that does **not**
exist on disk), per-line paths are mangled to
`logs/2026:5:01/app.log:10:`, 19 of 20 files effectively vanish from the
output, and `lines_unparsed: 0` means **nothing signals the
corruption**. On the patched build, same input and same API:
`files_affected: 20` (matches reality), every path in the compressed
output exists on disk (`all_exist=True`), and per-file match counts and
line numbers are correct.
- Not tested: the end-to-end proxy path (`headroom-proxy` against a live
LLM provider) — I exercised the `SearchCompressor` public API directly,
which is the surface `SearchOffload` and the MCP `headroom_compress`
tool wrap. I also did not test Windows path behavior on an actual
Windows host (the existing `scan_start` drive-letter logic is untouched,
and its tests still pass).
**Observed on the SHIPPED 0.26.0 build (the bug, in the released
product):**
```text
SHIPPED headroom 0.26.0 | real `rg -n ERROR logs` output, 240 lines
lines_unparsed : 0 <-- corruption is SILENT: nothing reported as unparsed
original_match_count: 240
files_affected : 1 <-- 20 distinct real files collapsed into ONE bucket
=== compressed output actually handed to the model ===
logs/2026:5:01/app.log:10:ERROR failure 1 connection refused upstream timeout on 2026-05-01 ...
logs/2026:5:20/app.log:21:ERROR failure 12 connection refused upstream timeout on 2026-05-20 ...
logs/2026:5:01/app.log:11:ERROR failure 2 connection refused upstream timeout on 2026-05-01 ...
[... and 235 more matches in logs/2026]
[240 matches compressed to 5. Retrieve more: hash=39c894009014d42b856ddd8a]
=== do the file paths in that output exist on disk? ===
logs/2026 exists_on_disk=False
```
**Observed on the PATCHED build (same input, same API, only the patch
differs):**
```text
PATCHED headroom-core | same real `rg` output, 240 lines
lines_unparsed : 0
original_match_count: 240
files_affected : 20 <-- was 1 (bogus) on the shipped build
=== compressed output handed to the model ===
logs/2026-05-01/app.log:10:ERROR failure 1 connection refused upstream timeout on 2026-05-01 ...
logs/2026-05-01/app.log:11:ERROR failure 2 connection refused upstream timeout on 2026-05-01 ...
[... and 7 more matches in logs/2026-05-01/app.log]
logs/2026-05-02/app.log:10:ERROR failure 1 connection refused upstream timeout on 2026-05-02 ...
=== do the file paths in that output exist on disk? ===
logs/2026-05-01/app.log exists_on_disk=True
logs/2026-05-02/app.log exists_on_disk=True
...all distinct paths referenced, all_exist=True
```
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
## Additional Notes
**Known residual ambiguity (stating it rather than hiding it).** grep
output is inherently ambiguous — `logs/2026-05-03/x:12:y` *could*
legitimately be a file literally named `logs/2026` with context line 5.
The tiers pick the overwhelmingly more likely reading. Two contrived
cases still parse the old way, both preserved deliberately:
1. a path containing a whitespace character;
2. a `-`-context line whose body is a whitespace-free token containing
its own `-N-` triplet.
If you'd prefer a different disambiguation policy (e.g. only trusting
`:` and treating all `-` context lines as unparseable, or gating on
filesystem existence), I'm happy to rework — the tiering is deliberately
isolated to one function so the policy is easy to swap.
N/A checklist items: no documentation or CHANGELOG change (internal
parser fix, no public API or behavior contract change); no screenshots
(no UI surface).
---------
Signed-off-by: dosthcpp <drakedog19@gmail.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Parideboy
|
c46cd8f950
|
fix(core): load ONNX Runtime dynamically so headroom._core imports on non-AVX2 x86-64 (#1715)
## Description
`import headroom._core` dies with SIGILL (`Illegal instruction`) on
x86-64 CPUs without AVX2 (Pentium N4200, Celeron N4500, AMD FX 8350 —
all reported on the issue). The repo sets no `RUSTFLAGS`/`target-cpu`
anywhere, so first-party Rust code is baseline x86-64; the AVX2 code
comes from Microsoft's prebuilt ONNX Runtime, statically linked into the
extension by fastembed's `ort-download-binaries-rustls-tls` feature on
non-Windows targets. Because it is statically linked, its code is mapped
and initialized when the extension module loads — **before** the runtime
AVX2 guard from #1162 can run, which is why that fix helped Magika init
but not the import-time crash.
Fix, mirroring what Windows already does for its own reasons (DirectML
link libs): build with `ort-load-dynamic` on every platform, so ONNX
Runtime is only `dlopen`'d at first use, where the #1162 AVX2 guard
falls back to the non-ONNX detection tiers on unsupported CPUs. Since
both target blocks became identical, they are collapsed into one
platform-independent `fastembed` dependency.
To keep Magika/fastembed working out of the box on Linux/macOS, the
existing `ORT_DYLIB_PATH` auto-pin (`headroom/_ort.py`, previously
Windows-only) now resolves the pip `onnxruntime` package's shared
library on all platforms (`onnxruntime.dll` / `libonnxruntime.so*` /
`libonnxruntime*.dylib`). The pip `onnxruntime` CPU wheels use runtime
CPU dispatch, so they also work on pre-AVX2 machines — non-AVX2 users
get working ML detection instead of a crash. Without the `onnxruntime`
package, ML detection degrades gracefully to the non-ONNX tiers exactly
as it already does on Windows.
Fixes #1278
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `crates/headroom-core/Cargo.toml`: replaced the per-target `fastembed`
blocks (`ort-download-binaries-rustls-tls` on non-Windows,
`ort-load-dynamic` on Windows) with a single platform-independent
dependency on `ort-load-dynamic`, with a comment documenting both the
DirectML and the AVX2/#1278 rationale.
- `Cargo.lock`: regenerated — `ort-sys` drops its static-download
dependencies (`hmac-sha256`, `lzma-rust2`, `ureq`); no version bumps.
- `headroom/_ort.py`: `ORT_DYLIB_PATH` auto-pin extended from
Windows-only to all platforms via a small `_find_dylib` helper that
resolves the platform's shared-library name inside the pip `onnxruntime`
package.
- `tests/test_transforms/test_ort_dylib.py`: replaced the obsolete
`test_noop_on_non_windows` with Linux (versioned `.so`) and macOS
(`.dylib`) pin tests; module docstring updated.
- `docs/content/docs/configuration.mdx`: `ORT_DYLIB_PATH` row updated
from Windows-only wording to the cross-platform behavior.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ cargo fmt --all -- --check && cargo clippy --workspace -- -D warnings && cargo test -p headroom-core --lib
clean
test result: 844 passed; 0 failed; 1 ignored
$ python -m pytest tests/test_transforms/test_ort_dylib.py -q
8 passed
$ ruff check headroom/_ort.py tests/test_transforms/test_ort_dylib.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11 (AVX2-capable — the SIGILL itself is not
reproducible on this machine), Python 3.13, Rust 1.95.0, local checkout
branched from `upstream/main` (
|
||
|
JD Davis
|
f00833654f
|
fix(proxy): satisfy rustfmt import ordering (#2158)
## Summary Fixes the Rust workflow failure from https://github.com/headroomlabs-ai/headroom/actions/runs/29294598252/job/86965269576 by applying rustfmt's import ordering in `crates/headroom-proxy/src/proxy.rs`. ## Testing ```text cargo fmt --all -- --check # passed git diff --check # no output ``` |
||
|
Abhishek Mittal
|
52a024d28c
|
fix(proxy): strip [1m] model suffix before upstream forwarding (#2027)
## Description Scopes the `[1m]` context-window tier suffix sanitizer to Anthropic `/v1/messages` requests only (addresses PR #2027 review feedback). The original patch applied the rewrite to every buffered compressible endpoint, which would have silently mutated OpenAI Chat Completions and OpenAI Responses request model IDs. The `[1m]` marker is an Anthropic/Claude Code compatibility signal emitted by the Headroom CLI; the existing Python parity behavior (`sanitize_anthropic_model_id()`) is Anthropic-specific and must not leak onto OpenAI shapes. Refactors the helper into `compression::sanitize_anthropic_model_id_in_body`, drops the dead `sanitize_model_id` helper in `sse/anthropic.rs`, and adds 8 unit tests + 5 wiremock-backed integration tests that pin the scope. All 420 `headroom-proxy` tests pass; `cargo fmt` and `cargo clippy -D warnings` clean. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Move `sanitize_request_model_id` out of `proxy.rs` and into `compression::sanitize_anthropic_model_id_in_body` (Anthropic-specific name; private `trim_anthropic_model_id_suffix` helper for unit-testable pure behavior). - Gate the call site on `CompressibleEndpoint::AnthropicMessages` **after** classification. The OpenAI Chat Completions and OpenAI Responses arms get an explicit no-op match so the sanitizer cannot re-apply to those paths. - Drop the dead `sanitize_model_id` helper in `sse/anthropic.rs` (it was `#[allow(dead_code)]` with no callers). - 8 new unit tests in `compression/mod.rs`: trailing `[1m]` stripped, Claude-style suffix stripped, no-suffix passthrough (byte-equal), non-string model, missing `model` field, non-JSON body, `[1m]` mid-string, and the pure trim helper. - 5 new integration tests in `tests/integration_anthropic_model_sanitize.rs` that boot a real Rust proxy in front of a wiremock upstream. ## Testing - [x] Unit tests pass (`cargo test -p headroom-proxy` → 420 passed, 35 suites) - [x] Linting passes (`cargo clippy -p headroom-proxy --tests --all-features -- -D warnings` clean) - [x] Type checking passes (`cargo check -p headroom-proxy --tests --all-features` clean) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ cargo test -p headroom-proxy --test integration_anthropic_model_sanitize Compiling headroom-proxy v0.x.x Finished `test` profile [unoptimized + debuginfo] target(s) Running tests/integration_anthropic_model_sanitize.rs test anthropic_messages_strips_1m_suffix_glm ... ok test anthropic_messages_strips_1m_suffix_claude ... ok test anthropic_messages_passthrough_when_no_suffix ... ok test openai_chat_completions_passthrough_with_1m_model ... ok test openai_responses_passthrough_with_1m_model ... ok test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out ``` ```text $ cargo test -p headroom-proxy test result: ok. 420 passed; 0 failed; 0 ignored; 0 measured; 235 filtered out finished in 10.93s ``` ```text $ cargo clippy -p headroom-proxy --tests --all-features -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) ``` ## Real Behavior Proof - **Environment:** macOS 14.x; `rustc` pinned via `rust-toolchain.toml`; `cargo` 1.x. No network access required (wiremock upstream). - **Exact command / steps:** 1. `cargo test -p headroom-proxy --test integration_anthropic_model_sanitize` — confirms `/v1/messages` strips `glm-5.2[1m]` and `claude-3-7-sonnet[1m]`; confirms `/v1/chat/completions` and `/v1/responses` leave the body byte-equal (SHA-256 asserted). 2. `cargo test -p headroom-proxy` — full suite green (420 passed). 3. `cargo clippy -p headroom-proxy --tests --all-features -- -D warnings` — clean. 4. `cargo fmt -p headroom-proxy --check` — clean. 5. Source inspection of `crates/headroom-proxy/src/proxy.rs` after the change: the call site is now in a `match endpoint` arm that explicitly returns `buffered` for the OpenAI variants, so the sanitizer cannot re-apply to those paths. - **Observed result:** all 5 new integration tests pass, all 420 crate tests pass, clippy and fmt clean. The OpenAI tests assert SHA-256 byte equality on a body whose `model` field ends in `[1m]`; if the sanitizer were to re-leak onto OpenAI shapes these would fail loudly with a length delta. - **Not tested:** a live Anthropic API call (would require real credentials and is not required to prove the byte-level scope fix). The Python proxy's `sanitize_anthropic_model_id()` is the documented parity reference (Python PR #1840, issue #1812). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (N/A — no user-facing docs change; the Python proxy's `sanitize_anthropic_model_id` is the parity reference cited in code comments) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable (project uses git log + PR titles; this PR's title follows the conventional commit shape) ## Screenshots (if applicable) N/A — backend behavior, no UI change. ## Additional Notes - The OpenAI integration tests rely on a JWT-style `Authorization: Bearer` header to classify the request as `AuthMode::OAuth` and short-circuit the PR-E4 `prompt_cache_key` injector. This is the same control variable the existing `integration_chat_completions.rs` tests use to isolate dispatcher byte-fidelity from the E4 hook. Comments in each test explain the relationship. - The dead helper in `sse/anthropic.rs` is removed, so the diff is net negative on LoC for the SSE module. - The Python parity reference is `sanitize_anthropic_model_id()` (Python PR #1840, issue #1812); the function name and the call-site scope are the explicit parity contract. - Branch was rebased onto `upstream/main` (91 commits behind) before force-push to the fork; conflict-free rebase. The original PR commit and the fix are the only two commits on the PR. --------- Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com> Co-authored-by: Abhishek Mittal <abhishek.mittal@users.noreply.github.com> Co-authored-by: Tejas Chopra <chopratejas@gmail.com> |
||
|
JD Davis
|
2c9eb7c5f1
|
feat(simulators): add provider simulator service (#2014)
## Description
Adds a Rust-only `headroom-simulators` workspace crate: a deterministic
local upstream simulator service for Headroom proxy and pipeline
validation. It supplies configurable stubs plus bottled provider-shaped
responses for supported provider/path surfaces without calling real
LLMs.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Added `crates/headroom-simulators` Rust crate with library and
`headroom-simulators` binary.
- Added clean domain classification for supported surfaces: Anthropic
`/v1/messages`, OpenAI chat/responses/conversations, Bedrock
invoke/stream routes, Vertex raw/stream predict, health, and generic
fallback.
- Added JSON-configured stub matching by method, path, body substring,
and JSON pointer.
- Added bottled provider-shaped JSON, SSE, and Bedrock EventStream
responses for unconfigured requests.
- Added a container `Dockerfile` and README for local/GitHub Actions
usage.
- Added unit and HTTP integration tests for defaults, configured stubs,
SSE, Vertex, and Bedrock EventStream behavior.
- Added proxy-level simulator-backed E2E tests that run Headroom against
the simulator across Anthropic, OpenAI Chat, OpenAI Responses, OpenAI
Conversations, Bedrock invoke/converse/streaming, Vertex raw/stream
predict, and upstream health.
- Added simulator-backed provider error-path E2E coverage for OpenAI
429, Anthropic 529, Bedrock 502, and Vertex 503 responses flowing
through Headroom unchanged.
- Added Headroom-owned preflight error E2E coverage proving Bedrock
missing credentials and invalid Vertex envelopes stop inside the proxy
instead of silently falling through to the simulator/provider.
- Fixed direct Rust `headroom-core` binaries/tests on Windows so Magika
initializes ONNX Runtime via `ort::init_from` from an explicit pip
`onnxruntime` library path, with fail-fast fallback only when no safe
runtime is discoverable.
- Added a Rust CI `simulator-e2e` matrix for `ubuntu-latest`,
`macos-latest`, and `windows-latest` that runs `cargo test -p
headroom-proxy --test e2e_simulators`.
- Gated dynamic Magika `Path`/`PathBuf` imports to Windows and x86_64
macOS so Linux clippy does not see unused dynamic-ORT-only imports.
## Testing
- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
cargo fmt --all -- --check
# passed
cargo clippy --workspace -- -D warnings
# passed
$env:ORT_DYLIB_PATH=$null
cargo test -p headroom-core transforms::magika_detector::tests:: --lib
# 17 passed, 0 failed; Magika initialized from discovered pip
onnxruntime DLL
$env:ORT_DYLIB_PATH=$null
cargo test --workspace
# passed
gitleaks protect --staged --no-banner --redact
# no leaks found
gitleaks git --log-opts="headroomlabs/main..HEAD" --no-banner --redact
# 5 commits scanned; no leaks found
## Real Behavior Proof
- **Environment:** Windows PowerShell, Rust toolchain `1.95.0`, clean
worktree from `headroomlabs/main` at `
|
||
|
dependabot[bot]
|
5229c98228
|
deps: bump prometheus from 0.13.4 to 0.14.0 (#1518)
Bumps [prometheus](https://github.com/tikv/rust-prometheus) from 0.13.4 to 0.14.0. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/tikv/rust-prometheus/blob/master/CHANGELOG.md">prometheus's changelog</a>.</em></p> <blockquote> <h2>0.14.0</h2> <ul> <li> <p>API change: Use <code>AsRef<str></code> for owned label values (<a href="https://redirect.github.com/tikv/rust-prometheus/issues/537">#537</a>)</p> </li> <li> <p>Improvement: Hashing improvements (<a href="https://redirect.github.com/tikv/rust-prometheus/issues/532">#532</a>)</p> </li> <li> <p>Dependency upgrade: Update <code>hyper</code> to 1.6 (<a href="https://redirect.github.com/tikv/rust-prometheus/issues/524">#524</a>)</p> </li> <li> <p>Dependency upgrade: Update <code>procfs</code> to 0.17 (<a href="https://redirect.github.com/tikv/rust-prometheus/issues/543">#543</a>)</p> </li> <li> <p>Dependency upgrade: Update <code>protobuf</code> to 3.7.2 for RUSTSEC-2024-0437 (<a href="https://redirect.github.com/tikv/rust-prometheus/issues/541">#541</a>)</p> </li> <li> <p>Dependency upgrade: Update <code>thiserror</code> to 2.0 (<a href="https://redirect.github.com/tikv/rust-prometheus/issues/534">#534</a>)</p> </li> <li> <p>Internal change: Fix LSP and Clippy warnings (<a href="https://redirect.github.com/tikv/rust-prometheus/issues/540">#540</a>)</p> </li> <li> <p>Internal change: Bump MSRV to 1.81 (<a href="https://redirect.github.com/tikv/rust-prometheus/issues/539">#539</a>)</p> </li> <li> <p>Documentation: Fix <code>register_histogram_vec_with_registry</code> docstring (<a href="https://redirect.github.com/tikv/rust-prometheus/issues/528">#528</a>)</p> </li> <li> <p>Documentation: Fix typos in static-metric docstrings (<a href="https://redirect.github.com/tikv/rust-prometheus/issues/479">#479</a>)</p> </li> <li> <p>Documentation: Add missing <code>protobuf</code> feature to README list (<a href="https://redirect.github.com/tikv/rust-prometheus/issues/531">#531</a>)</p> </li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
98f7f1c2a3
|
deps: bump tower-http from 0.6.11 to 0.7.0 (#1520)
Bumps [tower-http](https://github.com/tower-rs/tower-http) from 0.6.11 to 0.7.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/tower-rs/tower-http/releases">tower-http's releases</a>.</em></p> <blockquote> <h2>tower-http-0.7.0</h2> <p><a href="https://github.com/tower-rs/tower-http/compare/tower-http-0.6.11...tower-http-0.7.0">Changes since 0.6.11</a></p> <h2>Added</h2> <ul> <li> <p><code>csrf</code>: add cross-site request forgery (CSRF) protection middleware, porting the cross-origin protection scheme introduced in Go 1.25 (<a href="https://redirect.github.com/tower-rs/tower-http/issues/699">#699</a>)</p> <pre lang="rust"><code>use tower::ServiceBuilder; use tower_http::csrf::CsrfLayer; <p>// Rejects cross-origin state-changing requests using <code>Sec-Fetch-Site</code>,<br /> // an <code>Origin</code> allow-list, and an <code>Origin</code>/<code>Host</code> fallback. No per-request<br /> // token state required.<br /> let layer = CsrfLayer::new().add_trusted_origin("<a href="https://example.com">https://example.com</a>")?;</p> <p>let service = ServiceBuilder::new().layer(layer).service_fn(handler);<br /> </code></pre></p> </li> <li> <p><code>timeout</code>: add <code>DeadlineBody</code> for non-resetting body timeouts, applied via the new <code>RequestBodyDeadlineLayer</code> and <code>ResponseBodyDeadlineLayer</code> (<a href="https://redirect.github.com/tower-rs/tower-http/issues/688">#688</a>)</p> <p>Unlike <code>TimeoutBody</code>, which resets its deadline on every frame, <code>DeadlineBody</code> caps the total time of a body transfer. A slow client trickling one byte at a time never trips an idle timeout but will trip a deadline.</p> <pre lang="rust"><code>use std::time::Duration; use tower::ServiceBuilder; use tower_http::timeout::RequestBodyDeadlineLayer; <p>// Abort the request body transfer after 30s total, regardless of how<br /> // frequently data arrives.<br /> let service = ServiceBuilder::new()<br /> .layer(RequestBodyDeadlineLayer::new(Duration::from_secs(30)))<br /> .service_fn(handler);<br /> </code></pre></p> </li> <li> <p><code>fs</code>: add strong <code>ETag</code> support to <code>ServeDir</code>, including <code>If-Match</code> and <code>If-None-Match</code> precondition handling per RFC 9110. <code>304 Not Modified</code> responses now carry the <code>ETag</code> and <code>Last-Modified</code> validators (<a href="https://redirect.github.com/tower-rs/tower-http/issues/691">#691</a>)</p> </li> <li> <p><code>fs</code>: add a <code>Backend</code> trait to make <code>ServeDir</code> work with non-filesystem sources (e.g. embedded assets or object storage). The default <code>TokioBackend</code> preserves existing behavior. Use <code>ServeDir::with_backend()</code> to plug in custom implementations (<a href="https://redirect.github.com/tower-rs/tower-http/issues/684">#684</a>)</p> <pre lang="rust"><code>use tower_http::services::fs::ServeDir; <p>// <code>MyBackend</code> implements <code>tower_http::services::fs::Backend</code>.<br /> // The default <code>ServeDir::new()</code> continues to use <code>TokioBackend</code> (local FS).<br /> let service = ServeDir::with_backend("assets", MyBackend::new());<br /> </code></pre></p> </li> <li> <p><code>fs</code>: add <code>html_as_default_extension</code> option to <code>ServeDir</code>, appending <code>.html</code> when the request path has no extension (<a href="https://redirect.github.com/tower-rs/tower-http/issues/519">#519</a>)</p> </li> <li> <p><code>fs</code>: add <code>redirect_path_prefix</code> option to <code>ServeDir</code>, prepending a prefix on trailing-slash redirects so the service can be mounted under a sub-path (<a href="https://redirect.github.com/tower-rs/tower-http/issues/486">#486</a>)</p> </li> <li> <p><code>validate-request</code>: add <code>ValidateRequestHeaderLayer::has_header_value()</code> to reject requests when a header does not have an expected value (<a href="https://redirect.github.com/tower-rs/tower-http/issues/360">#360</a>)</p> </li> <li> <p><code>body</code>: <code>UnsyncBoxBody::new()</code> constructor and <code>From<ServeFileSystemResponseBody></code> conversion to avoid double-boxing when combining <code>ServeDir</code> responses with other body types (<a href="https://redirect.github.com/tower-rs/tower-http/issues/537">#537</a>)</p> </li> <li> <p><code>limit</code>: implement <code>Default</code> for <code>limit::ResponseBody</code> when the wrapped body also implements <code>Default</code> (<a href="https://redirect.github.com/tower-rs/tower-http/issues/679">#679</a>)</p> </li> </ul> <h2>Changed</h2> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
6c705b4066
|
deps: bump toml from 0.8.23 to 1.1.2+spec-1.1.0 (#1517)
Bumps [toml](https://github.com/toml-rs/toml) from 0.8.23 to 1.1.2+spec-1.1.0. <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
Zhenjia ZHOU
|
8879c50dbe
|
fix(adaptive-sizer): char bigrams for spaceless CJK items (#1748)
## Description `compute_unique_bigram_curve` — the adaptive sizer's coverage-curve builder, mirrored in Rust and Python — word-splits each item on whitespace to form word bigrams. A spaceless CJK item has no whitespace, so it collapsed into one `(whole_string, "")` pseudo-bigram: the coverage curve then grew ~1 per item, the kneedle knee detector found no knee, and CJK lists under-compressed. Spaceless CJK items now use character bigrams, producing a real coverage curve. Mirrored byte-exactly in Rust and Python (identical reference-test curve values). Non-CJK items — anything whitespace-bearing or spaceless-ASCII — are byte-identical to before, so the `smart_crusher` parity fixtures are unchanged. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `crates/headroom-core/src/transforms/adaptive_sizer.rs` + `headroom/transforms/adaptive_sizer.py`: add `is_cjk_char`/`_is_cjk_char` (identical code-point ranges) and a spaceless-CJK character-bigram branch in `compute_unique_bigram_curve`. - Rust unit tests + `tests/test_adaptive_sizer.py`: CJK curve, single-char CJK, ASCII-unchanged, empty-item — the Rust and Python reference values are identical. ## Testing - [x] Unit tests pass (`cargo test` + `pytest`) - [x] Linting passes (`cargo clippy` / `cargo fmt` / `ruff` / `mypy`) - [x] New tests added for new functionality - [x] Manual testing performed (see Real Behavior Proof) ### Test Output ```text $ cargo test -p headroom-core --lib adaptive_sizer test result: ok. 35 passed; 0 failed $ .venv/bin/python -m pytest tests/test_adaptive_sizer.py 20 passed $ .venv/bin/python -m pytest -k "smart_crusher and parity" 18 passed, 6 skipped # non-CJK fixtures unchanged ``` ## Real Behavior Proof - Environment: macOS (Darwin), Rust via cargo, Python in a uv venv, branch `feat/adaptive-sizer-cjk` off `main`. - Exact command / steps: called `compute_unique_bigram_curve` on a CJK list and on ASCII lists, in both implementations. - Observed result: `compute_unique_bigram_curve(["数据库连接失败", "数据库连接成功"])` returns `[6, 8]` in **both** Rust and Python (before: ~`[1, 2]` — one pseudo-bigram per item, no coverage signal). ASCII curves are unchanged: `["the cat", "the dog", "a fish"]` → `[1, 2, 3]`. The `smart_crusher` parity suite (all-ASCII fixtures) stays green, confirming non-CJK output is byte-identical. - Byte-exact parity: the Rust reference test (`vec![6, 8]`) and the Python test (`[6, 8]`) use the same inputs and the same expected values, so the two implementations are pinned to agree. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation — N/A (internal sizing heuristic) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A: internal sizing-heuristic fix, no user-facing surface change ## Additional Notes - This is a parity-locked function (Rust and Python must agree byte-for-byte). The fix is CJK-gated, so non-CJK output is byte-identical and the `smart_crusher` parity fixtures need no re-recording. |
||
|
Zhenjia ZHOU
|
985621d60e
|
fix(search-compressor): CJK-aware relevance + harden Rust/Python parity (#1749)
## Description
The search compressor's relevance scorer (`score_matches`, present in
both the Rust runtime path and the Python legacy mirror) split the query
on whitespace. A spaceless CJK query therefore matched a result line
only when the WHOLE query was a literal substring of that line — partial
overlaps never boosted relevant lines, so correct matches got dropped
when the result set was over budget.
This adds CJK character bigrams to the query match set, so a longer CJK
query boosts lines that share a substring. It also fixes two latent
Rust/Python parity divergences the ASCII-only fixtures had masked:
- **Length filter**: Rust counted word length in BYTES (`w.len()`),
Python in codepoints (`len(w)`), so a CJK word crossed the `> 2`
threshold differently. Rust now uses `chars().count()`.
- **Dedup**: Rust collected words into a `Vec` (no dedup), Python into a
`set`, so a repeated query word double-counted in Rust. Rust now uses a
`BTreeSet`.
Both scorers are byte-exact now; non-CJK output is unchanged (the 53
existing tests and the parity fixtures stay green).
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `crates/headroom-core/src/transforms/search_compressor.rs` +
`headroom/transforms/search_compressor.py`: add
`is_cjk_char`/`_is_cjk_char` and `cjk_bigrams`/`_cjk_bigrams` (identical
ranges + logic), union CJK bigrams into the query match set, and align
the Rust word set to Python (`chars().count()` length, `BTreeSet`
dedup).
- `tests/test_search_compressor_cjk.py` + a Rust unit test: CJK bigram
extraction (same input/expected in both languages) and a CJK query
boosting a partially-overlapping line.
- Corrected a stale `_score_matches` docstring that referenced a
non-existent parity assertion; it now states honestly how the two sides
are pinned (test-equal for word-overlap + CJK bigrams; a few error-boost
keywords still diverge, fixed only Rust-side).
## Testing
- [x] Unit tests pass (`cargo test` + `pytest`)
- [x] Linting passes (`cargo clippy` / `cargo fmt` / `ruff` / `mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed (see Real Behavior Proof)
### Test Output
```text
$ cargo test -p headroom-core --lib search_compressor
test result: ok. 16 passed; 0 failed
$ .venv/bin/python -m pytest tests/test_search_compressor_cjk.py \
tests/test_transforms_search_compressor.py tests/test_search_compressor.py
55 passed # 2 new CJK tests + 53 existing (no regression)
```
## Real Behavior Proof
- Environment: macOS (Darwin), Rust via cargo, Python in a uv venv
(`_core` rebuilt on this branch), branch `feat/search-compressor-cjk`
off `main`.
- Exact command / steps: scored a CJK content line against a longer CJK
query whose whole form is not a substring of the line.
- Observed result: for content `src/a.py:10:认证令牌已过期需要重新登录` and query
`认证令牌缓存淘汰策略` (the whole query is NOT a substring of the line, but its
bigrams are), the line now scores `> 0` (bigrams 认证 / 证令 / 令牌 match);
before, it scored `0`. An ASCII-only line still scores `0`. All 53
existing search-compressor tests are unchanged. `cjk_bigrams("认证令牌")`
returns `{认证, 证令, 令牌}` in **both** Rust and Python.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — N/A
(internal relevance scoring)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md — N/A: internal relevance-scoring
fix, no user-facing surface change
## Additional Notes
- The two parity divergences (byte-vs-codepoint length, `Vec`-vs-`set`
dedup) were pre-existing and only reachable with non-ASCII or
repeated-word queries — the all-ASCII fixtures never exercised them.
This PR brings both sides back to byte-exact for the word-overlap +
CJK-bigram scoring. The remaining error-boost keyword divergence is
pre-existing (fixed only Rust-side in the 3e.1 port) and is now
documented in the code rather than glossed over.
|
||
|
Rod Boev
|
be51008c70
|
fix(toin): publish skip compression recommendations (#1782)
## Description TOIN already learns when a tool-output slice should skip compression, but the published recommendation artifact drops that signal. A high full-retrieval row can therefore still publish an ordinary compressor strategy even though TOIN marked it as skip-worthy. This change carries `skip_compression_recommended` into `recommendations.toml`, keeps Rust parsing backward compatible for older files, and makes skip rows publish a skip-oriented strategy hint instead of misleading compressor guidance. Refs #1775 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Publishes `skip_compression_recommended` in generated recommendation rows. - Uses retrieval-aware strategy output for rows TOIN already marked as skip-worthy. - Extends the Rust recommendation schema with a backward-compatible default for older TOML files. - Adds focused publish and schema coverage for skip and non-skip rows. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_toin_publish.py -q`) - [x] Linting passes (`uv run ruff check headroom/cli/toin_publish.py headroom/telemetry/toin.py tests/test_toin_publish.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed - [ ] I have made corresponding changes to the documentation ### Test Output ```text uv run pytest tests/test_toin_publish.py -q: 8 passed uv run ruff check headroom/cli/toin_publish.py headroom/telemetry/toin.py tests/test_toin_publish.py: passed cargo fmt --all -- --check: passed cargo check -p headroom-core: passed cargo test -p headroom-core --lib transforms::recommendations: 6 passed cargo clippy --workspace -- -D warnings: passed ``` ## Real Behavior Proof - Environment: Windows for Python validation through the headless runner; Rust validation via focused local cargo commands where available. - Exact command / steps: `uv run pytest tests/test_toin_publish.py -q`, `uv run ruff check headroom/cli/toin_publish.py headroom/telemetry/toin.py tests/test_toin_publish.py`, `cargo fmt --all -- --check`, `cargo check -p headroom-core`, `cargo test -p headroom-core --lib transforms::recommendations`, and `cargo clippy --workspace -- -D warnings`. - Observed result: Skip-worthy rows carry `skip_compression_recommended = true` and a skip strategy hint; normal rows carry `false` and preserve their ordinary strategy. - Not tested: Live runtime dispatcher skip behavior and full Rust workspace tests. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes This PR fixes the published recommendation artifact. Runtime dispatcher enforcement remains a separate follow-up because it needs a dedicated consumer proof matrix. Documentation and changelog are left unchecked because this changes generated recommendation data and Headroom's changelog is generated from conventional commits. |
||
|
Rob Francis
|
32ce99e4b4
|
fix(build): enable Intel macOS pip installs via ort-load-dynamic (#1538)
## Description
Fix `pip install headroom-ai` on Intel Mac (`x86_64-apple-darwin`).
Source installs failed because `ort-sys 2.0.0-rc.12` (transitive via
`fastembed`) does not ship prebuilt ONNX Runtime binaries for that
target, causing maturin/cargo to exit during the wheel build.
This PR mirrors the existing Windows fix: build the Rust core with
`ort-load-dynamic`, pin `ORT_DYLIB_PATH` to the pip `onnxruntime` native
library at import time, and publish Intel macOS wheels from CI.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `crates/headroom-core/Cargo.toml`: use `ort-load-dynamic` for
`x86_64-apple-darwin` instead of `ort-download-binaries-rustls-tls`.
- `headroom/_ort.py`: extend the ORT dylib pin hook to Intel macOS
(`darwin` + `x86_64`), resolving `libonnxruntime*.dylib` from the pip
`onnxruntime` package.
- `.github/workflows/release.yml` and `.github/workflows/rust.yml`: add
`macos-15-intel` / `x86_64-apple-darwin` wheel matrix entries.
- `tests/test_release_workflows.py` and
`tests/test_transforms/test_ort_dylib.py`: update/add coverage for the
new target and dylib pin behavior.
- `README.md`: note that prebuilt wheels are published for Intel macOS.
## Testing
- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_transforms/test_ort_dylib.py \
tests/test_release_workflows.py::test_fastembed_uses_dynamic_ort_on_windows \
tests/test_release_workflows.py::test_build_wheels_matrix_includes_intel_macos_with_dynamic_ort -q
.......................... [100%]
10 passed in 0.18s
$ maturin build --release -o /tmp/headroom-dist
📦 Built wheel for abi3 Python ≥ 3.10 to /tmp/headroom-dist/headroom_ai-0.27.0-cp310-abi3-macosx_10_12_x86_64.whl
$ python3.11 -m venv /tmp/hr-venv && /tmp/hr-venv/bin/pip install .
Successfully built headroom-ai
Successfully installed headroom-ai-0.27.0
$ cd /tmp && /tmp/hr-venv/bin/python -c "import headroom; import headroom._core; print('ok')"
version 0.27.0
_core ok
```
## Real Behavior Proof
- Environment: macOS `x86_64-apple-darwin`, Python 3.11.5, Rust 1.95.0
- Exact command / steps: Reproduced the reported failure with `pip
install headroom-ai` (sdist build dies in `ort-sys` for
`x86_64-apple-darwin`); after this patch ran `maturin build --release`,
then `pip install .` in a clean venv, then `python -c "import
headroom._core"`.
- Observed result: Before fix, cargo/maturin exit 101 on missing ORT
prebuilts; after fix, wheel build succeeds and `headroom._core` imports
cleanly (`version 0.27.0`, `_core ok`).
- Not tested: `macos-15-intel` GitHub Actions wheel matrix row (will be
validated by CI after merge).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
- ML features (magika detection, fastembed embeddings) still require
`onnxruntime` at runtime on Intel Mac. Users should install
`headroom-ai[proxy]` or `pip install onnxruntime`; `_ort.py` auto-pins
`ORT_DYLIB_PATH` when that package is present.
- Apple Silicon (`aarch64-apple-darwin`) behavior is unchanged: it
continues to bundle ORT via `ort-download-binaries-rustls-tls`.
- Lint/mypy not re-run locally in this pass; targeted pytest +
maturin/pip install proof covers the changed surface.
---------
Co-authored-by: Bor <you@example.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
Parideboy
|
728b33088b
|
fix(relevance): gate ONNX embedding backend behind AVX2 to avoid SIGILL (#1723) (#1765)
## Description Fixes the `SIGILL` / Illegal instruction crash in `headroom.compress` on CPUs without AVX2 (Docker / QEMU / older cloud VMs). The precompiled ONNX Runtime binary shipped by `ort-sys` (via fastembed's `ort-download-binaries*` feature) contains AVX2-family instructions on x86; running it on a non-AVX2 CPU traps with SIGILL — an uncatchable native fault that kills the whole host process. Magika detection was already guarded (#1162, landed after `v0.28.0`); the embedding relevance scorer shared the same `ort-sys` binary with no guard. This PR closes that remaining entry point and documents the requirement. Closes #1723 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add shared `onnx_cpu::onnx_runtime_supported_by_cpu()` helper (AVX2 check on x86/x86_64, `true` on other arches) as the single source of truth. - Route `magika_detector` through the shared helper (no behavior change). - Gate `EmbeddingScorer::try_new*` on the helper: unsupported CPU returns `Err` before touching ONNX, so callers fall back to BM25/stub instead of crashing. - Document the x86 AVX2 requirement + auto-fallback in the README. - Add offline tests (no network / no `RUN_FASTEMBED_TESTS`). ## Testing - [x] Unit tests pass (Rust: `cargo test -p headroom-core`) - [x] Linting passes (`cargo clippy -p headroom-core --all-targets`, `cargo fmt --check`) - [ ] Type checking passes (`mypy headroom`) — N/A, Rust-only change - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ cargo test -p headroom-core --lib relevance::embedding cargo test: 13 passed, 834 filtered out (1 suite, 0.00s) $ cargo test -p headroom-core --lib magika cargo test: 16 passed, 831 filtered out (1 suite, 0.16s) $ cargo clippy -p headroom-core --all-targets (no warnings, no errors) $ cargo fmt --check -p headroom-core (clean) $ cargo build --workspace cargo build (225 crates compiled) Finished `dev` profile [unoptimized + debuginfo] target(s) in 6m 57s ``` ## Real Behavior Proof - Environment: `headroom-core` workspace, Rust stable, x86_64 (AVX2-capable dev host). - Exact command / steps: added `onnx_guard_matches_cpu_features` and `try_new_errors_on_unsupported_cpu_instead_of_sigill` tests; ran the suites above. On a no-AVX2 host the guard makes `EmbeddingScorer::try_new()` return `Err(... "AVX2" ...)` instead of executing the AVX2 ONNX binary; callers fall back to BM25 relevance rather than crashing. - Observed result: guard returns `false` only when the CPU lacks AVX2; embedding + magika ONNX paths both short-circuit to non-ONNX fallbacks; no SIGILL. All suites green. - Not tested: end-to-end `pip install` run on a physically AVX2-less machine (dev host has AVX2); guard behavior is unit-tested via the shared `onnx_cpu` helper and mirrors the already-shipped magika guard (#1162). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable — N/A (release-please generates the changelog) ## Additional Notes Rust-only change, so the Python `pytest`/`ruff`/`mypy` items are N/A; equivalent Rust `cargo test`/`clippy`/`fmt` were run and pasted above. The fix is defense-in-depth parity with the existing magika AVX2 guard (#1162), applied to the second ONNX entry point (embedding relevance). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Abhay Singh
|
8cddf9b58e
|
fix(proxy/auth): match real Anthropic OAuth token prefix (sk-ant-oat) (#1672)
## Description
`classify_auth_mode` (in `headroom/proxy/auth_mode.py`) checks for
Anthropic
OAuth tokens with:
```python
if token.startswith("sk-ant-oat-"):
return AuthMode.OAUTH
if token.startswith("sk-ant-api") or token.startswith("sk-"):
return AuthMode.PAYG
```
But real Anthropic OAuth access tokens are **`sk-ant-oat01-...`** — a
version
number right after `oat`, **no dash**. So the `sk-ant-oat-` check never
matches a
real token; it falls through to the broad `sk-` rule and gets classified
**`PAYG`**.
That's exactly the misclassification the module is built to prevent: a
subscription/OAuth-bound request tagged `PAYG` gets the
aggressive-compression
policy — lossy compression, auto `cache_control`, `prompt_cache_key`
injection —
instead of the passthrough-prefer path OAuth is meant to get.
The existing tests didn't catch it because they use a synthetic
`sk-ant-oat-01-`
fixture (dashed) that happens to match the buggy prefix. Corroboration
that the
real shape is dash-less:
- `.gitguardian.yaml` fixture: `sk-ant-oat01-oauth-fixture`
- `tests/test_oauth_bearer_routing.py`: `sk-ant-oat01-xxx`
- the sibling helper `headroom/proxy/helpers.py` matches on `sk-ant-`
(no `oat-`)
## Fix
Match the dash-less `sk-ant-oat` prefix. It still matches the legacy
dashed
shape, and ordering relative to `sk-ant-api` / `sk-` is unchanged (OAuth
is
still checked first).
```python
if token.startswith("sk-ant-oat"):
return AuthMode.OAUTH
```
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/proxy/auth_mode.py`: match OAuth tokens on the dash-less
`sk-ant-oat` prefix.
- `tests/test_auth_mode.py`: add a regression test using the real
`sk-ant-oat01-...` format (the existing test keeps the legacy dashed
fixture, which still classifies correctly).
- `CHANGELOG.md`: Bug Fixes entry under Unreleased.
## Testing
- [x] New regression test added (`tests/test_auth_mode.py`)
- [x] Linting passes (`ruff check`) and formatting is clean (`ruff
format --check`)
- [ ] Full `pytest` deferred to CI (local-OOM reason below).
```text
$ uv run ruff check headroom/proxy/auth_mode.py tests/test_auth_mode.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, headroom from this branch.
Importing `headroom` loads the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the classification
logic with a dependency-free script and left the full pytest to CI.
- Exact command / steps: replicated the Bearer-token branch of
`classify_auth_mode` in a standalone script (only stdlib, no `headroom`
import) and ran the real and legacy token shapes plus PAYG keys through
it.
- Observed result: the real `sk-ant-oat01-...` now classifies OAUTH (was
PAYG before the change); the legacy dashed fixture still classifies
OAUTH; `sk-ant-api*` / `sk-*` keys still classify PAYG:
```text
OK: sk-ant-oat01-... -> OAUTH (was PAYG before fix)
OK: sk-ant-oat-01-... -> OAUTH (legacy fixture still matches)
OK: sk-ant-api* / sk-* -> PAYG (unchanged)
AUTH LOGIC VERIFIED
```
- Not tested: a live proxied Anthropic OAuth request end-to-end (needs a
real subscription token); the classification is pure and covered by the
regression test. Full local `pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- `crates/headroom-core/src/auth_mode.rs` carries the identical dashed
prefix (its Rust test matrix uses the same synthetic dashed fixture). I
scoped this PR to the Python runtime classifier since that's the
request-time path; happy to mirror the one-line fix in Rust in the same
PR or a follow-up — I just couldn't `cargo build` locally to verify, so
I left it out rather than push an unverified Rust edit.
- @JerrettDavis tagging you since you've been triaging these — small,
contained fix with a regression test if you have a moment.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
Manmit Singh
|
e386c097d6
|
fix(detection): contain unidiff panic on orphaned +++ target line (#1548)
## Description `headroom._core.detect_content_type()` panics with `pyo3_runtime.PanicException: called Option::unwrap() on a None value` on any text containing a `+++ ` target line with no preceding `--- ` source line — e.g. `set -x` xtrace output or a partial `git diff` quoted out of context. The panic originates in the bundled `unidiff` 0.4.0 parser (`lib.rs:665`): on a target-file header it does `source_file.clone().unwrap()`, but `source_file` is still `None` when no source header was seen. The crate's only guard there checks `current_file`, not `source_file`, so it falls through and unwraps `None` instead of returning `Err`. Because detection runs inside a `ThreadPoolExecutor` worker on the Python side, the native panic surfaces as an uncaught `PanicException`, bypasses the compression error handling, and returns **HTTP 500** for the whole request. The failure is deterministic on payload content, so client retries fail until the offending text leaves the context window. `is_diff()` in `unidiff_detector.rs` is the single entry point that drives `PatchSet::parse`, so the fix is contained there: wrap the parse in `catch_unwind` and treat an unparseable fragment as "not a diff". This matches the workspace's deliberate no-`panic = "abort"` policy (Cargo.toml) of surviving bad input rather than taking the long-lived proxy down. Closes #1547 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `crates/headroom-core/src/transforms/unidiff_detector.rs`: contain any `unidiff` parser panic inside `is_diff()` via `catch_unwind`, returning `false` (not a diff) on panic. Added regression test `orphaned_target_line_does_not_panic`. - `CHANGELOG.md`: note under Unreleased → Fixed. ## Testing - [x] Unit tests pass (`cargo test -p headroom-core`) - [x] Linting passes (`cargo fmt --check`, `cargo clippy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output Before the fix (regression test reproduces the exact panic): ```text running 1 test test transforms::unidiff_detector::tests::orphaned_target_line_does_not_panic ... FAILED ---- transforms::unidiff_detector::tests::orphaned_target_line_does_not_panic stdout ---- thread '...' panicked at unidiff-0.4.0/src/lib.rs:665:54: called `Option::unwrap()` on a `None` value test result: FAILED. 0 passed; 1 failed; ... ``` After the fix: ```text running 15 tests test transforms::unidiff_detector::tests::orphaned_target_line_does_not_panic ... ok test transforms::unidiff_detector::tests::standard_git_diff_detected ... ok ... test result: ok. 15 passed; 0 failed; 0 ignored # whole transforms suite test result: ok. 700 passed; 0 failed; 0 ignored ``` ## Real Behavior Proof - Environment: macOS (arm64), Rust stable, `cargo test -p headroom-core`. - Exact command / steps: `cargo test -p headroom-core --lib unidiff_detector` then `cargo test -p headroom-core`. (1) Added a test calling `is_diff("+++ x")` / `detect_diff("+++ x")` and ran it → reproduced the panic at `unidiff-0.4.0/src/lib.rs:665:54` (output above), confirming the same crash path as the report. (2) Applied the `catch_unwind` containment in `is_diff()`. (3) Re-ran the test and the full transforms suite → all green (output above). - Observed result: the orphaned-`+++ ` input is now classified as "not a diff" (plain text) and returns normally instead of panicking. Real diffs (`standard_git_diff_detected`, `naked_hunk_without_git_header_detected`, multi-file, added/removed-only) still detect correctly, so the containment does not weaken detection. - Not tested: I exercised the Rust layer directly (the sole `unidiff` caller, which the `headroom._core.detect_content_type` binding routes through) rather than rebuilding the Python wheel; I did not run the live proxy against a real provider. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md |
||
|
DM
|
64783d8824
|
fix: skip Magika backend on x86 CPUs without AVX2 (#1162)
## Description Adds a narrow runtime AVX2 guard before initializing the Magika/ONNX Runtime detector on x86/x86_64. On x86/x86_64 CPUs without AVX2, Headroom falls back to existing non-Magika detection tiers instead of crashing during ONNX Runtime initialization. AVX2-capable systems retain existing behavior. Refs #1005 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Adds a Magika/ONNX Runtime CPU support guard before `Session::new()`. - Returns a normal Magika init error on x86/x86_64 hosts without AVX2, allowing the existing detection chain to fall through to non-Magika tiers. - Keeps AVX2-capable x86/x86_64 behavior unchanged. - Does not apply the x86-specific AVX2 gate on non-x86 targets. - Adds CPU-aware Rust tests and a short troubleshooting note. ## Testing - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ cargo test -p headroom-core --lib --locked 833 passed; 0 failed; 1 ignored $ cargo test --workspace --locked passed $ cargo clippy -p headroom-core --locked -- -D warnings clean ``` ## Real Behavior Proof - Environment: x86_64 Linux host with AVX but no AVX2 (Intel Xeon E5-2697 v2 on Proxmox), local build from this branch. - Exact command / steps: `python -X faulthandler -c 'from headroom._core import detect_content_type; print(detect_content_type("hello world"))'` - Observed result: before — process exited with `Fatal Python error: Illegal instruction`; after — command completed successfully returning `DetectionResult(content_type="text", ...)`, and full `cargo test -p headroom-core --lib --locked` passed with 833/0/1. - Not tested: generic no-AVX CPUs, alternate ONNX Runtime builds, non-x86 platforms. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes This partially addresses #1005 by handling one concrete native crash class: the Magika detector initializes ONNX Runtime through ort/ort-sys, whose precompiled runtime can contain AVX2-family instructions. On AVX-only x86_64 hosts, that initialization can SIGILL before Headroom can fall back. Scope: - This does not introduce generic no-AVX wheels. - This does not redesign Rust-core packaging. - This does not disable the Rust core globally. - This only prevents the Magika/ONNX detector tier from loading on x86/x86_64 CPUs where AVX2 is unavailable. - Non-Magika detection tiers continue to run. - On non-x86 targets, this x86-specific AVX2 gate is not applied. Changelog omitted: small native detector fallback fix with no public API change. Co-authored-by: AI Agent <ai-agent@homelab.internal> |
||
|
Tejas Chopra
|
5771a8020e
|
fix(deps): remediate dependency CVEs and publish SBOM (#1509)
## Description
Supply-chain hardening: takes the **shipped** dependency surface from
**26 known CVEs to 0**. `pip install headroom-ai[all]` now resolves with
no known vulnerabilities (verified with Anchore syft + grype). Also
publishes a checked-in SBOM package (`sbom/`) so any user — especially
pilots running their own security review — can verify what's inside and
that we track it.
This addresses the Dependabot alerts on `main` (9 high / 4 moderate / 7
low at time of writing).
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
**Rust**
- `pyo3` 0.24 → 0.29 (GHSA-36hh-v3qg-5jq4 High, GHSA-chgr-c6px-7xpp
Med). Migrated `Python::allow_threads` → `Python::detach` (10 sites) and
added `from_py_object` to the `Clone`-deriving `#[pyclass]` types (both
required by the 0.25+ API).
- `pyo3-log` 0.12 → 0.13; `lru` 0.12 → 0.18 (GHSA-rhfx-m35p-ff5j).
**Python**
- `torch` → 2.12.1, `mem0ai` → 2.x.
- Floor-pinned transitive CVE deps via `[tool.uv]
constraint-dependencies`: `pygments>=2.20.0`,
`pydantic-settings>=2.14.2`, `gitpython>=3.1.50`, `langsmith>=0.9.0`.
- **Removed `benchmark` from the `[all]` aggregate** so the default
install is CVE-free. `lm-eval` is invoked as an external subprocess
(`python -m lm_eval`) and never imported, so it is not a true runtime
dep — it remains available via the opt-in `[benchmark]` extra. See
[Accepted Risks](#additional-notes).
**npm (build/test tooling — never shipped in the
wheel/container/published SDK)**
- `esbuild` override `>=0.28.1` in `sdk/typescript` + `plugins/openclaw`
(GHSA-g7r4-m6w7-qqqr).
- `docs/`: `@anthropic-ai/sdk` → `^0.106.0` (GHSA-p7fg-763f-g4gf),
`postcss` override to force Next.js's bundled copy ≥8.5.10
(GHSA-qx2v-qp2m-jg93); regenerated a stale `bun.lock` that carried a
**Critical** vitest/vite.
**CI**
- Pinned `pypa/gh-action-pypi-publish` `@release/v1` → `@v1.13.0`
(GHSA-vxmw-7h4f-hqxh) in `release.yml` + `publish.yml`.
**SBOM**
- New `sbom/` directory: CycloneDX 1.7 + SPDX 2.3 SBOMs, grype scan
evidence, 330-package license inventory, and a regeneration guide.
## Testing
- [ ] Unit tests pass (`pytest`) — N/A, no Python source changed
(deps/config only)
- [x] Linting passes — `cargo fmt --check` + `cargo clippy` clean on the
changed crate; 0 `.py` files changed so `ruff`/`mypy` scope is
unaffected
- [x] Type checking passes — `cargo check --workspace` (0 errors)
- [ ] New tests added — N/A (dependency bumps; covered by existing
suites)
- [x] Manual testing performed — see Real Behavior Proof
### Test Output
```text
# headroom-ai[all] product surface — the number that matters
$ grype sbom:sbom/headroom-sbom-all-extra.cdx.json
No vulnerabilities found
# full repo scan (universal lock incl. opt-in [benchmark] + dev)
$ grype sbom:sbom/headroom-sbom.cdx.json
NAME INSTALLED TYPE VULNERABILITY SEVERITY
sqlitedict 2.1.0 python GHSA-g4r7-86gm-pgqc High # [benchmark]-only, unpatchable, accepted
nltk 3.9.4 python GHSA-p4gq-832x-fm9v High # [benchmark]-only, unpatchable, accepted
# pyo3 0.29 migration — extension builds + imports + runs
$ cargo check --workspace
Finished `dev` profile [unoptimized + debuginfo] target(s)
$ maturin develop && python -c "from headroom._core import DiffCompressor, SmartCrusher; ..."
extension OK — detach + from_py_object paths exercised
# lru 0.18 — eviction path
$ cargo test -p headroom-proxy --lib drift
14 passed, 213 filtered out
# per-ecosystem npm audits
$ (cd sdk/typescript && npm audit) -> found 0 vulnerabilities
$ (cd plugins/openclaw && npm audit) -> found 0 vulnerabilities
$ (cd docs && npm audit && bun audit) -> found 0 vulnerabilities / No vulnerabilities found
```
## Real Behavior Proof
- Environment: macOS (darwin 25.4.0, arm64), Python 3.12 `.venv`, Rust
1.95 toolchain, syft 1.46.0, grype 0.115.0, bun 1.3.14, maturin 1.13.3.
- Exact command / steps: (1) `uv export --extra all --no-dev
--no-emit-project | syft → grype` for the product surface; (2) `cargo
check --workspace` + `maturin develop` + extension import/compress smoke
test; (3) `cargo test -p headroom-proxy --lib drift`; (4) `cargo fmt
--check` + `cargo clippy -p headroom-py`; (5) `npm audit` in
sdk/openclaw/docs + `bun audit` in docs.
- Observed result: `headroom-ai[all]` resolution scans clean — "No
vulnerabilities found" (179 pkgs); full/prod SBOM shows only the 2
documented accepted CVEs; pyo3 0.29 extension imports and runs (detach +
from_py_object paths exercised); drift tests 14/14 pass; cargo fmt +
clippy clean; all npm/bun audits report 0.
- Not tested: full `pytest` suite (no Python source changed);
release-profile wheel build (used dev-profile `maturin develop` for the
import proof — the extension is semantically identical).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
(`sbom/README.md`)
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective — N/A
(dependency bumps; existing suites + scans cover it)
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md — N/A (Release Please
auto-generates from the conventional commit)
## Additional Notes
**Accepted risks (the 2 residual CVEs).** Both originate solely from the
EleutherAI `lm-evaluation-harness` under the **opt-in `[benchmark]`
extra**, which Headroom invokes as a subprocess (never imports):
- `sqlitedict` CVE-2024-35515 (High) — pickle deserialization; package
abandoned (last release 2021), **no upstream fix exists**.
- `nltk` CVE-2026-54293 (High) — path traversal in `nltk.data.load()`;
affects ≤3.9.4 (current latest), **no patched release**.
Neither is in `[all]`, the published wheel, or the container. They are
documented in `sbom/README.md` and will be picked up automatically once
upstream ships fixes.
**Release/CHANGELOG:** N/A items above are because this is a
dependency/security PR with no Python source changes; CHANGELOG is
Release-Please-managed via the conventional commit message.
|