akadmin/headroom
1
0
Fork 0
mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
2556 commits 410 branches 169 tags 142 MiB
Commit graph

13 commits

Author SHA1 Message Date
Ruben A.
e530de5ad2
feat(rust): port CodeCompressor AST compressor to Rust (parity-only) (#1154) 
Adds crates/headroom-core/src/transforms/code_compressor.rs (1,882 lines): the
AST-aware CodeCompressor ported to Rust on tree-sitter, with grammars for
Python, JavaScript, TypeScript, Go, Rust, Java, C and C++.

Parity-only, like #1153. Nothing calls it: the only references outside the
module are the pub mod / pub use declarations in transforms/mod.rs, and
live_zone.rs still routes SourceCode to a no-op. The pyo3 bridge is untouched
and no Python source changes, so the engine is unreachable from the shipped
package. #1155 wires it into live-zone dispatch.

Every grammar is pinned with '=' to the exact version of the corresponding
Python tree-sitter-<lang> PyPI wheel. Same version on crates.io and PyPI means
the same grammar.js, hence the same generated parser.c, hence node-for-node
identical ASTs — the precondition for byte-parity. A canary over 9 samples x 8
languages confirmed identical node-type and line-span trees at these pins;
bumping any pin requires re-running it and re-recording the fixtures.

Ships 30 recorded parity fixtures, a CodeCompressorComparator in
headroom-parity, and scripts/record_code_compressor_fixtures.py.

Verified byte-identical to the recorded Python output:

  [code_aware_compressor] total=30 matched=30 skipped=0 diffed=0

Full harness on the merge result: 227 fixtures, 182 matched, 45 skipped
(cache_aligner + ccr stubs), 0 diffed, exit 0 — with kompress at 21/21 under
ONNX Runtime 1.24.4 (see #2591).

Also verified cargo check -p headroom-core --no-default-features passes, so the
static-musl path stays intact.
 2026-07-27 09:21:57 -07:00
Ruben A.
83e27e5036
feat(rust): port Kompress ML prose compressor to Rust (parity-only) (#1153) 
Adds crates/headroom-core/src/transforms/kompress.rs (672 lines): the Kompress
ML prose compressor ported to Rust, running the ModernBERT tokenizer plus the
kompress-v2-base ONNX model through ort with a cache-only loader that never
touches the network.

Parity-only. Nothing calls it: the only references outside the module are the
pub mod / pub use declarations in transforms/mod.rs. live_zone.rs still carries
TODO(PR-B4), so PlainText dispatch remains a no-op and Python continues to serve
prose compression. The pyo3 bridge is untouched and no Python source changes, so
the new engine is unreachable from the shipped package. #1155 wires it up.

Ships 21 recorded parity fixtures, a KompressComparator in headroom-parity, and
scripts/record_kompress_fixtures.py.

Verified byte-identical to the recorded Python output:

  [kompress] total=21 matched=21 skipped=0 diffed=0

That required ONNX Runtime >= 1.24 (see #2591) — below it ort deadlocks instead
of erroring, which is why these fixtures had never been run. In CI the model is
absent from the HF cache, so the comparator errors and the fixtures report
Skipped rather than hanging.

Also gates the module behind the ml feature, matching magika_detector: kompress.rs
uses ort, which is optional = true, so an unconditional pub mod broke
cargo check --no-default-features (the static-musl path). CI does not catch that
class of break because cargo test --workspace only builds default features.
 2026-07-27 08:17:53 -07:00
Zhenjia ZHOU
4035c04187
feat(text-crusher): CJK-aware segmentation + relevance via ICU (#1504) 
## Description

`TextCrusher` (the native extractive prose compressor added in #1171)
only handled ASCII: `split_segments` split on `.!?`+whitespace and
`tokens` split on whitespace/alphanumeric runs. CJK
(Chinese/Japanese/Korean) has neither spaces nor ASCII terminators, so a
whole CJK paragraph collapsed into **one segment / one token** — it
passed through at ~0% compression, and BM25 relevance + salience scored
zero terms.

This makes `TextCrusher` CJK-aware. CJK-bearing content takes an ICU
(`icu_segmenter`, UAX#29 sentence + dictionary word) segmentation path,
with a length fallback for terminator-sparse runs, a local BM25 over the
ICU word tokens, and ICU-token salience. Dispatch is on **content
only**, so pure-ASCII text is byte-identical to before — the shared
`BM25Scorer` and the ASCII path are untouched.

It also adds a committed, reproducible answer-retention eval
(`benchmarks/i18n_compression_eval.py`) with a deterministic zh/ja/ko CI
regression gate, so the improvement below is permanently verifiable
rather than a one-off measurement.

Extends #1171.

## Type of Change

- [x] Bug fix (CJK passed through near-uncompressed)
- [x] New feature (CJK segmentation / relevance support)
- [x] Performance improvement (CJK now compresses; ICU segmenters
cached, not rebuilt per call)

## Changes Made

- `is_cjk` predicate gates a CJK path (ideographs, kana, Hangul, CJK
punctuation, full/half-width forms).
- `split_segments` → ICU `SentenceSegmenter` for CJK + a mandatory
length fallback (whitespace / CJK punctuation / hard cap) for
terminator-sparse runs; ASCII path unchanged.
- `tokens` → ICU `WordSegmenter` (dictionary) for CJK; ASCII path
unchanged.
- `relevance_cjk`: a local BM25 over ICU word tokens — the shared ASCII
`BM25Scorer` scores zero terms for CJK and is parity-locked, so this is
an intentional separate scorer (documented in code).
- CJK salience uses ICU tokens (whitespace-split gave one giant "word" →
zero salience).
- `count_tokens`: CJK-aware so `compression_ratio` isn't nonsense for
space-free text.
- ICU segmenters resolved once in `static LazyLock` (compiled_data is
static) instead of rebuilt per call.
- New dep `icu_segmenter` 2.2, `compiled_data` only (see Dependency
below).
- `benchmarks/i18n_compression_eval.py` +
`tests/test_transforms/test_text_crusher_cjk_eval.py`: a zh/ja/ko
answer-retention eval — a deterministic needle CI gate (always-runs, no
external data), real-transcript fidelity with CJK-aware salient, and
optional `multi-wiki-qa` natural-data retention (loaded via the
`[evals]` `datasets` extra, skipped if absent; data never vendored —
CC-BY-NC-SA).

## Testing

- [x] Unit tests pass (`pytest` + `cargo test`)
- [x] Linting passes (`ruff check`/`format` on the new eval + test —
clean)
- [ ] Type checking passes (`mypy headroom`) — N/A, the only Python
added is a benchmark + test, not `headroom/` source
- [x] New tests added for new functionality
- [x] Manual testing performed (see Real Behavior Proof)

### Test Output

```text
$ cargo test -p headroom-core --lib text_crusher
running 12 tests
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 841 filtered out

$ .venv/bin/python -m pytest tests/test_transforms/test_text_crusher*.py
15 passed

$ .venv/bin/python -m pytest tests/test_transforms/test_text_crusher_cjk_eval.py
6 passed   # deterministic zh/ja/ko needle CI gate

$ cargo clippy -p headroom-core && ruff check benchmarks/i18n_compression_eval.py   # both clean
```

## Real Behavior Proof

- Environment: macOS (Darwin 25.3.0), Python in a uv venv,
`headroom-core` built via `uv pip install -e .` (maturin), branch
`feat/cjk-text-compression`.
- Exact command / steps: built `_core`, then ran a mixed
Chinese+Japanese doc (no spaces, `。` terminators) through
`TextCrusher().compress(doc, "认证令牌缓存策略", 0.3)`; separately evaluated
answer-retention on the public CMRC2018 Chinese QA dev set (bury the
gold-answer paragraph among 25 distractors, query = the question,
compress to 30%, check the gold answer survives), and end-to-end through
`ContentRouter`.
- Observed result: a mixed Chinese+Japanese doc compressed 189 → 78
tokens (ratio 0.41, kept 3/8 segments) with the query-relevant sentence
surviving — before this change the same doc was a single segment → 100%
passthrough. On the public CMRC2018 Chinese QA dev set, answer-retention
under 30% compression rose 34% → 93% (multiple seeds). End-to-end
through `ContentRouter` on real CJK content, aggregate savings rose 16%
→ 40%. Pure-ASCII (English) output stayed byte-identical (the English
parity fixtures did not move). Demo terminal output:

    ```text
    ORIGINAL  tokens= 189  chars=189
    COMPRESS  tokens=  78  ratio=0.41  segments kept 3/8
    QUERY-RELEVANT sentence survived: True
    --- compressed output (verbatim kept CJK sentences) ---
    认证令牌的缓存策略采用最近最少使用淘汰算法来管理过期条目。
    请求重试使用指数退避并设置最大次数上限。
    数据备份每天凌晨执行并保留最近三十天的快照。
    ```
The committed eval now demonstrates this across all three CJK languages.
The deterministic needle gate (in CI via
`tests/test_transforms/test_text_crusher_cjk_eval.py`, 6 passed) has
TextCrusher keep the query-relevant needle while truncate/random drop it
in zh, ja, and ko. On real `multi-wiki-qa` natural data (n=80/lang),
query-aware answer-retention is **zh 74% / ja 70% / ko 50%** vs
**25–41%** for the truncate/random baselines:

    ```text
=== Part A: multi-wiki-qa answer-retention (n=80/lang, target_ratio=0.3)
===
      lang    text_crusher  truncate  random
      zh-cn           74%       25%     38%
      ja              70%       31%     39%
      ko              50%       26%     41%
    ```
Korean is measurably weaker (ICU has no Korean dictionary and falls back
to UAX#29 word-breaking) — still well above baselines, and scoped as a
follow-up.
- Not tested: the live proxy HTTP path (validated at the `ContentRouter`
/ `TextCrusher` layer, not via a running proxy); no-space Korean
(standard Korean is space-delimited and is covered); non-CJK SE-Asian
scripts (out of scope).

## Dependency (per CONTRIBUTING supply-chain policy)

`icu_segmenter` 2.2 (ICU4X), `features = ["compiled_data"]`:

- **Why this package (vs. ourselves / existing deps):** CJK needs
dictionary/UAX#29 segmentation. A hand-rolled char-bigram scored
slightly worse on real data (CMRC2018 answer-retention: 92.5% ICU vs 91%
bigram, 4 seeds); jieba/lindera are ZH-only or 13–207 MB dicts. ICU4X
covers zh/ja/ko in one crate. The existing `unicode-segmentation` does
UAX#29 only (no CJK dictionary), so it can't word-segment space-free
CJK.
- **Who maintains it:** the official `unicode-org` ICU4X project; active
release cadence (2.2 in 2025); no known CVEs.
- **Install surface:** ~13 new pure-Rust crates, no build scripts, no
native code, no build/runtime network. `compiled_data` bundles locale
data at compile time (hermetic). `auto`/`lstm` deliberately NOT enabled
— LSTM covers SE-Asian scripts (Thai/Lao), not CJK, and would pull in
`libm` for nothing.
- **Why this version:** 2.x is the stabilized ICU4X API (1.x used a
different data-provider model); floored at 2.2 (Cargo.lock pins the
patch) since segmenter boundaries are observable in output and bumps
should be deliberate.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation (CHANGELOG)
- [x] My changes generate no new warnings (clippy + fmt clean)
- [x] I have added tests that prove my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md

## Additional Notes

- **Parity:** the shared `BM25Scorer` (byte-exact parity-locked with
`headroom/relevance/bm25.py`) is untouched. `relevance_cjk` is a
separate local scorer because the shared one's tokenizer is ASCII-only.
The whole CJK path lives in Rust (`text_crusher.py` is a thin wrapper
over `_core`), so there is no Python mirror to keep in sync; the parity
fixtures stay green (only the CJK `unicode` fixture was re-recorded,
intentionally; English fixtures unchanged).
- **Known by-design gap (not a bug):** CJK content + a pure-ASCII query
yields no token overlap, so relevance falls back to recency + salience
(cross-script query matching is unsupported).
- The Python added is a benchmark
(`benchmarks/i18n_compression_eval.py`) plus its test, not `headroom/`
runtime source — both are `ruff`-clean; `mypy headroom` is unaffected.
- **License:** the optional Part A pulls `alexandrainst/multi-wiki-qa`
(CC-BY-NC-SA-4.0) at run time via the `[evals]` extra and is skipped if
absent — the dataset is never vendored into the repo, and the always-run
CI gate (Part C) uses only our own deterministic data.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 19:58:48 +00:00
Zhenjia ZHOU
6c68ff4e9f
perf(compression): take large cold-start contexts off the synchronous kompress path (#1171) (#1298) 
## Description

On a cold-start large context, kompress (ModernBERT ONNX) runs
**synchronously on the request thread** — ~200–300s for ~1M tokens. It
blows the 30s compression budget, leaks a non-preemptible worker, and
cascades (executor saturation → queue timeouts on healthy requests); on
timeout the request is forwarded **uncompressed** after eating 30s. This
adds four layered, **default-off, fail-open** mitigations so the request
path is never blocked on ML compression.

Closes #1171

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **Phase 0 — size gate** (`HEADROOM_KOMPRESS_MAX_TOKENS`, default
50000): route oversized text away from ModernBERT (→ LogCompressor /
TextCrusher / passthrough) at the single `_try_ml_compressor` boundary.
- **Phase 1 — cooperative deadline**
(`HEADROOM_COMPRESSION_DEADLINE_MS`, default 20000): any kompress run
self-terminates at the next chunk boundary past the budget, keeping the
unprocessed tail verbatim.
- **Phase 2 — TextCrusher** (`HEADROOM_TEXT_CRUSHER`): a new **native
Rust** extractive prose compressor in
`crates/headroom-core/src/transforms/text_crusher/`, exposed via PyO3 as
`headroom._core.TextCrusher` with a thin Python wrapper. It **reuses the
shared `crate::relevance::BM25Scorer`** rather than reimplementing BM25,
and ships record/replay parity fixtures (mirroring the SmartCrusher
Rust-core + Python-shim pattern).
- **Phase 3 — off-path compression**
(`HEADROOM_BACKGROUND_COMPRESSION`): forward uncompressed immediately
and compress in a per-process background drain; a byte-identical cache
hit on a later turn means the request never blocks on ML.
- Benchmark (`benchmarks/text_crusher_quality_eval.py`), CHANGELOG
entry, and docstrings documenting the fail-open limits.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`, new modules)
- [x] New tests added for new functionality
- [ ] Manual testing performed (Phase 0/1 gate-fire + deadline observed
on real traffic in earlier iterations; Phase 3 off-path is unit- +
byte-identity-tested, not yet live-validated)

### Test Output

```text
$ pytest tests/test_transforms/ tests/test_cache/ \
    tests/test_proxy/test_background_compression.py tests/test_proxy/test_phase3_byte_identity.py -q
501 passed, 37 skipped in 40.33s

$ cargo test -p headroom-core --lib text_crusher
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 834 filtered out

$ ruff check <changed files>
All checks passed!

$ mypy headroom/proxy/background_compression.py headroom/transforms/text_crusher.py
Success: no issues found in 2 source files
```

New coverage: size-gate incl. the strategy-dispatch funnel (KOMPRESS +
TEXT); partial-run deadline (chunk-0 compressed + chunk-1 verbatim
tail); BackgroundCompressor (dedup / queue-full / fail-open); Phase 3
byte-identity round-trip; TextCrusher unit + parity.

## Real Behavior Proof

- Environment: macOS, local dev — `uv` venv, Rust `_core` built via `uv
pip install -e .`.
- Exact command / steps: the `pytest` / `cargo test` / `ruff` / `mypy`
commands shown under Test Output; quality eval `python
benchmarks/text_crusher_quality_eval.py /tmp/squad_dev.json`.
- Observed result: 501 Python + 3 Rust tests pass; ruff + mypy clean on
changed/new modules. Quality eval: TextCrusher keeps ~94% of buried
SQuAD answers at 30% size vs ~36% truncate/random; self-contained speed
run ~333k words in ~76ms (one O(n) pass) — sub-second where ModernBERT
takes minutes (fast-vs-slow contrast, not a same-input run).
- Not tested: Phase 3 off-path on live traffic; multi-worker
(per-process by design — see Additional Notes).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

- **All four features are off by default and fail-open** — with the env
flags unset the paths are no-ops for realistic inputs; on any error the
request is forwarded (compressed if possible, else verbatim), never
dropped. A full background queue / duplicate key surfaces as
`deferred:dropped`.
- **Known limits (documented in `background_compression.py`):** Phase 3
is per-process, in-memory, and token-mode-only — these are
**lost-savings, never lost-correctness**, and consistent with the
project's existing per-process compression cache + sticky-session
multi-worker model. The startup multi-worker warning now names off-path
background compression.
- Phase 2 reuses the existing BM25 scorer; reuse did not improve
answer-retention over a Python prototype (query-awareness dominates) —
its value is the Rust speed + repo-conventional Rust-core/Python-shim
shape.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-23 10:48:06 -05:00
gglucass
0ce68dedd7
fix(proxy): restore Codex usage headers on WS and streaming SSE transports (#577) (#794) 
## Description

Codex's subscription/rate-limit window (the `x-codex-*` headers) was
being
**stripped on every transport Codex actually uses**, so session/weekly
usage
never reached the Codex CLI's own `/status` display, Headroom
`/stats`/dashboard,
or any consumer that sniffs the client-facing handshake. This PR
restores it on
**both** the WebSocket and streaming-SSE paths — the two halves of #577
— in one
place.

Fixes #577

**Supersedes #582 and #590.** This PR incorporates #582's SSE fix
(carried verbatim
with a `Co-authored-by` trailer) and additionally forwards the window
onto the client
`101` on the WS path, which #582/#590's capture-only WS code cannot do.
Both can be
closed as superseded once this merges — GitHub closing keywords only
auto-close
issues (hence `Fixes #577` above), not PRs, so #582/#590 need a manual
close.

### WebSocket (`gpt-5.4+`)

OpenAI delivers `x-codex-*` **only** on the upstream WS handshake
response, never
in data frames. `handle_openai_responses_ws` accepted the client WS
*before* it
connected upstream and never read `upstream.response.headers`, so the
window was
dropped. This reorders the handler to **connect upstream first**,
extract the
`x-codex-*` subset, then **accept the client WS with those headers
attached** to
the `101`, and refresh the Python state for `/stats` parity.

### Streaming SSE (incorporated from #582, @m16khb)

Codex CLI almost always streams. `streaming.py` neither captured
`x-codex-*` into
`CodexRateLimitState` nor forwarded it — the forwarded-header filter
matched only
the substring `"ratelimit"`, which `x-codex-*` does not contain. This
calls
`update_from_headers()` **before** the `>=400` early-return (so a
streaming 429/5xx
still refreshes the window, matching the non-streaming handlers) and
widens the
forward filter to pass `x-codex-*`.

> Credit: the SSE fix is @m16khb's work from #582, carried here verbatim
with a
> `Co-authored-by` trailer so the maintainer gets a single PR covering
both
> transports. This supersedes #582/#590's **WS** capture (which only
writes
> `/stats`); the connect-before-accept reorder additionally forwards the
window to
> the client `101`, which capture-only cannot do. #590's optional
snapshot
> persistence is intentionally left out (separable; hot-path sync write;
doesn't
> help the `101`-sniff consumers).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `openai.py`: add `_extract_codex_handshake_headers()` (strictly
`x-codex-*`, via
`raw_items()` to avoid `MultipleValuesError`; never
`set-cookie`/`authorization`).
- `openai.py`: reorder `handle_openai_responses_ws` — connect-only retry
loop runs
before `accept()`; `accept(headers=...)` carries the forwarded window;
first
client frame read afterward. HTTP fallback preserved; it now also
refreshes
  `/stats` from the HTTP response headers.
- `streaming.py`: capture `x-codex-*` on all statuses + widen the
forwarded-header
  filter (from #582).

### Diff-size note

The bulk of the `openai.py` line count is **whitespace-only
relocation**: the relay
block dedents one level out of the old per-attempt `async with`. Logical
change is
~290 lines. **Review with `?w=1`.** In API-key mode the handshake
carries no
`x-codex-*`, so the accept-header list is empty and the path behaves
exactly as
before — the fix only activates for ChatGPT-subscription auth.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

- WS: `test_ws_connect_happens_before_accept`,
`test_ws_forwards_codex_headers_to_client_accept`
(only `x-codex-*` forwarded; `set-cookie`/`authorization` excluded;
`/stats` refreshed),
`test_ws_connect_failure_falls_back_to_http`,
`test_ws_first_frame_timeout_after_connect_closes_upstream`.
- Fallback: `test_fallback_refreshes_codex_rate_limit_state`.
- SSE:
`test_codex_rate_limit_headers_captured_and_forwarded_in_streaming`,
  `test_codex_rate_limit_captured_on_streaming_429` (from #582).
- Wire-level e2e: `tests/e2e_ws_codex_usage_headers.py` boots the real
proxy + fake
upstream + real `websockets` client and reads the client `101` — closes
the gap
the unit tests stub (that uvicorn/starlette actually write
`accept(headers=...)`).

## Test Output

```
$ uv run pytest tests/test_proxy_streaming_ratelimit_headers.py \
                tests/test_ws_http_fallback.py \
                tests/test_openai_codex_ws_lifecycle.py \
                tests/test_openai_codex_ws_timings.py \
                tests/test_codex_rate_limits.py -q
63 passed in 0.83s

$ .venv/bin/python tests/e2e_ws_codex_usage_headers.py
[codex-hdr-e2e] client 101 headers:
    x-codex-primary-used-percent: 42
    x-codex-primary-window-minutes: 300
    x-codex-secondary-used-percent: 7
    x-codex-secondary-window-minutes: 10080
[codex-hdr-e2e] /stats reflects codex window (primary-used=42)
=== CODEX-HDR E2E ALL GREEN ===

$ uv run ruff check . && uv run ruff format --check <touched files>
All checks passed!
```

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

- **Why connect-before-accept (not capture-only).** Once `accept()`
sends the `101`,
headers can no longer be added; the `x-codex-*` window only exists after
we connect
upstream. Capturing into Python state (as #582/#590's WS code does)
fixes `/stats`
but not the Codex CLI's native display or any `101`-sniffing consumer —
those need
  the headers *on the client handshake*, which requires the reorder.
- **Security.** Forwarding is filtered strictly to `x-codex-*`;
`set-cookie`,
`authorization`, and all other upstream headers are never forwarded to
the client
  (asserted by both the unit test and the e2e).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

## Contract Schemas

Per maintainer request: a JSON Schema (draft 2020-12) artifact
enshrining the OpenAI
interaction expectations this changeset relies on, so drift is
detectable later.

Committed following the repo's parity convention:
- schema:
`tests/parity/fixtures/codex_openai_contracts/codex-openai-interaction.schema.json`
- test: `tests/test_codex_openai_contract_parity.py` binds the schema to
the **live code**
in both directions, so drift fails CI rather than living only in this
description -
every declared `x-codex-*` header must be consumed by
`parse_codex_rate_limits`, and
`_extract_codex_handshake_headers` must forward exactly the declared
subset and never
`set-cookie`/`authorization`. No new dependency (does not pull in
`jsonschema`).

It covers, as `$defs`:

- `WSUpstreamHandshakeResponse` / `StreamingUpstreamResponseHeaders` -
the upstream
`x-codex-*` header family (full superset, with per-header wire pattern +
the parsed
semantic type) the WS and SSE captures read. Source of truth:
`parse_codex_rate_limits`.
- `ClientForwardedHandshakeHeaders` - the WS-101 **allow/deny**
contract: only
`x-codex-*` may be forwarded; `set-cookie`/`authorization` are
explicitly forbidden
  (`propertyNames` + `not`).
- `ClientForwardedStreamingHeaders` - the wider SSE forward set
(`*ratelimit*` OR `x-codex*`).
- `WSClientRequestFrame` / `WSRelayEvent` / `HTTPFallbackRequestBody` -
the WS frame
  envelopes and the unwrapped HTTP-fallback POST body.
- `CodexRateLimitStatsOutput` - the headroom `/stats` shape the parity
tests assert.

Validated with `jsonschema` (Draft202012 `check_schema` passes; positive
instances from
the e2e validate; negative instances - a leaked `set-cookie`, a fallback
body still
carrying a top-level `type` - are correctly rejected).

<details>
<summary><code>codex-openai-interaction.schema.json</code> (draft
2020-12)</summary>

```json
{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://github.com/chopratejas/headroom/contracts/codex-openai-interaction.schema.json",
  "title": "Codex <-> OpenAI interaction contracts (PR #794)",
  "description": "Enshrines the OpenAI interaction expectations this changeset depends on, so drift is detectable. Header values are transported as strings on the wire; the `x-headroom-parsed-type` annotation on each records the semantic type the parser (headroom/subscription/codex_rate_limits.py) coerces them to. Sources: codex_rate_limits.parse_codex_rate_limits (header family + gating), openai._extract_codex_handshake_headers (WS-101 forward filter), streaming.py (SSE forward filter).",
  "$defs": {
    "OpenAICodexWindowHeaders": {
      "title": "x-codex-*-{primary,secondary} window headers",
      "description": "A rolling rate-limit/subscription window. A window is materialized iff its `*-used-percent` header is present and numeric; `*-window-minutes` and `*-reset-at` are optional. `primary` and `secondary` are independent and either may be absent.",
      "type": "object",
      "properties": {
        "x-codex-primary-used-percent": {
          "type": "string",
          "pattern": "^\\d+(?:\\.\\d+)?$",
          "x-headroom-parsed-type": "float (0-100, NaN-guarded)",
          "description": "Percent of the primary window consumed. Gates creation of the primary window."
        },
        "x-codex-primary-window-minutes": {
          "type": "string",
          "pattern": "^\\d+$",
          "x-headroom-parsed-type": "int",
          "description": "Primary window size in minutes."
        },
        "x-codex-primary-reset-at": {
          "type": "string",
          "pattern": "^\\d+$",
          "x-headroom-parsed-type": "int (Unix epoch seconds)",
          "description": "Absolute reset time of the primary window."
        },
        "x-codex-secondary-used-percent": {
          "type": "string",
          "pattern": "^\\d+(?:\\.\\d+)?$",
          "x-headroom-parsed-type": "float (0-100, NaN-guarded)",
          "description": "Percent of the secondary window consumed. Gates creation of the secondary window."
        },
        "x-codex-secondary-window-minutes": {
          "type": "string",
          "pattern": "^\\d+$",
          "x-headroom-parsed-type": "int"
        },
        "x-codex-secondary-reset-at": {
          "type": "string",
          "pattern": "^\\d+$",
          "x-headroom-parsed-type": "int (Unix epoch seconds)"
        }
      },
      "additionalProperties": true
    },
    "OpenAICodexCreditsHeaders": {
      "title": "x-codex-credits-* headers",
      "description": "OpenAI credits balance. A credits snapshot is materialized iff `x-codex-credits-has-credits` is present; `unlimited` defaults to false; `balance` is optional.",
      "type": "object",
      "properties": {
        "x-codex-credits-has-credits": {
          "type": "string",
          "pattern": "^(?:[Tt][Rr][Uu][Ee]|[Ff][Aa][Ll][Ss][Ee]|[01])$",
          "x-headroom-parsed-type": "bool (true|false|1|0, case-insensitive)",
          "description": "Gates creation of the credits snapshot."
        },
        "x-codex-credits-unlimited": {
          "type": "string",
          "pattern": "^(?:[Tt][Rr][Uu][Ee]|[Ff][Aa][Ll][Ss][Ee]|[01])$",
          "x-headroom-parsed-type": "bool (defaults false when absent/unparseable)"
        },
        "x-codex-credits-balance": {
          "type": "string",
          "x-headroom-parsed-type": "str (empty -> null)",
          "description": "Free-form server string, e.g. \"$5.00\"."
        }
      },
      "additionalProperties": true
    },
    "OpenAICodexMetaHeaders": {
      "title": "x-codex meta headers",
      "type": "object",
      "properties": {
        "x-codex-limit-name": {
          "type": "string",
          "x-headroom-parsed-type": "str (empty -> null)",
          "description": "Active limit/model label, e.g. \"gpt-5.2-codex-sonic\"."
        },
        "x-codex-promo-message": {
          "type": "string",
          "x-headroom-parsed-type": "str (empty -> null)",
          "description": "Server announcement. Also gates snapshot creation when present."
        }
      },
      "additionalProperties": true
    },
    "OpenAICodexRateLimitHeaders": {
      "title": "Full x-codex-* header family OpenAI may emit",
      "description": "Superset of every x-codex-* header headroom reads. parse_codex_rate_limits returns a snapshot iff at least one of: a primary window, a secondary window, a credits snapshot, or a non-empty promo message is present; otherwise null (treated as a non-Codex response). All members are individually optional.",
      "type": "object",
      "allOf": [
        { "$ref": "#/$defs/OpenAICodexWindowHeaders" },
        { "$ref": "#/$defs/OpenAICodexCreditsHeaders" },
        { "$ref": "#/$defs/OpenAICodexMetaHeaders" }
      ],
      "additionalProperties": true
    },
    "WSUpstreamHandshakeResponse": {
      "title": "OpenAI WS handshake (101) response headers consumed by the WS fix",
      "description": "On the Codex WebSocket transport the x-codex-* window is delivered ONLY on the upstream handshake response (never in data frames). handle_openai_responses_ws reads upstream.response.headers here. This is the contract the connect-before-accept reorder depends on: if OpenAI ever moves these headers off the handshake (e.g. into a frame), the WS half of the fix goes stale.",
      "$ref": "#/$defs/OpenAICodexRateLimitHeaders"
    },
    "StreamingUpstreamResponseHeaders": {
      "title": "OpenAI streaming/HTTP response headers consumed by the SSE fix",
      "description": "On the streaming SSE/HTTP transport the same x-codex-* headers ride the HTTP response. streaming.py captures them on ALL statuses (including >=400) via update_from_headers, and forwards a wider set to the client (see ClientForwardedStreamingHeaders).",
      "$ref": "#/$defs/OpenAICodexRateLimitHeaders"
    },
    "ClientForwardedHandshakeHeaders": {
      "title": "Headers forwarded onto the CLIENT-facing WS 101 (allow/deny contract)",
      "description": "_extract_codex_handshake_headers forwards ONLY headers whose (lowercased) name starts with `x-codex-`. Every other upstream handshake header - notably set-cookie and authorization - MUST NOT appear on the client 101. Enforced by propertyNames below and asserted by the unit tests + tests/e2e_ws_codex_usage_headers.py.",
      "type": "object",
      "propertyNames": {
        "pattern": "^[Xx]-[Cc][Oo][Dd][Ee][Xx]-"
      },
      "not": {
        "anyOf": [
          { "required": ["set-cookie"] },
          { "required": ["Set-Cookie"] },
          { "required": ["authorization"] },
          { "required": ["Authorization"] }
        ]
      },
      "additionalProperties": { "type": "string" }
    },
    "ClientForwardedStreamingHeaders": {
      "title": "Headers forwarded to the client on the streaming SSE path",
      "description": "streaming.py forwards a header iff `\"ratelimit\" in name.lower()` OR `name.lower().startswith(\"x-codex\")`. This is a SUPERSET of the WS allow-list: it additionally passes generic *ratelimit* headers (e.g. the Anthropic streaming path) which do not contain the x-codex prefix.",
      "type": "object",
      "propertyNames": {
        "pattern": "(?:[Rr][Aa][Tt][Ee][Ll][Ii][Mm][Ii][Tt])|^[Xx]-[Cc][Oo][Dd][Ee][Xx]"
      },
      "additionalProperties": { "type": "string" }
    },
    "WSClientRequestFrame": {
      "title": "Client -> proxy WS data frame (Responses API over WS)",
      "description": "Codex sends the request as a response.create envelope. The HTTP fallback unwraps `.response` for the POST body, forces stream=true, and strips any top-level `type`. A flattened variant (no envelope, fields at top level) is also tolerated by the fallback.",
      "type": "object",
      "properties": {
        "type": { "const": "response.create" },
        "response": {
          "type": "object",
          "properties": {
            "model": { "type": "string", "description": "e.g. gpt-5.4" },
            "input": {
              "description": "String prompt or Responses-API structured input array.",
              "type": ["string", "array"]
            },
            "stream": { "type": "boolean" }
          },
          "required": ["model"],
          "additionalProperties": true
        }
      },
      "required": ["type", "response"],
      "additionalProperties": true
    },
    "WSRelayEvent": {
      "title": "proxy -> client WS data frame (relayed Responses API event)",
      "description": "SSE `data:` payloads relayed verbatim as WS text frames. `[DONE]` sentinels are dropped (not relayed). Every relayed event is a JSON object carrying a `type`. response.completed additionally carries usage under `response.usage`. anyOf (not oneOf): an error event also satisfies the looser lifecycle shape, which is fine.",
      "anyOf": [
        {
          "title": "lifecycle event",
          "type": "object",
          "properties": {
            "type": {
              "type": "string",
              "examples": [
                "response.created",
                "response.output_item.added",
                "response.completed"
              ]
            },
            "response": { "type": "object", "additionalProperties": true }
          },
          "required": ["type"],
          "additionalProperties": true
        },
        {
          "title": "error event",
          "type": "object",
          "properties": {
            "type": { "const": "error" },
            "error": {
              "type": "object",
              "properties": { "message": { "type": "string" } },
              "required": ["message"],
              "additionalProperties": true
            }
          },
          "required": ["type", "error"],
          "additionalProperties": true
        }
      ]
    },
    "HTTPFallbackRequestBody": {
      "title": "proxy -> OpenAI HTTP POST body on WS->HTTP fallback",
      "description": "Derived from WSClientRequestFrame: the inner `.response` object, with `stream` forced to true and any top-level `type` removed.",
      "type": "object",
      "properties": {
        "model": { "type": "string" },
        "stream": { "const": true },
        "input": { "type": ["string", "array"] }
      },
      "required": ["model", "stream"],
      "not": { "required": ["type"] },
      "additionalProperties": true
    },
    "CodexRateLimitStatsOutput": {
      "title": "headroom /stats output for the codex tracker (CodexRateLimitSnapshot.to_dict)",
      "description": "Internal (headroom-emitted) shape produced from the headers above; the WS and SSE update_from_headers parity tests assert this is refreshed. Included so drift in our own surface is also caught.",
      "type": "object",
      "properties": {
        "limit_id": { "const": "codex" },
        "limit_name": { "type": ["string", "null"] },
        "primary": { "$ref": "#/$defs/CodexWindowDict" },
        "secondary": { "$ref": "#/$defs/CodexWindowDict" },
        "credits": {
          "oneOf": [
            { "type": "null" },
            {
              "type": "object",
              "properties": {
                "has_credits": { "type": "boolean" },
                "unlimited": { "type": "boolean" },
                "balance": { "type": ["string", "null"] }
              },
              "required": ["has_credits", "unlimited", "balance"],
              "additionalProperties": false
            }
          ]
        },
        "promo_message": { "type": ["string", "null"] },
        "captured_at": { "type": "number", "description": "Unix epoch seconds (float)." }
      },
      "required": ["limit_id", "limit_name", "primary", "secondary", "credits", "promo_message", "captured_at"],
      "additionalProperties": false
    },
    "CodexWindowDict": {
      "oneOf": [
        { "type": "null" },
        {
          "type": "object",
          "properties": {
            "used_percent": { "type": "number" },
            "window_minutes": { "type": ["integer", "null"] },
            "window_label": { "type": "string", "description": "e.g. \"5h\", \"7d\"-style label; \"unknown\" when window_minutes is null." },
            "resets_at": { "type": ["integer", "null"], "description": "Unix epoch seconds." },
            "seconds_until_reset": { "type": ["integer", "null"] }
          },
          "required": ["used_percent", "window_minutes", "window_label", "resets_at", "seconds_until_reset"],
          "additionalProperties": false
        }
      ]
    }
  }
}
```

</details>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: m16khb <m16khb@gmail.com>
 2026-06-09 15:55:53 -05:00
chopratejas
967b0db439 fix: B1 — retire ICM, RollingWindow, scoring, relevance + dependents 
Phase B step 1 of the live-zone-only realignment. Removes ~10K LOC of
"drop messages from history" machinery that became unreachable after
PR-A1 made `/v1/messages` a passthrough on the proxy. Live-zone-only
compression (PR-B2..B7) operates on content blocks within messages;
message-list mutation no longer happens in the pipeline.

Python deletes:
- headroom/transforms/intelligent_context.py (1077 LOC)
- headroom/transforms/rolling_window.py (395 LOC)
- headroom/transforms/progressive_summarizer.py (508 LOC)
- headroom/transforms/scoring.py (459 LOC)
- headroom/transforms/tool_crusher.py (338 LOC)
- 5 corresponding tests/test_transforms/* and tests/test_proxy_intelligent_context.py

Rust deletes:
- crates/headroom-core/src/context/* (manager, config, workspace,
  candidate, ccr_drop, strategy/, mod) + safety.rs replaced
- crates/headroom-core/src/scoring/* (mod, score, scorer, traits, weights)
- MessageScorerComparator from crates/headroom-parity (PR #338/#343
  becomes deletable; sunk cost stays sunk)
- 13 message_scorer fixtures + record_message_scorer.py

Rust adds (move + rewrite):
- crates/headroom-core/src/transforms/safety.rs — `tool_pair_indices`
  preserves the OpenAI/Anthropic tool_use ↔ tool_result pairing rule
  the live-zone dispatcher (PR-B2) needs. No IcmConfig dependency.

Surface refactors:
- HeadroomConfig: drop `tool_crusher`, `rolling_window`,
  `intelligent_context` fields; hoist `output_buffer_tokens` to top
  level (used by client.py).
- ProxyConfig: drop `intelligent_context*` fields.
- `headroom wrap` proxy server: retire IntelligentContextManager
  and RollingWindow imports + branch; pipeline is CacheAligner →
  ContentRouter (smart_routing) or CacheAligner → SmartCrusher
  (legacy).
- CLI: drop `--no-intelligent-context`, `--no-intelligent-scoring`,
  `--no-compress-first` flags.
- LangChain memory integration: rename `_apply_rolling_window` →
  `_apply_compression`, drop RollingWindowConfig dep. Threshold is
  now advisory — B6 will rework the contract.
- TransformPipeline.create_pipeline now takes only cache_aligner_config.
- headroom/__init__.py + headroom/transforms/__init__.py: strip
  exports of deleted symbols.

Bug fixes uncovered by full pytest sweep:
- providers/copilot/wrap.py: `environ or os.environ` collapsed
  empty-dict to falsy → callers passing `environ={}` accidentally
  pulled from os.environ. Use `environ if environ is not None else
  os.environ`.

Test correctness fixes:
- _DummyAnthropicHandler._retry_request gains **_kwargs to match
  the real handler signature post-A8.
- test_ws_http_fallback extracts JSON from `content=` (post-A3
  byte-faithful) rather than the obsolete `json=` kwarg.
- test_ccr_response_handler_extra fixture joins SSE events with
  `\n\n` per spec (post-A8 byte-buffer parser requirement).
- test_proxy_responses_phase_preservation: capture via direct
  handler attached to the named logger, so the assertion is
  order-independent (proxy `_setup_file_logging` flips
  `headroom.propagate=False` once any earlier test triggers it).
- conftest.py autouse fixture resets `headroom.propagate=True`
  before each test as a defensive measure for the same pollution.
- test_wrap_copilot_translated_backend_still_requires_byok:
  monkeypatch.delenv every provider key so the BYOK error
  actually fires.
- test_native_installers: skip when system bash < 4.3 (macOS ships 3.2).
- TestGeminiEmbedContent / TestGeminiBatchEmbedContents:
  pytest.mark.skip — proxy currently has no :embedContent route;
  feature gap, not regression.

Acceptance:
- cargo build --workspace + cargo clippy + cargo fmt --check: green.
- cargo test --workspace --exclude headroom-py: 777 passed.
- pytest: 4892 passed, 240 skipped, 0 failed.
- git grep returns only intentional comments referencing the deletion.

Per-PR-B1 plan: REALIGNMENT/04-phase-B-live-zone.md.
 2026-05-02 12:23:17 -07:00
chopratejas
21989e3640 fix(rust): port MessageScorer to Rust + parity harness (PR-A) 
Direct port of `headroom.transforms.scoring.MessageScorer` (459 LOC).
Foundation piece for the IntelligentContext port (PR-B onward).

What's wired:
- Deterministic factors fully ported: recency (exp-decay), forward
  references (tool_call_id graph), token density (unique/total).
- External-dep factors gated behind traits: `EmbeddingProvider` and
  `ToinProvider`. No concrete impls yet — both default to neutral
  values matching Python's `embedding_provider=None` / `toin=None`.
  PR-A1 wires fastembed; PR-A2 plugs in a PyO3 ToinProvider.
- ScoringWeights + MessageScore with serde + BTreeMap-ordered
  breakdown for stable JSON.

Parity:
- 13 fixtures recorded from Python, byte-equal under the comparator.
- Floats rounded to 5 decimals on both sides — absorbs f32-vs-f64
  drift in the weighted sum without masking real bugs.

Drive-by: re-fix three pre-existing clippy errors in
smart_crusher/crusher.rs that re-emerged with new test additions
(field_reassign_with_default + dead hash_array_for_ccr).
 2026-05-01 10:06:56 -07:00
chopratejas
625290cfbc chore(rust): port ContentDetector to Rust + parity harness + PyO3 bridge 
Faithful port of `headroom/transforms/content_detector.py` into
`headroom-core`. Same regex patterns, dispatch order, confidence
formulas, and line-count caps; lockstep with Python via 21 recorded
parity fixtures (every dispatch branch exercised).

- crates/headroom-core/src/transforms/content_detector.rs: regex-only
  detector (no ML); ContentType/DetectionResult mirror Python's enum +
  dataclass surface; metadata uses serde_json::Map for clean PyO3
  bridging.
- Tie-break in code detection: track scores in first-match insertion
  order (matches Python dict iteration semantics on `max()` ties).
- TypeScript second pattern is start-anchored — Python's
  `pattern.match(line)` is start-anchored, but the regex crate's
  `is_match` is unanchored, so the literal `:` prefix is required for
  parity.
- crates/headroom-parity: ContentDetectorComparator + universal f64
  normalization in `compare_fixture` (serde_json's lossy parse vs
  full-precision serialize creates a 1-ULP asymmetry that broke
  comparison; round-trip the actual through to_string/from_str).
- crates/headroom-py: detect_content_type/is_json_array_of_dicts and
  PyDetectionResult exposed via PyO3 with GIL released during scan.
- tests/parity/recorder.py: new `_wrap_function` for free-function
  recording; content_detector hook + 21 varied inputs covering JSON
  arrays, diffs, HTML, search, build/log, six languages, and fallbacks.

Sets up PR2 (ContentRouter scaffold) to call into Rust ContentDetector
in-process via the bridge.
 2026-04-27 23:57:39 -07:00
chopratejas
da7716a95a chore(rust): SmartCrusher CCR marker injection + walker unification 
Closes four gaps in the Rust SmartCrusher pipeline that, together,
wire CCR storage end-to-end so the LLM can actually retrieve dropped
data:

1. CCR-Dropped marker is now injected into process_value's lossy-path
   output as a sentinel object {"_ccr_dropped": "<<ccr:HASH N_rows_offloaded>>"}
   appended to the kept-items array. Previously the store held the
   original but no pointer reached the prompt -- the retrieval contract
   was data-on-server, no-way-to-ask. Sentinel-as-object preserves the
   array-of-dicts shape so downstream iteration with x.get(...) keeps
   working.

2. Walker / process_value drift removed. process_value gains a
   Value::String arm that handles stringified-JSON containers (parse,
   recurse, re-encode) and opaque blobs (CCR marker + store) -- same
   semantics walker.rs has always had, now reachable from the main
   crush() pipeline.

3. Opaque-string CCR now stores originals. DocumentCompactor gains an
   Option<Arc<dyn CcrStore>> field; emit_opaque_ccr_marker calls
   store.put when one is configured. Same hash regardless of store
   presence -- runtime contract is stable across configurations.
   Same wiring is shared between walker.rs and process_value via the
   extracted helper.

5. PyO3 surface adds SmartCrusher.compact_document_json(doc_json) ->
   compacted-json string. Routes through the crusher's existing CCR
   store, so ccr_get resolves both row-drop and opaque-string hashes.

Tests:
- 5 new Rust integration tests in ccr_roundtrip.rs (marker visibility,
  nested-array marker, opaque-string roundtrip, stringified-JSON
  recursion, walker-with-store)
- 4 new Python tests covering the marker visible-to-LLM contract via
  both the native PyO3 surface and the Python shim
- 5 legacy parity fixtures re-recorded (dict_array_*, duplicate_dicts_40)
  -- their lossy outputs now carry the sentinel; Rust + Python both
  match the new bytes (parity-run smart_crusher: 17/17)
 2026-04-27 20:25:22 -07:00
chopratejas
43d1aa0329 parity(smart_crusher): byte-equal harness — 17 fixtures green 
Adds the SmartCrusher half of the Rust-vs-Python parity harness. Path A
from the Stage 3c.1 plan: record fixtures from Python (with real
fastembed embeddings + the post-bug-fix code), drive the Rust port over
the same inputs, and assert byte-equal output on every recorded
scenario.

What's in:
- `tests/parity/record_smart_crusher.py`: standalone recorder for
  `SmartCrusher.crush(content, query, bias)`. The generic recorder
  framework only captures one positional, so this script writes its
  own JSON envelope `{input: {content, query, bias}, config, output}`.
  17 scenarios cover the planning paths exercised by ContentRouter
  in production: passthrough, smart_sample, top_n, time-series,
  duplicates, unicode (`ensure_ascii=False`), nested-3-deep, empties,
  bias above and below 1.0.
- `crates/headroom-parity/src/lib.rs`: `SmartCrusherComparator`
  reconstructs `SmartCrusherConfig` from the fixture's config block,
  runs Rust `SmartCrusher::crush()`, emits the same JSON shape Python
  serialized.
- `crates/headroom-parity/examples/diff_fixture.rs`: diagnostic CLI
  that prints expected-vs-actual for one fixture (used during the
  iteration that found the serializer bug below).

Serializer fix — found by the harness:
SmartCrusher uses `safe_json_dumps` (compact `(",", ":")` separators
+ `ensure_ascii=False`) for the wire bytes. The Rust port was using
`python_json_dumps` (default Python: `(", ", ": ")` + `ensure_ascii=
True`), which is the right choice for hashing but wrong for the
output. Refactored `anchor_selector.rs` to take a small
`JsonFmt { sort_keys, compact, ensure_ascii }` config so the three
flavors share one writer, added `python_safe_json_dumps`, and
switched `_smart_crush_content` to call it. All three flavors now
have byte-exact tests.

Two unit tests in `crusher.rs` were pinning the old (wrong) format
and have been re-pinned to the compact form.

Cross-language status:
- All 17 empty-query fixtures: byte-equal.
- Embedding-driven (non-empty-query) fixtures deferred until the
  ~0.0002 numeric drift between Python `onnxruntime` and Rust `ort`
  is resolved (or until we accept the drift via a tolerance — none of
  the 17 fixtures exercises a borderline relevance_threshold call,
  and downstream code only branches at the 0.3 threshold).

Tests: cargo test --workspace (388 + supporting) green.
 2026-04-27 00:01:26 -07:00
chopratejas
48c13245c5 fix(diff): close ContentRouter routing gaps for merge diffs and long preambles 
User audit caught three gaps that prevented DiffCompressor from being
invoked even when the input was a real diff. These complement the four
emit-time bugs fixed in the previous commit — those fixes only kick in
once DiffCompressor receives the input. Without these gap fixes, real
merge-commit diffs and `git log -p` outputs with long commit messages
were misrouted away from DiffCompressor entirely.

# The three gaps (each fixed in Python; gap 3 also fixed in Rust)

1. Detector scan window was hardcoded to first 50 lines.
   `_try_detect_diff` in content_detector.py only inspected
   `content.split("\n")[:50]`. `git log -p` outputs commonly have
   commit messages longer than 50 lines (releases, squashed commits,
   bots), pushing the `diff --git` header out of the detection window.
   Result: input was returned with `content_type=PLAIN_TEXT` and routed
   to the text compressor, never reaching DiffCompressor. Fix: window
   widened to 500 lines.

2. Detector regex didn't recognize merge-commit headers.
   `_DIFF_HEADER_PATTERN` matched `diff --git`, `--- a/`, and the
   regular `@@ -A,B +C,D @@` hunk header. Merge-commit diffs from
   `git log -p` use `diff --combined <path>`, `diff --cc <path>`, and
   combined-diff hunk headers `@@@+`. The shared `--- a/` line still
   triggered the detector with low confidence, but only barely. Fix:
   extended the regex to recognize all four merge-shaped header forms.

3. DiffCompressor parser only matched `^diff --git`.
   Even after fixing detection, the parser's `_DIFF_GIT_PATTERN`
   wouldn't match `diff --combined` or `diff --cc`, so merge diffs
   reached DiffCompressor and were treated as one giant pre-diff blob —
   passed through unchanged after the previous PR's pre-diff
   preservation fix. Fix: added `_DIFF_COMBINED_PATTERN` and
   `_DIFF_CC_PATTERN`; `_parse_diff` starts a new file section on any
   of the three header forms. Mirrored in Rust as `is_diff_header`
   helper that checks all three regexes.

# Why this matters end-to-end

DiffCompressor's value comes from being routed to. Detection +
parser-level coverage are upstream of the compressor — without them,
the compressor never sees the input. The previous PR's four bug fixes
(rename, combined-diff hunks, no-newline marker, pre-diff content) are
correct and necessary, but for merge commits and long-preamble diffs,
they were only firing on the rare cases where the detector misclicked
into DiffCompressor anyway. With these three gaps closed, the
ContentRouter→DiffCompressor pipeline actually engages on:
- `git log -p` outputs of any commit-message length
- Merge-commit diffs (`diff --combined`, `diff --cc`)
- Combined-diff snippets (`@@@`+ hunk-only inputs)

# New fixtures (3 added to the existing 24)

- `066bc82…` — `diff --combined` merge diff (3-way)
- `5d950a94…` — `diff --cc` merge diff (alternate form)
- `66c86f64…` — long pre-diff content (60-line commit message)
  followed by a rename diff (exercises detector scan widening +
  pre-diff preservation in tandem)

Parity: total=27 matched=27 skipped=0 diffed=0.

# Tests

- Python: 4 new tests across 2 new test classes —
  `TestRoutingGapMergeDiffs` (combined / cc parser) and
  `TestRoutingGapDetectorScanWindow` (long preamble detection +
  combined-diff regex recognition).
- Rust: 2 new tests covering combined / cc parser sections.

# Verification

- 27/27 parity fixtures byte-equal.
- Python: 41/41 tests pass (was 37).
- Rust: 18/18 transforms tests; 62/62 workspace; 5/5 proptests.
- `cargo fmt --check` clean; `cargo clippy --workspace --all-targets
  -- -D warnings` clean.
 2026-04-26 09:13:08 -07:00
chopratejas
6d47a0cd00 fix(diff_compressor): four silent information-loss paths in Python AND Rust 
Audit caught four bugs that the byte-equal parity harness can't catch on
its own — both Python and Rust were faithfully emitting the buggy output.
Fixed in lockstep so parity is maintained while the underlying behavior
is now correct on inputs the existing 20 fixtures didn't exercise.

# The four bugs (each fixed in both Python and Rust)

1. Renames silently dropped from output. Parser captured `is_renamed=True`
   but the emitter never emitted ANY rename markers. Output of a rename
   looked exactly like a plain modification of the old path. Fix: capture
   `rename from` / `rename to` / `similarity index N%` / `dissimilarity
   index N%` / `copy from` / `copy to` lines in a new `rename_lines` field
   on `DiffFile`; emit them after `diff --git` in canonical git ordering.

2. Combined diff hunks (`@@@`) silently dropped. Hunk-header regex only
   matched `@@`, so 3-way merge hunks had `current_hunk` never set and
   ALL their content fell through to the no-op branch. Fix in Python:
   regex switched to `^(@@+) ... \1` (backreferences match any number of
   `@`s on each side). Fix in Rust: alternation over `@@`, `@@@`, `@@@@`
   since `regex` is RE2-based and rejects backreferences. n>3 octopus
   merges still fall through; rare in practice.

3. `\ No newline at end of file` markers can be context-trimmed away.
   Treated as ordinary "other" lines — if more than `max_context_lines`
   from a `+`/`-` change, dropped. Round-trip-breaking for patches; can
   change whether the trailing line has a newline. Fix: in
   `_reduce_context`, force-add any line starting with `\` to the keep
   set regardless of distance.

4. Pre-diff content silently dropped. Anything before the first `diff
   --git` — commit messages from `git log -p`, email headers from `git
   format-patch`, fork-and-rebase metadata — was discarded. Fix:
   `_parse_diff` now returns `(pre_diff_lines, files)`; `format_output`
   prepends pre-diff content verbatim when present.

# Hidden parity bug found during the work

`_compress_files` constructed a fresh `DiffFile` from the parsed one but
only copied a subset of the fields by name. The new `rename_lines` and
`original_*_line` fields were silently dropped here, so the parser
populated them correctly but the emitter saw an empty `rename_lines`
list. Caught by writing a real test instead of a smoke test — the smoke
test passed because it hit the no-diff-found short-circuit, not the
parser/emitter pipeline. Constructor now copies all fields explicitly.

# Parity status

- Existing 20 fixtures: still byte-equal between fixed Python and fixed
  Rust. None of them exercised the buggy paths.
- 4 NEW fixtures recorded against fixed Python, exercising each bug-fix
  path: rename, 3-way combined diff, `\ No newline` marker far from
  changes, pre-diff commit headers. All 4 byte-equal between Python and
  Rust.
- Parity harness: total=24 matched=24 skipped=0 diffed=0.

# Observability

Some normalizations remain parity-bound (file mode `100644` hardcode,
`Binary files differ` simplification). Those are surfaced in
`DiffCompressorStats::file_mode_normalizations` /
`binary_files_simplified` (Rust) and via `logger.warning` (Python's new
`_log_loss_signals` helper, called once per compress).

# Tests

- Python: 4 new test classes (11 tests) covering rename markers,
  combined diffs, no-newline preservation, pre-diff content. Edge case:
  no pre-diff content must NOT add a leading blank line.
- Rust: 4 new `bugfix_*` unit tests with the same scenarios.
- Existing Python tests calling `_parse_diff` directly were updated for
  the new `(pre_diff, files)` tuple return.

# Verification

- Python: 37/37 tests pass (was 26).
- Rust: 16/16 transforms tests; 60/60 workspace unit tests; 5/5
  proptests; 1/1 doctest.
- Parity: 24/24 byte-equal.
- `cargo fmt --check` clean; `cargo clippy --workspace --all-targets
  -- -D warnings` clean.
 2026-04-26 09:13:08 -07:00
chopratejas
0414cb70e4 feat(rust): scaffold workspace + parity harness (phase-0) 
Bootstrap the Rust port of Headroom. Additive only — no existing Python
code modified. Ships the workshop, not the widgets.

Layout
  Cargo.toml (workspace) + rust-toolchain.toml
  crates/headroom-core    — transform library, stub only
  crates/headroom-proxy   — axum binary, /healthz only
  crates/headroom-py      — PyO3 cdylib, exposes headroom._core.hello()
  crates/headroom-parity  — Rust-vs-Python oracle harness + parity-run CLI

Tooling
  Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint
  .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly
  deny.toml for cargo-deny

Parity corpus
  tests/parity/recorder.py + scripts/record_fixtures.py
  125 recorded fixtures across 5 leaf transforms (ccr, tokenizer,
  log_compressor, diff_compressor, cache_aligner)

Docs
  RUST_DEV.md — developer setup and workspace reference
  docs/spec/022-rust-migration.md — migration plan and stage breakdown

.gitignore: whitelist scripts/record_fixtures.py; ignore target/

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-24 13:39:48 -07:00