`maturin develop` requires a virtualenv (it errors with "Couldn't find a
virtualenv or conda environment"). CI's setup-python provides a bare
system Python without a venv, so the dev script's build path doesn't
work there.
This switches CI to build a release wheel via `maturin build` and
install it with `pip install --force-reinstall --no-deps`. Then symlink
the installed `.so` into the in-tree `headroom/` package so the
editable install resolves `import headroom._core` past the source-dir
shadowing of site-packages.
`scripts/build_rust_extension.sh` is unchanged — it stays optimized for
local dev (where there IS a venv).
The python `DiffCompressor` was retired in this PR's stage-3b commit;
the public class now delegates to `headroom._core` (built from
`crates/headroom-py`). Without the wheel installed in CI, every test
that constructs a `DiffCompressor` fails with `ModuleNotFoundError:
No module named 'headroom._core'`.
This adds a build step to the main `test` job in `ci.yml` that:
1. Installs the stable Rust toolchain (`dtolnay/rust-toolchain@stable`).
2. Caches the cargo registry and build output (`Swatinem/rust-cache@v2`).
3. Installs maturin.
4. Runs `scripts/build_rust_extension.sh`, which calls `maturin develop`
and symlinks the built `.so` into the in-tree `headroom/` package
so the editable install resolves `import headroom._core`.
Only the main `test` job needs this — `test-extras` and `test-agno`
run narrow subsets that don't construct `DiffCompressor`. The existing
`rust.yml` workflow continues to handle wheel builds for distribution
and `cargo test` for the Rust workspace.
# The bug
Several test modules and two production modules loaded the project `.env`
at *import time*. During pytest collection (where every test module is
imported once), this populated `os.environ` with API keys from `.env`.
The skipif guards in `test_proxy_passthrough_integration.py` (and
others) evaluate at collection time:
@pytest.mark.skipif(not os.environ.get("OPENAI_API_KEY"), reason="...")
If the polluter module was collected *before* the guard, the guard saw
the leaked key, decided not to skip, and the integration tests ran
live against a fake key and failed. In a fresh local-dev venv with
`.env` + full `[dev]` extras, this manifested as ~16 spurious test
failures plus a misleading test runtime of 6+ minutes (live HTTP).
# Why now
CI does not see this (no `.env`). It only manifests when:
1. `litellm` (and friends) are installed — they run `dotenv.load_dotenv()`
on import, populating `os.environ` from `.env`.
2. A `.env` file with real API keys exists locally.
Until the venv was provisioned with the full `[dev]` extras during
recent test work, `pytest.importorskip("litellm")` and
`from headroom.pricing import litellm_pricing` both silently no-op'd
(via try/except ImportError → `LITELLM_AVAILABLE=False`), so the leak
never triggered. With litellm now installed, the latent bug surfaced.
# The fix — three patterns
1. **Production modules** (`headroom/pricing/litellm_pricing.py`,
`headroom/backends/litellm.py`): wrap the eager `import litellm` with
a snapshot/restore of `os.environ`. Any keys litellm's bundled
`python-dotenv` adds during import are deleted immediately. The
module is fully imported and cached in `sys.modules` so subsequent
imports hit the cache without re-running the side effect.
2. **Test modules using `pytest.importorskip("litellm")`**
(`test_backend_bugs.py`, `test_bedrock_region.py`,
`test_cost_tracker_counterfactual.py`): replace with
`tests._dotenv.importorskip_no_env_leak("litellm")`, which does the
same snapshot/restore around `importlib.import_module`.
3. **Test modules that intentionally need `.env` values for skipif
guards** (`test_compression_summary_*.py`, `test_query_echo.py`,
`test_cost_tracker_counterfactual.py`, `test_memory_usage_integration.py`,
`test_bundled_tools_savings.py`): replace module-level
`os.environ.setdefault(...)` / `dotenv.load_dotenv()` with
`tests._dotenv.load_env_overrides()` (returns a local dict — does
NOT mutate `os.environ`) plus `autouse_apply_env(...)` (function-
scoped fixture that applies via `monkeypatch.setenv`, auto-cleaned
at teardown). The skipif still works because
`ANTHROPIC_KEY = os.environ.get(...) or _env_overrides.get(...)`
reads from the local dict as fallback.
# Helper module
New `tests/_dotenv.py` exposes:
- `load_env_overrides() -> dict[str, str]` — read `.env` into a dict.
- `autouse_apply_env(overrides) -> fixture` — function-scoped autouse
fixture that applies via `monkeypatch.setenv`.
- `importorskip_no_env_leak(module) -> module` — drop-in
`pytest.importorskip` substitute that quarantines env mutations.
# Results
Local full-suite (excluding live-LLM and live-feed tests):
- Before: 46 failed, 4830 passed, 387s
- After: 2 failed, 4672 passed, 134s
The remaining 2 failures are unrelated environment-dependent tests
(missing `PIL` / Docker daemon).
The python `DiffCompressor` is replaced by a thin pyo3-backed shim that
delegates to `headroom._core.DiffCompressor`. There is no python
implementation and no env-var fallback — the wheel is a hard import.
Why now: opt-in defaults don't drive python retirement. Byte-equal
parity was already proven across 27 fixtures (stage 3a); keeping a
shadow python impl behind a flag is a permanent maintenance cost with
no operational benefit. Stage 3b deletes ~700 lines of python parser /
scorer / formatter code; the rust crate has its own coverage.
Surface preserved:
- `headroom.transforms.diff_compressor.DiffCompressor` — same class
name, same `__init__`, same `compress(content, context)` shape.
Returns python `DiffCompressionResult` dataclasses so call sites that
destructure with `asdict()` work unchanged.
- `DiffCompressorConfig` and `DiffCompressionResult` dataclasses kept.
- Sidecar `compress_with_stats(...)` exposes the rust-only
`DiffCompressorStats` (per-file hunk drops, context lines trimmed,
file_mode normalizations) for observability.
Removed:
- Python parser / scorer / formatter (~700 lines).
- Internal `DiffHunk` / `DiffFile` parser dataclasses (rust crate has
parallel coverage).
- 12 tests that probed `_parse_diff` / `_score_hunks` or the deleted
parser dataclasses. The 29 public-API tests in `test_diff_compressor.py`
remain and now exercise the rust backend through the same import path.
- `HEADROOM_DIFF_COMPRESSOR_BACKEND` env var — no longer meaningful.
Build:
- `scripts/build_rust_extension.sh` runs `maturin develop` and symlinks
the built `.so` into `headroom/` so `import headroom._core` resolves
past the in-tree package shadowing the maturin overlay.
- `.gitignore` excludes the symlinks and allowlists the build script.
Tests:
- 544 transforms pass; 33 rust-vs-fixture parity tests guard the pyo3
bridge against regressions (`tests/test_transforms/test_diff_compressor_rust_parity.py`).
- Mypy clean.
User audit caught three gaps that prevented DiffCompressor from being
invoked even when the input was a real diff. These complement the four
emit-time bugs fixed in the previous commit — those fixes only kick in
once DiffCompressor receives the input. Without these gap fixes, real
merge-commit diffs and `git log -p` outputs with long commit messages
were misrouted away from DiffCompressor entirely.
# The three gaps (each fixed in Python; gap 3 also fixed in Rust)
1. Detector scan window was hardcoded to first 50 lines.
`_try_detect_diff` in content_detector.py only inspected
`content.split("\n")[:50]`. `git log -p` outputs commonly have
commit messages longer than 50 lines (releases, squashed commits,
bots), pushing the `diff --git` header out of the detection window.
Result: input was returned with `content_type=PLAIN_TEXT` and routed
to the text compressor, never reaching DiffCompressor. Fix: window
widened to 500 lines.
2. Detector regex didn't recognize merge-commit headers.
`_DIFF_HEADER_PATTERN` matched `diff --git`, `--- a/`, and the
regular `@@ -A,B +C,D @@` hunk header. Merge-commit diffs from
`git log -p` use `diff --combined <path>`, `diff --cc <path>`, and
combined-diff hunk headers `@@@+`. The shared `--- a/` line still
triggered the detector with low confidence, but only barely. Fix:
extended the regex to recognize all four merge-shaped header forms.
3. DiffCompressor parser only matched `^diff --git`.
Even after fixing detection, the parser's `_DIFF_GIT_PATTERN`
wouldn't match `diff --combined` or `diff --cc`, so merge diffs
reached DiffCompressor and were treated as one giant pre-diff blob —
passed through unchanged after the previous PR's pre-diff
preservation fix. Fix: added `_DIFF_COMBINED_PATTERN` and
`_DIFF_CC_PATTERN`; `_parse_diff` starts a new file section on any
of the three header forms. Mirrored in Rust as `is_diff_header`
helper that checks all three regexes.
# Why this matters end-to-end
DiffCompressor's value comes from being routed to. Detection +
parser-level coverage are upstream of the compressor — without them,
the compressor never sees the input. The previous PR's four bug fixes
(rename, combined-diff hunks, no-newline marker, pre-diff content) are
correct and necessary, but for merge commits and long-preamble diffs,
they were only firing on the rare cases where the detector misclicked
into DiffCompressor anyway. With these three gaps closed, the
ContentRouter→DiffCompressor pipeline actually engages on:
- `git log -p` outputs of any commit-message length
- Merge-commit diffs (`diff --combined`, `diff --cc`)
- Combined-diff snippets (`@@@`+ hunk-only inputs)
# New fixtures (3 added to the existing 24)
- `066bc82…` — `diff --combined` merge diff (3-way)
- `5d950a94…` — `diff --cc` merge diff (alternate form)
- `66c86f64…` — long pre-diff content (60-line commit message)
followed by a rename diff (exercises detector scan widening +
pre-diff preservation in tandem)
Parity: total=27 matched=27 skipped=0 diffed=0.
# Tests
- Python: 4 new tests across 2 new test classes —
`TestRoutingGapMergeDiffs` (combined / cc parser) and
`TestRoutingGapDetectorScanWindow` (long preamble detection +
combined-diff regex recognition).
- Rust: 2 new tests covering combined / cc parser sections.
# Verification
- 27/27 parity fixtures byte-equal.
- Python: 41/41 tests pass (was 37).
- Rust: 18/18 transforms tests; 62/62 workspace; 5/5 proptests.
- `cargo fmt --check` clean; `cargo clippy --workspace --all-targets
-- -D warnings` clean.
Audit caught four bugs that the byte-equal parity harness can't catch on
its own — both Python and Rust were faithfully emitting the buggy output.
Fixed in lockstep so parity is maintained while the underlying behavior
is now correct on inputs the existing 20 fixtures didn't exercise.
# The four bugs (each fixed in both Python and Rust)
1. Renames silently dropped from output. Parser captured `is_renamed=True`
but the emitter never emitted ANY rename markers. Output of a rename
looked exactly like a plain modification of the old path. Fix: capture
`rename from` / `rename to` / `similarity index N%` / `dissimilarity
index N%` / `copy from` / `copy to` lines in a new `rename_lines` field
on `DiffFile`; emit them after `diff --git` in canonical git ordering.
2. Combined diff hunks (`@@@`) silently dropped. Hunk-header regex only
matched `@@`, so 3-way merge hunks had `current_hunk` never set and
ALL their content fell through to the no-op branch. Fix in Python:
regex switched to `^(@@+) ... \1` (backreferences match any number of
`@`s on each side). Fix in Rust: alternation over `@@`, `@@@`, `@@@@`
since `regex` is RE2-based and rejects backreferences. n>3 octopus
merges still fall through; rare in practice.
3. `\ No newline at end of file` markers can be context-trimmed away.
Treated as ordinary "other" lines — if more than `max_context_lines`
from a `+`/`-` change, dropped. Round-trip-breaking for patches; can
change whether the trailing line has a newline. Fix: in
`_reduce_context`, force-add any line starting with `\` to the keep
set regardless of distance.
4. Pre-diff content silently dropped. Anything before the first `diff
--git` — commit messages from `git log -p`, email headers from `git
format-patch`, fork-and-rebase metadata — was discarded. Fix:
`_parse_diff` now returns `(pre_diff_lines, files)`; `format_output`
prepends pre-diff content verbatim when present.
# Hidden parity bug found during the work
`_compress_files` constructed a fresh `DiffFile` from the parsed one but
only copied a subset of the fields by name. The new `rename_lines` and
`original_*_line` fields were silently dropped here, so the parser
populated them correctly but the emitter saw an empty `rename_lines`
list. Caught by writing a real test instead of a smoke test — the smoke
test passed because it hit the no-diff-found short-circuit, not the
parser/emitter pipeline. Constructor now copies all fields explicitly.
# Parity status
- Existing 20 fixtures: still byte-equal between fixed Python and fixed
Rust. None of them exercised the buggy paths.
- 4 NEW fixtures recorded against fixed Python, exercising each bug-fix
path: rename, 3-way combined diff, `\ No newline` marker far from
changes, pre-diff commit headers. All 4 byte-equal between Python and
Rust.
- Parity harness: total=24 matched=24 skipped=0 diffed=0.
# Observability
Some normalizations remain parity-bound (file mode `100644` hardcode,
`Binary files differ` simplification). Those are surfaced in
`DiffCompressorStats::file_mode_normalizations` /
`binary_files_simplified` (Rust) and via `logger.warning` (Python's new
`_log_loss_signals` helper, called once per compress).
# Tests
- Python: 4 new test classes (11 tests) covering rename markers,
combined diffs, no-newline preservation, pre-diff content. Edge case:
no pre-diff content must NOT add a leading blank line.
- Rust: 4 new `bugfix_*` unit tests with the same scenarios.
- Existing Python tests calling `_parse_diff` directly were updated for
the new `(pre_diff, files)` tuple return.
# Verification
- Python: 37/37 tests pass (was 26).
- Rust: 16/16 transforms tests; 60/60 workspace unit tests; 5/5
proptests; 1/1 doctest.
- Parity: 24/24 byte-equal.
- `cargo fmt --check` clean; `cargo clippy --workspace --all-targets
-- -D warnings` clean.
Audit follow-up to the diff_compressor port. The parity-bound port had
several issues that the byte-equal harness can't catch: hardcoded magic
numbers, score-weight literals scattered through the scorer, and silent
information-loss paths (file mode + binary detail) with no observability
trail. Parity is preserved (still 20/20 byte-equal); these are quality
and observability fixes only.
# Changes
1. **Hardcoded `0.8` savings threshold → config knob.** New
`DiffCompressorConfig::min_compression_ratio_for_ccr: f64` (default
0.8) replaces the literal. Rust-only — Python hardcodes 0.8 too;
fixtures don't carry the field, parity comparator defaults to 0.8.
2. **Score-weight magic numbers → documented `pub const`.** The scorer's
`0.03`, `0.3`, `0.2`, `0.3`, `1.0`, and `len > 2` filter are now
`SCORE_CHANGE_DENSITY_WEIGHT`, `SCORE_CHANGE_DENSITY_CAP`,
`SCORE_CONTEXT_WORD_WEIGHT`, `SCORE_CONTEXT_MIN_WORD_LEN`,
`SCORE_PRIORITY_PATTERN_BOOST`, `SCORE_TOTAL_CAP`. Each carries a doc
comment explaining what it biases toward. A pinning test fails if
anyone changes the value without updating the test.
3. **File mode normalization is now visible.** Parity forces emit to
hardcode `100644` regardless of input mode (`100755` executable,
`120000` symlink, `160000` submodule). The parser now captures the
ORIGINAL mode line; if it's not `100644`, the loss is recorded in
`DiffCompressorStats::file_mode_normalizations: Vec<(label,
original_mode_line)>`. Empty when no normalization happened.
4. **Binary file detail loss is now visible.** Same pattern: parity
forces `Binary files differ` on emit, dropping the original `Binary
files X and Y differ` filenames. Captured in
`DiffCompressorStats::binary_files_simplified: Vec<String>` whenever
the original line carried richer detail.
5. **`min_lines_for_ccr` doc clarified.** Despite the name, it gates the
ENTIRE compression path, not just the CCR marker. A 49-line diff is
returned unchanged regardless of how compressible it is. Doc comment
now flags this misnomer prominently.
6. **6 new unit tests cover the lossy paths.** Previously the test
suite only exercised pass-through and the in-cap synthetic. New
tests:
- `max_hunks_per_file_cap_drops_excess_and_records_stats` (15
hunks, cap 10, verify 5 dropped + per-file accounting)
- `max_files_cap_drops_files_and_records_names_in_stats` (25
files, cap 20, verify 5 dropped names recorded)
- `file_mode_normalization_is_recorded_for_executable_bit` (input
`new file mode 100755`, verify stats capture original)
- `binary_files_simplification_is_recorded` (input `Binary files
a/x and b/x differ`, verify original captured)
- `min_compression_ratio_for_ccr_is_configurable` (default 0.8 vs
custom 0.5; same input, different CCR emit decision)
- `score_constants_match_inline_values` (pins constants so future
changes are intentional)
Stats fields are also reflected in the existing `tracing::info!` event
so OTel scrapers see the loss counts in production.
# Why these matter — the user-stated principle
"Information preservation > aggressive compression." When parity forces
us to drop bytes (file mode, binary detail, fully-dropped files,
trimmed context), the loss must be observable. Adding it to the
compressed output bytes would break parity; surfacing it in
`DiffCompressorStats` + tracing spans is the right escape hatch.
# Verification
- 20/20 parity fixtures still byte-equal
- 12/12 unit tests pass (6 new + 6 prior)
- 56 total tests in headroom-core, all green
- `cargo fmt --check` clean
- `cargo clippy --workspace --all-targets -- -D warnings` clean
Stage 3a: first real transform port. Faithful Rust port of
`headroom.transforms.diff_compressor` with byte-equal parity against all
20 recorded fixtures.
# Algorithm (matching Python)
1. Hand-rolled unified-diff parser (state machine over `diff --git`,
`index`, `--- a/`, `+++ b/`, `@@`, mode/binary/rename markers, +/- /
space lines, "other" lines like `\ No newline at end of file`).
2. File cap (`max_files=20`): when fired, sort by total changes (most
first) and keep top N.
3. Per-file hunk cap (`max_hunks_per_file=10`): keep first + last + top
relevance-scored middle, then resort by hunk-header start line to
restore appearance order.
4. Relevance scoring: change-density base + user-query word overlap
+ priority patterns (ERROR / IMPORTANCE / SECURITY regexes —
matches `error_detection.PRIORITY_PATTERNS_DIFF`).
5. Per-hunk context trim: keep `max_context_lines=2` lines either side
of each `+`/`-` line.
6. CCR cache_key: `md5(original)[:24]` (matches
`compression_store.CompressionStore.store`). Emitted only when
compression saved >20% of lines.
Parity result: `[diff_compressor ] total=20 matched=20 skipped=0 diffed=0`.
# Information preservation hardening
Three pass-through paths inherited from Python that we keep deliberate
(would lose info if we changed them):
- Below `min_lines_for_ccr` (50): return input unchanged.
- No diff sections parsed: return input unchanged.
- Below 20% compression savings: emit compressed output but no CCR
marker (the original is the cheaper representation anyway).
Plus a parity-bound subtlety: `compressed_line_count` is captured BEFORE
the CCR retrieval marker is appended, both for the marker text
(`compressed to N`) and the result field. The output string therefore
ends up with one more line than the field reports — by design, matching
Python exactly. An off-by-one bug from recounting after appending the
CCR marker was caught and pinned by a synthetic 8-file diff test.
# Observability — the Rust escape hatch
Python's `DiffCompressionResult` has thin observability: input/output
line counts, additions/deletions, hunks_kept/removed, files_affected,
cache_key. The Rust port adds a sidecar `DiffCompressorStats` struct
with metrics Python doesn't emit:
- `files_dropped: Vec<String>` — names (old → new path) of files
silently discarded by the `max_files` cap. Python loses these.
- `hunks_dropped_per_file: BTreeMap<String, usize>` — per-file hunk
drops, stable iteration via `BTreeMap`.
- `context_lines_input` / `context_lines_kept` / `context_lines_trimmed`
— directly proxies info loss from the context trim.
- `largest_hunk_kept_lines` / `largest_hunk_dropped_lines` — outlier
detection (a single huge dropped hunk is much worse than many small).
- `parse_warnings: Vec<String>` — surfaces malformed input rather than
dropping silently.
- `processing_duration_us` — latency budget.
- `cache_key_emitted` + `ccr_skipped_reason: Option<String>` — explicit
signal for "we chose not to emit CCR and this is why".
A `tracing::info!(target: "diff_compressor", ...)` event is emitted on
every call, carrying these fields for OTel scraping in prod. The
sidecar struct is returned alongside via `compress_with_stats`; the
parity-only `compress` API discards it.
# Module layout
- `crates/headroom-core/src/transforms/mod.rs` — namespace, doc comment
with the guiding principle ("information preservation > aggressive
compression") so future ports inherit the philosophy.
- `crates/headroom-core/src/transforms/diff_compressor.rs` — full port
(parser, scorer, hunk selector, context trimmer, formatter, CCR layer,
stats, tracing).
# Dependencies added to headroom-core
- `md-5 = "0.10"` — for the CCR cache_key (matches Python MD5[:24]).
- `regex = "1"` — was a transitive dep via tokenizers; now a direct
dependency for the hunk-header parser and priority patterns.
# Tests
6 unit tests covering pass-through paths, MD5 hex truncation, the
Python `split("\n")` line-count semantics, sidecar stats emission,
and a synthetic 8-file diff that locks the byte-equal behavior found
in the parity fixtures.
Stage 2.1: closes the loop on the HuggingFace tokenizer story. Stage 2
shipped `HfTokenizer::from_bytes`/`from_file`, which required callers to
manage their own tokenizer.json files. This adds the third constructor:
let t = HfTokenizer::from_pretrained("CohereForAI/c4ai-command-r-v01")?;
register_hf("command-", t);
`from_pretrained` is a thin wrapper around the `hf-hub` crate's blocking
`ureq` API. First call downloads `tokenizer.json` to `~/.cache/huggingface/
hub` (or `$HF_HOME` if set); subsequent calls reuse the on-disk cache. Uses
the `main` revision; gated repos (Llama, Mistral) require `HF_TOKEN` in env
or `~/.cache/huggingface/token`.
Also adds `try_register_hf(prefix, repo)` as the obvious one-liner for
proxy startup code:
let _ = try_register_hf("command-", "CohereForAI/c4ai-command-r-v01");
let _ = try_register_hf("mistral-", "mistralai/Mistral-7B-v0.1");
Each call is independent — a download failure for one model (e.g. gated
without a token) does not affect others.
`HfTokenizerError` gains a new `Hub` variant so callers can distinguish
"couldn't fetch" from "fetched but malformed" — relevant when deciding
whether to retry, surface to the user, or fall back to the estimator.
Why blocking, not async: `from_pretrained` is called once at startup. A
sync API works from `main()`, from a `OnceLock` initializer, or from
`tokio::task::spawn_blocking` if a tokio caller needs it later. The async
hf-hub backend would force callers to await at startup, which doesn't fit
the `register_hf` registry pattern.
Why rustls, not native-tls: keeps the binary statically linkable for AWS
deploys (no system OpenSSL dependency).
Tests: a network-dependent integration test (`#[ignore]`d in CI; hits HF
for `gpt2`, ~1.4 MB) verifies the real download + load + count path. A
non-network negative test verifies that an invalid repo name surfaces as
`HfTokenizerError::Hub`, not a panic. 44 unit tests + 5 proptests +
1 doctest pass; parity stays 40/40 byte-equal.
The `qdrant-env-vars` change in d3c37d7 (PR #266) added `qdrant_url` and
`qdrant_api_key` keys to the kwargs that `MemoryHandler` passes into
`DirectMem0Adapter.__init__`. The corresponding assertion in
`test_ensure_initialized_fast_paths_and_qdrant_variants` was missed in that
PR and has been failing on `main` ever since. Surfacing here because it
fails on every PR's CI; not caused by the Rust tokenizer work this branch
adds.
The two new keys are both `None` when the corresponding `HEADROOM_QDRANT_*`
env vars are unset, which is the case in this test.
Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the
Python `headroom.tokenizers` surface, with three backends behind a single
`Tokenizer` trait.
Backends, in dispatch order:
1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any
public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the
Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen,
BERT, T5, etc. Construct from bytes or a file path; register against a
model-name prefix via `register_hf` for automatic dispatch. No `hf-hub`
auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out
of core. Longest-prefix wins; lookups are RwLock-protected.
2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series
families. Byte-identical to Python `tiktoken` for ordinary text. Lazy
shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base,
r50k_base).
3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback.
Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up
formula (a self-review caught and fixed an earlier `ceil`-based version
that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt).
Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal.
Bench: criterion baseline on small/medium/large inputs.
Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`.
No proxy wiring. Library-only; production behavior unchanged.
GitHub Actions deprecated the macos-13 runner label. The validate-workflows
actionlint step in CI fails because macos-13 is no longer in the available
labels list. macos-15-intel is the current x86_64 macOS runner.
(Bumped from macos-14 to macos-15 for arm64 was unnecessary; macos-14 is
still valid and we keep it for cache-warmth.)
cargo fmt --check failed in CI: import order in proxy.rs (cfg(test)
attributes before/after non-attr imports) and a few line-wrapping
nits in e2e_real.rs. Ran cargo fmt --all to fix.
maturin-action@v1 does not have a 'manifest-path' input — the action
warned 'Unexpected input(s) manifest-path' and proceeded to invoke
maturin from the repo root, which sees the workspace Cargo.toml with
no [package] section and bails. Move -m crates/headroom-py/Cargo.toml
back inside the 'args' string.
Cargo.lock: pick up tokio-util added in the WS half-close fix.
RUST_DEV.md: document how to run headroom-proxy in passthrough mode
(listen + upstream flags, e2e test gate, env vars).
Previously, any package registered under the headroom.proxy_extension
entry-point group auto-loaded at proxy startup. A user pip-installing a
plugin (or pulling one in transitively) would get its middleware running
in front of all their LLM traffic with zero opt-in or visibility — the
same mechanism that masked the Shield Enterprise streaming bug.
Change: install_all() now takes an explicit enabled set (or reads
HEADROOM_PROXY_EXTENSIONS). Discovery still runs to enumerate what's
available, but only names the operator opted into actually install.
The literal '*' is a wildcard for trusted environments.
CLI: headroom proxy --proxy-extension shield_enterprise
headroom proxy --proxy-extension shield_enterprise,mypkg
headroom proxy --proxy-extension '*'
Env: HEADROOM_PROXY_EXTENSIONS=shield_enterprise
The startup banner now shows discovered + enabled extensions:
Extensions: discovered=shield_enterprise (opt-in: --proxy-extension ...)
Extensions: ENABLED shield_enterprise (available: shield_enterprise)
Extensions: ENABLED (wildcard) shield_enterprise
Names that were requested but not found are logged as warnings.
Adds proxy_extensions: list[str] | None to ProxyConfig. Plumbs it
through CLI -> ProxyConfig -> install_all(enabled=...).
This is a behavior change for users who relied on auto-loading.
Existing Shield/extension users must add --proxy-extension or set
HEADROOM_PROXY_EXTENSIONS to keep their middleware running.
Adds 0.10.7-ab46594 (root) and 0.10.7-<variant>-<sha> (variants) so
images can be referenced by an exact version+commit pair without
relying on the moving variant or :latest tags.
CodeQL alert #61 (CWE-275, actions/missing-workflow-permissions):
add explicit `permissions: contents: read` to the rust workflow root.
Defaults the GITHUB_TOKEN to read-only across all jobs, so even if the
repo policy changes, this workflow stays at least-privilege. No job in
this workflow needs write — wheels/audit/parity all read-only.
Add real end-to-end test suite at tests/e2e_real.rs gated behind
HEADROOM_E2E=1. Spawns the actual Python Headroom proxy as a subprocess,
runs the Rust proxy in-process in front of it, and exercises:
- health endpoints across the full chain
- Anthropic non-streaming (real API call)
- Anthropic streaming SSE (real API call) with chunk-level validation
- OpenAI non-streaming (real API call)
- X-Request-Id generation and pass-through
Adds tokio-process feature for Command/Child usage. Loads .env at the
repo root for API keys (does not log values). Tests skip cleanly when
HEADROOM_E2E is unset, so cargo test stays fast.
When the docker workflow is triggered directly by release.published
(rather than via workflow_call from the Release parent), inputs.enable_ref_tags
is null and produced an empty enable= attribute that the metadata-action
rejected. Default to true on non-release triggers and skip ref/pr tags
on release events where they don't apply anyway.
- Replace full-sha image tags with type=sha,format=short (7-char) so the
primary package versions list stops accumulating long sha-only entries.
- Route cosign signatures into a sibling GHCR package via
COSIGN_REPOSITORY=<image>-signatures, so the main image's package
version list stays clean. GHCR does not yet implement the OCI 1.1
Distribution Referrers API (community discussion #163029, June 2025),
so legacy signature mode is used here -- OCI 1.1 mode would force the
signature manifest's subject into the same repo as the image and
override COSIGN_REPOSITORY. Verifiers must export the same
COSIGN_REPOSITORY value when running 'cosign verify'.
- Add a promote-latest job that runs after the variant matrix and
re-pushes the :latest tag pointing at the root image with a unique
index annotation. This forces a fresh manifest digest, generating a
new GHCR package version with current timestamp so :latest sits at
the top of the version listing instead of whichever variant happened
to finish last.
Bug 1 (HIGH) health.rs: Url::join('healthz') used relative resolution,
stripping non-trailing-slash base paths. Fixed with set_path('/healthz').
Bug 2 (HIGH) main.rs: graceful_shutdown_timeout was configured and logged
but never enforced. Now sleeps for the configured duration after signal
before axum exits, giving in-flight LLM streams time to drain.
Bug 3 (MEDIUM) websocket.rs: WS pump half-close could hang forever if
close() on one side failed. Replaced tokio::join! on async blocks with
spawned tasks + CancellationToken so either direction cancels the other.
Bug 4 (MEDIUM) proxy.rs/websocket.rs: URL path-join logic was copy-pasted
verbatim in two places. Extracted to join_upstream_path() helper; websocket
now calls it instead of duplicating the 15-line block.
Bug 5 (MEDIUM) proxy.rs: mid-stream upstream errors were silently swallowed
by Body::from_stream. Added a .map() wrapper that logs before re-raising.
Bug 6 (LOW) websocket.rs: WS session log was missing the request path,
making it hard to correlate logs with client sessions. Added path field.
Bug 7 (LOW) websocket.rs: scheme match arm 'ws'|'wss' borrowed joined
immutably while set_scheme needed a mutable borrow. Fixed by using literal
'ws' (set_scheme on an already-ws URL is a no-op for the ws case).
maturin>=1.5 requires -m to point to Cargo.toml, not pyproject.toml.
Fixes wheel build job failure in CI (all three matrix targets).
Also switches to manifest-path: action param for cleaner workflow syntax.
Applies same fix to Makefile build-wheel and develop targets.
Adds `HEADROOM_QDRANT_URL`, `_HOST`, `_PORT`, `_API_KEY`, `_HTTPS`,
`_PREFER_GRPC`, `_GRPC_PORT` support across the memory stack:
- `headroom/memory/qdrant_env.py`: shared resolver helper with
explicit-arg > env > default precedence (URL wins over host/port;
booleans parsed via standard truthy set).
- `memory/easy.py`, `backends/{mem0,direct_mem0}.py`,
`proxy/memory_handler.py`: call the resolver so
`Memory(backend="qdrant-neo4j")`, `Mem0Config`, and the proxy's
`MemoryConfig` all honor the same env keys.
- `proxy/models.py` + `proxy/server.py`: `ProxyConfig` picks up the
same keys so hosted Qdrant (e.g. Qdrant Cloud) works without code
changes.
- `cli/proxy.py`: adds `--memory-qdrant-{url,host,port,api-key}`
flags that override the env when present.
- `tests/test_memory/test_qdrant_env.py`: unit coverage for
precedence, URL-vs-host/port, boolean parsing, and unset defaults.
- `CHANGELOG.md`: documented under [Unreleased] / Added.
Explicit constructor arguments still win; unset env keeps the existing
localhost:6333 defaults, so this is backwards-compatible.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
15 integration tests across five suites that spin up the proxy on an
ephemeral port pointed at a per-test mock upstream:
- integration_http: all 7 methods round-trip with body, status passthrough
for 404/500/502, query strings preserved, 1MB POST streams through.
- integration_sse: a 10-event in-process hyper SSE upstream emits at 50ms
cadence; chunks reach the client with max gap < 500ms (loose CI bound)
and a client disconnect propagates to the upstream within 2s.
- integration_ws: 5 text + 5 binary messages echo through a tungstenite
upstream byte-equal; client-initiated close propagates.
- integration_headers: hop-by-hop strip both directions, X-Forwarded-*
injection, X-Forwarded-For appends to existing value, multi-valued
response headers preserved.
- integration_body: 5MB POST round-trips byte-equal; streaming response
yields first byte before the upstream finishes sending.
- integration_health: own /healthz always 200; /healthz/upstream is 200
when upstream healthy and 503 when down.
The Sec-WebSocket-Protocol forwarding is exercised implicitly by the WS
tests via tungstenite handshake. The harness lives at tests/common/mod.rs
and is shared by every integration suite.
/healthz returns 200 unconditionally (own health). /healthz/upstream
proxies a GET to the upstream's /healthz and returns 200 when reachable
+ 2xx, 503 otherwise. Both endpoints are intercepted in axum and never
forwarded; documented in RUST_DEV.md as reserved paths.
When the catch-all sees an Upgrade: websocket request, hand it to the ws
module: axum upgrades the client side, tokio-tungstenite connects to the
upstream (rewriting http->ws / https->wss while preserving path + query),
and two pumps shovel messages until either side closes. Forwarded headers
exclude what tungstenite manages (Host, Upgrade, Connection, Sec-*) but
preserve Authorization, Sec-WebSocket-Protocol, etc. Supports text,
binary, ping, pong, and close frames in both directions.
Implements RFC 7230 6.1 hop-by-hop filtering on both request and response
sides (Connection, Keep-Alive, Proxy-Authenticate, Proxy-Authorization,
TE, Trailers, Transfer-Encoding, Upgrade), plus the additional headers
listed inside any incoming Connection: header. Injects X-Forwarded-For
(appending to existing value if any), X-Forwarded-Proto, X-Forwarded-Host,
and X-Request-Id. The proxy module wires these in for both HTTP and WS.
Builds out crates/headroom-proxy from a /healthz stub into a transparent
reverse proxy: catch-all router that forwards every method/path/query to
--upstream verbatim, streaming both request and response bodies through
reqwest without buffering. Adds clap-based config (CLI + env), thiserror
error type with sane upstream-status mapping, JSON tracing-subscriber
logging, and graceful shutdown. The library surface (build_app, AppState,
Config) is reused by the integration tests.
Lists the line-ending renormalization commit so `git blame` and GitHub's
blame UI skip it. Contributors can opt in locally with:
git config blame.ignoreRevsFile .git-blame-ignore-revs
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`.gitattributes` declares `*.py text eol=lf` and `*.sh text eol=lf`, but
74 files (73 .py, 1 .sh) are stored in the index with CRLF line endings,
violating that contract. Every macOS/Linux clone reports these files as
"modified" on fresh checkout because git's diff engine sees the stored
bytes don't match the attribute contract, even though the working tree
and index match byte-for-byte.
Running `git add --renormalize .` rewrites each affected blob so the
stored form matches the attribute declaration. No semantic changes —
every affected file's diff is "N insertions, N deletions" with inserts
and deletes being the same lines modulo line endings.
Follow-up commit adds `.git-blame-ignore-revs` so `git blame` / GitHub
blame skip this mechanical commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Learned: error recovery section was bloating with stale, near-duplicate,
and contradictory entries because the dedup key was the literal rendered
bullet text and there was no TTL or re-validation.
- Normalize the hash key for error_recovery patterns. Read recoveries key
on (basename(error_path), basename(success_path)); Bash recoveries strip
volatile suffixes (| tail -N, 2>&1, etc.) and hash only the primary
command before the first | or &&. Non-error-recovery categories keep
literal-content hashing.
- Stamp first_seen_at / last_seen_at on every pattern; bump both in
_bump_persisted_evidence via json_set. Stored in metadata JSON — no
schema change.
- Refine at render time (error_recovery only): drop rows not re-observed
in 21 days, re-validate Read success paths against the filesystem,
collapse same-error_path-with-multiple-targets into one "use Glob/Grep
first" bullet, rank by evidence_count * 0.5 ** (days/5), cap at 15
bullets.
15 new tests (TestNormalizedHash, TestRefineErrorRecovery). Full suite:
526 passed, 1 skipped. Ruff + mypy clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Make the ONNX + sqlite-vec memory path truly batched.
Batch ONNX embed_batch calls, batch sqlite-vec index/remove work under a single cached connection, and update MCP warm-up to use batch embed/save/index flows.
Add focused regression tests for ONNX batching, sqlite-vec single-connection batch behavior, and MCP warm-up batching.
Skip the MCP-specific test when optional MCP dependencies are not installed.
Refs #240
Two fixes for the init-native-e2e matrix surfaced on PR #256:
1. Composite action installed `headroom` without extras, but
`headroom/cli/__init__.py` eagerly imports `proxy.server` (via
`cli/proxy.py`), which requires `fastapi`. All 6 POSIX jobs hit
`ModuleNotFoundError: No module named 'fastapi'` before `init` ran.
Fix: install `-e .[proxy]` to match the Docker e2e image.
2. On Windows, shims are `.cmd` files and Git Bash's `which` cannot
resolve them (exact-match only). Python's `shutil.which` (used by
`headroom init`) honors PATHEXT and finds the shim fine, but the
pre-flight `which` step failed first. Fix: use `Get-Command` via
`pwsh` for the Windows verification step.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
