akadmin/headroom
1
0
Fork 0
mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
headroom/Cargo.lock

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

5358 lines
128 KiB
Text
Raw Normal View History

feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 3
feat(rust): HfTokenizer::from_pretrained — HuggingFace Hub auto-download Stage 2.1: closes the loop on the HuggingFace tokenizer story. Stage 2 shipped `HfTokenizer::from_bytes`/`from_file`, which required callers to manage their own tokenizer.json files. This adds the third constructor: let t = HfTokenizer::from_pretrained("CohereForAI/c4ai-command-r-v01")?; register_hf("command-", t); `from_pretrained` is a thin wrapper around the `hf-hub` crate's blocking `ureq` API. First call downloads `tokenizer.json` to `~/.cache/huggingface/ hub` (or `$HF_HOME` if set); subsequent calls reuse the on-disk cache. Uses the `main` revision; gated repos (Llama, Mistral) require `HF_TOKEN` in env or `~/.cache/huggingface/token`. Also adds `try_register_hf(prefix, repo)` as the obvious one-liner for proxy startup code: let _ = try_register_hf("command-", "CohereForAI/c4ai-command-r-v01"); let _ = try_register_hf("mistral-", "mistralai/Mistral-7B-v0.1"); Each call is independent — a download failure for one model (e.g. gated without a token) does not affect others. `HfTokenizerError` gains a new `Hub` variant so callers can distinguish "couldn't fetch" from "fetched but malformed" — relevant when deciding whether to retry, surface to the user, or fall back to the estimator. Why blocking, not async: `from_pretrained` is called once at startup. A sync API works from `main()`, from a `OnceLock` initializer, or from `tokio::task::spawn_blocking` if a tokio caller needs it later. The async hf-hub backend would force callers to await at startup, which doesn't fit the `register_hf` registry pattern. Why rustls, not native-tls: keeps the binary statically linkable for AWS deploys (no system OpenSSL dependency). Tests: a network-dependent integration test (`#[ignore]`d in CI; hits HF for `gpt2`, ~1.4 MB) verifies the real download + load + count path. A non-network negative test verifies that an invalid repo name surfaces as `HfTokenizerError::Hub`, not a panic. 44 unit tests + 5 proptests + 1 doctest pass; parity stays 40/40 byte-equal.
2026-04-25 15:05:09 -07:00
[[package]]
name = "adler2"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "ahash"
version = "0.8.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75"
dependencies = [
"cfg-if",
"getrandom 0.3.4",
"once_cell",
"serde",
"version_check",
"zerocopy",
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "aho-corasick"
version = "1.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301"
dependencies = [
"memchr",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "aligned"
version = "0.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ee4508988c62edf04abd8d92897fca0c2995d907ce1dfeaf369dac3716a40685"
dependencies = [
"as-slice",
]
[[package]]
name = "aligned-vec"
version = "0.6.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dc890384c8602f339876ded803c97ad529f3842aba97f6392b3dba0dd171769b"
dependencies = [
"equator",
]
[[package]]
name = "allocator-api2"
version = "0.2.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "683d7910e743518b0e34f1186f92494becacb047c7b6bf616c96772180fef923"
fix(proxy): PR-D4 native Vertex publisher path + ADC bearer auth Adds a Rust-native Vertex AI publisher route ahead of the LiteLLM Python converter (which dropped `thinking`, `redacted_thinking`, `document`, `image`, `server_tool_use`, `mcp_tool_use` block kinds — the P4-37 / P4-38 bug). After this PR the Vertex `:rawPredict` and `:streamRawPredict` calls survive byte-equal upstream and benefit from the live-zone Anthropic dispatcher (PR-B-series) running over the body — same behaviour as `/v1/messages`. New module `crates/headroom-proxy/src/vertex/`: - `mod.rs` — single dispatch handler at the `/v1beta1/.../models/:model_action` route. Splits the trailing `:<verb>` segment with `str::rsplit_once(':')` (no regex) and flips an `attach_sse_tee` flag to dispatch to the streaming or non-streaming arm. Both verbs share one axum route shape because matchit can't distinguish two patterns that overlap on a parameter. - `envelope.rs` — `VertexEnvelope` parser. Confirms `anthropic_version` present + `model` field absent (the two fingerprints of the Vertex envelope vs `/v1/messages`). - `adc.rs` — `TokenSource` trait + `GcpAdcTokenSource` (production, `gcp_auth` 0.12) + `StaticTokenSource` (tests). Caches tokens with a 60s refresh-ahead-of-expiry window. Emits structured `event = "vertex_adc_token_refreshed"` per refresh. - `raw_predict.rs` — POST handler + shared `forward_vertex_request`. Buffers body, parses envelope, runs live-zone Anthropic compression, fetches ADC bearer, attaches `Authorization: Bearer <token>` (overwrites client-supplied Authorization header), forwards. SSE telemetry tee for the streaming verb reuses PR-C1's `AnthropicStreamState` directly (Vertex streams plain SSE, unlike Bedrock's binary EventStream). - `stream_raw_predict.rs` — module-level docs + alias to the shared dispatcher (the streaming-vs-non-streaming difference is one boolean flag inside the shared forwarder). Modifications: - `proxy.rs::build_app` — registers the single Vertex route. - `proxy.rs::AppState` — new `vertex_token_source: Arc<dyn TokenSource>` field. Production constructs `GcpAdcTokenSource` lazily (no GCP call until first `bearer()`); tests inject `StaticTokenSource` via the new `AppState::with_token_source` helper. - `config.rs` — adds `--vertex-region` / `HEADROOM_PROXY_VERTEX_REGION` (default `us-central1`, observability tag only — the upstream URL is `--upstream`) and `--vertex-adc-scope` / `HEADROOM_PROXY_VERTEX_ADC_SCOPE` (default `cloud-platform`). - `Cargo.toml` (workspace + proxy) — adds `gcp_auth = "0.12"` and `async-trait = "0.1"`. - `tests/common/mod.rs` — `start_proxy_with_state` accepts both config + state customizers; `install_static_token_source` helper for tests. `crates/headroom-proxy/tests/integration_vertex_raw_predict.rs` — all five tests pass: 1. `native_envelope_round_trip_byte_equal` — Vertex-shape body (with `anthropic_version`, no `model`) round-trips SHA-256 byte-equal upstream. 2. `adc_bearer_token_signed_correctly` — `Authorization: Bearer <static-test-token>` reaches upstream verbatim and OVERWRITES a client-supplied Authorization header. 3. `thinking_block_preserved` — request with `thinking` (incl. signature) + `redacted_thinking` (incl. opaque `data`) blocks round-trips byte-equal even with `LiveZone` compression mode enabled. This is the P4-37 / P4-38 teeth. 4. `stream_raw_predict_sse_handled` — `:streamRawPredict` proxies an Anthropic SSE response (full `message_start` → `content_block_delta` → `message_stop` sequence) back to the client without corruption; SSE content-type preserved end-to-end; bearer attached. 5. (bonus, no-silent-fallback contract) `adc_failure_returns_5xx_no_silent_forward` — when the token source returns `Err`, the proxy returns 5xx and never reaches upstream. Verifies the `event = "vertex_adc_fetch_failed"` error path. Workspace: `cargo test --workspace` green; `cargo clippy --workspace -- -D warnings` clean; `make ci-precheck` passes. - No silent fallbacks: ADC failure → structured 5xx, never an unauthenticated forward. - No hardcodes: every knob (region, ADC scope, upstream URL) is CLI-flag + env-var configurable. - No regexes: axum path parameters + `str::rsplit_once` only. - Comprehensive structured logs: `event` field on every decision point — `vertex_envelope_parsed`, `vertex_envelope_invalid`, `vertex_compression_skipped`, `vertex_compression_applied`, `vertex_adc_token_refreshed`, `vertex_adc_fetch_failed`, `vertex_streaming_pipeline_active`, `vertex_sse_stream_closed`, `vertex_forwarded`, `vertex_unknown_verb`, etc. - Performant: no body clone; ADC token cached + refreshed ahead-of-expiry, not fetched per request. - Comprehensive tests: realistic Anthropic block content (signature payload, redacted_thinking opaque blob) in `thinking_block_preserved`. The local `gcloud auth application-default print-access-token` returns no credentials, so manual validation against a real Vertex endpoint is not possible in this PR. Follow-up: the user runs `gcloud auth application-default login` once and exercises a live Vertex request — should be a no-code-change check. PR-D1 (Bedrock native) is running concurrently and will land its own envelope module at `crates/headroom-proxy/src/bedrock/envelope.rs`. The two envelope modules are intentionally siblings (not a shared trait) — the shapes differ (Bedrock has a different `anthropic_version` value, no `model` field, AWS SigV4 instead of GCP ADC), and a premature shared abstraction would obscure the provider-specific contracts. Whichever PR merges second rebases without conflict. Retires P4-38 (and the Vertex parts of P4-39); marketplace BYOC pitch (per project memory) gets one more native provider.
2026-05-03 16:22:07 -07:00
[[package]]
name = "android_system_properties"
version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311"
dependencies = [
"libc",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "anes"
version = "0.1.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4b46cbb362ab8752921c97e041f5e366ee6297bd428a31275b9fcf1e380f7299"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "anstream"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "824a212faf96e9acacdbd09febd34438f8f711fb84e09a8916013cd7815ca28d"
dependencies = [
"anstyle",
"anstyle-parse",
"anstyle-query",
"anstyle-wincon",
"colorchoice",
"is_terminal_polyfill",
"utf8parse",
]
[[package]]
name = "anstyle"
version = "1.0.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000"
[[package]]
name = "anstyle-parse"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "52ce7f38b242319f7cabaa6813055467063ecdc9d355bbb4ce0c68908cd8130e"
dependencies = [
"utf8parse",
]
[[package]]
name = "anstyle-query"
version = "1.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc"
dependencies = [
"windows-sys 0.61.2",
]
[[package]]
name = "anstyle-wincon"
version = "3.0.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d"
dependencies = [
"anstyle",
"once_cell_polyfill",
"windows-sys 0.61.2",
]
[[package]]
name = "anyhow"
version = "1.0.102"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "arbitrary"
version = "1.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c3d036a3c4ab069c7b410a2ce876bd74808d2d0888a82667669f8e783a898bf1"
fix: B7 — CCR hardening: persistent backends + always-on tool P2-25, P2-26: CCR (Compress-Cache-Retrieve) used an in-memory store that fragmented across uvicorn workers and was wiped on restart, and the `headroom_retrieve` tool was registered/unregistered per-request based on whether the latest body happened to contain compression markers — every flip busted the prompt cache. Both are sticky side-channels: once a session has done CCR, the tool list bytes and the retrieval store must stay stable. This PR fixes both. Rust: * Split `ccr.rs` into `ccr/` with `backends/` submodule (`in_memory.rs`, `sqlite.rs`, `redis.rs` cfg-gated). * `SqliteCcrStore` (production default): WAL mode, prepared upsert, lazy TTL purge on read, persistent across worker restarts and shareable across workers on the same host via SQLite file locking. * `RedisCcrStore` (cfg-gated behind `feature = "redis"`): SETEX with startup PING smoke-test, no key-prefix collision risk, no sticky session required at the LB. * `CcrBackendConfig::{InMemory, Sqlite, Redis}` + `from_config(...)` factory — every init failure surfaces (no silent fallback per `feedback_no_silent_fallbacks.md`). * `ccr::compute_key` (BLAKE3 → first 24 hex chars) and `ccr::marker_for("HASH") -> "<<ccr:HASH>>"` centralize the hash + marker format; one definition for the live-zone dispatcher and the Python regex (`headroom/ccr/tool_injection.py:211`). * `compress_anthropic_live_zone_with_ccr` accepts `Option<&dyn CcrStore>`. When wired, every accepted compression puts the original bytes into the backend and appends `<<ccr:HASH>>` to the compressed string. The token-validation gate runs on the marker-augmented string so the `compressed_tokens >= original_tokens` rejection stays honest. Python: * `SessionCcrTracker` + `apply_session_sticky_ccr_tool` mirror the PR-A7 `SessionToolTracker` / `apply_session_sticky_memory_tools` pattern: once a session has done CCR, every subsequent request injects the recorded golden tool-definition bytes. Tool list bytes are byte-stable across turns (snapshot test pins them). * `headroom/ccr/tool_injection.py::inject_tool_definition` accepts a new `session_has_done_ccr` kwarg per the PR-B7 spec change at line 302-328. The legacy per-request path stays intact for callers that don't yet thread a session id (e.g. Google handler). * Anthropic + OpenAI handlers route their CCR tool-list updates through `apply_session_sticky_ccr_tool`, keyed off the existing `session_tracker_store.compute_session_id(...)` plumbing. Backend selection model: `CcrBackendConfig::Sqlite { path }` is the production default — single host, persistent, multi-worker safe with sticky session. `CcrBackendConfig::Redis { url }` is the multi-host scale-out option — no stickiness needed. `InMemory` is for tests and single-worker dev only. RUST_DEV.md "Multi-worker deployment — CCR fragmentation" rewritten around this matrix. Tests: * `crates/headroom-core/tests/ccr_backends.rs` — 7 tests covering SQLite round-trip, TTL purge, proxy-restart survival, cross-backend byte-equal keys, `from_config` paths, and the no-redis-feature loud-failure check (+ 2 redis tests gated behind the feature). * `crates/headroom-core/tests/live_zone_ccr.rs` — confirms `<<ccr:HASH>>` marker injection, store population, and no-marker-when-no-store invariants end-to-end. * `tests/test_ccr_tool_always_on.py` — 12 tests pinning the always-on behaviour, session/provider isolation, LRU bound, no- session-id fallback, and (per-acceptance-criterion) the byte-stable tool-definition snapshot. Per-PR-B7 plan: REALIGNMENT/04-phase-B-live-zone.md.
2026-05-02 16:49:08 -07:00
[[package]]
name = "arc-swap"
version = "1.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6a3a1fd6f75306b68087b831f025c712524bcb19aad54e557b1129cfa0a2b207"
dependencies = [
"rustversion",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "arg_enum_proc_macro"
version = "0.3.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0ae92a5119aa49cdbcf6b9f893fe4e1d98b04ccbf82ee0584ad948a44a734dea"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
fix: B7 — CCR hardening: persistent backends + always-on tool P2-25, P2-26: CCR (Compress-Cache-Retrieve) used an in-memory store that fragmented across uvicorn workers and was wiped on restart, and the `headroom_retrieve` tool was registered/unregistered per-request based on whether the latest body happened to contain compression markers — every flip busted the prompt cache. Both are sticky side-channels: once a session has done CCR, the tool list bytes and the retrieval store must stay stable. This PR fixes both. Rust: * Split `ccr.rs` into `ccr/` with `backends/` submodule (`in_memory.rs`, `sqlite.rs`, `redis.rs` cfg-gated). * `SqliteCcrStore` (production default): WAL mode, prepared upsert, lazy TTL purge on read, persistent across worker restarts and shareable across workers on the same host via SQLite file locking. * `RedisCcrStore` (cfg-gated behind `feature = "redis"`): SETEX with startup PING smoke-test, no key-prefix collision risk, no sticky session required at the LB. * `CcrBackendConfig::{InMemory, Sqlite, Redis}` + `from_config(...)` factory — every init failure surfaces (no silent fallback per `feedback_no_silent_fallbacks.md`). * `ccr::compute_key` (BLAKE3 → first 24 hex chars) and `ccr::marker_for("HASH") -> "<<ccr:HASH>>"` centralize the hash + marker format; one definition for the live-zone dispatcher and the Python regex (`headroom/ccr/tool_injection.py:211`). * `compress_anthropic_live_zone_with_ccr` accepts `Option<&dyn CcrStore>`. When wired, every accepted compression puts the original bytes into the backend and appends `<<ccr:HASH>>` to the compressed string. The token-validation gate runs on the marker-augmented string so the `compressed_tokens >= original_tokens` rejection stays honest. Python: * `SessionCcrTracker` + `apply_session_sticky_ccr_tool` mirror the PR-A7 `SessionToolTracker` / `apply_session_sticky_memory_tools` pattern: once a session has done CCR, every subsequent request injects the recorded golden tool-definition bytes. Tool list bytes are byte-stable across turns (snapshot test pins them). * `headroom/ccr/tool_injection.py::inject_tool_definition` accepts a new `session_has_done_ccr` kwarg per the PR-B7 spec change at line 302-328. The legacy per-request path stays intact for callers that don't yet thread a session id (e.g. Google handler). * Anthropic + OpenAI handlers route their CCR tool-list updates through `apply_session_sticky_ccr_tool`, keyed off the existing `session_tracker_store.compute_session_id(...)` plumbing. Backend selection model: `CcrBackendConfig::Sqlite { path }` is the production default — single host, persistent, multi-worker safe with sticky session. `CcrBackendConfig::Redis { url }` is the multi-host scale-out option — no stickiness needed. `InMemory` is for tests and single-worker dev only. RUST_DEV.md "Multi-worker deployment — CCR fragmentation" rewritten around this matrix. Tests: * `crates/headroom-core/tests/ccr_backends.rs` — 7 tests covering SQLite round-trip, TTL purge, proxy-restart survival, cross-backend byte-equal keys, `from_config` paths, and the no-redis-feature loud-failure check (+ 2 redis tests gated behind the feature). * `crates/headroom-core/tests/live_zone_ccr.rs` — confirms `<<ccr:HASH>>` marker injection, store population, and no-marker-when-no-store invariants end-to-end. * `tests/test_ccr_tool_always_on.py` — 12 tests pinning the always-on behaviour, session/provider isolation, LRU bound, no- session-id fallback, and (per-acceptance-criterion) the byte-stable tool-definition snapshot. Per-PR-B7 plan: REALIGNMENT/04-phase-B-live-zone.md.
2026-05-02 16:49:08 -07:00
[[package]]
name = "arrayref"
version = "0.3.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "76a2e8124351fda1ef8aaaa3bbd7ebbcb486bbcd4225aca0aa0d84bb2db8fecb"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "arrayvec"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.7.7"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "f02882884d3e1bc524fb12c79f107f6ad0e1cfd498c536ffb494301740995dfe"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "as-slice"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "516b6b4f0e40d50dcda9365d53964ec74560ad4284da2e7fc97122cd83174516"
dependencies = [
"stable_deref_trait",
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "assert-json-diff"
version = "2.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "47e4f2b81832e72834d7518d8487a0396a28cc408186a2e8854c0f98011faf12"
dependencies = [
"serde",
"serde_json",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "async-trait"
version = "0.1.89"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9035ad2d096bed7955a320ee7e2230574d28fd3c3a0f186cbea1ff3c7eed5dbb"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "atomic-waker"
version = "1.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0"
[[package]]
name = "autocfg"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.5.1"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "av-scenechange"
version = "0.14.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0f321d77c20e19b92c39e7471cf986812cbb46659d2af674adc4331ef3f18394"
dependencies = [
"aligned",
"anyhow",
"arg_enum_proc_macro",
"arrayvec",
"log",
"num-rational",
"num-traits",
"pastey",
"rayon",
"thiserror 2.0.18",
"v_frame",
"y4m",
]
[[package]]
name = "av1-grain"
version = "0.2.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8cfddb07216410377231960af4fcab838eaa12e013417781b78bd95ee22077f8"
dependencies = [
"anyhow",
"arrayvec",
"log",
"nom 8.0.0",
"num-rational",
"v_frame",
]
[[package]]
name = "avif-serialize"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.8.9"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "e7178fe5f7d460b13895ebb9dcb28a3a6216d2df2574a0806cb51b555d297f38"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
dependencies = [
"arrayvec",
]
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
[[package]]
name = "aws-config"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.8.18"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "e33f815b73a3899c03b380d543532e5865f230dce9678d108dc10732a8682275"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
dependencies = [
"aws-credential-types",
"aws-runtime",
feat(bedrock): cross-region + Converse compression; bundle proxy binary in images (#999) ## Description The native Bedrock path (Phase D) compresses + signs Anthropic-on-Bedrock requests, but two real-world cases slipped through, and the native binary that powers it was never shipped. This PR closes those gaps as a focused set of give-backs. Aligns with the Rust migration plan (see below). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Cross-region inference-profile detection** via a new `bedrock::vendor` module (`canonical_vendor()`), following the design proposed in #953: strip a known geo prefix (`eu.`/`us.`/`apac.`/`global.`) then match the canonical vendor. Geo-prefixed Anthropic profiles (`eu.anthropic.…`) now get live-zone compression instead of being silently skipped; geo-prefixed non-Anthropic vendors stay correctly excluded. - **Converse-body compression (two parts)**: 1. `run_anthropic_compression` no longer bails to passthrough when the body lacks an InvokeModel `anthropic_version` envelope; envelope re-emit stays gated on successful parse. 2. The **live-zone dispatcher now recognizes Bedrock Converse content blocks**. Converse blocks carry no `type` discriminator (the variant is the key: `{"text": …}` vs Anthropic's `{"type":"text","text":…}`), so real Converse user-message text was still passing through uncompressed. A typeless block whose `text` is a JSON string now routes through the same surgical text path. Anthropic blocks always carry `type`, so the Anthropic path is byte-for-byte unchanged; non-text Converse blocks (`{"image":…}`, `{"toolUse":…}`) stay unrecognized and no-op. - **Correct `/converse` upstream routing**: the non-streaming handler resolved the upstream action from a hard-coded `"invoke"`, so `/converse` requests were forwarded to Bedrock's `/invoke` endpoint. It now resolves the action from the inbound path (`extract_invoke_action`), mirroring the streaming handler's `extract_streaming_action`. SigV4 signs the same URL it forwards, so the signature stays consistent. - **`aws-config` `sso` feature**: SSO profiles now resolve through the default credential chain for SigV4 — the credential chain in `docs/bedrock.md` already promised SSO; this makes the code match. - **Ship the `headroom-proxy` binary in published images** (`Dockerfile`): built in the builder stage (`--locked`, with the cargo registry cache mounted at `CARGO_HOME`) and copied into both the debian and distroless runtime images. - **Docs** (`docs/bedrock.md`): document cross-region inference profiles and a "Running the proxy" section. AWS credentials mount at `/home/nonroot/.aws` (the default nonroot image home) where the SDK looks for `~/.aws`, with a note on the root-image alternative. ## Related issues - Closes #976 — ship the `headroom-proxy` binary in published images (this PR implements the exact fix proposed there). - Addresses the **cross-region inference-profile** half of #953 via its proposed `canonical_vendor()` design. Non-Anthropic vendor compression parity (Nova/GLM/MiniMax/ Kimi) is the natural follow-up — `bedrock::vendor` is the shared resolver it can build on. - Extends the native Bedrock InvokeModel compression requested in #734 (the Bedrock slice of #510) to cross-region profiles and Converse bodies. - Partially enables #181 (native, Python-free packaging): the native binary now ships in the images, though full Python-free distribution remains out of scope. ## Alignment with the Rust migration plan Per `docs/spec/022-rust-migration.md`, the migration is **proxy-first**: `headroom-proxy` is the deployable Rust artifact, native routes replace Python passthroughs one at a time (Stage 4 = provider expansion, Bedrock included), and the binary is meant to be "built, tested, and **released together with the Python package**." Two ways this PR advances that: - The binary-in-images change makes the codebase do what the spec already states (ship the artifact) — closing the gap that forced downstreams to build from source. - Hardening the native Bedrock route (cross-region, Converse routing + body compression) is exactly the Stage-4 provider-expansion work, keeping the native path at parity with real traffic so it can be the default rather than a passthrough. ## Testing - [x] Unit tests pass (`cargo test -p headroom-core -p headroom-proxy` — full suites, 0 failures) - [x] Linting passes (`cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings`) - [x] Formatting passes (`cargo fmt -- --check`) - [x] New tests added — `bedrock::vendor` (foundation + inference-profile matching), `extract_invoke_action` + converse upstream URL, and live-zone Converse text-block routing (`block_has_string_text_field`, converse-vs-anthropic dispatch equivalence). - [x] Manual testing performed ### Test Output ```text $ cargo test -p headroom-core -p headroom-proxy # all suites: ok, 0 failed $ cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings # Finished, no warnings $ cargo fmt -- --check # clean # image validation (local, proxy/code extras): $ docker build --target runtime ... # debian: /usr/local/bin/headroom-proxy, --help OK $ docker build --target runtime-slim ... # distroless: binary links + --help OK ``` ## Real Behavior Proof - Environment: native Bedrock proxy against `bedrock-runtime.eu-west-2`, SSO profile, model `eu.anthropic.claude-haiku-4-5-20251001-v1:0`. - Exact command / steps: POST a large multi-turn Converse body to `/model/eu.anthropic.claude-haiku-4-5-20251001-v1:0/converse`; separately build the `runtime` + `runtime-slim` targets and run `/usr/local/bin/headroom-proxy --help`. - Observed result: before — `bedrock_compression_skipped` (geo-prefixed id not recognized), forwarded uncompressed to the wrong `/invoke` upstream; after — geo-prefixed id recognized, `/converse` forwarded to the `/converse` upstream, live-zone dispatcher compresses the Converse user-message text, measurable token savings. Images contain a runnable `headroom-proxy` in both variants. - Not tested: non-Anthropic vendor compression parity (#953 follow-up); Converse `toolResult` nested-text compression (follow-up — only top-level Converse text blocks compress today). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - An earlier revision flipped the EventStream `Accept` default (`*/*`/absent → passthrough); **dropped** — `*/*` is what most clients (incl. reqwest and the proxy's own metrics tests) send while expecting SSE, so forcing passthrough breaks the standard SSE path. - The binary build adds the native-proxy compile to the image build; happy to gate it behind a build arg if maintainers prefer it opt-in. - Addressed a Copilot review round: corrected the `/converse` upstream routing, the stale `run_anthropic_compression` comment, the Dockerfile cargo cache mount + `--locked`, and the nonroot AWS-credentials docs example.
2026-06-16 16:45:24 +02:00
"aws-sdk-sso",
"aws-sdk-ssooidc",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"aws-sdk-sts",
"aws-smithy-async",
"aws-smithy-http",
"aws-smithy-json",
"aws-smithy-runtime",
"aws-smithy-runtime-api",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"aws-smithy-schema",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"aws-smithy-types",
"aws-types",
"bytes",
"fastrand",
feat(bedrock): cross-region + Converse compression; bundle proxy binary in images (#999) ## Description The native Bedrock path (Phase D) compresses + signs Anthropic-on-Bedrock requests, but two real-world cases slipped through, and the native binary that powers it was never shipped. This PR closes those gaps as a focused set of give-backs. Aligns with the Rust migration plan (see below). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Cross-region inference-profile detection** via a new `bedrock::vendor` module (`canonical_vendor()`), following the design proposed in #953: strip a known geo prefix (`eu.`/`us.`/`apac.`/`global.`) then match the canonical vendor. Geo-prefixed Anthropic profiles (`eu.anthropic.…`) now get live-zone compression instead of being silently skipped; geo-prefixed non-Anthropic vendors stay correctly excluded. - **Converse-body compression (two parts)**: 1. `run_anthropic_compression` no longer bails to passthrough when the body lacks an InvokeModel `anthropic_version` envelope; envelope re-emit stays gated on successful parse. 2. The **live-zone dispatcher now recognizes Bedrock Converse content blocks**. Converse blocks carry no `type` discriminator (the variant is the key: `{"text": …}` vs Anthropic's `{"type":"text","text":…}`), so real Converse user-message text was still passing through uncompressed. A typeless block whose `text` is a JSON string now routes through the same surgical text path. Anthropic blocks always carry `type`, so the Anthropic path is byte-for-byte unchanged; non-text Converse blocks (`{"image":…}`, `{"toolUse":…}`) stay unrecognized and no-op. - **Correct `/converse` upstream routing**: the non-streaming handler resolved the upstream action from a hard-coded `"invoke"`, so `/converse` requests were forwarded to Bedrock's `/invoke` endpoint. It now resolves the action from the inbound path (`extract_invoke_action`), mirroring the streaming handler's `extract_streaming_action`. SigV4 signs the same URL it forwards, so the signature stays consistent. - **`aws-config` `sso` feature**: SSO profiles now resolve through the default credential chain for SigV4 — the credential chain in `docs/bedrock.md` already promised SSO; this makes the code match. - **Ship the `headroom-proxy` binary in published images** (`Dockerfile`): built in the builder stage (`--locked`, with the cargo registry cache mounted at `CARGO_HOME`) and copied into both the debian and distroless runtime images. - **Docs** (`docs/bedrock.md`): document cross-region inference profiles and a "Running the proxy" section. AWS credentials mount at `/home/nonroot/.aws` (the default nonroot image home) where the SDK looks for `~/.aws`, with a note on the root-image alternative. ## Related issues - Closes #976 — ship the `headroom-proxy` binary in published images (this PR implements the exact fix proposed there). - Addresses the **cross-region inference-profile** half of #953 via its proposed `canonical_vendor()` design. Non-Anthropic vendor compression parity (Nova/GLM/MiniMax/ Kimi) is the natural follow-up — `bedrock::vendor` is the shared resolver it can build on. - Extends the native Bedrock InvokeModel compression requested in #734 (the Bedrock slice of #510) to cross-region profiles and Converse bodies. - Partially enables #181 (native, Python-free packaging): the native binary now ships in the images, though full Python-free distribution remains out of scope. ## Alignment with the Rust migration plan Per `docs/spec/022-rust-migration.md`, the migration is **proxy-first**: `headroom-proxy` is the deployable Rust artifact, native routes replace Python passthroughs one at a time (Stage 4 = provider expansion, Bedrock included), and the binary is meant to be "built, tested, and **released together with the Python package**." Two ways this PR advances that: - The binary-in-images change makes the codebase do what the spec already states (ship the artifact) — closing the gap that forced downstreams to build from source. - Hardening the native Bedrock route (cross-region, Converse routing + body compression) is exactly the Stage-4 provider-expansion work, keeping the native path at parity with real traffic so it can be the default rather than a passthrough. ## Testing - [x] Unit tests pass (`cargo test -p headroom-core -p headroom-proxy` — full suites, 0 failures) - [x] Linting passes (`cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings`) - [x] Formatting passes (`cargo fmt -- --check`) - [x] New tests added — `bedrock::vendor` (foundation + inference-profile matching), `extract_invoke_action` + converse upstream URL, and live-zone Converse text-block routing (`block_has_string_text_field`, converse-vs-anthropic dispatch equivalence). - [x] Manual testing performed ### Test Output ```text $ cargo test -p headroom-core -p headroom-proxy # all suites: ok, 0 failed $ cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings # Finished, no warnings $ cargo fmt -- --check # clean # image validation (local, proxy/code extras): $ docker build --target runtime ... # debian: /usr/local/bin/headroom-proxy, --help OK $ docker build --target runtime-slim ... # distroless: binary links + --help OK ``` ## Real Behavior Proof - Environment: native Bedrock proxy against `bedrock-runtime.eu-west-2`, SSO profile, model `eu.anthropic.claude-haiku-4-5-20251001-v1:0`. - Exact command / steps: POST a large multi-turn Converse body to `/model/eu.anthropic.claude-haiku-4-5-20251001-v1:0/converse`; separately build the `runtime` + `runtime-slim` targets and run `/usr/local/bin/headroom-proxy --help`. - Observed result: before — `bedrock_compression_skipped` (geo-prefixed id not recognized), forwarded uncompressed to the wrong `/invoke` upstream; after — geo-prefixed id recognized, `/converse` forwarded to the `/converse` upstream, live-zone dispatcher compresses the Converse user-message text, measurable token savings. Images contain a runnable `headroom-proxy` in both variants. - Not tested: non-Anthropic vendor compression parity (#953 follow-up); Converse `toolResult` nested-text compression (follow-up — only top-level Converse text blocks compress today). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - An earlier revision flipped the EventStream `Accept` default (`*/*`/absent → passthrough); **dropped** — `*/*` is what most clients (incl. reqwest and the proxy's own metrics tests) send while expecting SSE, so forcing passthrough breaks the standard SSE path. - The binary build adds the native-proxy compile to the image build; happy to gate it behind a build arg if maintainers prefer it opt-in. - Addressed a Copilot review round: corrected the `/converse` upstream routing, the stale `run_anthropic_compression` comment, the Dockerfile cargo cache mount + `--locked`, and the nonroot AWS-credentials docs example.
2026-06-16 16:45:24 +02:00
"hex",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
feat(bedrock): cross-region + Converse compression; bundle proxy binary in images (#999) ## Description The native Bedrock path (Phase D) compresses + signs Anthropic-on-Bedrock requests, but two real-world cases slipped through, and the native binary that powers it was never shipped. This PR closes those gaps as a focused set of give-backs. Aligns with the Rust migration plan (see below). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Cross-region inference-profile detection** via a new `bedrock::vendor` module (`canonical_vendor()`), following the design proposed in #953: strip a known geo prefix (`eu.`/`us.`/`apac.`/`global.`) then match the canonical vendor. Geo-prefixed Anthropic profiles (`eu.anthropic.…`) now get live-zone compression instead of being silently skipped; geo-prefixed non-Anthropic vendors stay correctly excluded. - **Converse-body compression (two parts)**: 1. `run_anthropic_compression` no longer bails to passthrough when the body lacks an InvokeModel `anthropic_version` envelope; envelope re-emit stays gated on successful parse. 2. The **live-zone dispatcher now recognizes Bedrock Converse content blocks**. Converse blocks carry no `type` discriminator (the variant is the key: `{"text": …}` vs Anthropic's `{"type":"text","text":…}`), so real Converse user-message text was still passing through uncompressed. A typeless block whose `text` is a JSON string now routes through the same surgical text path. Anthropic blocks always carry `type`, so the Anthropic path is byte-for-byte unchanged; non-text Converse blocks (`{"image":…}`, `{"toolUse":…}`) stay unrecognized and no-op. - **Correct `/converse` upstream routing**: the non-streaming handler resolved the upstream action from a hard-coded `"invoke"`, so `/converse` requests were forwarded to Bedrock's `/invoke` endpoint. It now resolves the action from the inbound path (`extract_invoke_action`), mirroring the streaming handler's `extract_streaming_action`. SigV4 signs the same URL it forwards, so the signature stays consistent. - **`aws-config` `sso` feature**: SSO profiles now resolve through the default credential chain for SigV4 — the credential chain in `docs/bedrock.md` already promised SSO; this makes the code match. - **Ship the `headroom-proxy` binary in published images** (`Dockerfile`): built in the builder stage (`--locked`, with the cargo registry cache mounted at `CARGO_HOME`) and copied into both the debian and distroless runtime images. - **Docs** (`docs/bedrock.md`): document cross-region inference profiles and a "Running the proxy" section. AWS credentials mount at `/home/nonroot/.aws` (the default nonroot image home) where the SDK looks for `~/.aws`, with a note on the root-image alternative. ## Related issues - Closes #976 — ship the `headroom-proxy` binary in published images (this PR implements the exact fix proposed there). - Addresses the **cross-region inference-profile** half of #953 via its proposed `canonical_vendor()` design. Non-Anthropic vendor compression parity (Nova/GLM/MiniMax/ Kimi) is the natural follow-up — `bedrock::vendor` is the shared resolver it can build on. - Extends the native Bedrock InvokeModel compression requested in #734 (the Bedrock slice of #510) to cross-region profiles and Converse bodies. - Partially enables #181 (native, Python-free packaging): the native binary now ships in the images, though full Python-free distribution remains out of scope. ## Alignment with the Rust migration plan Per `docs/spec/022-rust-migration.md`, the migration is **proxy-first**: `headroom-proxy` is the deployable Rust artifact, native routes replace Python passthroughs one at a time (Stage 4 = provider expansion, Bedrock included), and the binary is meant to be "built, tested, and **released together with the Python package**." Two ways this PR advances that: - The binary-in-images change makes the codebase do what the spec already states (ship the artifact) — closing the gap that forced downstreams to build from source. - Hardening the native Bedrock route (cross-region, Converse routing + body compression) is exactly the Stage-4 provider-expansion work, keeping the native path at parity with real traffic so it can be the default rather than a passthrough. ## Testing - [x] Unit tests pass (`cargo test -p headroom-core -p headroom-proxy` — full suites, 0 failures) - [x] Linting passes (`cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings`) - [x] Formatting passes (`cargo fmt -- --check`) - [x] New tests added — `bedrock::vendor` (foundation + inference-profile matching), `extract_invoke_action` + converse upstream URL, and live-zone Converse text-block routing (`block_has_string_text_field`, converse-vs-anthropic dispatch equivalence). - [x] Manual testing performed ### Test Output ```text $ cargo test -p headroom-core -p headroom-proxy # all suites: ok, 0 failed $ cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings # Finished, no warnings $ cargo fmt -- --check # clean # image validation (local, proxy/code extras): $ docker build --target runtime ... # debian: /usr/local/bin/headroom-proxy, --help OK $ docker build --target runtime-slim ... # distroless: binary links + --help OK ``` ## Real Behavior Proof - Environment: native Bedrock proxy against `bedrock-runtime.eu-west-2`, SSO profile, model `eu.anthropic.claude-haiku-4-5-20251001-v1:0`. - Exact command / steps: POST a large multi-turn Converse body to `/model/eu.anthropic.claude-haiku-4-5-20251001-v1:0/converse`; separately build the `runtime` + `runtime-slim` targets and run `/usr/local/bin/headroom-proxy --help`. - Observed result: before — `bedrock_compression_skipped` (geo-prefixed id not recognized), forwarded uncompressed to the wrong `/invoke` upstream; after — geo-prefixed id recognized, `/converse` forwarded to the `/converse` upstream, live-zone dispatcher compresses the Converse user-message text, measurable token savings. Images contain a runnable `headroom-proxy` in both variants. - Not tested: non-Anthropic vendor compression parity (#953 follow-up); Converse `toolResult` nested-text compression (follow-up — only top-level Converse text blocks compress today). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - An earlier revision flipped the EventStream `Accept` default (`*/*`/absent → passthrough); **dropped** — `*/*` is what most clients (incl. reqwest and the proxy's own metrics tests) send while expecting SSE, so forcing passthrough breaks the standard SSE path. - The binary build adds the native-proxy compile to the image build; happy to gate it behind a build arg if maintainers prefer it opt-in. - Addressed a Copilot review round: corrected the `/converse` upstream routing, the stale `run_anthropic_compression` comment, the Dockerfile cargo cache mount + `--locked`, and the nonroot AWS-credentials docs example.
2026-06-16 16:45:24 +02:00
"sha1",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"time",
"tokio",
"tracing",
"url",
feat(bedrock): cross-region + Converse compression; bundle proxy binary in images (#999) ## Description The native Bedrock path (Phase D) compresses + signs Anthropic-on-Bedrock requests, but two real-world cases slipped through, and the native binary that powers it was never shipped. This PR closes those gaps as a focused set of give-backs. Aligns with the Rust migration plan (see below). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Cross-region inference-profile detection** via a new `bedrock::vendor` module (`canonical_vendor()`), following the design proposed in #953: strip a known geo prefix (`eu.`/`us.`/`apac.`/`global.`) then match the canonical vendor. Geo-prefixed Anthropic profiles (`eu.anthropic.…`) now get live-zone compression instead of being silently skipped; geo-prefixed non-Anthropic vendors stay correctly excluded. - **Converse-body compression (two parts)**: 1. `run_anthropic_compression` no longer bails to passthrough when the body lacks an InvokeModel `anthropic_version` envelope; envelope re-emit stays gated on successful parse. 2. The **live-zone dispatcher now recognizes Bedrock Converse content blocks**. Converse blocks carry no `type` discriminator (the variant is the key: `{"text": …}` vs Anthropic's `{"type":"text","text":…}`), so real Converse user-message text was still passing through uncompressed. A typeless block whose `text` is a JSON string now routes through the same surgical text path. Anthropic blocks always carry `type`, so the Anthropic path is byte-for-byte unchanged; non-text Converse blocks (`{"image":…}`, `{"toolUse":…}`) stay unrecognized and no-op. - **Correct `/converse` upstream routing**: the non-streaming handler resolved the upstream action from a hard-coded `"invoke"`, so `/converse` requests were forwarded to Bedrock's `/invoke` endpoint. It now resolves the action from the inbound path (`extract_invoke_action`), mirroring the streaming handler's `extract_streaming_action`. SigV4 signs the same URL it forwards, so the signature stays consistent. - **`aws-config` `sso` feature**: SSO profiles now resolve through the default credential chain for SigV4 — the credential chain in `docs/bedrock.md` already promised SSO; this makes the code match. - **Ship the `headroom-proxy` binary in published images** (`Dockerfile`): built in the builder stage (`--locked`, with the cargo registry cache mounted at `CARGO_HOME`) and copied into both the debian and distroless runtime images. - **Docs** (`docs/bedrock.md`): document cross-region inference profiles and a "Running the proxy" section. AWS credentials mount at `/home/nonroot/.aws` (the default nonroot image home) where the SDK looks for `~/.aws`, with a note on the root-image alternative. ## Related issues - Closes #976 — ship the `headroom-proxy` binary in published images (this PR implements the exact fix proposed there). - Addresses the **cross-region inference-profile** half of #953 via its proposed `canonical_vendor()` design. Non-Anthropic vendor compression parity (Nova/GLM/MiniMax/ Kimi) is the natural follow-up — `bedrock::vendor` is the shared resolver it can build on. - Extends the native Bedrock InvokeModel compression requested in #734 (the Bedrock slice of #510) to cross-region profiles and Converse bodies. - Partially enables #181 (native, Python-free packaging): the native binary now ships in the images, though full Python-free distribution remains out of scope. ## Alignment with the Rust migration plan Per `docs/spec/022-rust-migration.md`, the migration is **proxy-first**: `headroom-proxy` is the deployable Rust artifact, native routes replace Python passthroughs one at a time (Stage 4 = provider expansion, Bedrock included), and the binary is meant to be "built, tested, and **released together with the Python package**." Two ways this PR advances that: - The binary-in-images change makes the codebase do what the spec already states (ship the artifact) — closing the gap that forced downstreams to build from source. - Hardening the native Bedrock route (cross-region, Converse routing + body compression) is exactly the Stage-4 provider-expansion work, keeping the native path at parity with real traffic so it can be the default rather than a passthrough. ## Testing - [x] Unit tests pass (`cargo test -p headroom-core -p headroom-proxy` — full suites, 0 failures) - [x] Linting passes (`cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings`) - [x] Formatting passes (`cargo fmt -- --check`) - [x] New tests added — `bedrock::vendor` (foundation + inference-profile matching), `extract_invoke_action` + converse upstream URL, and live-zone Converse text-block routing (`block_has_string_text_field`, converse-vs-anthropic dispatch equivalence). - [x] Manual testing performed ### Test Output ```text $ cargo test -p headroom-core -p headroom-proxy # all suites: ok, 0 failed $ cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings # Finished, no warnings $ cargo fmt -- --check # clean # image validation (local, proxy/code extras): $ docker build --target runtime ... # debian: /usr/local/bin/headroom-proxy, --help OK $ docker build --target runtime-slim ... # distroless: binary links + --help OK ``` ## Real Behavior Proof - Environment: native Bedrock proxy against `bedrock-runtime.eu-west-2`, SSO profile, model `eu.anthropic.claude-haiku-4-5-20251001-v1:0`. - Exact command / steps: POST a large multi-turn Converse body to `/model/eu.anthropic.claude-haiku-4-5-20251001-v1:0/converse`; separately build the `runtime` + `runtime-slim` targets and run `/usr/local/bin/headroom-proxy --help`. - Observed result: before — `bedrock_compression_skipped` (geo-prefixed id not recognized), forwarded uncompressed to the wrong `/invoke` upstream; after — geo-prefixed id recognized, `/converse` forwarded to the `/converse` upstream, live-zone dispatcher compresses the Converse user-message text, measurable token savings. Images contain a runnable `headroom-proxy` in both variants. - Not tested: non-Anthropic vendor compression parity (#953 follow-up); Converse `toolResult` nested-text compression (follow-up — only top-level Converse text blocks compress today). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - An earlier revision flipped the EventStream `Accept` default (`*/*`/absent → passthrough); **dropped** — `*/*` is what most clients (incl. reqwest and the proxy's own metrics tests) send while expecting SSE, so forcing passthrough breaks the standard SSE path. - The binary build adds the native-proxy compile to the image build; happy to gate it behind a build arg if maintainers prefer it opt-in. - Addressed a Copilot review round: corrected the `/converse` upstream routing, the stale `run_anthropic_compression` comment, the Dockerfile cargo cache mount + `--locked`, and the nonroot AWS-credentials docs example.
2026-06-16 16:45:24 +02:00
"zeroize",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
]
[[package]]
name = "aws-credential-types"
version = "1.2.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8f20799b373a1be121fe3005fba0c2090af9411573878f224df44b42727fcaf7"
dependencies = [
"aws-smithy-async",
"aws-smithy-runtime-api",
"aws-smithy-types",
"zeroize",
]
[[package]]
name = "aws-lc-rs"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.17.0"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "5ec2f1fc3ec205783a5da9a7e6c1509cc69dedf09a1949e412c1e18469326d00"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
dependencies = [
"aws-lc-sys",
"zeroize",
]
[[package]]
name = "aws-lc-sys"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.41.0"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "1a2f9779ce85b93ab6170dd940ad0169b5766ff848247aff13bb788b832fe3f4"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
dependencies = [
"cc",
"cmake",
"dunce",
"fs_extra",
]
[[package]]
name = "aws-runtime"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.7.5"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "6c9b9de216a988dd54b754a82a7660cfe14cee4f6782ae4524470972fa0ccb39"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
dependencies = [
"aws-credential-types",
"aws-sigv4",
"aws-smithy-async",
"aws-smithy-http",
"aws-smithy-runtime",
"aws-smithy-runtime-api",
"aws-smithy-types",
"aws-types",
"bytes",
"bytes-utils",
"fastrand",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"http-body 1.0.1",
"percent-encoding",
"pin-project-lite",
"tracing",
"uuid",
]
feat(bedrock): cross-region + Converse compression; bundle proxy binary in images (#999) ## Description The native Bedrock path (Phase D) compresses + signs Anthropic-on-Bedrock requests, but two real-world cases slipped through, and the native binary that powers it was never shipped. This PR closes those gaps as a focused set of give-backs. Aligns with the Rust migration plan (see below). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Cross-region inference-profile detection** via a new `bedrock::vendor` module (`canonical_vendor()`), following the design proposed in #953: strip a known geo prefix (`eu.`/`us.`/`apac.`/`global.`) then match the canonical vendor. Geo-prefixed Anthropic profiles (`eu.anthropic.…`) now get live-zone compression instead of being silently skipped; geo-prefixed non-Anthropic vendors stay correctly excluded. - **Converse-body compression (two parts)**: 1. `run_anthropic_compression` no longer bails to passthrough when the body lacks an InvokeModel `anthropic_version` envelope; envelope re-emit stays gated on successful parse. 2. The **live-zone dispatcher now recognizes Bedrock Converse content blocks**. Converse blocks carry no `type` discriminator (the variant is the key: `{"text": …}` vs Anthropic's `{"type":"text","text":…}`), so real Converse user-message text was still passing through uncompressed. A typeless block whose `text` is a JSON string now routes through the same surgical text path. Anthropic blocks always carry `type`, so the Anthropic path is byte-for-byte unchanged; non-text Converse blocks (`{"image":…}`, `{"toolUse":…}`) stay unrecognized and no-op. - **Correct `/converse` upstream routing**: the non-streaming handler resolved the upstream action from a hard-coded `"invoke"`, so `/converse` requests were forwarded to Bedrock's `/invoke` endpoint. It now resolves the action from the inbound path (`extract_invoke_action`), mirroring the streaming handler's `extract_streaming_action`. SigV4 signs the same URL it forwards, so the signature stays consistent. - **`aws-config` `sso` feature**: SSO profiles now resolve through the default credential chain for SigV4 — the credential chain in `docs/bedrock.md` already promised SSO; this makes the code match. - **Ship the `headroom-proxy` binary in published images** (`Dockerfile`): built in the builder stage (`--locked`, with the cargo registry cache mounted at `CARGO_HOME`) and copied into both the debian and distroless runtime images. - **Docs** (`docs/bedrock.md`): document cross-region inference profiles and a "Running the proxy" section. AWS credentials mount at `/home/nonroot/.aws` (the default nonroot image home) where the SDK looks for `~/.aws`, with a note on the root-image alternative. ## Related issues - Closes #976 — ship the `headroom-proxy` binary in published images (this PR implements the exact fix proposed there). - Addresses the **cross-region inference-profile** half of #953 via its proposed `canonical_vendor()` design. Non-Anthropic vendor compression parity (Nova/GLM/MiniMax/ Kimi) is the natural follow-up — `bedrock::vendor` is the shared resolver it can build on. - Extends the native Bedrock InvokeModel compression requested in #734 (the Bedrock slice of #510) to cross-region profiles and Converse bodies. - Partially enables #181 (native, Python-free packaging): the native binary now ships in the images, though full Python-free distribution remains out of scope. ## Alignment with the Rust migration plan Per `docs/spec/022-rust-migration.md`, the migration is **proxy-first**: `headroom-proxy` is the deployable Rust artifact, native routes replace Python passthroughs one at a time (Stage 4 = provider expansion, Bedrock included), and the binary is meant to be "built, tested, and **released together with the Python package**." Two ways this PR advances that: - The binary-in-images change makes the codebase do what the spec already states (ship the artifact) — closing the gap that forced downstreams to build from source. - Hardening the native Bedrock route (cross-region, Converse routing + body compression) is exactly the Stage-4 provider-expansion work, keeping the native path at parity with real traffic so it can be the default rather than a passthrough. ## Testing - [x] Unit tests pass (`cargo test -p headroom-core -p headroom-proxy` — full suites, 0 failures) - [x] Linting passes (`cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings`) - [x] Formatting passes (`cargo fmt -- --check`) - [x] New tests added — `bedrock::vendor` (foundation + inference-profile matching), `extract_invoke_action` + converse upstream URL, and live-zone Converse text-block routing (`block_has_string_text_field`, converse-vs-anthropic dispatch equivalence). - [x] Manual testing performed ### Test Output ```text $ cargo test -p headroom-core -p headroom-proxy # all suites: ok, 0 failed $ cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings # Finished, no warnings $ cargo fmt -- --check # clean # image validation (local, proxy/code extras): $ docker build --target runtime ... # debian: /usr/local/bin/headroom-proxy, --help OK $ docker build --target runtime-slim ... # distroless: binary links + --help OK ``` ## Real Behavior Proof - Environment: native Bedrock proxy against `bedrock-runtime.eu-west-2`, SSO profile, model `eu.anthropic.claude-haiku-4-5-20251001-v1:0`. - Exact command / steps: POST a large multi-turn Converse body to `/model/eu.anthropic.claude-haiku-4-5-20251001-v1:0/converse`; separately build the `runtime` + `runtime-slim` targets and run `/usr/local/bin/headroom-proxy --help`. - Observed result: before — `bedrock_compression_skipped` (geo-prefixed id not recognized), forwarded uncompressed to the wrong `/invoke` upstream; after — geo-prefixed id recognized, `/converse` forwarded to the `/converse` upstream, live-zone dispatcher compresses the Converse user-message text, measurable token savings. Images contain a runnable `headroom-proxy` in both variants. - Not tested: non-Anthropic vendor compression parity (#953 follow-up); Converse `toolResult` nested-text compression (follow-up — only top-level Converse text blocks compress today). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - An earlier revision flipped the EventStream `Accept` default (`*/*`/absent → passthrough); **dropped** — `*/*` is what most clients (incl. reqwest and the proxy's own metrics tests) send while expecting SSE, so forcing passthrough breaks the standard SSE path. - The binary build adds the native-proxy compile to the image build; happy to gate it behind a build arg if maintainers prefer it opt-in. - Addressed a Copilot review round: corrected the `/converse` upstream routing, the stale `run_anthropic_compression` comment, the Dockerfile cargo cache mount + `--locked`, and the nonroot AWS-credentials docs example.
2026-06-16 16:45:24 +02:00
[[package]]
name = "aws-sdk-sso"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.102.0"
feat(bedrock): cross-region + Converse compression; bundle proxy binary in images (#999) ## Description The native Bedrock path (Phase D) compresses + signs Anthropic-on-Bedrock requests, but two real-world cases slipped through, and the native binary that powers it was never shipped. This PR closes those gaps as a focused set of give-backs. Aligns with the Rust migration plan (see below). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Cross-region inference-profile detection** via a new `bedrock::vendor` module (`canonical_vendor()`), following the design proposed in #953: strip a known geo prefix (`eu.`/`us.`/`apac.`/`global.`) then match the canonical vendor. Geo-prefixed Anthropic profiles (`eu.anthropic.…`) now get live-zone compression instead of being silently skipped; geo-prefixed non-Anthropic vendors stay correctly excluded. - **Converse-body compression (two parts)**: 1. `run_anthropic_compression` no longer bails to passthrough when the body lacks an InvokeModel `anthropic_version` envelope; envelope re-emit stays gated on successful parse. 2. The **live-zone dispatcher now recognizes Bedrock Converse content blocks**. Converse blocks carry no `type` discriminator (the variant is the key: `{"text": …}` vs Anthropic's `{"type":"text","text":…}`), so real Converse user-message text was still passing through uncompressed. A typeless block whose `text` is a JSON string now routes through the same surgical text path. Anthropic blocks always carry `type`, so the Anthropic path is byte-for-byte unchanged; non-text Converse blocks (`{"image":…}`, `{"toolUse":…}`) stay unrecognized and no-op. - **Correct `/converse` upstream routing**: the non-streaming handler resolved the upstream action from a hard-coded `"invoke"`, so `/converse` requests were forwarded to Bedrock's `/invoke` endpoint. It now resolves the action from the inbound path (`extract_invoke_action`), mirroring the streaming handler's `extract_streaming_action`. SigV4 signs the same URL it forwards, so the signature stays consistent. - **`aws-config` `sso` feature**: SSO profiles now resolve through the default credential chain for SigV4 — the credential chain in `docs/bedrock.md` already promised SSO; this makes the code match. - **Ship the `headroom-proxy` binary in published images** (`Dockerfile`): built in the builder stage (`--locked`, with the cargo registry cache mounted at `CARGO_HOME`) and copied into both the debian and distroless runtime images. - **Docs** (`docs/bedrock.md`): document cross-region inference profiles and a "Running the proxy" section. AWS credentials mount at `/home/nonroot/.aws` (the default nonroot image home) where the SDK looks for `~/.aws`, with a note on the root-image alternative. ## Related issues - Closes #976 — ship the `headroom-proxy` binary in published images (this PR implements the exact fix proposed there). - Addresses the **cross-region inference-profile** half of #953 via its proposed `canonical_vendor()` design. Non-Anthropic vendor compression parity (Nova/GLM/MiniMax/ Kimi) is the natural follow-up — `bedrock::vendor` is the shared resolver it can build on. - Extends the native Bedrock InvokeModel compression requested in #734 (the Bedrock slice of #510) to cross-region profiles and Converse bodies. - Partially enables #181 (native, Python-free packaging): the native binary now ships in the images, though full Python-free distribution remains out of scope. ## Alignment with the Rust migration plan Per `docs/spec/022-rust-migration.md`, the migration is **proxy-first**: `headroom-proxy` is the deployable Rust artifact, native routes replace Python passthroughs one at a time (Stage 4 = provider expansion, Bedrock included), and the binary is meant to be "built, tested, and **released together with the Python package**." Two ways this PR advances that: - The binary-in-images change makes the codebase do what the spec already states (ship the artifact) — closing the gap that forced downstreams to build from source. - Hardening the native Bedrock route (cross-region, Converse routing + body compression) is exactly the Stage-4 provider-expansion work, keeping the native path at parity with real traffic so it can be the default rather than a passthrough. ## Testing - [x] Unit tests pass (`cargo test -p headroom-core -p headroom-proxy` — full suites, 0 failures) - [x] Linting passes (`cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings`) - [x] Formatting passes (`cargo fmt -- --check`) - [x] New tests added — `bedrock::vendor` (foundation + inference-profile matching), `extract_invoke_action` + converse upstream URL, and live-zone Converse text-block routing (`block_has_string_text_field`, converse-vs-anthropic dispatch equivalence). - [x] Manual testing performed ### Test Output ```text $ cargo test -p headroom-core -p headroom-proxy # all suites: ok, 0 failed $ cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings # Finished, no warnings $ cargo fmt -- --check # clean # image validation (local, proxy/code extras): $ docker build --target runtime ... # debian: /usr/local/bin/headroom-proxy, --help OK $ docker build --target runtime-slim ... # distroless: binary links + --help OK ``` ## Real Behavior Proof - Environment: native Bedrock proxy against `bedrock-runtime.eu-west-2`, SSO profile, model `eu.anthropic.claude-haiku-4-5-20251001-v1:0`. - Exact command / steps: POST a large multi-turn Converse body to `/model/eu.anthropic.claude-haiku-4-5-20251001-v1:0/converse`; separately build the `runtime` + `runtime-slim` targets and run `/usr/local/bin/headroom-proxy --help`. - Observed result: before — `bedrock_compression_skipped` (geo-prefixed id not recognized), forwarded uncompressed to the wrong `/invoke` upstream; after — geo-prefixed id recognized, `/converse` forwarded to the `/converse` upstream, live-zone dispatcher compresses the Converse user-message text, measurable token savings. Images contain a runnable `headroom-proxy` in both variants. - Not tested: non-Anthropic vendor compression parity (#953 follow-up); Converse `toolResult` nested-text compression (follow-up — only top-level Converse text blocks compress today). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - An earlier revision flipped the EventStream `Accept` default (`*/*`/absent → passthrough); **dropped** — `*/*` is what most clients (incl. reqwest and the proxy's own metrics tests) send while expecting SSE, so forcing passthrough breaks the standard SSE path. - The binary build adds the native-proxy compile to the image build; happy to gate it behind a build arg if maintainers prefer it opt-in. - Addressed a Copilot review round: corrected the `/converse` upstream routing, the stale `run_anthropic_compression` comment, the Dockerfile cargo cache mount + `--locked`, and the nonroot AWS-credentials docs example.
2026-06-16 16:45:24 +02:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "8c82b3ac19f1431854f7ace3a7531674633e286bfdde21976893bfee36fd493b"
feat(bedrock): cross-region + Converse compression; bundle proxy binary in images (#999) ## Description The native Bedrock path (Phase D) compresses + signs Anthropic-on-Bedrock requests, but two real-world cases slipped through, and the native binary that powers it was never shipped. This PR closes those gaps as a focused set of give-backs. Aligns with the Rust migration plan (see below). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Cross-region inference-profile detection** via a new `bedrock::vendor` module (`canonical_vendor()`), following the design proposed in #953: strip a known geo prefix (`eu.`/`us.`/`apac.`/`global.`) then match the canonical vendor. Geo-prefixed Anthropic profiles (`eu.anthropic.…`) now get live-zone compression instead of being silently skipped; geo-prefixed non-Anthropic vendors stay correctly excluded. - **Converse-body compression (two parts)**: 1. `run_anthropic_compression` no longer bails to passthrough when the body lacks an InvokeModel `anthropic_version` envelope; envelope re-emit stays gated on successful parse. 2. The **live-zone dispatcher now recognizes Bedrock Converse content blocks**. Converse blocks carry no `type` discriminator (the variant is the key: `{"text": …}` vs Anthropic's `{"type":"text","text":…}`), so real Converse user-message text was still passing through uncompressed. A typeless block whose `text` is a JSON string now routes through the same surgical text path. Anthropic blocks always carry `type`, so the Anthropic path is byte-for-byte unchanged; non-text Converse blocks (`{"image":…}`, `{"toolUse":…}`) stay unrecognized and no-op. - **Correct `/converse` upstream routing**: the non-streaming handler resolved the upstream action from a hard-coded `"invoke"`, so `/converse` requests were forwarded to Bedrock's `/invoke` endpoint. It now resolves the action from the inbound path (`extract_invoke_action`), mirroring the streaming handler's `extract_streaming_action`. SigV4 signs the same URL it forwards, so the signature stays consistent. - **`aws-config` `sso` feature**: SSO profiles now resolve through the default credential chain for SigV4 — the credential chain in `docs/bedrock.md` already promised SSO; this makes the code match. - **Ship the `headroom-proxy` binary in published images** (`Dockerfile`): built in the builder stage (`--locked`, with the cargo registry cache mounted at `CARGO_HOME`) and copied into both the debian and distroless runtime images. - **Docs** (`docs/bedrock.md`): document cross-region inference profiles and a "Running the proxy" section. AWS credentials mount at `/home/nonroot/.aws` (the default nonroot image home) where the SDK looks for `~/.aws`, with a note on the root-image alternative. ## Related issues - Closes #976 — ship the `headroom-proxy` binary in published images (this PR implements the exact fix proposed there). - Addresses the **cross-region inference-profile** half of #953 via its proposed `canonical_vendor()` design. Non-Anthropic vendor compression parity (Nova/GLM/MiniMax/ Kimi) is the natural follow-up — `bedrock::vendor` is the shared resolver it can build on. - Extends the native Bedrock InvokeModel compression requested in #734 (the Bedrock slice of #510) to cross-region profiles and Converse bodies. - Partially enables #181 (native, Python-free packaging): the native binary now ships in the images, though full Python-free distribution remains out of scope. ## Alignment with the Rust migration plan Per `docs/spec/022-rust-migration.md`, the migration is **proxy-first**: `headroom-proxy` is the deployable Rust artifact, native routes replace Python passthroughs one at a time (Stage 4 = provider expansion, Bedrock included), and the binary is meant to be "built, tested, and **released together with the Python package**." Two ways this PR advances that: - The binary-in-images change makes the codebase do what the spec already states (ship the artifact) — closing the gap that forced downstreams to build from source. - Hardening the native Bedrock route (cross-region, Converse routing + body compression) is exactly the Stage-4 provider-expansion work, keeping the native path at parity with real traffic so it can be the default rather than a passthrough. ## Testing - [x] Unit tests pass (`cargo test -p headroom-core -p headroom-proxy` — full suites, 0 failures) - [x] Linting passes (`cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings`) - [x] Formatting passes (`cargo fmt -- --check`) - [x] New tests added — `bedrock::vendor` (foundation + inference-profile matching), `extract_invoke_action` + converse upstream URL, and live-zone Converse text-block routing (`block_has_string_text_field`, converse-vs-anthropic dispatch equivalence). - [x] Manual testing performed ### Test Output ```text $ cargo test -p headroom-core -p headroom-proxy # all suites: ok, 0 failed $ cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings # Finished, no warnings $ cargo fmt -- --check # clean # image validation (local, proxy/code extras): $ docker build --target runtime ... # debian: /usr/local/bin/headroom-proxy, --help OK $ docker build --target runtime-slim ... # distroless: binary links + --help OK ``` ## Real Behavior Proof - Environment: native Bedrock proxy against `bedrock-runtime.eu-west-2`, SSO profile, model `eu.anthropic.claude-haiku-4-5-20251001-v1:0`. - Exact command / steps: POST a large multi-turn Converse body to `/model/eu.anthropic.claude-haiku-4-5-20251001-v1:0/converse`; separately build the `runtime` + `runtime-slim` targets and run `/usr/local/bin/headroom-proxy --help`. - Observed result: before — `bedrock_compression_skipped` (geo-prefixed id not recognized), forwarded uncompressed to the wrong `/invoke` upstream; after — geo-prefixed id recognized, `/converse` forwarded to the `/converse` upstream, live-zone dispatcher compresses the Converse user-message text, measurable token savings. Images contain a runnable `headroom-proxy` in both variants. - Not tested: non-Anthropic vendor compression parity (#953 follow-up); Converse `toolResult` nested-text compression (follow-up — only top-level Converse text blocks compress today). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - An earlier revision flipped the EventStream `Accept` default (`*/*`/absent → passthrough); **dropped** — `*/*` is what most clients (incl. reqwest and the proxy's own metrics tests) send while expecting SSE, so forcing passthrough breaks the standard SSE path. - The binary build adds the native-proxy compile to the image build; happy to gate it behind a build arg if maintainers prefer it opt-in. - Addressed a Copilot review round: corrected the `/converse` upstream routing, the stale `run_anthropic_compression` comment, the Dockerfile cargo cache mount + `--locked`, and the nonroot AWS-credentials docs example.
2026-06-16 16:45:24 +02:00
dependencies = [
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"arc-swap",
feat(bedrock): cross-region + Converse compression; bundle proxy binary in images (#999) ## Description The native Bedrock path (Phase D) compresses + signs Anthropic-on-Bedrock requests, but two real-world cases slipped through, and the native binary that powers it was never shipped. This PR closes those gaps as a focused set of give-backs. Aligns with the Rust migration plan (see below). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Cross-region inference-profile detection** via a new `bedrock::vendor` module (`canonical_vendor()`), following the design proposed in #953: strip a known geo prefix (`eu.`/`us.`/`apac.`/`global.`) then match the canonical vendor. Geo-prefixed Anthropic profiles (`eu.anthropic.…`) now get live-zone compression instead of being silently skipped; geo-prefixed non-Anthropic vendors stay correctly excluded. - **Converse-body compression (two parts)**: 1. `run_anthropic_compression` no longer bails to passthrough when the body lacks an InvokeModel `anthropic_version` envelope; envelope re-emit stays gated on successful parse. 2. The **live-zone dispatcher now recognizes Bedrock Converse content blocks**. Converse blocks carry no `type` discriminator (the variant is the key: `{"text": …}` vs Anthropic's `{"type":"text","text":…}`), so real Converse user-message text was still passing through uncompressed. A typeless block whose `text` is a JSON string now routes through the same surgical text path. Anthropic blocks always carry `type`, so the Anthropic path is byte-for-byte unchanged; non-text Converse blocks (`{"image":…}`, `{"toolUse":…}`) stay unrecognized and no-op. - **Correct `/converse` upstream routing**: the non-streaming handler resolved the upstream action from a hard-coded `"invoke"`, so `/converse` requests were forwarded to Bedrock's `/invoke` endpoint. It now resolves the action from the inbound path (`extract_invoke_action`), mirroring the streaming handler's `extract_streaming_action`. SigV4 signs the same URL it forwards, so the signature stays consistent. - **`aws-config` `sso` feature**: SSO profiles now resolve through the default credential chain for SigV4 — the credential chain in `docs/bedrock.md` already promised SSO; this makes the code match. - **Ship the `headroom-proxy` binary in published images** (`Dockerfile`): built in the builder stage (`--locked`, with the cargo registry cache mounted at `CARGO_HOME`) and copied into both the debian and distroless runtime images. - **Docs** (`docs/bedrock.md`): document cross-region inference profiles and a "Running the proxy" section. AWS credentials mount at `/home/nonroot/.aws` (the default nonroot image home) where the SDK looks for `~/.aws`, with a note on the root-image alternative. ## Related issues - Closes #976 — ship the `headroom-proxy` binary in published images (this PR implements the exact fix proposed there). - Addresses the **cross-region inference-profile** half of #953 via its proposed `canonical_vendor()` design. Non-Anthropic vendor compression parity (Nova/GLM/MiniMax/ Kimi) is the natural follow-up — `bedrock::vendor` is the shared resolver it can build on. - Extends the native Bedrock InvokeModel compression requested in #734 (the Bedrock slice of #510) to cross-region profiles and Converse bodies. - Partially enables #181 (native, Python-free packaging): the native binary now ships in the images, though full Python-free distribution remains out of scope. ## Alignment with the Rust migration plan Per `docs/spec/022-rust-migration.md`, the migration is **proxy-first**: `headroom-proxy` is the deployable Rust artifact, native routes replace Python passthroughs one at a time (Stage 4 = provider expansion, Bedrock included), and the binary is meant to be "built, tested, and **released together with the Python package**." Two ways this PR advances that: - The binary-in-images change makes the codebase do what the spec already states (ship the artifact) — closing the gap that forced downstreams to build from source. - Hardening the native Bedrock route (cross-region, Converse routing + body compression) is exactly the Stage-4 provider-expansion work, keeping the native path at parity with real traffic so it can be the default rather than a passthrough. ## Testing - [x] Unit tests pass (`cargo test -p headroom-core -p headroom-proxy` — full suites, 0 failures) - [x] Linting passes (`cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings`) - [x] Formatting passes (`cargo fmt -- --check`) - [x] New tests added — `bedrock::vendor` (foundation + inference-profile matching), `extract_invoke_action` + converse upstream URL, and live-zone Converse text-block routing (`block_has_string_text_field`, converse-vs-anthropic dispatch equivalence). - [x] Manual testing performed ### Test Output ```text $ cargo test -p headroom-core -p headroom-proxy # all suites: ok, 0 failed $ cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings # Finished, no warnings $ cargo fmt -- --check # clean # image validation (local, proxy/code extras): $ docker build --target runtime ... # debian: /usr/local/bin/headroom-proxy, --help OK $ docker build --target runtime-slim ... # distroless: binary links + --help OK ``` ## Real Behavior Proof - Environment: native Bedrock proxy against `bedrock-runtime.eu-west-2`, SSO profile, model `eu.anthropic.claude-haiku-4-5-20251001-v1:0`. - Exact command / steps: POST a large multi-turn Converse body to `/model/eu.anthropic.claude-haiku-4-5-20251001-v1:0/converse`; separately build the `runtime` + `runtime-slim` targets and run `/usr/local/bin/headroom-proxy --help`. - Observed result: before — `bedrock_compression_skipped` (geo-prefixed id not recognized), forwarded uncompressed to the wrong `/invoke` upstream; after — geo-prefixed id recognized, `/converse` forwarded to the `/converse` upstream, live-zone dispatcher compresses the Converse user-message text, measurable token savings. Images contain a runnable `headroom-proxy` in both variants. - Not tested: non-Anthropic vendor compression parity (#953 follow-up); Converse `toolResult` nested-text compression (follow-up — only top-level Converse text blocks compress today). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - An earlier revision flipped the EventStream `Accept` default (`*/*`/absent → passthrough); **dropped** — `*/*` is what most clients (incl. reqwest and the proxy's own metrics tests) send while expecting SSE, so forcing passthrough breaks the standard SSE path. - The binary build adds the native-proxy compile to the image build; happy to gate it behind a build arg if maintainers prefer it opt-in. - Addressed a Copilot review round: corrected the `/converse` upstream routing, the stale `run_anthropic_compression` comment, the Dockerfile cargo cache mount + `--locked`, and the nonroot AWS-credentials docs example.
2026-06-16 16:45:24 +02:00
"aws-credential-types",
"aws-runtime",
"aws-smithy-async",
"aws-smithy-http",
"aws-smithy-json",
"aws-smithy-observability",
"aws-smithy-runtime",
"aws-smithy-runtime-api",
"aws-smithy-types",
"aws-types",
"bytes",
"fastrand",
"http 0.2.12",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
feat(bedrock): cross-region + Converse compression; bundle proxy binary in images (#999) ## Description The native Bedrock path (Phase D) compresses + signs Anthropic-on-Bedrock requests, but two real-world cases slipped through, and the native binary that powers it was never shipped. This PR closes those gaps as a focused set of give-backs. Aligns with the Rust migration plan (see below). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Cross-region inference-profile detection** via a new `bedrock::vendor` module (`canonical_vendor()`), following the design proposed in #953: strip a known geo prefix (`eu.`/`us.`/`apac.`/`global.`) then match the canonical vendor. Geo-prefixed Anthropic profiles (`eu.anthropic.…`) now get live-zone compression instead of being silently skipped; geo-prefixed non-Anthropic vendors stay correctly excluded. - **Converse-body compression (two parts)**: 1. `run_anthropic_compression` no longer bails to passthrough when the body lacks an InvokeModel `anthropic_version` envelope; envelope re-emit stays gated on successful parse. 2. The **live-zone dispatcher now recognizes Bedrock Converse content blocks**. Converse blocks carry no `type` discriminator (the variant is the key: `{"text": …}` vs Anthropic's `{"type":"text","text":…}`), so real Converse user-message text was still passing through uncompressed. A typeless block whose `text` is a JSON string now routes through the same surgical text path. Anthropic blocks always carry `type`, so the Anthropic path is byte-for-byte unchanged; non-text Converse blocks (`{"image":…}`, `{"toolUse":…}`) stay unrecognized and no-op. - **Correct `/converse` upstream routing**: the non-streaming handler resolved the upstream action from a hard-coded `"invoke"`, so `/converse` requests were forwarded to Bedrock's `/invoke` endpoint. It now resolves the action from the inbound path (`extract_invoke_action`), mirroring the streaming handler's `extract_streaming_action`. SigV4 signs the same URL it forwards, so the signature stays consistent. - **`aws-config` `sso` feature**: SSO profiles now resolve through the default credential chain for SigV4 — the credential chain in `docs/bedrock.md` already promised SSO; this makes the code match. - **Ship the `headroom-proxy` binary in published images** (`Dockerfile`): built in the builder stage (`--locked`, with the cargo registry cache mounted at `CARGO_HOME`) and copied into both the debian and distroless runtime images. - **Docs** (`docs/bedrock.md`): document cross-region inference profiles and a "Running the proxy" section. AWS credentials mount at `/home/nonroot/.aws` (the default nonroot image home) where the SDK looks for `~/.aws`, with a note on the root-image alternative. ## Related issues - Closes #976 — ship the `headroom-proxy` binary in published images (this PR implements the exact fix proposed there). - Addresses the **cross-region inference-profile** half of #953 via its proposed `canonical_vendor()` design. Non-Anthropic vendor compression parity (Nova/GLM/MiniMax/ Kimi) is the natural follow-up — `bedrock::vendor` is the shared resolver it can build on. - Extends the native Bedrock InvokeModel compression requested in #734 (the Bedrock slice of #510) to cross-region profiles and Converse bodies. - Partially enables #181 (native, Python-free packaging): the native binary now ships in the images, though full Python-free distribution remains out of scope. ## Alignment with the Rust migration plan Per `docs/spec/022-rust-migration.md`, the migration is **proxy-first**: `headroom-proxy` is the deployable Rust artifact, native routes replace Python passthroughs one at a time (Stage 4 = provider expansion, Bedrock included), and the binary is meant to be "built, tested, and **released together with the Python package**." Two ways this PR advances that: - The binary-in-images change makes the codebase do what the spec already states (ship the artifact) — closing the gap that forced downstreams to build from source. - Hardening the native Bedrock route (cross-region, Converse routing + body compression) is exactly the Stage-4 provider-expansion work, keeping the native path at parity with real traffic so it can be the default rather than a passthrough. ## Testing - [x] Unit tests pass (`cargo test -p headroom-core -p headroom-proxy` — full suites, 0 failures) - [x] Linting passes (`cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings`) - [x] Formatting passes (`cargo fmt -- --check`) - [x] New tests added — `bedrock::vendor` (foundation + inference-profile matching), `extract_invoke_action` + converse upstream URL, and live-zone Converse text-block routing (`block_has_string_text_field`, converse-vs-anthropic dispatch equivalence). - [x] Manual testing performed ### Test Output ```text $ cargo test -p headroom-core -p headroom-proxy # all suites: ok, 0 failed $ cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings # Finished, no warnings $ cargo fmt -- --check # clean # image validation (local, proxy/code extras): $ docker build --target runtime ... # debian: /usr/local/bin/headroom-proxy, --help OK $ docker build --target runtime-slim ... # distroless: binary links + --help OK ``` ## Real Behavior Proof - Environment: native Bedrock proxy against `bedrock-runtime.eu-west-2`, SSO profile, model `eu.anthropic.claude-haiku-4-5-20251001-v1:0`. - Exact command / steps: POST a large multi-turn Converse body to `/model/eu.anthropic.claude-haiku-4-5-20251001-v1:0/converse`; separately build the `runtime` + `runtime-slim` targets and run `/usr/local/bin/headroom-proxy --help`. - Observed result: before — `bedrock_compression_skipped` (geo-prefixed id not recognized), forwarded uncompressed to the wrong `/invoke` upstream; after — geo-prefixed id recognized, `/converse` forwarded to the `/converse` upstream, live-zone dispatcher compresses the Converse user-message text, measurable token savings. Images contain a runnable `headroom-proxy` in both variants. - Not tested: non-Anthropic vendor compression parity (#953 follow-up); Converse `toolResult` nested-text compression (follow-up — only top-level Converse text blocks compress today). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - An earlier revision flipped the EventStream `Accept` default (`*/*`/absent → passthrough); **dropped** — `*/*` is what most clients (incl. reqwest and the proxy's own metrics tests) send while expecting SSE, so forcing passthrough breaks the standard SSE path. - The binary build adds the native-proxy compile to the image build; happy to gate it behind a build arg if maintainers prefer it opt-in. - Addressed a Copilot review round: corrected the `/converse` upstream routing, the stale `run_anthropic_compression` comment, the Dockerfile cargo cache mount + `--locked`, and the nonroot AWS-credentials docs example.
2026-06-16 16:45:24 +02:00
"regex-lite",
"tracing",
]
[[package]]
name = "aws-sdk-ssooidc"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.104.0"
feat(bedrock): cross-region + Converse compression; bundle proxy binary in images (#999) ## Description The native Bedrock path (Phase D) compresses + signs Anthropic-on-Bedrock requests, but two real-world cases slipped through, and the native binary that powers it was never shipped. This PR closes those gaps as a focused set of give-backs. Aligns with the Rust migration plan (see below). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Cross-region inference-profile detection** via a new `bedrock::vendor` module (`canonical_vendor()`), following the design proposed in #953: strip a known geo prefix (`eu.`/`us.`/`apac.`/`global.`) then match the canonical vendor. Geo-prefixed Anthropic profiles (`eu.anthropic.…`) now get live-zone compression instead of being silently skipped; geo-prefixed non-Anthropic vendors stay correctly excluded. - **Converse-body compression (two parts)**: 1. `run_anthropic_compression` no longer bails to passthrough when the body lacks an InvokeModel `anthropic_version` envelope; envelope re-emit stays gated on successful parse. 2. The **live-zone dispatcher now recognizes Bedrock Converse content blocks**. Converse blocks carry no `type` discriminator (the variant is the key: `{"text": …}` vs Anthropic's `{"type":"text","text":…}`), so real Converse user-message text was still passing through uncompressed. A typeless block whose `text` is a JSON string now routes through the same surgical text path. Anthropic blocks always carry `type`, so the Anthropic path is byte-for-byte unchanged; non-text Converse blocks (`{"image":…}`, `{"toolUse":…}`) stay unrecognized and no-op. - **Correct `/converse` upstream routing**: the non-streaming handler resolved the upstream action from a hard-coded `"invoke"`, so `/converse` requests were forwarded to Bedrock's `/invoke` endpoint. It now resolves the action from the inbound path (`extract_invoke_action`), mirroring the streaming handler's `extract_streaming_action`. SigV4 signs the same URL it forwards, so the signature stays consistent. - **`aws-config` `sso` feature**: SSO profiles now resolve through the default credential chain for SigV4 — the credential chain in `docs/bedrock.md` already promised SSO; this makes the code match. - **Ship the `headroom-proxy` binary in published images** (`Dockerfile`): built in the builder stage (`--locked`, with the cargo registry cache mounted at `CARGO_HOME`) and copied into both the debian and distroless runtime images. - **Docs** (`docs/bedrock.md`): document cross-region inference profiles and a "Running the proxy" section. AWS credentials mount at `/home/nonroot/.aws` (the default nonroot image home) where the SDK looks for `~/.aws`, with a note on the root-image alternative. ## Related issues - Closes #976 — ship the `headroom-proxy` binary in published images (this PR implements the exact fix proposed there). - Addresses the **cross-region inference-profile** half of #953 via its proposed `canonical_vendor()` design. Non-Anthropic vendor compression parity (Nova/GLM/MiniMax/ Kimi) is the natural follow-up — `bedrock::vendor` is the shared resolver it can build on. - Extends the native Bedrock InvokeModel compression requested in #734 (the Bedrock slice of #510) to cross-region profiles and Converse bodies. - Partially enables #181 (native, Python-free packaging): the native binary now ships in the images, though full Python-free distribution remains out of scope. ## Alignment with the Rust migration plan Per `docs/spec/022-rust-migration.md`, the migration is **proxy-first**: `headroom-proxy` is the deployable Rust artifact, native routes replace Python passthroughs one at a time (Stage 4 = provider expansion, Bedrock included), and the binary is meant to be "built, tested, and **released together with the Python package**." Two ways this PR advances that: - The binary-in-images change makes the codebase do what the spec already states (ship the artifact) — closing the gap that forced downstreams to build from source. - Hardening the native Bedrock route (cross-region, Converse routing + body compression) is exactly the Stage-4 provider-expansion work, keeping the native path at parity with real traffic so it can be the default rather than a passthrough. ## Testing - [x] Unit tests pass (`cargo test -p headroom-core -p headroom-proxy` — full suites, 0 failures) - [x] Linting passes (`cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings`) - [x] Formatting passes (`cargo fmt -- --check`) - [x] New tests added — `bedrock::vendor` (foundation + inference-profile matching), `extract_invoke_action` + converse upstream URL, and live-zone Converse text-block routing (`block_has_string_text_field`, converse-vs-anthropic dispatch equivalence). - [x] Manual testing performed ### Test Output ```text $ cargo test -p headroom-core -p headroom-proxy # all suites: ok, 0 failed $ cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings # Finished, no warnings $ cargo fmt -- --check # clean # image validation (local, proxy/code extras): $ docker build --target runtime ... # debian: /usr/local/bin/headroom-proxy, --help OK $ docker build --target runtime-slim ... # distroless: binary links + --help OK ``` ## Real Behavior Proof - Environment: native Bedrock proxy against `bedrock-runtime.eu-west-2`, SSO profile, model `eu.anthropic.claude-haiku-4-5-20251001-v1:0`. - Exact command / steps: POST a large multi-turn Converse body to `/model/eu.anthropic.claude-haiku-4-5-20251001-v1:0/converse`; separately build the `runtime` + `runtime-slim` targets and run `/usr/local/bin/headroom-proxy --help`. - Observed result: before — `bedrock_compression_skipped` (geo-prefixed id not recognized), forwarded uncompressed to the wrong `/invoke` upstream; after — geo-prefixed id recognized, `/converse` forwarded to the `/converse` upstream, live-zone dispatcher compresses the Converse user-message text, measurable token savings. Images contain a runnable `headroom-proxy` in both variants. - Not tested: non-Anthropic vendor compression parity (#953 follow-up); Converse `toolResult` nested-text compression (follow-up — only top-level Converse text blocks compress today). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - An earlier revision flipped the EventStream `Accept` default (`*/*`/absent → passthrough); **dropped** — `*/*` is what most clients (incl. reqwest and the proxy's own metrics tests) send while expecting SSE, so forcing passthrough breaks the standard SSE path. - The binary build adds the native-proxy compile to the image build; happy to gate it behind a build arg if maintainers prefer it opt-in. - Addressed a Copilot review round: corrected the `/converse` upstream routing, the stale `run_anthropic_compression` comment, the Dockerfile cargo cache mount + `--locked`, and the nonroot AWS-credentials docs example.
2026-06-16 16:45:24 +02:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "321000d2b4c5519ee573f73167f612efd7329322d9b26969ad1979f0427f1913"
feat(bedrock): cross-region + Converse compression; bundle proxy binary in images (#999) ## Description The native Bedrock path (Phase D) compresses + signs Anthropic-on-Bedrock requests, but two real-world cases slipped through, and the native binary that powers it was never shipped. This PR closes those gaps as a focused set of give-backs. Aligns with the Rust migration plan (see below). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Cross-region inference-profile detection** via a new `bedrock::vendor` module (`canonical_vendor()`), following the design proposed in #953: strip a known geo prefix (`eu.`/`us.`/`apac.`/`global.`) then match the canonical vendor. Geo-prefixed Anthropic profiles (`eu.anthropic.…`) now get live-zone compression instead of being silently skipped; geo-prefixed non-Anthropic vendors stay correctly excluded. - **Converse-body compression (two parts)**: 1. `run_anthropic_compression` no longer bails to passthrough when the body lacks an InvokeModel `anthropic_version` envelope; envelope re-emit stays gated on successful parse. 2. The **live-zone dispatcher now recognizes Bedrock Converse content blocks**. Converse blocks carry no `type` discriminator (the variant is the key: `{"text": …}` vs Anthropic's `{"type":"text","text":…}`), so real Converse user-message text was still passing through uncompressed. A typeless block whose `text` is a JSON string now routes through the same surgical text path. Anthropic blocks always carry `type`, so the Anthropic path is byte-for-byte unchanged; non-text Converse blocks (`{"image":…}`, `{"toolUse":…}`) stay unrecognized and no-op. - **Correct `/converse` upstream routing**: the non-streaming handler resolved the upstream action from a hard-coded `"invoke"`, so `/converse` requests were forwarded to Bedrock's `/invoke` endpoint. It now resolves the action from the inbound path (`extract_invoke_action`), mirroring the streaming handler's `extract_streaming_action`. SigV4 signs the same URL it forwards, so the signature stays consistent. - **`aws-config` `sso` feature**: SSO profiles now resolve through the default credential chain for SigV4 — the credential chain in `docs/bedrock.md` already promised SSO; this makes the code match. - **Ship the `headroom-proxy` binary in published images** (`Dockerfile`): built in the builder stage (`--locked`, with the cargo registry cache mounted at `CARGO_HOME`) and copied into both the debian and distroless runtime images. - **Docs** (`docs/bedrock.md`): document cross-region inference profiles and a "Running the proxy" section. AWS credentials mount at `/home/nonroot/.aws` (the default nonroot image home) where the SDK looks for `~/.aws`, with a note on the root-image alternative. ## Related issues - Closes #976 — ship the `headroom-proxy` binary in published images (this PR implements the exact fix proposed there). - Addresses the **cross-region inference-profile** half of #953 via its proposed `canonical_vendor()` design. Non-Anthropic vendor compression parity (Nova/GLM/MiniMax/ Kimi) is the natural follow-up — `bedrock::vendor` is the shared resolver it can build on. - Extends the native Bedrock InvokeModel compression requested in #734 (the Bedrock slice of #510) to cross-region profiles and Converse bodies. - Partially enables #181 (native, Python-free packaging): the native binary now ships in the images, though full Python-free distribution remains out of scope. ## Alignment with the Rust migration plan Per `docs/spec/022-rust-migration.md`, the migration is **proxy-first**: `headroom-proxy` is the deployable Rust artifact, native routes replace Python passthroughs one at a time (Stage 4 = provider expansion, Bedrock included), and the binary is meant to be "built, tested, and **released together with the Python package**." Two ways this PR advances that: - The binary-in-images change makes the codebase do what the spec already states (ship the artifact) — closing the gap that forced downstreams to build from source. - Hardening the native Bedrock route (cross-region, Converse routing + body compression) is exactly the Stage-4 provider-expansion work, keeping the native path at parity with real traffic so it can be the default rather than a passthrough. ## Testing - [x] Unit tests pass (`cargo test -p headroom-core -p headroom-proxy` — full suites, 0 failures) - [x] Linting passes (`cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings`) - [x] Formatting passes (`cargo fmt -- --check`) - [x] New tests added — `bedrock::vendor` (foundation + inference-profile matching), `extract_invoke_action` + converse upstream URL, and live-zone Converse text-block routing (`block_has_string_text_field`, converse-vs-anthropic dispatch equivalence). - [x] Manual testing performed ### Test Output ```text $ cargo test -p headroom-core -p headroom-proxy # all suites: ok, 0 failed $ cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings # Finished, no warnings $ cargo fmt -- --check # clean # image validation (local, proxy/code extras): $ docker build --target runtime ... # debian: /usr/local/bin/headroom-proxy, --help OK $ docker build --target runtime-slim ... # distroless: binary links + --help OK ``` ## Real Behavior Proof - Environment: native Bedrock proxy against `bedrock-runtime.eu-west-2`, SSO profile, model `eu.anthropic.claude-haiku-4-5-20251001-v1:0`. - Exact command / steps: POST a large multi-turn Converse body to `/model/eu.anthropic.claude-haiku-4-5-20251001-v1:0/converse`; separately build the `runtime` + `runtime-slim` targets and run `/usr/local/bin/headroom-proxy --help`. - Observed result: before — `bedrock_compression_skipped` (geo-prefixed id not recognized), forwarded uncompressed to the wrong `/invoke` upstream; after — geo-prefixed id recognized, `/converse` forwarded to the `/converse` upstream, live-zone dispatcher compresses the Converse user-message text, measurable token savings. Images contain a runnable `headroom-proxy` in both variants. - Not tested: non-Anthropic vendor compression parity (#953 follow-up); Converse `toolResult` nested-text compression (follow-up — only top-level Converse text blocks compress today). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - An earlier revision flipped the EventStream `Accept` default (`*/*`/absent → passthrough); **dropped** — `*/*` is what most clients (incl. reqwest and the proxy's own metrics tests) send while expecting SSE, so forcing passthrough breaks the standard SSE path. - The binary build adds the native-proxy compile to the image build; happy to gate it behind a build arg if maintainers prefer it opt-in. - Addressed a Copilot review round: corrected the `/converse` upstream routing, the stale `run_anthropic_compression` comment, the Dockerfile cargo cache mount + `--locked`, and the nonroot AWS-credentials docs example.
2026-06-16 16:45:24 +02:00
dependencies = [
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"arc-swap",
feat(bedrock): cross-region + Converse compression; bundle proxy binary in images (#999) ## Description The native Bedrock path (Phase D) compresses + signs Anthropic-on-Bedrock requests, but two real-world cases slipped through, and the native binary that powers it was never shipped. This PR closes those gaps as a focused set of give-backs. Aligns with the Rust migration plan (see below). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Cross-region inference-profile detection** via a new `bedrock::vendor` module (`canonical_vendor()`), following the design proposed in #953: strip a known geo prefix (`eu.`/`us.`/`apac.`/`global.`) then match the canonical vendor. Geo-prefixed Anthropic profiles (`eu.anthropic.…`) now get live-zone compression instead of being silently skipped; geo-prefixed non-Anthropic vendors stay correctly excluded. - **Converse-body compression (two parts)**: 1. `run_anthropic_compression` no longer bails to passthrough when the body lacks an InvokeModel `anthropic_version` envelope; envelope re-emit stays gated on successful parse. 2. The **live-zone dispatcher now recognizes Bedrock Converse content blocks**. Converse blocks carry no `type` discriminator (the variant is the key: `{"text": …}` vs Anthropic's `{"type":"text","text":…}`), so real Converse user-message text was still passing through uncompressed. A typeless block whose `text` is a JSON string now routes through the same surgical text path. Anthropic blocks always carry `type`, so the Anthropic path is byte-for-byte unchanged; non-text Converse blocks (`{"image":…}`, `{"toolUse":…}`) stay unrecognized and no-op. - **Correct `/converse` upstream routing**: the non-streaming handler resolved the upstream action from a hard-coded `"invoke"`, so `/converse` requests were forwarded to Bedrock's `/invoke` endpoint. It now resolves the action from the inbound path (`extract_invoke_action`), mirroring the streaming handler's `extract_streaming_action`. SigV4 signs the same URL it forwards, so the signature stays consistent. - **`aws-config` `sso` feature**: SSO profiles now resolve through the default credential chain for SigV4 — the credential chain in `docs/bedrock.md` already promised SSO; this makes the code match. - **Ship the `headroom-proxy` binary in published images** (`Dockerfile`): built in the builder stage (`--locked`, with the cargo registry cache mounted at `CARGO_HOME`) and copied into both the debian and distroless runtime images. - **Docs** (`docs/bedrock.md`): document cross-region inference profiles and a "Running the proxy" section. AWS credentials mount at `/home/nonroot/.aws` (the default nonroot image home) where the SDK looks for `~/.aws`, with a note on the root-image alternative. ## Related issues - Closes #976 — ship the `headroom-proxy` binary in published images (this PR implements the exact fix proposed there). - Addresses the **cross-region inference-profile** half of #953 via its proposed `canonical_vendor()` design. Non-Anthropic vendor compression parity (Nova/GLM/MiniMax/ Kimi) is the natural follow-up — `bedrock::vendor` is the shared resolver it can build on. - Extends the native Bedrock InvokeModel compression requested in #734 (the Bedrock slice of #510) to cross-region profiles and Converse bodies. - Partially enables #181 (native, Python-free packaging): the native binary now ships in the images, though full Python-free distribution remains out of scope. ## Alignment with the Rust migration plan Per `docs/spec/022-rust-migration.md`, the migration is **proxy-first**: `headroom-proxy` is the deployable Rust artifact, native routes replace Python passthroughs one at a time (Stage 4 = provider expansion, Bedrock included), and the binary is meant to be "built, tested, and **released together with the Python package**." Two ways this PR advances that: - The binary-in-images change makes the codebase do what the spec already states (ship the artifact) — closing the gap that forced downstreams to build from source. - Hardening the native Bedrock route (cross-region, Converse routing + body compression) is exactly the Stage-4 provider-expansion work, keeping the native path at parity with real traffic so it can be the default rather than a passthrough. ## Testing - [x] Unit tests pass (`cargo test -p headroom-core -p headroom-proxy` — full suites, 0 failures) - [x] Linting passes (`cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings`) - [x] Formatting passes (`cargo fmt -- --check`) - [x] New tests added — `bedrock::vendor` (foundation + inference-profile matching), `extract_invoke_action` + converse upstream URL, and live-zone Converse text-block routing (`block_has_string_text_field`, converse-vs-anthropic dispatch equivalence). - [x] Manual testing performed ### Test Output ```text $ cargo test -p headroom-core -p headroom-proxy # all suites: ok, 0 failed $ cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings # Finished, no warnings $ cargo fmt -- --check # clean # image validation (local, proxy/code extras): $ docker build --target runtime ... # debian: /usr/local/bin/headroom-proxy, --help OK $ docker build --target runtime-slim ... # distroless: binary links + --help OK ``` ## Real Behavior Proof - Environment: native Bedrock proxy against `bedrock-runtime.eu-west-2`, SSO profile, model `eu.anthropic.claude-haiku-4-5-20251001-v1:0`. - Exact command / steps: POST a large multi-turn Converse body to `/model/eu.anthropic.claude-haiku-4-5-20251001-v1:0/converse`; separately build the `runtime` + `runtime-slim` targets and run `/usr/local/bin/headroom-proxy --help`. - Observed result: before — `bedrock_compression_skipped` (geo-prefixed id not recognized), forwarded uncompressed to the wrong `/invoke` upstream; after — geo-prefixed id recognized, `/converse` forwarded to the `/converse` upstream, live-zone dispatcher compresses the Converse user-message text, measurable token savings. Images contain a runnable `headroom-proxy` in both variants. - Not tested: non-Anthropic vendor compression parity (#953 follow-up); Converse `toolResult` nested-text compression (follow-up — only top-level Converse text blocks compress today). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - An earlier revision flipped the EventStream `Accept` default (`*/*`/absent → passthrough); **dropped** — `*/*` is what most clients (incl. reqwest and the proxy's own metrics tests) send while expecting SSE, so forcing passthrough breaks the standard SSE path. - The binary build adds the native-proxy compile to the image build; happy to gate it behind a build arg if maintainers prefer it opt-in. - Addressed a Copilot review round: corrected the `/converse` upstream routing, the stale `run_anthropic_compression` comment, the Dockerfile cargo cache mount + `--locked`, and the nonroot AWS-credentials docs example.
2026-06-16 16:45:24 +02:00
"aws-credential-types",
"aws-runtime",
"aws-smithy-async",
"aws-smithy-http",
"aws-smithy-json",
"aws-smithy-observability",
"aws-smithy-runtime",
"aws-smithy-runtime-api",
"aws-smithy-types",
"aws-types",
"bytes",
"fastrand",
"http 0.2.12",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
feat(bedrock): cross-region + Converse compression; bundle proxy binary in images (#999) ## Description The native Bedrock path (Phase D) compresses + signs Anthropic-on-Bedrock requests, but two real-world cases slipped through, and the native binary that powers it was never shipped. This PR closes those gaps as a focused set of give-backs. Aligns with the Rust migration plan (see below). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Cross-region inference-profile detection** via a new `bedrock::vendor` module (`canonical_vendor()`), following the design proposed in #953: strip a known geo prefix (`eu.`/`us.`/`apac.`/`global.`) then match the canonical vendor. Geo-prefixed Anthropic profiles (`eu.anthropic.…`) now get live-zone compression instead of being silently skipped; geo-prefixed non-Anthropic vendors stay correctly excluded. - **Converse-body compression (two parts)**: 1. `run_anthropic_compression` no longer bails to passthrough when the body lacks an InvokeModel `anthropic_version` envelope; envelope re-emit stays gated on successful parse. 2. The **live-zone dispatcher now recognizes Bedrock Converse content blocks**. Converse blocks carry no `type` discriminator (the variant is the key: `{"text": …}` vs Anthropic's `{"type":"text","text":…}`), so real Converse user-message text was still passing through uncompressed. A typeless block whose `text` is a JSON string now routes through the same surgical text path. Anthropic blocks always carry `type`, so the Anthropic path is byte-for-byte unchanged; non-text Converse blocks (`{"image":…}`, `{"toolUse":…}`) stay unrecognized and no-op. - **Correct `/converse` upstream routing**: the non-streaming handler resolved the upstream action from a hard-coded `"invoke"`, so `/converse` requests were forwarded to Bedrock's `/invoke` endpoint. It now resolves the action from the inbound path (`extract_invoke_action`), mirroring the streaming handler's `extract_streaming_action`. SigV4 signs the same URL it forwards, so the signature stays consistent. - **`aws-config` `sso` feature**: SSO profiles now resolve through the default credential chain for SigV4 — the credential chain in `docs/bedrock.md` already promised SSO; this makes the code match. - **Ship the `headroom-proxy` binary in published images** (`Dockerfile`): built in the builder stage (`--locked`, with the cargo registry cache mounted at `CARGO_HOME`) and copied into both the debian and distroless runtime images. - **Docs** (`docs/bedrock.md`): document cross-region inference profiles and a "Running the proxy" section. AWS credentials mount at `/home/nonroot/.aws` (the default nonroot image home) where the SDK looks for `~/.aws`, with a note on the root-image alternative. ## Related issues - Closes #976 — ship the `headroom-proxy` binary in published images (this PR implements the exact fix proposed there). - Addresses the **cross-region inference-profile** half of #953 via its proposed `canonical_vendor()` design. Non-Anthropic vendor compression parity (Nova/GLM/MiniMax/ Kimi) is the natural follow-up — `bedrock::vendor` is the shared resolver it can build on. - Extends the native Bedrock InvokeModel compression requested in #734 (the Bedrock slice of #510) to cross-region profiles and Converse bodies. - Partially enables #181 (native, Python-free packaging): the native binary now ships in the images, though full Python-free distribution remains out of scope. ## Alignment with the Rust migration plan Per `docs/spec/022-rust-migration.md`, the migration is **proxy-first**: `headroom-proxy` is the deployable Rust artifact, native routes replace Python passthroughs one at a time (Stage 4 = provider expansion, Bedrock included), and the binary is meant to be "built, tested, and **released together with the Python package**." Two ways this PR advances that: - The binary-in-images change makes the codebase do what the spec already states (ship the artifact) — closing the gap that forced downstreams to build from source. - Hardening the native Bedrock route (cross-region, Converse routing + body compression) is exactly the Stage-4 provider-expansion work, keeping the native path at parity with real traffic so it can be the default rather than a passthrough. ## Testing - [x] Unit tests pass (`cargo test -p headroom-core -p headroom-proxy` — full suites, 0 failures) - [x] Linting passes (`cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings`) - [x] Formatting passes (`cargo fmt -- --check`) - [x] New tests added — `bedrock::vendor` (foundation + inference-profile matching), `extract_invoke_action` + converse upstream URL, and live-zone Converse text-block routing (`block_has_string_text_field`, converse-vs-anthropic dispatch equivalence). - [x] Manual testing performed ### Test Output ```text $ cargo test -p headroom-core -p headroom-proxy # all suites: ok, 0 failed $ cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings # Finished, no warnings $ cargo fmt -- --check # clean # image validation (local, proxy/code extras): $ docker build --target runtime ... # debian: /usr/local/bin/headroom-proxy, --help OK $ docker build --target runtime-slim ... # distroless: binary links + --help OK ``` ## Real Behavior Proof - Environment: native Bedrock proxy against `bedrock-runtime.eu-west-2`, SSO profile, model `eu.anthropic.claude-haiku-4-5-20251001-v1:0`. - Exact command / steps: POST a large multi-turn Converse body to `/model/eu.anthropic.claude-haiku-4-5-20251001-v1:0/converse`; separately build the `runtime` + `runtime-slim` targets and run `/usr/local/bin/headroom-proxy --help`. - Observed result: before — `bedrock_compression_skipped` (geo-prefixed id not recognized), forwarded uncompressed to the wrong `/invoke` upstream; after — geo-prefixed id recognized, `/converse` forwarded to the `/converse` upstream, live-zone dispatcher compresses the Converse user-message text, measurable token savings. Images contain a runnable `headroom-proxy` in both variants. - Not tested: non-Anthropic vendor compression parity (#953 follow-up); Converse `toolResult` nested-text compression (follow-up — only top-level Converse text blocks compress today). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - An earlier revision flipped the EventStream `Accept` default (`*/*`/absent → passthrough); **dropped** — `*/*` is what most clients (incl. reqwest and the proxy's own metrics tests) send while expecting SSE, so forcing passthrough breaks the standard SSE path. - The binary build adds the native-proxy compile to the image build; happy to gate it behind a build arg if maintainers prefer it opt-in. - Addressed a Copilot review round: corrected the `/converse` upstream routing, the stale `run_anthropic_compression` comment, the Dockerfile cargo cache mount + `--locked`, and the nonroot AWS-credentials docs example.
2026-06-16 16:45:24 +02:00
"regex-lite",
"tracing",
]
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
[[package]]
name = "aws-sdk-sts"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.107.0"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "3d0d328ba962af23ecfa3c9f23b98d3d35e325fa218d7f13d17a6bf522f8a560"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
dependencies = [
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"arc-swap",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"aws-credential-types",
"aws-runtime",
"aws-smithy-async",
"aws-smithy-http",
"aws-smithy-json",
"aws-smithy-observability",
"aws-smithy-query",
"aws-smithy-runtime",
"aws-smithy-runtime-api",
"aws-smithy-types",
"aws-smithy-xml",
"aws-types",
"fastrand",
"http 0.2.12",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"regex-lite",
"tracing",
]
[[package]]
name = "aws-sigv4"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.4.5"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "bae38512beae0ffee7010fc24e7a8a123c53efdfef42a61e80fda4882418dc71"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
dependencies = [
"aws-credential-types",
"aws-smithy-http",
"aws-smithy-runtime-api",
"aws-smithy-types",
"bytes",
"form_urlencoded",
"hex",
"hmac",
"http 0.2.12",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"percent-encoding",
"sha2 0.11.0",
"time",
"tracing",
]
[[package]]
name = "aws-smithy-async"
version = "1.2.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2ffcaf626bdda484571968400c326a244598634dc75fd451325a54ad1a59acfc"
dependencies = [
"futures-util",
"pin-project-lite",
"tokio",
]
[[package]]
name = "aws-smithy-http"
version = "0.63.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ba1ab2dc1c2c3749ead27180d333c42f11be8b0e934058fb4b2258ee8dbe5231"
dependencies = [
"aws-smithy-runtime-api",
"aws-smithy-types",
"bytes",
"bytes-utils",
"futures-core",
"futures-util",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"http-body 1.0.1",
"http-body-util",
"percent-encoding",
"pin-project-lite",
"pin-utils",
"tracing",
]
[[package]]
name = "aws-smithy-http-client"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.1.13"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "5c3ef8931ad1c98aa6a55b4256f847f3116090819844e0dd41ea682cac5dd2d3"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
dependencies = [
"aws-smithy-async",
"aws-smithy-runtime-api",
"aws-smithy-types",
"h2",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"hyper",
"hyper-rustls",
"hyper-util",
"pin-project-lite",
"rustls",
"rustls-native-certs",
"rustls-pki-types",
"tokio",
"tokio-rustls",
"tower",
"tracing",
]
[[package]]
name = "aws-smithy-json"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.62.7"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "701a947f4797e52a911e114a898667c746c39feea467bbd1abd7b3721f702ffa"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
dependencies = [
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"aws-smithy-runtime-api",
"aws-smithy-schema",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"aws-smithy-types",
]
[[package]]
name = "aws-smithy-observability"
version = "0.2.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a06c2315d173edbf1920da8ba3a7189695827002e4c0fc961973ab1c54abca9c"
dependencies = [
"aws-smithy-runtime-api",
]
[[package]]
name = "aws-smithy-query"
version = "0.60.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1a56d79744fb3edb5d722ef79d86081e121d3b9422cb209eb03aea6aa4f21ebd"
dependencies = [
"aws-smithy-types",
"urlencoding",
]
[[package]]
name = "aws-smithy-runtime"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.11.3"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "b8e6f5caf6fea86f8c2206541ab5857cfcda9013426cdbe8fa0098b9e2d32182"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
dependencies = [
"aws-smithy-async",
"aws-smithy-http",
"aws-smithy-http-client",
"aws-smithy-observability",
"aws-smithy-runtime-api",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"aws-smithy-schema",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"aws-smithy-types",
"bytes",
"fastrand",
"http 0.2.12",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"http-body 0.4.6",
"http-body 1.0.1",
"http-body-util",
"pin-project-lite",
"pin-utils",
"tokio",
"tracing",
]
[[package]]
name = "aws-smithy-runtime-api"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.12.3"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "9db177daa6ba8afb9ee1aefcf548c907abcf52065e394ee11a92780057fe0e8c"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
dependencies = [
"aws-smithy-async",
"aws-smithy-runtime-api-macros",
"aws-smithy-types",
"bytes",
"http 0.2.12",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"pin-project-lite",
"tokio",
"tracing",
"zeroize",
]
[[package]]
name = "aws-smithy-runtime-api-macros"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8d7396fd9500589e62e460e987ecb671bad374934e55ec3b5f498cc7a8a8a7b7"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
[[package]]
name = "aws-smithy-schema"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7442cb268338f0eb8278140a107c046756aa01093d8ef5e99628d34ae09c94f5"
dependencies = [
"aws-smithy-runtime-api",
"aws-smithy-types",
"http 1.4.2",
]
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
[[package]]
name = "aws-smithy-types"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.5.0"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "32b42fcf341259d85ca10fac9a2f6448a8ec691c6955a18e45bc3b71a85fab85"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
dependencies = [
"base64-simd",
"bytes",
"bytes-utils",
"http 0.2.12",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"http-body 0.4.6",
"http-body 1.0.1",
"http-body-util",
"itoa",
"num-integer",
"pin-project-lite",
"pin-utils",
"ryu",
"serde",
"time",
]
[[package]]
name = "aws-smithy-xml"
version = "0.60.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0ce02add1aa3677d022f8adf81dcbe3046a95f17a1b1e8979c145cd21d3d22b3"
dependencies = [
"xmlparser",
]
[[package]]
name = "aws-types"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.3.16"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "d16bf10b03a3c01e6b3b7d47cd964e873ffe9e7d4e80fad16bd4c077cb068531"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
dependencies = [
"aws-credential-types",
"aws-smithy-async",
"aws-smithy-runtime-api",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"aws-smithy-schema",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"aws-smithy-types",
"rustc_version",
"tracing",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "axum"
version = "0.7.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "edca88bc138befd0323b20752846e6587272d3b03b0343c8ea28a6f819e6e71f"
dependencies = [
"async-trait",
"axum-core",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"axum-macros",
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
"base64 0.22.1",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"bytes",
"futures-util",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"http-body 1.0.1",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"http-body-util",
"hyper",
"hyper-util",
"itoa",
"matchit",
"memchr",
"mime",
"percent-encoding",
"pin-project-lite",
"rustversion",
"serde",
"serde_json",
"serde_path_to_error",
"serde_urlencoded",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"sha1",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"sync_wrapper",
"tokio",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"tokio-tungstenite",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"tower",
"tower-layer",
"tower-service",
"tracing",
]
[[package]]
name = "axum-core"
version = "0.4.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09f2bd6146b97ae3359fa0cc6d6b376d9539582c7b4220f041a33ec24c226199"
dependencies = [
"async-trait",
"bytes",
"futures-util",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"http-body 1.0.1",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"http-body-util",
"mime",
"pin-project-lite",
"rustversion",
"sync_wrapper",
"tower-layer",
"tower-service",
"tracing",
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "axum-macros"
version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "57d123550fa8d071b7255cb0cc04dc302baa6c8c4a79f55701552684d8399bce"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "base64"
version = "0.13.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9e1b586273c5702936fe7b7d6896644d8be71e6314cfe09d3167c95f712589e8"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "base64"
version = "0.22.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
[[package]]
name = "base64-simd"
version = "0.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "339abbe78e73178762e23bea9dfd08e697eb3f3301cd4be981c0f78ba5859195"
dependencies = [
"outref",
"vsimd",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "bit-set"
version = "0.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "08807e080ed7f9d5433fa9b275196cfc35414f66a0c79d864dc51a0d825231a3"
dependencies = [
"bit-vec",
]
[[package]]
name = "bit-vec"
version = "0.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5e764a1d40d510daf35e07be9eb06e75770908c27d411ee6c92109c9840eaaf7"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "bit_field"
version = "0.10.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e4b40c7323adcfc0a41c4b88143ed58346ff65a288fc144329c5c45e05d70c6"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "bitflags"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "2.13.0"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "bitstream-io"
version = "4.10.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7eff00be299a18769011411c9def0d827e8f2d7bf0c3dbf53633147a8867fd1f"
dependencies = [
"no_std_io2",
]
fix: B7 — CCR hardening: persistent backends + always-on tool P2-25, P2-26: CCR (Compress-Cache-Retrieve) used an in-memory store that fragmented across uvicorn workers and was wiped on restart, and the `headroom_retrieve` tool was registered/unregistered per-request based on whether the latest body happened to contain compression markers — every flip busted the prompt cache. Both are sticky side-channels: once a session has done CCR, the tool list bytes and the retrieval store must stay stable. This PR fixes both. Rust: * Split `ccr.rs` into `ccr/` with `backends/` submodule (`in_memory.rs`, `sqlite.rs`, `redis.rs` cfg-gated). * `SqliteCcrStore` (production default): WAL mode, prepared upsert, lazy TTL purge on read, persistent across worker restarts and shareable across workers on the same host via SQLite file locking. * `RedisCcrStore` (cfg-gated behind `feature = "redis"`): SETEX with startup PING smoke-test, no key-prefix collision risk, no sticky session required at the LB. * `CcrBackendConfig::{InMemory, Sqlite, Redis}` + `from_config(...)` factory — every init failure surfaces (no silent fallback per `feedback_no_silent_fallbacks.md`). * `ccr::compute_key` (BLAKE3 → first 24 hex chars) and `ccr::marker_for("HASH") -> "<<ccr:HASH>>"` centralize the hash + marker format; one definition for the live-zone dispatcher and the Python regex (`headroom/ccr/tool_injection.py:211`). * `compress_anthropic_live_zone_with_ccr` accepts `Option<&dyn CcrStore>`. When wired, every accepted compression puts the original bytes into the backend and appends `<<ccr:HASH>>` to the compressed string. The token-validation gate runs on the marker-augmented string so the `compressed_tokens >= original_tokens` rejection stays honest. Python: * `SessionCcrTracker` + `apply_session_sticky_ccr_tool` mirror the PR-A7 `SessionToolTracker` / `apply_session_sticky_memory_tools` pattern: once a session has done CCR, every subsequent request injects the recorded golden tool-definition bytes. Tool list bytes are byte-stable across turns (snapshot test pins them). * `headroom/ccr/tool_injection.py::inject_tool_definition` accepts a new `session_has_done_ccr` kwarg per the PR-B7 spec change at line 302-328. The legacy per-request path stays intact for callers that don't yet thread a session id (e.g. Google handler). * Anthropic + OpenAI handlers route their CCR tool-list updates through `apply_session_sticky_ccr_tool`, keyed off the existing `session_tracker_store.compute_session_id(...)` plumbing. Backend selection model: `CcrBackendConfig::Sqlite { path }` is the production default — single host, persistent, multi-worker safe with sticky session. `CcrBackendConfig::Redis { url }` is the multi-host scale-out option — no stickiness needed. `InMemory` is for tests and single-worker dev only. RUST_DEV.md "Multi-worker deployment — CCR fragmentation" rewritten around this matrix. Tests: * `crates/headroom-core/tests/ccr_backends.rs` — 7 tests covering SQLite round-trip, TTL purge, proxy-restart survival, cross-backend byte-equal keys, `from_config` paths, and the no-redis-feature loud-failure check (+ 2 redis tests gated behind the feature). * `crates/headroom-core/tests/live_zone_ccr.rs` — confirms `<<ccr:HASH>>` marker injection, store population, and no-marker-when-no-store invariants end-to-end. * `tests/test_ccr_tool_always_on.py` — 12 tests pinning the always-on behaviour, session/provider isolation, LRU bound, no- session-id fallback, and (per-acceptance-criterion) the byte-stable tool-definition snapshot. Per-PR-B7 plan: REALIGNMENT/04-phase-B-live-zone.md.
2026-05-02 16:49:08 -07:00
[[package]]
name = "blake3"
version = "1.8.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0aa83c34e62843d924f905e0f5c866eb1dd6545fc4d719e803d9ba6030371fce"
dependencies = [
"arrayref",
"arrayvec",
"cc",
"cfg-if",
"constant_time_eq",
"cpufeatures 0.3.0",
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "block-buffer"
version = "0.10.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71"
dependencies = [
"generic-array",
]
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
[[package]]
name = "block-buffer"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.12.1"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "d2f6c7dbe95a6ed67ad9f18e57daf93a2f034c524b99fd2b76d18fdfeb6660aa"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
dependencies = [
"hybrid-array",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "bstr"
version = "1.12.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "63044e1ae8e69f3b5a92c736ca6269b8d12fa7efe39bf34ddb06d102cf0e2cab"
dependencies = [
"memchr",
"regex-automata",
"serde",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "built"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.8.1"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "5c0e531d93d39c34eef561e929e8a7f86d77a5af08aac4f6d6e39976c51858e9"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "bumpalo"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "3.20.3"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "bytemuck"
version = "1.25.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c8efb64bd706a16a1bdde310ae86b351e4d21550d98d056f22f8a7f7a2183fec"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "byteorder"
version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "byteorder-lite"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8f1fe948ff07f4bd06c30984e69f5b4899c516a3ef74f34df92a2df2ab535495"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "bytes"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.12.0"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "8ae3f5d315924270530207e2a68396c3cc547f6dca3fbdca317cfb1a51edb593"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
[[package]]
name = "bytes-utils"
version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7dafe3a8757b027e2be6e4e5601ed563c55989fcf1546e933c66c8eb3a058d35"
dependencies = [
"bytes",
"either",
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "bytesize"
version = "1.3.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2e93abca9e28e0a1b9877922aacb20576e05d4679ffa78c3d6dc22a26a216659"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "cast"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "37b2a672a2cb129a2e41c10b1224bb368f9f37a2b16b612598138befd7b37eb5"
[[package]]
name = "castaway"
version = "0.2.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dec551ab6e7578819132c713a93c022a05d60159dc86e7a7050223577484c55a"
dependencies = [
"rustversion",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "cc"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.2.65"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "e228eec9be7c17ccb640b59b36a5cd805ea2a564a4c5e162c2f659fea30d3b96"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"find-msvc-tools",
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
"jobserver",
"libc",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"shlex",
]
[[package]]
name = "cfg-if"
version = "1.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
[[package]]
name = "cfg_aliases"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724"
fix(proxy): PR-D4 native Vertex publisher path + ADC bearer auth Adds a Rust-native Vertex AI publisher route ahead of the LiteLLM Python converter (which dropped `thinking`, `redacted_thinking`, `document`, `image`, `server_tool_use`, `mcp_tool_use` block kinds — the P4-37 / P4-38 bug). After this PR the Vertex `:rawPredict` and `:streamRawPredict` calls survive byte-equal upstream and benefit from the live-zone Anthropic dispatcher (PR-B-series) running over the body — same behaviour as `/v1/messages`. New module `crates/headroom-proxy/src/vertex/`: - `mod.rs` — single dispatch handler at the `/v1beta1/.../models/:model_action` route. Splits the trailing `:<verb>` segment with `str::rsplit_once(':')` (no regex) and flips an `attach_sse_tee` flag to dispatch to the streaming or non-streaming arm. Both verbs share one axum route shape because matchit can't distinguish two patterns that overlap on a parameter. - `envelope.rs` — `VertexEnvelope` parser. Confirms `anthropic_version` present + `model` field absent (the two fingerprints of the Vertex envelope vs `/v1/messages`). - `adc.rs` — `TokenSource` trait + `GcpAdcTokenSource` (production, `gcp_auth` 0.12) + `StaticTokenSource` (tests). Caches tokens with a 60s refresh-ahead-of-expiry window. Emits structured `event = "vertex_adc_token_refreshed"` per refresh. - `raw_predict.rs` — POST handler + shared `forward_vertex_request`. Buffers body, parses envelope, runs live-zone Anthropic compression, fetches ADC bearer, attaches `Authorization: Bearer <token>` (overwrites client-supplied Authorization header), forwards. SSE telemetry tee for the streaming verb reuses PR-C1's `AnthropicStreamState` directly (Vertex streams plain SSE, unlike Bedrock's binary EventStream). - `stream_raw_predict.rs` — module-level docs + alias to the shared dispatcher (the streaming-vs-non-streaming difference is one boolean flag inside the shared forwarder). Modifications: - `proxy.rs::build_app` — registers the single Vertex route. - `proxy.rs::AppState` — new `vertex_token_source: Arc<dyn TokenSource>` field. Production constructs `GcpAdcTokenSource` lazily (no GCP call until first `bearer()`); tests inject `StaticTokenSource` via the new `AppState::with_token_source` helper. - `config.rs` — adds `--vertex-region` / `HEADROOM_PROXY_VERTEX_REGION` (default `us-central1`, observability tag only — the upstream URL is `--upstream`) and `--vertex-adc-scope` / `HEADROOM_PROXY_VERTEX_ADC_SCOPE` (default `cloud-platform`). - `Cargo.toml` (workspace + proxy) — adds `gcp_auth = "0.12"` and `async-trait = "0.1"`. - `tests/common/mod.rs` — `start_proxy_with_state` accepts both config + state customizers; `install_static_token_source` helper for tests. `crates/headroom-proxy/tests/integration_vertex_raw_predict.rs` — all five tests pass: 1. `native_envelope_round_trip_byte_equal` — Vertex-shape body (with `anthropic_version`, no `model`) round-trips SHA-256 byte-equal upstream. 2. `adc_bearer_token_signed_correctly` — `Authorization: Bearer <static-test-token>` reaches upstream verbatim and OVERWRITES a client-supplied Authorization header. 3. `thinking_block_preserved` — request with `thinking` (incl. signature) + `redacted_thinking` (incl. opaque `data`) blocks round-trips byte-equal even with `LiveZone` compression mode enabled. This is the P4-37 / P4-38 teeth. 4. `stream_raw_predict_sse_handled` — `:streamRawPredict` proxies an Anthropic SSE response (full `message_start` → `content_block_delta` → `message_stop` sequence) back to the client without corruption; SSE content-type preserved end-to-end; bearer attached. 5. (bonus, no-silent-fallback contract) `adc_failure_returns_5xx_no_silent_forward` — when the token source returns `Err`, the proxy returns 5xx and never reaches upstream. Verifies the `event = "vertex_adc_fetch_failed"` error path. Workspace: `cargo test --workspace` green; `cargo clippy --workspace -- -D warnings` clean; `make ci-precheck` passes. - No silent fallbacks: ADC failure → structured 5xx, never an unauthenticated forward. - No hardcodes: every knob (region, ADC scope, upstream URL) is CLI-flag + env-var configurable. - No regexes: axum path parameters + `str::rsplit_once` only. - Comprehensive structured logs: `event` field on every decision point — `vertex_envelope_parsed`, `vertex_envelope_invalid`, `vertex_compression_skipped`, `vertex_compression_applied`, `vertex_adc_token_refreshed`, `vertex_adc_fetch_failed`, `vertex_streaming_pipeline_active`, `vertex_sse_stream_closed`, `vertex_forwarded`, `vertex_unknown_verb`, etc. - Performant: no body clone; ADC token cached + refreshed ahead-of-expiry, not fetched per request. - Comprehensive tests: realistic Anthropic block content (signature payload, redacted_thinking opaque blob) in `thinking_block_preserved`. The local `gcloud auth application-default print-access-token` returns no credentials, so manual validation against a real Vertex endpoint is not possible in this PR. Follow-up: the user runs `gcloud auth application-default login` once and exercises a live Vertex request — should be a no-code-change check. PR-D1 (Bedrock native) is running concurrently and will land its own envelope module at `crates/headroom-proxy/src/bedrock/envelope.rs`. The two envelope modules are intentionally siblings (not a shared trait) — the shapes differ (Bedrock has a different `anthropic_version` value, no `model` field, AWS SigV4 instead of GCP ADC), and a premature shared abstraction would obscure the provider-specific contracts. Whichever PR merges second rebases without conflict. Retires P4-38 (and the Vertex parts of P4-39); marketplace BYOC pitch (per project memory) gets one more native provider.
2026-05-03 16:22:07 -07:00
[[package]]
name = "chrono"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.4.45"
fix(proxy): PR-D4 native Vertex publisher path + ADC bearer auth Adds a Rust-native Vertex AI publisher route ahead of the LiteLLM Python converter (which dropped `thinking`, `redacted_thinking`, `document`, `image`, `server_tool_use`, `mcp_tool_use` block kinds — the P4-37 / P4-38 bug). After this PR the Vertex `:rawPredict` and `:streamRawPredict` calls survive byte-equal upstream and benefit from the live-zone Anthropic dispatcher (PR-B-series) running over the body — same behaviour as `/v1/messages`. New module `crates/headroom-proxy/src/vertex/`: - `mod.rs` — single dispatch handler at the `/v1beta1/.../models/:model_action` route. Splits the trailing `:<verb>` segment with `str::rsplit_once(':')` (no regex) and flips an `attach_sse_tee` flag to dispatch to the streaming or non-streaming arm. Both verbs share one axum route shape because matchit can't distinguish two patterns that overlap on a parameter. - `envelope.rs` — `VertexEnvelope` parser. Confirms `anthropic_version` present + `model` field absent (the two fingerprints of the Vertex envelope vs `/v1/messages`). - `adc.rs` — `TokenSource` trait + `GcpAdcTokenSource` (production, `gcp_auth` 0.12) + `StaticTokenSource` (tests). Caches tokens with a 60s refresh-ahead-of-expiry window. Emits structured `event = "vertex_adc_token_refreshed"` per refresh. - `raw_predict.rs` — POST handler + shared `forward_vertex_request`. Buffers body, parses envelope, runs live-zone Anthropic compression, fetches ADC bearer, attaches `Authorization: Bearer <token>` (overwrites client-supplied Authorization header), forwards. SSE telemetry tee for the streaming verb reuses PR-C1's `AnthropicStreamState` directly (Vertex streams plain SSE, unlike Bedrock's binary EventStream). - `stream_raw_predict.rs` — module-level docs + alias to the shared dispatcher (the streaming-vs-non-streaming difference is one boolean flag inside the shared forwarder). Modifications: - `proxy.rs::build_app` — registers the single Vertex route. - `proxy.rs::AppState` — new `vertex_token_source: Arc<dyn TokenSource>` field. Production constructs `GcpAdcTokenSource` lazily (no GCP call until first `bearer()`); tests inject `StaticTokenSource` via the new `AppState::with_token_source` helper. - `config.rs` — adds `--vertex-region` / `HEADROOM_PROXY_VERTEX_REGION` (default `us-central1`, observability tag only — the upstream URL is `--upstream`) and `--vertex-adc-scope` / `HEADROOM_PROXY_VERTEX_ADC_SCOPE` (default `cloud-platform`). - `Cargo.toml` (workspace + proxy) — adds `gcp_auth = "0.12"` and `async-trait = "0.1"`. - `tests/common/mod.rs` — `start_proxy_with_state` accepts both config + state customizers; `install_static_token_source` helper for tests. `crates/headroom-proxy/tests/integration_vertex_raw_predict.rs` — all five tests pass: 1. `native_envelope_round_trip_byte_equal` — Vertex-shape body (with `anthropic_version`, no `model`) round-trips SHA-256 byte-equal upstream. 2. `adc_bearer_token_signed_correctly` — `Authorization: Bearer <static-test-token>` reaches upstream verbatim and OVERWRITES a client-supplied Authorization header. 3. `thinking_block_preserved` — request with `thinking` (incl. signature) + `redacted_thinking` (incl. opaque `data`) blocks round-trips byte-equal even with `LiveZone` compression mode enabled. This is the P4-37 / P4-38 teeth. 4. `stream_raw_predict_sse_handled` — `:streamRawPredict` proxies an Anthropic SSE response (full `message_start` → `content_block_delta` → `message_stop` sequence) back to the client without corruption; SSE content-type preserved end-to-end; bearer attached. 5. (bonus, no-silent-fallback contract) `adc_failure_returns_5xx_no_silent_forward` — when the token source returns `Err`, the proxy returns 5xx and never reaches upstream. Verifies the `event = "vertex_adc_fetch_failed"` error path. Workspace: `cargo test --workspace` green; `cargo clippy --workspace -- -D warnings` clean; `make ci-precheck` passes. - No silent fallbacks: ADC failure → structured 5xx, never an unauthenticated forward. - No hardcodes: every knob (region, ADC scope, upstream URL) is CLI-flag + env-var configurable. - No regexes: axum path parameters + `str::rsplit_once` only. - Comprehensive structured logs: `event` field on every decision point — `vertex_envelope_parsed`, `vertex_envelope_invalid`, `vertex_compression_skipped`, `vertex_compression_applied`, `vertex_adc_token_refreshed`, `vertex_adc_fetch_failed`, `vertex_streaming_pipeline_active`, `vertex_sse_stream_closed`, `vertex_forwarded`, `vertex_unknown_verb`, etc. - Performant: no body clone; ADC token cached + refreshed ahead-of-expiry, not fetched per request. - Comprehensive tests: realistic Anthropic block content (signature payload, redacted_thinking opaque blob) in `thinking_block_preserved`. The local `gcloud auth application-default print-access-token` returns no credentials, so manual validation against a real Vertex endpoint is not possible in this PR. Follow-up: the user runs `gcloud auth application-default login` once and exercises a live Vertex request — should be a no-code-change check. PR-D1 (Bedrock native) is running concurrently and will land its own envelope module at `crates/headroom-proxy/src/bedrock/envelope.rs`. The two envelope modules are intentionally siblings (not a shared trait) — the shapes differ (Bedrock has a different `anthropic_version` value, no `model` field, AWS SigV4 instead of GCP ADC), and a premature shared abstraction would obscure the provider-specific contracts. Whichever PR merges second rebases without conflict. Retires P4-38 (and the Vertex parts of P4-39); marketplace BYOC pitch (per project memory) gets one more native provider.
2026-05-03 16:22:07 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "1aa79e62e7697b8e29b513a68abacf485adcd1fe8284a4316c5ae868e6633327"
fix(proxy): PR-D4 native Vertex publisher path + ADC bearer auth Adds a Rust-native Vertex AI publisher route ahead of the LiteLLM Python converter (which dropped `thinking`, `redacted_thinking`, `document`, `image`, `server_tool_use`, `mcp_tool_use` block kinds — the P4-37 / P4-38 bug). After this PR the Vertex `:rawPredict` and `:streamRawPredict` calls survive byte-equal upstream and benefit from the live-zone Anthropic dispatcher (PR-B-series) running over the body — same behaviour as `/v1/messages`. New module `crates/headroom-proxy/src/vertex/`: - `mod.rs` — single dispatch handler at the `/v1beta1/.../models/:model_action` route. Splits the trailing `:<verb>` segment with `str::rsplit_once(':')` (no regex) and flips an `attach_sse_tee` flag to dispatch to the streaming or non-streaming arm. Both verbs share one axum route shape because matchit can't distinguish two patterns that overlap on a parameter. - `envelope.rs` — `VertexEnvelope` parser. Confirms `anthropic_version` present + `model` field absent (the two fingerprints of the Vertex envelope vs `/v1/messages`). - `adc.rs` — `TokenSource` trait + `GcpAdcTokenSource` (production, `gcp_auth` 0.12) + `StaticTokenSource` (tests). Caches tokens with a 60s refresh-ahead-of-expiry window. Emits structured `event = "vertex_adc_token_refreshed"` per refresh. - `raw_predict.rs` — POST handler + shared `forward_vertex_request`. Buffers body, parses envelope, runs live-zone Anthropic compression, fetches ADC bearer, attaches `Authorization: Bearer <token>` (overwrites client-supplied Authorization header), forwards. SSE telemetry tee for the streaming verb reuses PR-C1's `AnthropicStreamState` directly (Vertex streams plain SSE, unlike Bedrock's binary EventStream). - `stream_raw_predict.rs` — module-level docs + alias to the shared dispatcher (the streaming-vs-non-streaming difference is one boolean flag inside the shared forwarder). Modifications: - `proxy.rs::build_app` — registers the single Vertex route. - `proxy.rs::AppState` — new `vertex_token_source: Arc<dyn TokenSource>` field. Production constructs `GcpAdcTokenSource` lazily (no GCP call until first `bearer()`); tests inject `StaticTokenSource` via the new `AppState::with_token_source` helper. - `config.rs` — adds `--vertex-region` / `HEADROOM_PROXY_VERTEX_REGION` (default `us-central1`, observability tag only — the upstream URL is `--upstream`) and `--vertex-adc-scope` / `HEADROOM_PROXY_VERTEX_ADC_SCOPE` (default `cloud-platform`). - `Cargo.toml` (workspace + proxy) — adds `gcp_auth = "0.12"` and `async-trait = "0.1"`. - `tests/common/mod.rs` — `start_proxy_with_state` accepts both config + state customizers; `install_static_token_source` helper for tests. `crates/headroom-proxy/tests/integration_vertex_raw_predict.rs` — all five tests pass: 1. `native_envelope_round_trip_byte_equal` — Vertex-shape body (with `anthropic_version`, no `model`) round-trips SHA-256 byte-equal upstream. 2. `adc_bearer_token_signed_correctly` — `Authorization: Bearer <static-test-token>` reaches upstream verbatim and OVERWRITES a client-supplied Authorization header. 3. `thinking_block_preserved` — request with `thinking` (incl. signature) + `redacted_thinking` (incl. opaque `data`) blocks round-trips byte-equal even with `LiveZone` compression mode enabled. This is the P4-37 / P4-38 teeth. 4. `stream_raw_predict_sse_handled` — `:streamRawPredict` proxies an Anthropic SSE response (full `message_start` → `content_block_delta` → `message_stop` sequence) back to the client without corruption; SSE content-type preserved end-to-end; bearer attached. 5. (bonus, no-silent-fallback contract) `adc_failure_returns_5xx_no_silent_forward` — when the token source returns `Err`, the proxy returns 5xx and never reaches upstream. Verifies the `event = "vertex_adc_fetch_failed"` error path. Workspace: `cargo test --workspace` green; `cargo clippy --workspace -- -D warnings` clean; `make ci-precheck` passes. - No silent fallbacks: ADC failure → structured 5xx, never an unauthenticated forward. - No hardcodes: every knob (region, ADC scope, upstream URL) is CLI-flag + env-var configurable. - No regexes: axum path parameters + `str::rsplit_once` only. - Comprehensive structured logs: `event` field on every decision point — `vertex_envelope_parsed`, `vertex_envelope_invalid`, `vertex_compression_skipped`, `vertex_compression_applied`, `vertex_adc_token_refreshed`, `vertex_adc_fetch_failed`, `vertex_streaming_pipeline_active`, `vertex_sse_stream_closed`, `vertex_forwarded`, `vertex_unknown_verb`, etc. - Performant: no body clone; ADC token cached + refreshed ahead-of-expiry, not fetched per request. - Comprehensive tests: realistic Anthropic block content (signature payload, redacted_thinking opaque blob) in `thinking_block_preserved`. The local `gcloud auth application-default print-access-token` returns no credentials, so manual validation against a real Vertex endpoint is not possible in this PR. Follow-up: the user runs `gcloud auth application-default login` once and exercises a live Vertex request — should be a no-code-change check. PR-D1 (Bedrock native) is running concurrently and will land its own envelope module at `crates/headroom-proxy/src/bedrock/envelope.rs`. The two envelope modules are intentionally siblings (not a shared trait) — the shapes differ (Bedrock has a different `anthropic_version` value, no `model` field, AWS SigV4 instead of GCP ADC), and a premature shared abstraction would obscure the provider-specific contracts. Whichever PR merges second rebases without conflict. Retires P4-38 (and the Vertex parts of P4-39); marketplace BYOC pitch (per project memory) gets one more native provider.
2026-05-03 16:22:07 -07:00
dependencies = [
"iana-time-zone",
"js-sys",
"num-traits",
"serde",
"wasm-bindgen",
"windows-link",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "ciborium"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "42e69ffd6f0917f5c029256a24d0161db17cea3997d185db0d35926308770f0e"
dependencies = [
"ciborium-io",
"ciborium-ll",
"serde",
]
[[package]]
name = "ciborium-io"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "05afea1e0a06c9be33d539b876f1ce3692f4afea2cb41f740e7743225ed1c757"
[[package]]
name = "ciborium-ll"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "57663b653d948a338bfb3eeba9bb2fd5fcfaecb9e199e87e1eda4d9e8b240fd9"
dependencies = [
"ciborium-io",
"half",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "clap"
version = "4.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1ddb117e43bbf7dacf0a4190fef4d345b9bad68dfc649cb349e7d17d28428e51"
dependencies = [
"clap_builder",
"clap_derive",
]
[[package]]
name = "clap_builder"
version = "4.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "714a53001bf66416adb0e2ef5ac857140e7dc3a0c48fb28b2f10762fc4b5069f"
dependencies = [
"anstream",
"anstyle",
"clap_lex",
"strsim",
]
[[package]]
name = "clap_derive"
version = "4.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f2ce8604710f6733aa641a2b3731eaa1e8b3d9973d5e3565da11800813f997a9"
dependencies = [
"heck",
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "clap_lex"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
[[package]]
name = "cmake"
version = "0.1.58"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c0f78a02292a74a88ac736019ab962ece0bc380e3f977bf72e376c5d78ff0678"
dependencies = [
"cc",
]
[[package]]
name = "cmov"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.5.4"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "0c9ea0ac24bc397ab3c98583a3c9ba74fa56b09a4449bbe172b9b1ddb016027a"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "color_quant"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3d7b894f5411737b7867f4827955924d7c254fc9f4d91a6aad6b097804b1018b"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "colorchoice"
version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570"
fix: B7 — CCR hardening: persistent backends + always-on tool P2-25, P2-26: CCR (Compress-Cache-Retrieve) used an in-memory store that fragmented across uvicorn workers and was wiped on restart, and the `headroom_retrieve` tool was registered/unregistered per-request based on whether the latest body happened to contain compression markers — every flip busted the prompt cache. Both are sticky side-channels: once a session has done CCR, the tool list bytes and the retrieval store must stay stable. This PR fixes both. Rust: * Split `ccr.rs` into `ccr/` with `backends/` submodule (`in_memory.rs`, `sqlite.rs`, `redis.rs` cfg-gated). * `SqliteCcrStore` (production default): WAL mode, prepared upsert, lazy TTL purge on read, persistent across worker restarts and shareable across workers on the same host via SQLite file locking. * `RedisCcrStore` (cfg-gated behind `feature = "redis"`): SETEX with startup PING smoke-test, no key-prefix collision risk, no sticky session required at the LB. * `CcrBackendConfig::{InMemory, Sqlite, Redis}` + `from_config(...)` factory — every init failure surfaces (no silent fallback per `feedback_no_silent_fallbacks.md`). * `ccr::compute_key` (BLAKE3 → first 24 hex chars) and `ccr::marker_for("HASH") -> "<<ccr:HASH>>"` centralize the hash + marker format; one definition for the live-zone dispatcher and the Python regex (`headroom/ccr/tool_injection.py:211`). * `compress_anthropic_live_zone_with_ccr` accepts `Option<&dyn CcrStore>`. When wired, every accepted compression puts the original bytes into the backend and appends `<<ccr:HASH>>` to the compressed string. The token-validation gate runs on the marker-augmented string so the `compressed_tokens >= original_tokens` rejection stays honest. Python: * `SessionCcrTracker` + `apply_session_sticky_ccr_tool` mirror the PR-A7 `SessionToolTracker` / `apply_session_sticky_memory_tools` pattern: once a session has done CCR, every subsequent request injects the recorded golden tool-definition bytes. Tool list bytes are byte-stable across turns (snapshot test pins them). * `headroom/ccr/tool_injection.py::inject_tool_definition` accepts a new `session_has_done_ccr` kwarg per the PR-B7 spec change at line 302-328. The legacy per-request path stays intact for callers that don't yet thread a session id (e.g. Google handler). * Anthropic + OpenAI handlers route their CCR tool-list updates through `apply_session_sticky_ccr_tool`, keyed off the existing `session_tracker_store.compute_session_id(...)` plumbing. Backend selection model: `CcrBackendConfig::Sqlite { path }` is the production default — single host, persistent, multi-worker safe with sticky session. `CcrBackendConfig::Redis { url }` is the multi-host scale-out option — no stickiness needed. `InMemory` is for tests and single-worker dev only. RUST_DEV.md "Multi-worker deployment — CCR fragmentation" rewritten around this matrix. Tests: * `crates/headroom-core/tests/ccr_backends.rs` — 7 tests covering SQLite round-trip, TTL purge, proxy-restart survival, cross-backend byte-equal keys, `from_config` paths, and the no-redis-feature loud-failure check (+ 2 redis tests gated behind the feature). * `crates/headroom-core/tests/live_zone_ccr.rs` — confirms `<<ccr:HASH>>` marker injection, store population, and no-marker-when-no-store invariants end-to-end. * `tests/test_ccr_tool_always_on.py` — 12 tests pinning the always-on behaviour, session/provider isolation, LRU bound, no- session-id fallback, and (per-acceptance-criterion) the byte-stable tool-definition snapshot. Per-PR-B7 plan: REALIGNMENT/04-phase-B-live-zone.md.
2026-05-02 16:49:08 -07:00
[[package]]
name = "combine"
version = "4.6.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ba5a308b75df32fe02788e748662718f03fde005016435c444eea572398219fd"
dependencies = [
"bytes",
"memchr",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "compact_str"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.9.1"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "9dfdd1c2274d9aa354115b09dc9a901d6c5576818cdf70d14cae2bdb47df00ab"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
dependencies = [
"castaway",
"cfg-if",
"itoa",
"rustversion",
"ryu",
"serde",
"static_assertions",
]
[[package]]
name = "console"
version = "0.15.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "054ccb5b10f9f2cbf51eb355ca1d05c2d279ce1804688d0db74b4733a5aeafd8"
dependencies = [
"encode_unicode",
"libc",
"once_cell",
"unicode-width",
"windows-sys 0.59.0",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "console"
version = "0.16.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d64e8af5551369d19cf50138de61f1c42074ab970f74e99be916646777f8fc87"
dependencies = [
"encode_unicode",
"libc",
"unicode-width",
"windows-sys 0.61.2",
]
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
[[package]]
name = "const-oid"
version = "0.10.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c"
fix: B7 — CCR hardening: persistent backends + always-on tool P2-25, P2-26: CCR (Compress-Cache-Retrieve) used an in-memory store that fragmented across uvicorn workers and was wiped on restart, and the `headroom_retrieve` tool was registered/unregistered per-request based on whether the latest body happened to contain compression markers — every flip busted the prompt cache. Both are sticky side-channels: once a session has done CCR, the tool list bytes and the retrieval store must stay stable. This PR fixes both. Rust: * Split `ccr.rs` into `ccr/` with `backends/` submodule (`in_memory.rs`, `sqlite.rs`, `redis.rs` cfg-gated). * `SqliteCcrStore` (production default): WAL mode, prepared upsert, lazy TTL purge on read, persistent across worker restarts and shareable across workers on the same host via SQLite file locking. * `RedisCcrStore` (cfg-gated behind `feature = "redis"`): SETEX with startup PING smoke-test, no key-prefix collision risk, no sticky session required at the LB. * `CcrBackendConfig::{InMemory, Sqlite, Redis}` + `from_config(...)` factory — every init failure surfaces (no silent fallback per `feedback_no_silent_fallbacks.md`). * `ccr::compute_key` (BLAKE3 → first 24 hex chars) and `ccr::marker_for("HASH") -> "<<ccr:HASH>>"` centralize the hash + marker format; one definition for the live-zone dispatcher and the Python regex (`headroom/ccr/tool_injection.py:211`). * `compress_anthropic_live_zone_with_ccr` accepts `Option<&dyn CcrStore>`. When wired, every accepted compression puts the original bytes into the backend and appends `<<ccr:HASH>>` to the compressed string. The token-validation gate runs on the marker-augmented string so the `compressed_tokens >= original_tokens` rejection stays honest. Python: * `SessionCcrTracker` + `apply_session_sticky_ccr_tool` mirror the PR-A7 `SessionToolTracker` / `apply_session_sticky_memory_tools` pattern: once a session has done CCR, every subsequent request injects the recorded golden tool-definition bytes. Tool list bytes are byte-stable across turns (snapshot test pins them). * `headroom/ccr/tool_injection.py::inject_tool_definition` accepts a new `session_has_done_ccr` kwarg per the PR-B7 spec change at line 302-328. The legacy per-request path stays intact for callers that don't yet thread a session id (e.g. Google handler). * Anthropic + OpenAI handlers route their CCR tool-list updates through `apply_session_sticky_ccr_tool`, keyed off the existing `session_tracker_store.compute_session_id(...)` plumbing. Backend selection model: `CcrBackendConfig::Sqlite { path }` is the production default — single host, persistent, multi-worker safe with sticky session. `CcrBackendConfig::Redis { url }` is the multi-host scale-out option — no stickiness needed. `InMemory` is for tests and single-worker dev only. RUST_DEV.md "Multi-worker deployment — CCR fragmentation" rewritten around this matrix. Tests: * `crates/headroom-core/tests/ccr_backends.rs` — 7 tests covering SQLite round-trip, TTL purge, proxy-restart survival, cross-backend byte-equal keys, `from_config` paths, and the no-redis-feature loud-failure check (+ 2 redis tests gated behind the feature). * `crates/headroom-core/tests/live_zone_ccr.rs` — confirms `<<ccr:HASH>>` marker injection, store population, and no-marker-when-no-store invariants end-to-end. * `tests/test_ccr_tool_always_on.py` — 12 tests pinning the always-on behaviour, session/provider isolation, LRU bound, no- session-id fallback, and (per-acceptance-criterion) the byte-stable tool-definition snapshot. Per-PR-B7 plan: REALIGNMENT/04-phase-B-live-zone.md.
2026-05-02 16:49:08 -07:00
[[package]]
name = "constant_time_eq"
version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "cookie"
version = "0.18.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4ddef33a339a91ea89fb53151bd0a4689cfce27055c291dfa69945475d22c747"
dependencies = [
"percent-encoding",
"time",
"version_check",
]
[[package]]
name = "cookie_store"
version = "0.22.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "15b2c103cf610ec6cae3da84a766285b42fd16aad564758459e6ecf128c75206"
dependencies = [
"cookie",
"document-features",
"idna",
"indexmap",
"log",
"serde",
"serde_derive",
"serde_json",
"time",
"url",
]
[[package]]
name = "core-foundation"
version = "0.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b2a6cd9ae233e7f62ba4e9353e81a88df7fc8a5987b8d445b4d90c879bd156f6"
dependencies = [
"core-foundation-sys",
"libc",
]
[[package]]
name = "core-foundation-sys"
version = "0.8.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "cpufeatures"
version = "0.2.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280"
dependencies = [
"libc",
]
fix: B7 — CCR hardening: persistent backends + always-on tool P2-25, P2-26: CCR (Compress-Cache-Retrieve) used an in-memory store that fragmented across uvicorn workers and was wiped on restart, and the `headroom_retrieve` tool was registered/unregistered per-request based on whether the latest body happened to contain compression markers — every flip busted the prompt cache. Both are sticky side-channels: once a session has done CCR, the tool list bytes and the retrieval store must stay stable. This PR fixes both. Rust: * Split `ccr.rs` into `ccr/` with `backends/` submodule (`in_memory.rs`, `sqlite.rs`, `redis.rs` cfg-gated). * `SqliteCcrStore` (production default): WAL mode, prepared upsert, lazy TTL purge on read, persistent across worker restarts and shareable across workers on the same host via SQLite file locking. * `RedisCcrStore` (cfg-gated behind `feature = "redis"`): SETEX with startup PING smoke-test, no key-prefix collision risk, no sticky session required at the LB. * `CcrBackendConfig::{InMemory, Sqlite, Redis}` + `from_config(...)` factory — every init failure surfaces (no silent fallback per `feedback_no_silent_fallbacks.md`). * `ccr::compute_key` (BLAKE3 → first 24 hex chars) and `ccr::marker_for("HASH") -> "<<ccr:HASH>>"` centralize the hash + marker format; one definition for the live-zone dispatcher and the Python regex (`headroom/ccr/tool_injection.py:211`). * `compress_anthropic_live_zone_with_ccr` accepts `Option<&dyn CcrStore>`. When wired, every accepted compression puts the original bytes into the backend and appends `<<ccr:HASH>>` to the compressed string. The token-validation gate runs on the marker-augmented string so the `compressed_tokens >= original_tokens` rejection stays honest. Python: * `SessionCcrTracker` + `apply_session_sticky_ccr_tool` mirror the PR-A7 `SessionToolTracker` / `apply_session_sticky_memory_tools` pattern: once a session has done CCR, every subsequent request injects the recorded golden tool-definition bytes. Tool list bytes are byte-stable across turns (snapshot test pins them). * `headroom/ccr/tool_injection.py::inject_tool_definition` accepts a new `session_has_done_ccr` kwarg per the PR-B7 spec change at line 302-328. The legacy per-request path stays intact for callers that don't yet thread a session id (e.g. Google handler). * Anthropic + OpenAI handlers route their CCR tool-list updates through `apply_session_sticky_ccr_tool`, keyed off the existing `session_tracker_store.compute_session_id(...)` plumbing. Backend selection model: `CcrBackendConfig::Sqlite { path }` is the production default — single host, persistent, multi-worker safe with sticky session. `CcrBackendConfig::Redis { url }` is the multi-host scale-out option — no stickiness needed. `InMemory` is for tests and single-worker dev only. RUST_DEV.md "Multi-worker deployment — CCR fragmentation" rewritten around this matrix. Tests: * `crates/headroom-core/tests/ccr_backends.rs` — 7 tests covering SQLite round-trip, TTL purge, proxy-restart survival, cross-backend byte-equal keys, `from_config` paths, and the no-redis-feature loud-failure check (+ 2 redis tests gated behind the feature). * `crates/headroom-core/tests/live_zone_ccr.rs` — confirms `<<ccr:HASH>>` marker injection, store population, and no-marker-when-no-store invariants end-to-end. * `tests/test_ccr_tool_always_on.py` — 12 tests pinning the always-on behaviour, session/provider isolation, LRU bound, no- session-id fallback, and (per-acceptance-criterion) the byte-stable tool-definition snapshot. Per-PR-B7 plan: REALIGNMENT/04-phase-B-live-zone.md.
2026-05-02 16:49:08 -07:00
[[package]]
name = "cpufeatures"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201"
dependencies = [
"libc",
]
feat(rust): HfTokenizer::from_pretrained — HuggingFace Hub auto-download Stage 2.1: closes the loop on the HuggingFace tokenizer story. Stage 2 shipped `HfTokenizer::from_bytes`/`from_file`, which required callers to manage their own tokenizer.json files. This adds the third constructor: let t = HfTokenizer::from_pretrained("CohereForAI/c4ai-command-r-v01")?; register_hf("command-", t); `from_pretrained` is a thin wrapper around the `hf-hub` crate's blocking `ureq` API. First call downloads `tokenizer.json` to `~/.cache/huggingface/ hub` (or `$HF_HOME` if set); subsequent calls reuse the on-disk cache. Uses the `main` revision; gated repos (Llama, Mistral) require `HF_TOKEN` in env or `~/.cache/huggingface/token`. Also adds `try_register_hf(prefix, repo)` as the obvious one-liner for proxy startup code: let _ = try_register_hf("command-", "CohereForAI/c4ai-command-r-v01"); let _ = try_register_hf("mistral-", "mistralai/Mistral-7B-v0.1"); Each call is independent — a download failure for one model (e.g. gated without a token) does not affect others. `HfTokenizerError` gains a new `Hub` variant so callers can distinguish "couldn't fetch" from "fetched but malformed" — relevant when deciding whether to retry, surface to the user, or fall back to the estimator. Why blocking, not async: `from_pretrained` is called once at startup. A sync API works from `main()`, from a `OnceLock` initializer, or from `tokio::task::spawn_blocking` if a tokio caller needs it later. The async hf-hub backend would force callers to await at startup, which doesn't fit the `register_hf` registry pattern. Why rustls, not native-tls: keeps the binary statically linkable for AWS deploys (no system OpenSSL dependency). Tests: a network-dependent integration test (`#[ignore]`d in CI; hits HF for `gpt2`, ~1.4 MB) verifies the real download + load + count path. A non-network negative test verifies that an invalid repo name surfaces as `HfTokenizerError::Hub`, not a panic. 44 unit tests + 5 proptests + 1 doctest pass; parity stays 40/40 byte-equal.
2026-04-25 15:05:09 -07:00
[[package]]
name = "crc32fast"
version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9481c1c90cbf2ac953f07c8d4a58aa3945c425b7185c9154d67a65e4230da511"
dependencies = [
"cfg-if",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "criterion"
version = "0.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f2b12d017a929603d80db1831cd3a24082f8137ce19c69e6447f54f5fc8d692f"
dependencies = [
"anes",
"cast",
"ciborium",
"clap",
"criterion-plot",
"is-terminal",
"itertools 0.10.5",
"num-traits",
"once_cell",
"oorandom",
"plotters",
"rayon",
"regex",
"serde",
"serde_derive",
"serde_json",
"tinytemplate",
"walkdir",
]
[[package]]
name = "criterion-plot"
version = "0.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6b50826342786a51a89e2da3a28f1c32b06e387201bc2d19791f622c673706b1"
dependencies = [
"cast",
"itertools 0.10.5",
]
[[package]]
name = "crossbeam-deque"
version = "0.8.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9dd111b7b7f7d55b72c0a6ae361660ee5853c9af73f70c3c2ef6858b950e2e51"
dependencies = [
"crossbeam-epoch",
"crossbeam-utils",
]
[[package]]
name = "crossbeam-epoch"
version = "0.9.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5b82ac4a3c2ca9c3460964f020e1402edd5753411d7737aa39c3714ad1b5420e"
dependencies = [
"crossbeam-utils",
]
[[package]]
name = "crossbeam-utils"
version = "0.8.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d0a5c400df2834b80a4c3327b3aad3a4c4cd4de0629063962b03235697506a28"
[[package]]
name = "crunchy"
version = "0.2.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "crypto-common"
version = "0.1.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a"
dependencies = [
"generic-array",
"typenum",
]
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
[[package]]
name = "crypto-common"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.2.2"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "ce6e4c961d6cd6c9a86db418387425e8bdeaf05b3c8bc1411e6dca4c252f1453"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
dependencies = [
"hybrid-array",
]
[[package]]
name = "ctutils"
version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7d5515a3834141de9eafb9717ad39eea8247b5674e6066c404e8c4b365d2a29e"
dependencies = [
"cmov",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "darling"
version = "0.20.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fc7f46116c46ff9ab3eb1597a45688b6715c6e628b5c133e288e709a29bcb4ee"
dependencies = [
"darling_core",
"darling_macro",
]
[[package]]
name = "darling_core"
version = "0.20.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0d00b9596d185e565c2207a0b01f8bd1a135483d02d9b7b0a54b11da8d53412e"
dependencies = [
"fnv",
"ident_case",
"proc-macro2",
"quote",
"strsim",
"syn",
]
[[package]]
name = "darling_macro"
version = "0.20.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fc34b93ccb385b40dc71c6fceac4b2ad23662c7eeb248cf10d529b7e055b6ead"
dependencies = [
"darling_core",
"quote",
"syn",
]
[[package]]
name = "dary_heap"
version = "0.3.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8b1e3a325bc115f096c8b77bbf027a7c2592230e70be2d985be950d3d5e60ebe"
dependencies = [
"serde",
]
perf(rust): tier-1 multi-worker wins — GIL release, sharded CCR store, single-serialize CCR write Three orthogonal hot-path fixes targeting concurrent-request throughput. Each is independently bench-measured below; the proxy hot path benefits from all three at once. == 1. PyO3 GIL release on heavy compute == PyO3 methods (crush, smart_crush_content, crush_array_json, compact_document_json, compress, compress_with_stats) used to hold the GIL across the entire Rust call. Result: a 100ms compress() blocked EVERY other Python thread for 100ms — multi-worker uvicorn deployments serialized through SmartCrusher. Wrap each compute call in `py.allow_threads(|| ...)`. Inputs (`&str` from Python) are copied to owned `String` first because PyO3 ties them to the GIL hold. PyDict construction stays on the GIL side. Measured: 4 Python threads each running 20 crushes: before (GIL held): ~3.3s wall (serialized — equivalent to 4×0.83s) after (allow_threads): 826ms wall (4.01x speedup, perfect parallel) == 2. CcrStore: Mutex<HashMap> -> DashMap-backed sharded == Single Mutex was the dominant bottleneck under multi-worker load — every put/get serialized through one lock. Replace with DashMap (sharded concurrent map, lock-free reads within a shard) plus a separate small Mutex<VecDeque> for FIFO insertion-order eviction. Reads of distinct keys never contend; writes only contend during the brief order-queue push or capacity-sweep. A/B bench (200 mixed put/get ops × N threads, in benches/ccr_store.rs): Threads | DashMap Legacy Mutex Speedup ------------------------------------------- 1 | 63 µs 71 µs 1.13x 2 | 98 µs 194 µs 2.0x 4 | 178 µs 707 µs 4.0x 8 | 342 µs 1267 µs 3.7x Legacy degrades ~linearly with thread count; DashMap stays near-flat per-thread. Real multi-worker scaling. == 3. Single-serialize the lossy CCR payload == The lossy `crush_array` path used to serialize the full array TWICE: once in `hash_array_for_ccr` (allocates `Value::Array(items.to_vec())`, deep-clones every Value subtree, then serializes), and a second time in the store-write site. For a 50-item dict array that's ~MB of allocator pressure per crushed array. Introduce `canonical_array_json` (serializes `&[Value]` directly — same bytes as `Value::Array(items.to_vec())` but no wrapper allocation + no tree clone), call it ONCE per lossy path, then both hash and store from those same bytes. Hash-format stable — all 17 parity fixtures match byte-for-byte. == Tests == - 8 ccr.rs unit tests including a new concurrent-stress test (8 threads × 200 puts/gets, every key readable afterwards) - 14 ccr_roundtrip integration tests stay green - parity-run smart_crusher: 17/17 fixtures match - 479 lib + 14 integration + 185 Python tests all pass - New benches/ccr_store.rs runs the A/B and is committed for regression visibility == Dependencies added == - dashmap v6 (mature, widely-used in tokio/linkerd ecosystem)
2026-04-27 20:44:55 -07:00
[[package]]
name = "dashmap"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "6.2.1"
perf(rust): tier-1 multi-worker wins — GIL release, sharded CCR store, single-serialize CCR write Three orthogonal hot-path fixes targeting concurrent-request throughput. Each is independently bench-measured below; the proxy hot path benefits from all three at once. == 1. PyO3 GIL release on heavy compute == PyO3 methods (crush, smart_crush_content, crush_array_json, compact_document_json, compress, compress_with_stats) used to hold the GIL across the entire Rust call. Result: a 100ms compress() blocked EVERY other Python thread for 100ms — multi-worker uvicorn deployments serialized through SmartCrusher. Wrap each compute call in `py.allow_threads(|| ...)`. Inputs (`&str` from Python) are copied to owned `String` first because PyO3 ties them to the GIL hold. PyDict construction stays on the GIL side. Measured: 4 Python threads each running 20 crushes: before (GIL held): ~3.3s wall (serialized — equivalent to 4×0.83s) after (allow_threads): 826ms wall (4.01x speedup, perfect parallel) == 2. CcrStore: Mutex<HashMap> -> DashMap-backed sharded == Single Mutex was the dominant bottleneck under multi-worker load — every put/get serialized through one lock. Replace with DashMap (sharded concurrent map, lock-free reads within a shard) plus a separate small Mutex<VecDeque> for FIFO insertion-order eviction. Reads of distinct keys never contend; writes only contend during the brief order-queue push or capacity-sweep. A/B bench (200 mixed put/get ops × N threads, in benches/ccr_store.rs): Threads | DashMap Legacy Mutex Speedup ------------------------------------------- 1 | 63 µs 71 µs 1.13x 2 | 98 µs 194 µs 2.0x 4 | 178 µs 707 µs 4.0x 8 | 342 µs 1267 µs 3.7x Legacy degrades ~linearly with thread count; DashMap stays near-flat per-thread. Real multi-worker scaling. == 3. Single-serialize the lossy CCR payload == The lossy `crush_array` path used to serialize the full array TWICE: once in `hash_array_for_ccr` (allocates `Value::Array(items.to_vec())`, deep-clones every Value subtree, then serializes), and a second time in the store-write site. For a 50-item dict array that's ~MB of allocator pressure per crushed array. Introduce `canonical_array_json` (serializes `&[Value]` directly — same bytes as `Value::Array(items.to_vec())` but no wrapper allocation + no tree clone), call it ONCE per lossy path, then both hash and store from those same bytes. Hash-format stable — all 17 parity fixtures match byte-for-byte. == Tests == - 8 ccr.rs unit tests including a new concurrent-stress test (8 threads × 200 puts/gets, every key readable afterwards) - 14 ccr_roundtrip integration tests stay green - parity-run smart_crusher: 17/17 fixtures match - 479 lib + 14 integration + 185 Python tests all pass - New benches/ccr_store.rs runs the A/B and is committed for regression visibility == Dependencies added == - dashmap v6 (mature, widely-used in tokio/linkerd ecosystem)
2026-04-27 20:44:55 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "e6361d5c062261c78a176addb82d4c821ae42bed6089de0e12603cd25de2059c"
perf(rust): tier-1 multi-worker wins — GIL release, sharded CCR store, single-serialize CCR write Three orthogonal hot-path fixes targeting concurrent-request throughput. Each is independently bench-measured below; the proxy hot path benefits from all three at once. == 1. PyO3 GIL release on heavy compute == PyO3 methods (crush, smart_crush_content, crush_array_json, compact_document_json, compress, compress_with_stats) used to hold the GIL across the entire Rust call. Result: a 100ms compress() blocked EVERY other Python thread for 100ms — multi-worker uvicorn deployments serialized through SmartCrusher. Wrap each compute call in `py.allow_threads(|| ...)`. Inputs (`&str` from Python) are copied to owned `String` first because PyO3 ties them to the GIL hold. PyDict construction stays on the GIL side. Measured: 4 Python threads each running 20 crushes: before (GIL held): ~3.3s wall (serialized — equivalent to 4×0.83s) after (allow_threads): 826ms wall (4.01x speedup, perfect parallel) == 2. CcrStore: Mutex<HashMap> -> DashMap-backed sharded == Single Mutex was the dominant bottleneck under multi-worker load — every put/get serialized through one lock. Replace with DashMap (sharded concurrent map, lock-free reads within a shard) plus a separate small Mutex<VecDeque> for FIFO insertion-order eviction. Reads of distinct keys never contend; writes only contend during the brief order-queue push or capacity-sweep. A/B bench (200 mixed put/get ops × N threads, in benches/ccr_store.rs): Threads | DashMap Legacy Mutex Speedup ------------------------------------------- 1 | 63 µs 71 µs 1.13x 2 | 98 µs 194 µs 2.0x 4 | 178 µs 707 µs 4.0x 8 | 342 µs 1267 µs 3.7x Legacy degrades ~linearly with thread count; DashMap stays near-flat per-thread. Real multi-worker scaling. == 3. Single-serialize the lossy CCR payload == The lossy `crush_array` path used to serialize the full array TWICE: once in `hash_array_for_ccr` (allocates `Value::Array(items.to_vec())`, deep-clones every Value subtree, then serializes), and a second time in the store-write site. For a 50-item dict array that's ~MB of allocator pressure per crushed array. Introduce `canonical_array_json` (serializes `&[Value]` directly — same bytes as `Value::Array(items.to_vec())` but no wrapper allocation + no tree clone), call it ONCE per lossy path, then both hash and store from those same bytes. Hash-format stable — all 17 parity fixtures match byte-for-byte. == Tests == - 8 ccr.rs unit tests including a new concurrent-stress test (8 threads × 200 puts/gets, every key readable afterwards) - 14 ccr_roundtrip integration tests stay green - parity-run smart_crusher: 17/17 fixtures match - 479 lib + 14 integration + 185 Python tests all pass - New benches/ccr_store.rs runs the A/B and is committed for regression visibility == Dependencies added == - dashmap v6 (mature, widely-used in tokio/linkerd ecosystem)
2026-04-27 20:44:55 -07:00
dependencies = [
"cfg-if",
"crossbeam-utils",
"hashbrown 0.14.5",
"lock_api",
"once_cell",
"parking_lot_core",
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "data-encoding"
version = "2.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a4ae5f15dda3c708c0ade84bfee31ccab44a3da4f88015ed22f63732abe300c8"
[[package]]
name = "deadpool"
version = "0.12.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0be2b1d1d6ec8d846f05e137292d0b89133caf95ef33695424c09568bdd39b1b"
dependencies = [
"deadpool-runtime",
"lazy_static",
"num_cpus",
"tokio",
]
[[package]]
name = "deadpool-runtime"
version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "092966b41edc516079bdf31ec78a2e0588d1d0c08f78b91d8307215928642b2b"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "deranged"
version = "0.5.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "derive_builder"
version = "0.20.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "507dfb09ea8b7fa618fcf76e953f4f5e192547945816d5358edffe39f6f94947"
dependencies = [
"derive_builder_macro",
]
[[package]]
name = "derive_builder_core"
version = "0.20.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2d5bcf7b024d6835cfb3d473887cd966994907effbe9227e8c8219824d06c4e8"
dependencies = [
"darling",
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "derive_builder_macro"
version = "0.20.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ab63b0e2bf4d5928aff72e83a7dace85d7bba5fe12dcc3c5a572d78caffd3f3c"
dependencies = [
"derive_builder_core",
"syn",
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "digest"
version = "0.10.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
dependencies = [
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"block-buffer 0.10.4",
"crypto-common 0.1.7",
]
[[package]]
name = "digest"
version = "0.11.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2"
dependencies = [
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"block-buffer 0.12.1",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"const-oid",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"crypto-common 0.2.2",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"ctutils",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
]
feat(rust): HfTokenizer::from_pretrained — HuggingFace Hub auto-download Stage 2.1: closes the loop on the HuggingFace tokenizer story. Stage 2 shipped `HfTokenizer::from_bytes`/`from_file`, which required callers to manage their own tokenizer.json files. This adds the third constructor: let t = HfTokenizer::from_pretrained("CohereForAI/c4ai-command-r-v01")?; register_hf("command-", t); `from_pretrained` is a thin wrapper around the `hf-hub` crate's blocking `ureq` API. First call downloads `tokenizer.json` to `~/.cache/huggingface/ hub` (or `$HF_HOME` if set); subsequent calls reuse the on-disk cache. Uses the `main` revision; gated repos (Llama, Mistral) require `HF_TOKEN` in env or `~/.cache/huggingface/token`. Also adds `try_register_hf(prefix, repo)` as the obvious one-liner for proxy startup code: let _ = try_register_hf("command-", "CohereForAI/c4ai-command-r-v01"); let _ = try_register_hf("mistral-", "mistralai/Mistral-7B-v0.1"); Each call is independent — a download failure for one model (e.g. gated without a token) does not affect others. `HfTokenizerError` gains a new `Hub` variant so callers can distinguish "couldn't fetch" from "fetched but malformed" — relevant when deciding whether to retry, surface to the user, or fall back to the estimator. Why blocking, not async: `from_pretrained` is called once at startup. A sync API works from `main()`, from a `OnceLock` initializer, or from `tokio::task::spawn_blocking` if a tokio caller needs it later. The async hf-hub backend would force callers to await at startup, which doesn't fit the `register_hf` registry pattern. Why rustls, not native-tls: keeps the binary statically linkable for AWS deploys (no system OpenSSL dependency). Tests: a network-dependent integration test (`#[ignore]`d in CI; hits HF for `gpt2`, ~1.4 MB) verifies the real download + load + count path. A non-network negative test verifies that an invalid repo name surfaces as `HfTokenizerError::Hub`, not a panic. 44 unit tests + 5 proptests + 1 doctest pass; parity stays 40/40 byte-equal.
2026-04-25 15:05:09 -07:00
[[package]]
name = "dirs"
version = "6.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c3e8aa94d75141228480295a7d0e7feb620b1a5ad9f12bc40be62411e38cce4e"
dependencies = [
"dirs-sys",
]
[[package]]
name = "dirs-sys"
version = "0.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e01a3366d27ee9890022452ee61b2b63a67e6f13f58900b651ff5665f0bb1fab"
dependencies = [
"libc",
"option-ext",
"redox_users",
"windows-sys 0.61.2",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "displaydoc"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.2.6"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "1ac70aa55017e108007fbaf5aa0f54b021c98f92ff8af59d42eda9da96e3dd4f"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"proc-macro2",
"quote",
"syn",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "document-features"
version = "0.2.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d4b8a88685455ed29a21542a33abd9cb6510b6b129abadabdcef0f4c55bc8f61"
dependencies = [
"litrs",
]
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
[[package]]
name = "dunce"
version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92773504d58c093f6de2459af4af33faa518c13451eb8f2b5698ed3d36e7c813"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "either"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.16.0"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "91622ff5e7162018101f2fea40d6ebf4a78bbe5a49736a2020649edf9693679e"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "encode_unicode"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "34aa73646ffb006b8f5147f3dc182bd4bcb190227ce861fc4a4844bf8e3cb2c0"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "encoding_rs"
version = "0.8.35"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "75030f3c4f45dafd7586dd6780965a8c7e8e285a5ecb86713e63a79c5b2766f3"
dependencies = [
"cfg-if",
]
[[package]]
name = "equator"
version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4711b213838dfee0117e3be6ac926007d7f433d7bbe33595975d4190cb07e6fc"
dependencies = [
"equator-macro",
]
[[package]]
name = "equator-macro"
version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "44f23cf4b44bfce11a86ace86f8a73ffdec849c9fd00a386a53d278bd9e81fb3"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "equivalent"
version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "errno"
version = "0.3.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
dependencies = [
"libc",
"windows-sys 0.61.2",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "esaxx-rs"
version = "0.1.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d817e038c30374a4bcb22f94d0a8a0e216958d4c3dcde369b1439fec4bdda6e6"
dependencies = [
"cc",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "exr"
version = "1.74.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4300e043a56aa2cb633c01af81ca8f699a321879a7854d3896a0ba89056363be"
dependencies = [
"bit_field",
"half",
"lebe",
"miniz_oxide",
"rayon-core",
"smallvec",
"zune-inflate",
]
fix: B7 — CCR hardening: persistent backends + always-on tool P2-25, P2-26: CCR (Compress-Cache-Retrieve) used an in-memory store that fragmented across uvicorn workers and was wiped on restart, and the `headroom_retrieve` tool was registered/unregistered per-request based on whether the latest body happened to contain compression markers — every flip busted the prompt cache. Both are sticky side-channels: once a session has done CCR, the tool list bytes and the retrieval store must stay stable. This PR fixes both. Rust: * Split `ccr.rs` into `ccr/` with `backends/` submodule (`in_memory.rs`, `sqlite.rs`, `redis.rs` cfg-gated). * `SqliteCcrStore` (production default): WAL mode, prepared upsert, lazy TTL purge on read, persistent across worker restarts and shareable across workers on the same host via SQLite file locking. * `RedisCcrStore` (cfg-gated behind `feature = "redis"`): SETEX with startup PING smoke-test, no key-prefix collision risk, no sticky session required at the LB. * `CcrBackendConfig::{InMemory, Sqlite, Redis}` + `from_config(...)` factory — every init failure surfaces (no silent fallback per `feedback_no_silent_fallbacks.md`). * `ccr::compute_key` (BLAKE3 → first 24 hex chars) and `ccr::marker_for("HASH") -> "<<ccr:HASH>>"` centralize the hash + marker format; one definition for the live-zone dispatcher and the Python regex (`headroom/ccr/tool_injection.py:211`). * `compress_anthropic_live_zone_with_ccr` accepts `Option<&dyn CcrStore>`. When wired, every accepted compression puts the original bytes into the backend and appends `<<ccr:HASH>>` to the compressed string. The token-validation gate runs on the marker-augmented string so the `compressed_tokens >= original_tokens` rejection stays honest. Python: * `SessionCcrTracker` + `apply_session_sticky_ccr_tool` mirror the PR-A7 `SessionToolTracker` / `apply_session_sticky_memory_tools` pattern: once a session has done CCR, every subsequent request injects the recorded golden tool-definition bytes. Tool list bytes are byte-stable across turns (snapshot test pins them). * `headroom/ccr/tool_injection.py::inject_tool_definition` accepts a new `session_has_done_ccr` kwarg per the PR-B7 spec change at line 302-328. The legacy per-request path stays intact for callers that don't yet thread a session id (e.g. Google handler). * Anthropic + OpenAI handlers route their CCR tool-list updates through `apply_session_sticky_ccr_tool`, keyed off the existing `session_tracker_store.compute_session_id(...)` plumbing. Backend selection model: `CcrBackendConfig::Sqlite { path }` is the production default — single host, persistent, multi-worker safe with sticky session. `CcrBackendConfig::Redis { url }` is the multi-host scale-out option — no stickiness needed. `InMemory` is for tests and single-worker dev only. RUST_DEV.md "Multi-worker deployment — CCR fragmentation" rewritten around this matrix. Tests: * `crates/headroom-core/tests/ccr_backends.rs` — 7 tests covering SQLite round-trip, TTL purge, proxy-restart survival, cross-backend byte-equal keys, `from_config` paths, and the no-redis-feature loud-failure check (+ 2 redis tests gated behind the feature). * `crates/headroom-core/tests/live_zone_ccr.rs` — confirms `<<ccr:HASH>>` marker injection, store population, and no-marker-when-no-store invariants end-to-end. * `tests/test_ccr_tool_always_on.py` — 12 tests pinning the always-on behaviour, session/provider isolation, LRU bound, no- session-id fallback, and (per-acceptance-criterion) the byte-stable tool-definition snapshot. Per-PR-B7 plan: REALIGNMENT/04-phase-B-live-zone.md.
2026-05-02 16:49:08 -07:00
[[package]]
name = "fallible-iterator"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2acce4a10f12dc2fb14a218589d4f1f62ef011b2d0cc4b3cb1bba8e94da14649"
[[package]]
name = "fallible-streaming-iterator"
version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7360491ce676a36bf9bb3c56c1aa791658183a54d2744120f27285738d90465a"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "fancy-regex"
version = "0.17.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "72cf461f865c862bb7dc573f643dd6a2b6842f7c30b07882b56bd148cc2761b8"
dependencies = [
"bit-set",
"regex-automata",
"regex-syntax",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "fastembed"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "5.17.2"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "545e4fb17fc48768ff36c2a3854aa5b0b809d0ed595ab5530fa8ac94f31bd0ea"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
dependencies = [
"anyhow",
"hf-hub 0.5.0",
"image",
"ndarray",
"ort",
"safetensors",
"serde",
"serde_json",
fix(rust): audit cleanup — DiffCompressor CCR leak, CCR TOCTOU race, clippy debt, dep dedup Closes findings from the post-Phase-3g audit. Five surgical fixes plus telemetry-discoverability docs. PyO3 0.22 → 0.24 security upgrade is its own PR (issue #335). 1. DiffCompressor cache_key persistence (production bug) --------------------------------------------------------- Pre-fix: `RustDiffCompressor.compress()` minted a `cache_key`, embedded `[... hash=abc123]` in the wire marker, and returned without storing the original anywhere. Python ContentRouter then returned the compressed text with a dangling marker — every retrieval tool call from the LLM 404'd. Sibling compressors (LogCompressor, SearchCompressor) already had the right pattern: Rust mints the key, Python's `_persist_to_python_ccr` writes the original to the production `CompressionStore`. DiffCompressor was the asymmetric one. Fix: - Rust: add `DiffCompressor::compress_with_store(content, context, Option<&dyn CcrStore>)` mirroring siblings. Calls `store.put` when a key is minted; legacy `compress()` and `compress_with_stats()` delegate with `None` for parity. - Python: add `_persist_to_python_ccr` helper to `headroom/transforms/diff_compressor.py.compress()` mirroring `log_compressor.py` and `search_compressor.py`. - Pipeline `DiffOffload`: switch to `compress_with_store(Some(store))` and drop the post-hoc double-store hack that papered over this bug at the orchestrator boundary. 2. CCR store TOCTOU race in `get()` ----------------------------------- `InMemoryCcrStore::get()` checked TTL under a read lock, dropped the lock, then called `remove()`. Between drop and remove a concurrent `put()` of the same hash with fresh data could land — and our `remove` would then wipe that fresh entry. Under multi-worker proxy load this manifested as "I just stored it; why is it gone?" Fix: use `DashMap::remove_if`. Predicate runs under the shard write lock so check-and-remove is atomic. New regression test exercises a tight contention loop between writer and reader on the same key. 3. Pre-existing clippy debt in smart_crusher -------------------------------------------- - 3× `field_reassign_with_default` in `crusher.rs` test setup — switch to struct-update syntax `Config { field: x, ..Default }`. - `hash_array_for_ccr` was `#[cfg(test)]` but unused; deleted with a comment so a future test can reintroduce it as a one-liner. `cargo clippy --workspace --all-targets -- -D warnings` is now clean across the whole workspace; previous CI patches that allowed these warnings can be removed in a follow-up. 4. Tokenizers dependency dedup ------------------------------ `tokenizers 0.21` (direct dep) + `tokenizers 0.22` (transitive via fastembed) compiled twice into the binary. Bumped direct dep to `0.22` to align; API is compatible (verified by full tokenizer test suite). Saves compile time + binary bloat. 5. Telemetry-discoverability doc (no new code) ---------------------------------------------- The audit recommended a per-transform invocation counter to inform the next Python → Rust port. Discovered the infrastructure already exists at `/stats`: - `compressions_by_strategy` — invocation count per strategy - `pipeline_timing` — count + avg/max ms per transform name - `tokens_saved_by_strategy` — savings attribution Added a section to `RUST_DEV.md` showing the `curl + jq` recipes to read this data, with example output highlighting how to spot zero-invocation deferral candidates (e.g. `code_compressor`). Verification: workspace tests 734 + 14 + 5 + 4 + 6 + 5 + 2 + 2 + 3 + 4 + 2 + 2 + 1 = all green; cargo fmt clean; cargo clippy --all-targets clean; Python tests 185 pass; commitlint clean.
2026-04-30 20:54:22 -07:00
"tokenizers",
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "fastrand"
version = "2.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9f1f227452a390804cdb637b74a86990f2a7d7ba4b7d5693aac9b4dd6defd8d6"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "fax"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.2.7"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "caf1079563223d5d59d83c85886a56e586cfd5c1a26292e971a0fa266531ac5a"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "fdeflate"
version = "0.3.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e6853b52649d4ac5c0bd02320cddc5ba956bdb407c4b75a2c6b75bf51500f8c"
dependencies = [
"simd-adler32",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "find-msvc-tools"
version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582"
feat(rust): HfTokenizer::from_pretrained — HuggingFace Hub auto-download Stage 2.1: closes the loop on the HuggingFace tokenizer story. Stage 2 shipped `HfTokenizer::from_bytes`/`from_file`, which required callers to manage their own tokenizer.json files. This adds the third constructor: let t = HfTokenizer::from_pretrained("CohereForAI/c4ai-command-r-v01")?; register_hf("command-", t); `from_pretrained` is a thin wrapper around the `hf-hub` crate's blocking `ureq` API. First call downloads `tokenizer.json` to `~/.cache/huggingface/ hub` (or `$HF_HOME` if set); subsequent calls reuse the on-disk cache. Uses the `main` revision; gated repos (Llama, Mistral) require `HF_TOKEN` in env or `~/.cache/huggingface/token`. Also adds `try_register_hf(prefix, repo)` as the obvious one-liner for proxy startup code: let _ = try_register_hf("command-", "CohereForAI/c4ai-command-r-v01"); let _ = try_register_hf("mistral-", "mistralai/Mistral-7B-v0.1"); Each call is independent — a download failure for one model (e.g. gated without a token) does not affect others. `HfTokenizerError` gains a new `Hub` variant so callers can distinguish "couldn't fetch" from "fetched but malformed" — relevant when deciding whether to retry, surface to the user, or fall back to the estimator. Why blocking, not async: `from_pretrained` is called once at startup. A sync API works from `main()`, from a `OnceLock` initializer, or from `tokio::task::spawn_blocking` if a tokio caller needs it later. The async hf-hub backend would force callers to await at startup, which doesn't fit the `register_hf` registry pattern. Why rustls, not native-tls: keeps the binary statically linkable for AWS deploys (no system OpenSSL dependency). Tests: a network-dependent integration test (`#[ignore]`d in CI; hits HF for `gpt2`, ~1.4 MB) verifies the real download + load + count path. A non-network negative test verifies that an invalid repo name surfaces as `HfTokenizerError::Hub`, not a panic. 44 unit tests + 5 proptests + 1 doctest pass; parity stays 40/40 byte-equal.
2026-04-25 15:05:09 -07:00
[[package]]
name = "flate2"
version = "1.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c"
dependencies = [
"crc32fast",
"miniz_oxide",
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "fnv"
version = "1.0.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "foldhash"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "form_urlencoded"
version = "1.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cb4cb245038516f5f85277875cdaa4f7d2c9a0fa0468de06ed190163b1581fcf"
dependencies = [
"percent-encoding",
]
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
[[package]]
name = "fs_extra"
version = "1.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "futures"
version = "0.3.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8b147ee9d1f6d097cef9ce628cd2ee62288d963e16fb287bd9286455b241382d"
dependencies = [
"futures-channel",
"futures-core",
"futures-executor",
"futures-io",
"futures-sink",
"futures-task",
"futures-util",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "futures-channel"
version = "0.3.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "07bbe89c50d7a535e539b8c17bc0b49bdb77747034daa8087407d655f3f7cc1d"
dependencies = [
"futures-core",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"futures-sink",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
]
[[package]]
name = "futures-core"
version = "0.3.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7e3450815272ef58cec6d564423f6e755e25379b217b0bc688e295ba24df6b1d"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "futures-executor"
version = "0.3.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "baf29c38818342a3b26b5b923639e7b1f4a61fc5e76102d4b1981c6dc7a7579d"
dependencies = [
"futures-core",
"futures-task",
"futures-util",
]
[[package]]
name = "futures-io"
version = "0.3.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cecba35d7ad927e23624b22ad55235f2239cfa44fd10428eecbeba6d6a717718"
[[package]]
name = "futures-macro"
version = "0.3.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e835b70203e41293343137df5c0664546da5745f82ec9b84d40be8336958447b"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "futures-sink"
version = "0.3.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c39754e157331b013978ec91992bde1ac089843443c49cbc7f46150b0fad0893"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "futures-task"
version = "0.3.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "037711b3d59c33004d3856fbdc83b99d4ff37a24768fa1be9ce3538a1cde4393"
[[package]]
name = "futures-util"
version = "0.3.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "389ca41296e6190b48053de0321d02a77f32f8a5d2461dd38762c0593805c6d6"
dependencies = [
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"futures-channel",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"futures-core",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"futures-io",
"futures-macro",
"futures-sink",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"futures-task",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"memchr",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"pin-project-lite",
"slab",
]
fix(proxy): PR-D4 native Vertex publisher path + ADC bearer auth Adds a Rust-native Vertex AI publisher route ahead of the LiteLLM Python converter (which dropped `thinking`, `redacted_thinking`, `document`, `image`, `server_tool_use`, `mcp_tool_use` block kinds — the P4-37 / P4-38 bug). After this PR the Vertex `:rawPredict` and `:streamRawPredict` calls survive byte-equal upstream and benefit from the live-zone Anthropic dispatcher (PR-B-series) running over the body — same behaviour as `/v1/messages`. New module `crates/headroom-proxy/src/vertex/`: - `mod.rs` — single dispatch handler at the `/v1beta1/.../models/:model_action` route. Splits the trailing `:<verb>` segment with `str::rsplit_once(':')` (no regex) and flips an `attach_sse_tee` flag to dispatch to the streaming or non-streaming arm. Both verbs share one axum route shape because matchit can't distinguish two patterns that overlap on a parameter. - `envelope.rs` — `VertexEnvelope` parser. Confirms `anthropic_version` present + `model` field absent (the two fingerprints of the Vertex envelope vs `/v1/messages`). - `adc.rs` — `TokenSource` trait + `GcpAdcTokenSource` (production, `gcp_auth` 0.12) + `StaticTokenSource` (tests). Caches tokens with a 60s refresh-ahead-of-expiry window. Emits structured `event = "vertex_adc_token_refreshed"` per refresh. - `raw_predict.rs` — POST handler + shared `forward_vertex_request`. Buffers body, parses envelope, runs live-zone Anthropic compression, fetches ADC bearer, attaches `Authorization: Bearer <token>` (overwrites client-supplied Authorization header), forwards. SSE telemetry tee for the streaming verb reuses PR-C1's `AnthropicStreamState` directly (Vertex streams plain SSE, unlike Bedrock's binary EventStream). - `stream_raw_predict.rs` — module-level docs + alias to the shared dispatcher (the streaming-vs-non-streaming difference is one boolean flag inside the shared forwarder). Modifications: - `proxy.rs::build_app` — registers the single Vertex route. - `proxy.rs::AppState` — new `vertex_token_source: Arc<dyn TokenSource>` field. Production constructs `GcpAdcTokenSource` lazily (no GCP call until first `bearer()`); tests inject `StaticTokenSource` via the new `AppState::with_token_source` helper. - `config.rs` — adds `--vertex-region` / `HEADROOM_PROXY_VERTEX_REGION` (default `us-central1`, observability tag only — the upstream URL is `--upstream`) and `--vertex-adc-scope` / `HEADROOM_PROXY_VERTEX_ADC_SCOPE` (default `cloud-platform`). - `Cargo.toml` (workspace + proxy) — adds `gcp_auth = "0.12"` and `async-trait = "0.1"`. - `tests/common/mod.rs` — `start_proxy_with_state` accepts both config + state customizers; `install_static_token_source` helper for tests. `crates/headroom-proxy/tests/integration_vertex_raw_predict.rs` — all five tests pass: 1. `native_envelope_round_trip_byte_equal` — Vertex-shape body (with `anthropic_version`, no `model`) round-trips SHA-256 byte-equal upstream. 2. `adc_bearer_token_signed_correctly` — `Authorization: Bearer <static-test-token>` reaches upstream verbatim and OVERWRITES a client-supplied Authorization header. 3. `thinking_block_preserved` — request with `thinking` (incl. signature) + `redacted_thinking` (incl. opaque `data`) blocks round-trips byte-equal even with `LiveZone` compression mode enabled. This is the P4-37 / P4-38 teeth. 4. `stream_raw_predict_sse_handled` — `:streamRawPredict` proxies an Anthropic SSE response (full `message_start` → `content_block_delta` → `message_stop` sequence) back to the client without corruption; SSE content-type preserved end-to-end; bearer attached. 5. (bonus, no-silent-fallback contract) `adc_failure_returns_5xx_no_silent_forward` — when the token source returns `Err`, the proxy returns 5xx and never reaches upstream. Verifies the `event = "vertex_adc_fetch_failed"` error path. Workspace: `cargo test --workspace` green; `cargo clippy --workspace -- -D warnings` clean; `make ci-precheck` passes. - No silent fallbacks: ADC failure → structured 5xx, never an unauthenticated forward. - No hardcodes: every knob (region, ADC scope, upstream URL) is CLI-flag + env-var configurable. - No regexes: axum path parameters + `str::rsplit_once` only. - Comprehensive structured logs: `event` field on every decision point — `vertex_envelope_parsed`, `vertex_envelope_invalid`, `vertex_compression_skipped`, `vertex_compression_applied`, `vertex_adc_token_refreshed`, `vertex_adc_fetch_failed`, `vertex_streaming_pipeline_active`, `vertex_sse_stream_closed`, `vertex_forwarded`, `vertex_unknown_verb`, etc. - Performant: no body clone; ADC token cached + refreshed ahead-of-expiry, not fetched per request. - Comprehensive tests: realistic Anthropic block content (signature payload, redacted_thinking opaque blob) in `thinking_block_preserved`. The local `gcloud auth application-default print-access-token` returns no credentials, so manual validation against a real Vertex endpoint is not possible in this PR. Follow-up: the user runs `gcloud auth application-default login` once and exercises a live Vertex request — should be a no-code-change check. PR-D1 (Bedrock native) is running concurrently and will land its own envelope module at `crates/headroom-proxy/src/bedrock/envelope.rs`. The two envelope modules are intentionally siblings (not a shared trait) — the shapes differ (Bedrock has a different `anthropic_version` value, no `model` field, AWS SigV4 instead of GCP ADC), and a premature shared abstraction would obscure the provider-specific contracts. Whichever PR merges second rebases without conflict. Retires P4-38 (and the Vertex parts of P4-39); marketplace BYOC pitch (per project memory) gets one more native provider.
2026-05-03 16:22:07 -07:00
[[package]]
name = "gcp_auth"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.12.7"
fix(proxy): PR-D4 native Vertex publisher path + ADC bearer auth Adds a Rust-native Vertex AI publisher route ahead of the LiteLLM Python converter (which dropped `thinking`, `redacted_thinking`, `document`, `image`, `server_tool_use`, `mcp_tool_use` block kinds — the P4-37 / P4-38 bug). After this PR the Vertex `:rawPredict` and `:streamRawPredict` calls survive byte-equal upstream and benefit from the live-zone Anthropic dispatcher (PR-B-series) running over the body — same behaviour as `/v1/messages`. New module `crates/headroom-proxy/src/vertex/`: - `mod.rs` — single dispatch handler at the `/v1beta1/.../models/:model_action` route. Splits the trailing `:<verb>` segment with `str::rsplit_once(':')` (no regex) and flips an `attach_sse_tee` flag to dispatch to the streaming or non-streaming arm. Both verbs share one axum route shape because matchit can't distinguish two patterns that overlap on a parameter. - `envelope.rs` — `VertexEnvelope` parser. Confirms `anthropic_version` present + `model` field absent (the two fingerprints of the Vertex envelope vs `/v1/messages`). - `adc.rs` — `TokenSource` trait + `GcpAdcTokenSource` (production, `gcp_auth` 0.12) + `StaticTokenSource` (tests). Caches tokens with a 60s refresh-ahead-of-expiry window. Emits structured `event = "vertex_adc_token_refreshed"` per refresh. - `raw_predict.rs` — POST handler + shared `forward_vertex_request`. Buffers body, parses envelope, runs live-zone Anthropic compression, fetches ADC bearer, attaches `Authorization: Bearer <token>` (overwrites client-supplied Authorization header), forwards. SSE telemetry tee for the streaming verb reuses PR-C1's `AnthropicStreamState` directly (Vertex streams plain SSE, unlike Bedrock's binary EventStream). - `stream_raw_predict.rs` — module-level docs + alias to the shared dispatcher (the streaming-vs-non-streaming difference is one boolean flag inside the shared forwarder). Modifications: - `proxy.rs::build_app` — registers the single Vertex route. - `proxy.rs::AppState` — new `vertex_token_source: Arc<dyn TokenSource>` field. Production constructs `GcpAdcTokenSource` lazily (no GCP call until first `bearer()`); tests inject `StaticTokenSource` via the new `AppState::with_token_source` helper. - `config.rs` — adds `--vertex-region` / `HEADROOM_PROXY_VERTEX_REGION` (default `us-central1`, observability tag only — the upstream URL is `--upstream`) and `--vertex-adc-scope` / `HEADROOM_PROXY_VERTEX_ADC_SCOPE` (default `cloud-platform`). - `Cargo.toml` (workspace + proxy) — adds `gcp_auth = "0.12"` and `async-trait = "0.1"`. - `tests/common/mod.rs` — `start_proxy_with_state` accepts both config + state customizers; `install_static_token_source` helper for tests. `crates/headroom-proxy/tests/integration_vertex_raw_predict.rs` — all five tests pass: 1. `native_envelope_round_trip_byte_equal` — Vertex-shape body (with `anthropic_version`, no `model`) round-trips SHA-256 byte-equal upstream. 2. `adc_bearer_token_signed_correctly` — `Authorization: Bearer <static-test-token>` reaches upstream verbatim and OVERWRITES a client-supplied Authorization header. 3. `thinking_block_preserved` — request with `thinking` (incl. signature) + `redacted_thinking` (incl. opaque `data`) blocks round-trips byte-equal even with `LiveZone` compression mode enabled. This is the P4-37 / P4-38 teeth. 4. `stream_raw_predict_sse_handled` — `:streamRawPredict` proxies an Anthropic SSE response (full `message_start` → `content_block_delta` → `message_stop` sequence) back to the client without corruption; SSE content-type preserved end-to-end; bearer attached. 5. (bonus, no-silent-fallback contract) `adc_failure_returns_5xx_no_silent_forward` — when the token source returns `Err`, the proxy returns 5xx and never reaches upstream. Verifies the `event = "vertex_adc_fetch_failed"` error path. Workspace: `cargo test --workspace` green; `cargo clippy --workspace -- -D warnings` clean; `make ci-precheck` passes. - No silent fallbacks: ADC failure → structured 5xx, never an unauthenticated forward. - No hardcodes: every knob (region, ADC scope, upstream URL) is CLI-flag + env-var configurable. - No regexes: axum path parameters + `str::rsplit_once` only. - Comprehensive structured logs: `event` field on every decision point — `vertex_envelope_parsed`, `vertex_envelope_invalid`, `vertex_compression_skipped`, `vertex_compression_applied`, `vertex_adc_token_refreshed`, `vertex_adc_fetch_failed`, `vertex_streaming_pipeline_active`, `vertex_sse_stream_closed`, `vertex_forwarded`, `vertex_unknown_verb`, etc. - Performant: no body clone; ADC token cached + refreshed ahead-of-expiry, not fetched per request. - Comprehensive tests: realistic Anthropic block content (signature payload, redacted_thinking opaque blob) in `thinking_block_preserved`. The local `gcloud auth application-default print-access-token` returns no credentials, so manual validation against a real Vertex endpoint is not possible in this PR. Follow-up: the user runs `gcloud auth application-default login` once and exercises a live Vertex request — should be a no-code-change check. PR-D1 (Bedrock native) is running concurrently and will land its own envelope module at `crates/headroom-proxy/src/bedrock/envelope.rs`. The two envelope modules are intentionally siblings (not a shared trait) — the shapes differ (Bedrock has a different `anthropic_version` value, no `model` field, AWS SigV4 instead of GCP ADC), and a premature shared abstraction would obscure the provider-specific contracts. Whichever PR merges second rebases without conflict. Retires P4-38 (and the Vertex parts of P4-39); marketplace BYOC pitch (per project memory) gets one more native provider.
2026-05-03 16:22:07 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "26d27dbcc645b60b8e7f6e2868a9d7102ece97d1bb49c1288b5321fcc67f7260"
fix(proxy): PR-D4 native Vertex publisher path + ADC bearer auth Adds a Rust-native Vertex AI publisher route ahead of the LiteLLM Python converter (which dropped `thinking`, `redacted_thinking`, `document`, `image`, `server_tool_use`, `mcp_tool_use` block kinds — the P4-37 / P4-38 bug). After this PR the Vertex `:rawPredict` and `:streamRawPredict` calls survive byte-equal upstream and benefit from the live-zone Anthropic dispatcher (PR-B-series) running over the body — same behaviour as `/v1/messages`. New module `crates/headroom-proxy/src/vertex/`: - `mod.rs` — single dispatch handler at the `/v1beta1/.../models/:model_action` route. Splits the trailing `:<verb>` segment with `str::rsplit_once(':')` (no regex) and flips an `attach_sse_tee` flag to dispatch to the streaming or non-streaming arm. Both verbs share one axum route shape because matchit can't distinguish two patterns that overlap on a parameter. - `envelope.rs` — `VertexEnvelope` parser. Confirms `anthropic_version` present + `model` field absent (the two fingerprints of the Vertex envelope vs `/v1/messages`). - `adc.rs` — `TokenSource` trait + `GcpAdcTokenSource` (production, `gcp_auth` 0.12) + `StaticTokenSource` (tests). Caches tokens with a 60s refresh-ahead-of-expiry window. Emits structured `event = "vertex_adc_token_refreshed"` per refresh. - `raw_predict.rs` — POST handler + shared `forward_vertex_request`. Buffers body, parses envelope, runs live-zone Anthropic compression, fetches ADC bearer, attaches `Authorization: Bearer <token>` (overwrites client-supplied Authorization header), forwards. SSE telemetry tee for the streaming verb reuses PR-C1's `AnthropicStreamState` directly (Vertex streams plain SSE, unlike Bedrock's binary EventStream). - `stream_raw_predict.rs` — module-level docs + alias to the shared dispatcher (the streaming-vs-non-streaming difference is one boolean flag inside the shared forwarder). Modifications: - `proxy.rs::build_app` — registers the single Vertex route. - `proxy.rs::AppState` — new `vertex_token_source: Arc<dyn TokenSource>` field. Production constructs `GcpAdcTokenSource` lazily (no GCP call until first `bearer()`); tests inject `StaticTokenSource` via the new `AppState::with_token_source` helper. - `config.rs` — adds `--vertex-region` / `HEADROOM_PROXY_VERTEX_REGION` (default `us-central1`, observability tag only — the upstream URL is `--upstream`) and `--vertex-adc-scope` / `HEADROOM_PROXY_VERTEX_ADC_SCOPE` (default `cloud-platform`). - `Cargo.toml` (workspace + proxy) — adds `gcp_auth = "0.12"` and `async-trait = "0.1"`. - `tests/common/mod.rs` — `start_proxy_with_state` accepts both config + state customizers; `install_static_token_source` helper for tests. `crates/headroom-proxy/tests/integration_vertex_raw_predict.rs` — all five tests pass: 1. `native_envelope_round_trip_byte_equal` — Vertex-shape body (with `anthropic_version`, no `model`) round-trips SHA-256 byte-equal upstream. 2. `adc_bearer_token_signed_correctly` — `Authorization: Bearer <static-test-token>` reaches upstream verbatim and OVERWRITES a client-supplied Authorization header. 3. `thinking_block_preserved` — request with `thinking` (incl. signature) + `redacted_thinking` (incl. opaque `data`) blocks round-trips byte-equal even with `LiveZone` compression mode enabled. This is the P4-37 / P4-38 teeth. 4. `stream_raw_predict_sse_handled` — `:streamRawPredict` proxies an Anthropic SSE response (full `message_start` → `content_block_delta` → `message_stop` sequence) back to the client without corruption; SSE content-type preserved end-to-end; bearer attached. 5. (bonus, no-silent-fallback contract) `adc_failure_returns_5xx_no_silent_forward` — when the token source returns `Err`, the proxy returns 5xx and never reaches upstream. Verifies the `event = "vertex_adc_fetch_failed"` error path. Workspace: `cargo test --workspace` green; `cargo clippy --workspace -- -D warnings` clean; `make ci-precheck` passes. - No silent fallbacks: ADC failure → structured 5xx, never an unauthenticated forward. - No hardcodes: every knob (region, ADC scope, upstream URL) is CLI-flag + env-var configurable. - No regexes: axum path parameters + `str::rsplit_once` only. - Comprehensive structured logs: `event` field on every decision point — `vertex_envelope_parsed`, `vertex_envelope_invalid`, `vertex_compression_skipped`, `vertex_compression_applied`, `vertex_adc_token_refreshed`, `vertex_adc_fetch_failed`, `vertex_streaming_pipeline_active`, `vertex_sse_stream_closed`, `vertex_forwarded`, `vertex_unknown_verb`, etc. - Performant: no body clone; ADC token cached + refreshed ahead-of-expiry, not fetched per request. - Comprehensive tests: realistic Anthropic block content (signature payload, redacted_thinking opaque blob) in `thinking_block_preserved`. The local `gcloud auth application-default print-access-token` returns no credentials, so manual validation against a real Vertex endpoint is not possible in this PR. Follow-up: the user runs `gcloud auth application-default login` once and exercises a live Vertex request — should be a no-code-change check. PR-D1 (Bedrock native) is running concurrently and will land its own envelope module at `crates/headroom-proxy/src/bedrock/envelope.rs`. The two envelope modules are intentionally siblings (not a shared trait) — the shapes differ (Bedrock has a different `anthropic_version` value, no `model` field, AWS SigV4 instead of GCP ADC), and a premature shared abstraction would obscure the provider-specific contracts. Whichever PR merges second rebases without conflict. Retires P4-38 (and the Vertex parts of P4-39); marketplace BYOC pitch (per project memory) gets one more native provider.
2026-05-03 16:22:07 -07:00
dependencies = [
"async-trait",
"base64 0.22.1",
"bytes",
"chrono",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
fix(proxy): PR-D4 native Vertex publisher path + ADC bearer auth Adds a Rust-native Vertex AI publisher route ahead of the LiteLLM Python converter (which dropped `thinking`, `redacted_thinking`, `document`, `image`, `server_tool_use`, `mcp_tool_use` block kinds — the P4-37 / P4-38 bug). After this PR the Vertex `:rawPredict` and `:streamRawPredict` calls survive byte-equal upstream and benefit from the live-zone Anthropic dispatcher (PR-B-series) running over the body — same behaviour as `/v1/messages`. New module `crates/headroom-proxy/src/vertex/`: - `mod.rs` — single dispatch handler at the `/v1beta1/.../models/:model_action` route. Splits the trailing `:<verb>` segment with `str::rsplit_once(':')` (no regex) and flips an `attach_sse_tee` flag to dispatch to the streaming or non-streaming arm. Both verbs share one axum route shape because matchit can't distinguish two patterns that overlap on a parameter. - `envelope.rs` — `VertexEnvelope` parser. Confirms `anthropic_version` present + `model` field absent (the two fingerprints of the Vertex envelope vs `/v1/messages`). - `adc.rs` — `TokenSource` trait + `GcpAdcTokenSource` (production, `gcp_auth` 0.12) + `StaticTokenSource` (tests). Caches tokens with a 60s refresh-ahead-of-expiry window. Emits structured `event = "vertex_adc_token_refreshed"` per refresh. - `raw_predict.rs` — POST handler + shared `forward_vertex_request`. Buffers body, parses envelope, runs live-zone Anthropic compression, fetches ADC bearer, attaches `Authorization: Bearer <token>` (overwrites client-supplied Authorization header), forwards. SSE telemetry tee for the streaming verb reuses PR-C1's `AnthropicStreamState` directly (Vertex streams plain SSE, unlike Bedrock's binary EventStream). - `stream_raw_predict.rs` — module-level docs + alias to the shared dispatcher (the streaming-vs-non-streaming difference is one boolean flag inside the shared forwarder). Modifications: - `proxy.rs::build_app` — registers the single Vertex route. - `proxy.rs::AppState` — new `vertex_token_source: Arc<dyn TokenSource>` field. Production constructs `GcpAdcTokenSource` lazily (no GCP call until first `bearer()`); tests inject `StaticTokenSource` via the new `AppState::with_token_source` helper. - `config.rs` — adds `--vertex-region` / `HEADROOM_PROXY_VERTEX_REGION` (default `us-central1`, observability tag only — the upstream URL is `--upstream`) and `--vertex-adc-scope` / `HEADROOM_PROXY_VERTEX_ADC_SCOPE` (default `cloud-platform`). - `Cargo.toml` (workspace + proxy) — adds `gcp_auth = "0.12"` and `async-trait = "0.1"`. - `tests/common/mod.rs` — `start_proxy_with_state` accepts both config + state customizers; `install_static_token_source` helper for tests. `crates/headroom-proxy/tests/integration_vertex_raw_predict.rs` — all five tests pass: 1. `native_envelope_round_trip_byte_equal` — Vertex-shape body (with `anthropic_version`, no `model`) round-trips SHA-256 byte-equal upstream. 2. `adc_bearer_token_signed_correctly` — `Authorization: Bearer <static-test-token>` reaches upstream verbatim and OVERWRITES a client-supplied Authorization header. 3. `thinking_block_preserved` — request with `thinking` (incl. signature) + `redacted_thinking` (incl. opaque `data`) blocks round-trips byte-equal even with `LiveZone` compression mode enabled. This is the P4-37 / P4-38 teeth. 4. `stream_raw_predict_sse_handled` — `:streamRawPredict` proxies an Anthropic SSE response (full `message_start` → `content_block_delta` → `message_stop` sequence) back to the client without corruption; SSE content-type preserved end-to-end; bearer attached. 5. (bonus, no-silent-fallback contract) `adc_failure_returns_5xx_no_silent_forward` — when the token source returns `Err`, the proxy returns 5xx and never reaches upstream. Verifies the `event = "vertex_adc_fetch_failed"` error path. Workspace: `cargo test --workspace` green; `cargo clippy --workspace -- -D warnings` clean; `make ci-precheck` passes. - No silent fallbacks: ADC failure → structured 5xx, never an unauthenticated forward. - No hardcodes: every knob (region, ADC scope, upstream URL) is CLI-flag + env-var configurable. - No regexes: axum path parameters + `str::rsplit_once` only. - Comprehensive structured logs: `event` field on every decision point — `vertex_envelope_parsed`, `vertex_envelope_invalid`, `vertex_compression_skipped`, `vertex_compression_applied`, `vertex_adc_token_refreshed`, `vertex_adc_fetch_failed`, `vertex_streaming_pipeline_active`, `vertex_sse_stream_closed`, `vertex_forwarded`, `vertex_unknown_verb`, etc. - Performant: no body clone; ADC token cached + refreshed ahead-of-expiry, not fetched per request. - Comprehensive tests: realistic Anthropic block content (signature payload, redacted_thinking opaque blob) in `thinking_block_preserved`. The local `gcloud auth application-default print-access-token` returns no credentials, so manual validation against a real Vertex endpoint is not possible in this PR. Follow-up: the user runs `gcloud auth application-default login` once and exercises a live Vertex request — should be a no-code-change check. PR-D1 (Bedrock native) is running concurrently and will land its own envelope module at `crates/headroom-proxy/src/bedrock/envelope.rs`. The two envelope modules are intentionally siblings (not a shared trait) — the shapes differ (Bedrock has a different `anthropic_version` value, no `model` field, AWS SigV4 instead of GCP ADC), and a premature shared abstraction would obscure the provider-specific contracts. Whichever PR merges second rebases without conflict. Retires P4-38 (and the Vertex parts of P4-39); marketplace BYOC pitch (per project memory) gets one more native provider.
2026-05-03 16:22:07 -07:00
"http-body-util",
"hyper",
"hyper-rustls",
"hyper-util",
"ring",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"rustls",
fix(proxy): PR-D4 native Vertex publisher path + ADC bearer auth Adds a Rust-native Vertex AI publisher route ahead of the LiteLLM Python converter (which dropped `thinking`, `redacted_thinking`, `document`, `image`, `server_tool_use`, `mcp_tool_use` block kinds — the P4-37 / P4-38 bug). After this PR the Vertex `:rawPredict` and `:streamRawPredict` calls survive byte-equal upstream and benefit from the live-zone Anthropic dispatcher (PR-B-series) running over the body — same behaviour as `/v1/messages`. New module `crates/headroom-proxy/src/vertex/`: - `mod.rs` — single dispatch handler at the `/v1beta1/.../models/:model_action` route. Splits the trailing `:<verb>` segment with `str::rsplit_once(':')` (no regex) and flips an `attach_sse_tee` flag to dispatch to the streaming or non-streaming arm. Both verbs share one axum route shape because matchit can't distinguish two patterns that overlap on a parameter. - `envelope.rs` — `VertexEnvelope` parser. Confirms `anthropic_version` present + `model` field absent (the two fingerprints of the Vertex envelope vs `/v1/messages`). - `adc.rs` — `TokenSource` trait + `GcpAdcTokenSource` (production, `gcp_auth` 0.12) + `StaticTokenSource` (tests). Caches tokens with a 60s refresh-ahead-of-expiry window. Emits structured `event = "vertex_adc_token_refreshed"` per refresh. - `raw_predict.rs` — POST handler + shared `forward_vertex_request`. Buffers body, parses envelope, runs live-zone Anthropic compression, fetches ADC bearer, attaches `Authorization: Bearer <token>` (overwrites client-supplied Authorization header), forwards. SSE telemetry tee for the streaming verb reuses PR-C1's `AnthropicStreamState` directly (Vertex streams plain SSE, unlike Bedrock's binary EventStream). - `stream_raw_predict.rs` — module-level docs + alias to the shared dispatcher (the streaming-vs-non-streaming difference is one boolean flag inside the shared forwarder). Modifications: - `proxy.rs::build_app` — registers the single Vertex route. - `proxy.rs::AppState` — new `vertex_token_source: Arc<dyn TokenSource>` field. Production constructs `GcpAdcTokenSource` lazily (no GCP call until first `bearer()`); tests inject `StaticTokenSource` via the new `AppState::with_token_source` helper. - `config.rs` — adds `--vertex-region` / `HEADROOM_PROXY_VERTEX_REGION` (default `us-central1`, observability tag only — the upstream URL is `--upstream`) and `--vertex-adc-scope` / `HEADROOM_PROXY_VERTEX_ADC_SCOPE` (default `cloud-platform`). - `Cargo.toml` (workspace + proxy) — adds `gcp_auth = "0.12"` and `async-trait = "0.1"`. - `tests/common/mod.rs` — `start_proxy_with_state` accepts both config + state customizers; `install_static_token_source` helper for tests. `crates/headroom-proxy/tests/integration_vertex_raw_predict.rs` — all five tests pass: 1. `native_envelope_round_trip_byte_equal` — Vertex-shape body (with `anthropic_version`, no `model`) round-trips SHA-256 byte-equal upstream. 2. `adc_bearer_token_signed_correctly` — `Authorization: Bearer <static-test-token>` reaches upstream verbatim and OVERWRITES a client-supplied Authorization header. 3. `thinking_block_preserved` — request with `thinking` (incl. signature) + `redacted_thinking` (incl. opaque `data`) blocks round-trips byte-equal even with `LiveZone` compression mode enabled. This is the P4-37 / P4-38 teeth. 4. `stream_raw_predict_sse_handled` — `:streamRawPredict` proxies an Anthropic SSE response (full `message_start` → `content_block_delta` → `message_stop` sequence) back to the client without corruption; SSE content-type preserved end-to-end; bearer attached. 5. (bonus, no-silent-fallback contract) `adc_failure_returns_5xx_no_silent_forward` — when the token source returns `Err`, the proxy returns 5xx and never reaches upstream. Verifies the `event = "vertex_adc_fetch_failed"` error path. Workspace: `cargo test --workspace` green; `cargo clippy --workspace -- -D warnings` clean; `make ci-precheck` passes. - No silent fallbacks: ADC failure → structured 5xx, never an unauthenticated forward. - No hardcodes: every knob (region, ADC scope, upstream URL) is CLI-flag + env-var configurable. - No regexes: axum path parameters + `str::rsplit_once` only. - Comprehensive structured logs: `event` field on every decision point — `vertex_envelope_parsed`, `vertex_envelope_invalid`, `vertex_compression_skipped`, `vertex_compression_applied`, `vertex_adc_token_refreshed`, `vertex_adc_fetch_failed`, `vertex_streaming_pipeline_active`, `vertex_sse_stream_closed`, `vertex_forwarded`, `vertex_unknown_verb`, etc. - Performant: no body clone; ADC token cached + refreshed ahead-of-expiry, not fetched per request. - Comprehensive tests: realistic Anthropic block content (signature payload, redacted_thinking opaque blob) in `thinking_block_preserved`. The local `gcloud auth application-default print-access-token` returns no credentials, so manual validation against a real Vertex endpoint is not possible in this PR. Follow-up: the user runs `gcloud auth application-default login` once and exercises a live Vertex request — should be a no-code-change check. PR-D1 (Bedrock native) is running concurrently and will land its own envelope module at `crates/headroom-proxy/src/bedrock/envelope.rs`. The two envelope modules are intentionally siblings (not a shared trait) — the shapes differ (Bedrock has a different `anthropic_version` value, no `model` field, AWS SigV4 instead of GCP ADC), and a premature shared abstraction would obscure the provider-specific contracts. Whichever PR merges second rebases without conflict. Retires P4-38 (and the Vertex parts of P4-39); marketplace BYOC pitch (per project memory) gets one more native provider.
2026-05-03 16:22:07 -07:00
"rustls-pki-types",
"serde",
"serde_json",
"thiserror 2.0.18",
"tokio",
"tracing",
"tracing-futures",
"url",
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "generic-array"
version = "0.14.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a"
dependencies = [
"typenum",
"version_check",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "getrandom"
version = "0.2.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0"
dependencies = [
"cfg-if",
"js-sys",
"libc",
"wasi",
"wasm-bindgen",
]
[[package]]
name = "getrandom"
version = "0.3.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd"
dependencies = [
"cfg-if",
"js-sys",
"libc",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"r-efi 5.3.0",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"wasip2",
"wasm-bindgen",
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "getrandom"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.4.3"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
dependencies = [
"cfg-if",
"libc",
"r-efi 6.0.0",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "gif"
version = "0.14.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ee8cfcc411d9adbbaba82fb72661cc1bcca13e8bba98b364e62b2dba8f960159"
dependencies = [
"color_quant",
"weezl",
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "h2"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.4.15"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "6cb093c84e8bd9b188d4c4a8cb6579fc016968d14c99882163cd3ff402a4f155"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
dependencies = [
"atomic-waker",
"bytes",
"fnv",
"futures-core",
"futures-sink",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"indexmap",
"slab",
"tokio",
"tokio-util",
"tracing",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "half"
version = "2.7.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6ea2d84b969582b4b1864a92dc5d27cd2b77b622a8d79306834f1be5ba20d84b"
dependencies = [
"cfg-if",
"crunchy",
"zerocopy",
]
perf(rust): tier-1 multi-worker wins — GIL release, sharded CCR store, single-serialize CCR write Three orthogonal hot-path fixes targeting concurrent-request throughput. Each is independently bench-measured below; the proxy hot path benefits from all three at once. == 1. PyO3 GIL release on heavy compute == PyO3 methods (crush, smart_crush_content, crush_array_json, compact_document_json, compress, compress_with_stats) used to hold the GIL across the entire Rust call. Result: a 100ms compress() blocked EVERY other Python thread for 100ms — multi-worker uvicorn deployments serialized through SmartCrusher. Wrap each compute call in `py.allow_threads(|| ...)`. Inputs (`&str` from Python) are copied to owned `String` first because PyO3 ties them to the GIL hold. PyDict construction stays on the GIL side. Measured: 4 Python threads each running 20 crushes: before (GIL held): ~3.3s wall (serialized — equivalent to 4×0.83s) after (allow_threads): 826ms wall (4.01x speedup, perfect parallel) == 2. CcrStore: Mutex<HashMap> -> DashMap-backed sharded == Single Mutex was the dominant bottleneck under multi-worker load — every put/get serialized through one lock. Replace with DashMap (sharded concurrent map, lock-free reads within a shard) plus a separate small Mutex<VecDeque> for FIFO insertion-order eviction. Reads of distinct keys never contend; writes only contend during the brief order-queue push or capacity-sweep. A/B bench (200 mixed put/get ops × N threads, in benches/ccr_store.rs): Threads | DashMap Legacy Mutex Speedup ------------------------------------------- 1 | 63 µs 71 µs 1.13x 2 | 98 µs 194 µs 2.0x 4 | 178 µs 707 µs 4.0x 8 | 342 µs 1267 µs 3.7x Legacy degrades ~linearly with thread count; DashMap stays near-flat per-thread. Real multi-worker scaling. == 3. Single-serialize the lossy CCR payload == The lossy `crush_array` path used to serialize the full array TWICE: once in `hash_array_for_ccr` (allocates `Value::Array(items.to_vec())`, deep-clones every Value subtree, then serializes), and a second time in the store-write site. For a 50-item dict array that's ~MB of allocator pressure per crushed array. Introduce `canonical_array_json` (serializes `&[Value]` directly — same bytes as `Value::Array(items.to_vec())` but no wrapper allocation + no tree clone), call it ONCE per lossy path, then both hash and store from those same bytes. Hash-format stable — all 17 parity fixtures match byte-for-byte. == Tests == - 8 ccr.rs unit tests including a new concurrent-stress test (8 threads × 200 puts/gets, every key readable afterwards) - 14 ccr_roundtrip integration tests stay green - parity-run smart_crusher: 17/17 fixtures match - 479 lib + 14 integration + 185 Python tests all pass - New benches/ccr_store.rs runs the A/B and is committed for regression visibility == Dependencies added == - dashmap v6 (mature, widely-used in tokio/linkerd ecosystem)
2026-04-27 20:44:55 -07:00
[[package]]
name = "hashbrown"
version = "0.14.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1"
fix: B7 — CCR hardening: persistent backends + always-on tool P2-25, P2-26: CCR (Compress-Cache-Retrieve) used an in-memory store that fragmented across uvicorn workers and was wiped on restart, and the `headroom_retrieve` tool was registered/unregistered per-request based on whether the latest body happened to contain compression markers — every flip busted the prompt cache. Both are sticky side-channels: once a session has done CCR, the tool list bytes and the retrieval store must stay stable. This PR fixes both. Rust: * Split `ccr.rs` into `ccr/` with `backends/` submodule (`in_memory.rs`, `sqlite.rs`, `redis.rs` cfg-gated). * `SqliteCcrStore` (production default): WAL mode, prepared upsert, lazy TTL purge on read, persistent across worker restarts and shareable across workers on the same host via SQLite file locking. * `RedisCcrStore` (cfg-gated behind `feature = "redis"`): SETEX with startup PING smoke-test, no key-prefix collision risk, no sticky session required at the LB. * `CcrBackendConfig::{InMemory, Sqlite, Redis}` + `from_config(...)` factory — every init failure surfaces (no silent fallback per `feedback_no_silent_fallbacks.md`). * `ccr::compute_key` (BLAKE3 → first 24 hex chars) and `ccr::marker_for("HASH") -> "<<ccr:HASH>>"` centralize the hash + marker format; one definition for the live-zone dispatcher and the Python regex (`headroom/ccr/tool_injection.py:211`). * `compress_anthropic_live_zone_with_ccr` accepts `Option<&dyn CcrStore>`. When wired, every accepted compression puts the original bytes into the backend and appends `<<ccr:HASH>>` to the compressed string. The token-validation gate runs on the marker-augmented string so the `compressed_tokens >= original_tokens` rejection stays honest. Python: * `SessionCcrTracker` + `apply_session_sticky_ccr_tool` mirror the PR-A7 `SessionToolTracker` / `apply_session_sticky_memory_tools` pattern: once a session has done CCR, every subsequent request injects the recorded golden tool-definition bytes. Tool list bytes are byte-stable across turns (snapshot test pins them). * `headroom/ccr/tool_injection.py::inject_tool_definition` accepts a new `session_has_done_ccr` kwarg per the PR-B7 spec change at line 302-328. The legacy per-request path stays intact for callers that don't yet thread a session id (e.g. Google handler). * Anthropic + OpenAI handlers route their CCR tool-list updates through `apply_session_sticky_ccr_tool`, keyed off the existing `session_tracker_store.compute_session_id(...)` plumbing. Backend selection model: `CcrBackendConfig::Sqlite { path }` is the production default — single host, persistent, multi-worker safe with sticky session. `CcrBackendConfig::Redis { url }` is the multi-host scale-out option — no stickiness needed. `InMemory` is for tests and single-worker dev only. RUST_DEV.md "Multi-worker deployment — CCR fragmentation" rewritten around this matrix. Tests: * `crates/headroom-core/tests/ccr_backends.rs` — 7 tests covering SQLite round-trip, TTL purge, proxy-restart survival, cross-backend byte-equal keys, `from_config` paths, and the no-redis-feature loud-failure check (+ 2 redis tests gated behind the feature). * `crates/headroom-core/tests/live_zone_ccr.rs` — confirms `<<ccr:HASH>>` marker injection, store population, and no-marker-when-no-store invariants end-to-end. * `tests/test_ccr_tool_always_on.py` — 12 tests pinning the always-on behaviour, session/provider isolation, LRU bound, no- session-id fallback, and (per-acceptance-criterion) the byte-stable tool-definition snapshot. Per-PR-B7 plan: REALIGNMENT/04-phase-B-live-zone.md.
2026-05-02 16:49:08 -07:00
dependencies = [
"ahash",
]
perf(rust): tier-1 multi-worker wins — GIL release, sharded CCR store, single-serialize CCR write Three orthogonal hot-path fixes targeting concurrent-request throughput. Each is independently bench-measured below; the proxy hot path benefits from all three at once. == 1. PyO3 GIL release on heavy compute == PyO3 methods (crush, smart_crush_content, crush_array_json, compact_document_json, compress, compress_with_stats) used to hold the GIL across the entire Rust call. Result: a 100ms compress() blocked EVERY other Python thread for 100ms — multi-worker uvicorn deployments serialized through SmartCrusher. Wrap each compute call in `py.allow_threads(|| ...)`. Inputs (`&str` from Python) are copied to owned `String` first because PyO3 ties them to the GIL hold. PyDict construction stays on the GIL side. Measured: 4 Python threads each running 20 crushes: before (GIL held): ~3.3s wall (serialized — equivalent to 4×0.83s) after (allow_threads): 826ms wall (4.01x speedup, perfect parallel) == 2. CcrStore: Mutex<HashMap> -> DashMap-backed sharded == Single Mutex was the dominant bottleneck under multi-worker load — every put/get serialized through one lock. Replace with DashMap (sharded concurrent map, lock-free reads within a shard) plus a separate small Mutex<VecDeque> for FIFO insertion-order eviction. Reads of distinct keys never contend; writes only contend during the brief order-queue push or capacity-sweep. A/B bench (200 mixed put/get ops × N threads, in benches/ccr_store.rs): Threads | DashMap Legacy Mutex Speedup ------------------------------------------- 1 | 63 µs 71 µs 1.13x 2 | 98 µs 194 µs 2.0x 4 | 178 µs 707 µs 4.0x 8 | 342 µs 1267 µs 3.7x Legacy degrades ~linearly with thread count; DashMap stays near-flat per-thread. Real multi-worker scaling. == 3. Single-serialize the lossy CCR payload == The lossy `crush_array` path used to serialize the full array TWICE: once in `hash_array_for_ccr` (allocates `Value::Array(items.to_vec())`, deep-clones every Value subtree, then serializes), and a second time in the store-write site. For a 50-item dict array that's ~MB of allocator pressure per crushed array. Introduce `canonical_array_json` (serializes `&[Value]` directly — same bytes as `Value::Array(items.to_vec())` but no wrapper allocation + no tree clone), call it ONCE per lossy path, then both hash and store from those same bytes. Hash-format stable — all 17 parity fixtures match byte-for-byte. == Tests == - 8 ccr.rs unit tests including a new concurrent-stress test (8 threads × 200 puts/gets, every key readable afterwards) - 14 ccr_roundtrip integration tests stay green - parity-run smart_crusher: 17/17 fixtures match - 479 lib + 14 integration + 185 Python tests all pass - New benches/ccr_store.rs runs the A/B and is committed for regression visibility == Dependencies added == - dashmap v6 (mature, widely-used in tokio/linkerd ecosystem)
2026-04-27 20:44:55 -07:00
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "hashbrown"
version = "0.16.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100"
dependencies = [
"allocator-api2",
"equivalent",
fix(deps): remediate dependency CVEs and publish SBOM (#1509) ## Description Supply-chain hardening: takes the **shipped** dependency surface from **26 known CVEs to 0**. `pip install headroom-ai[all]` now resolves with no known vulnerabilities (verified with Anchore syft + grype). Also publishes a checked-in SBOM package (`sbom/`) so any user — especially pilots running their own security review — can verify what's inside and that we track it. This addresses the Dependabot alerts on `main` (9 high / 4 moderate / 7 low at time of writing). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made **Rust** - `pyo3` 0.24 → 0.29 (GHSA-36hh-v3qg-5jq4 High, GHSA-chgr-c6px-7xpp Med). Migrated `Python::allow_threads` → `Python::detach` (10 sites) and added `from_py_object` to the `Clone`-deriving `#[pyclass]` types (both required by the 0.25+ API). - `pyo3-log` 0.12 → 0.13; `lru` 0.12 → 0.18 (GHSA-rhfx-m35p-ff5j). **Python** - `torch` → 2.12.1, `mem0ai` → 2.x. - Floor-pinned transitive CVE deps via `[tool.uv] constraint-dependencies`: `pygments>=2.20.0`, `pydantic-settings>=2.14.2`, `gitpython>=3.1.50`, `langsmith>=0.9.0`. - **Removed `benchmark` from the `[all]` aggregate** so the default install is CVE-free. `lm-eval` is invoked as an external subprocess (`python -m lm_eval`) and never imported, so it is not a true runtime dep — it remains available via the opt-in `[benchmark]` extra. See [Accepted Risks](#additional-notes). **npm (build/test tooling — never shipped in the wheel/container/published SDK)** - `esbuild` override `>=0.28.1` in `sdk/typescript` + `plugins/openclaw` (GHSA-g7r4-m6w7-qqqr). - `docs/`: `@anthropic-ai/sdk` → `^0.106.0` (GHSA-p7fg-763f-g4gf), `postcss` override to force Next.js's bundled copy ≥8.5.10 (GHSA-qx2v-qp2m-jg93); regenerated a stale `bun.lock` that carried a **Critical** vitest/vite. **CI** - Pinned `pypa/gh-action-pypi-publish` `@release/v1` → `@v1.13.0` (GHSA-vxmw-7h4f-hqxh) in `release.yml` + `publish.yml`. **SBOM** - New `sbom/` directory: CycloneDX 1.7 + SPDX 2.3 SBOMs, grype scan evidence, 330-package license inventory, and a regeneration guide. ## Testing - [ ] Unit tests pass (`pytest`) — N/A, no Python source changed (deps/config only) - [x] Linting passes — `cargo fmt --check` + `cargo clippy` clean on the changed crate; 0 `.py` files changed so `ruff`/`mypy` scope is unaffected - [x] Type checking passes — `cargo check --workspace` (0 errors) - [ ] New tests added — N/A (dependency bumps; covered by existing suites) - [x] Manual testing performed — see Real Behavior Proof ### Test Output ```text # headroom-ai[all] product surface — the number that matters $ grype sbom:sbom/headroom-sbom-all-extra.cdx.json No vulnerabilities found # full repo scan (universal lock incl. opt-in [benchmark] + dev) $ grype sbom:sbom/headroom-sbom.cdx.json NAME INSTALLED TYPE VULNERABILITY SEVERITY sqlitedict 2.1.0 python GHSA-g4r7-86gm-pgqc High # [benchmark]-only, unpatchable, accepted nltk 3.9.4 python GHSA-p4gq-832x-fm9v High # [benchmark]-only, unpatchable, accepted # pyo3 0.29 migration — extension builds + imports + runs $ cargo check --workspace Finished `dev` profile [unoptimized + debuginfo] target(s) $ maturin develop && python -c "from headroom._core import DiffCompressor, SmartCrusher; ..." extension OK — detach + from_py_object paths exercised # lru 0.18 — eviction path $ cargo test -p headroom-proxy --lib drift 14 passed, 213 filtered out # per-ecosystem npm audits $ (cd sdk/typescript && npm audit) -> found 0 vulnerabilities $ (cd plugins/openclaw && npm audit) -> found 0 vulnerabilities $ (cd docs && npm audit && bun audit) -> found 0 vulnerabilities / No vulnerabilities found ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0, arm64), Python 3.12 `.venv`, Rust 1.95 toolchain, syft 1.46.0, grype 0.115.0, bun 1.3.14, maturin 1.13.3. - Exact command / steps: (1) `uv export --extra all --no-dev --no-emit-project | syft → grype` for the product surface; (2) `cargo check --workspace` + `maturin develop` + extension import/compress smoke test; (3) `cargo test -p headroom-proxy --lib drift`; (4) `cargo fmt --check` + `cargo clippy -p headroom-py`; (5) `npm audit` in sdk/openclaw/docs + `bun audit` in docs. - Observed result: `headroom-ai[all]` resolution scans clean — "No vulnerabilities found" (179 pkgs); full/prod SBOM shows only the 2 documented accepted CVEs; pyo3 0.29 extension imports and runs (detach + from_py_object paths exercised); drift tests 14/14 pass; cargo fmt + clippy clean; all npm/bun audits report 0. - Not tested: full `pytest` suite (no Python source changed); release-profile wheel build (used dev-profile `maturin develop` for the import proof — the extension is semantically identical). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (`sbom/README.md`) - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective — N/A (dependency bumps; existing suites + scans cover it) - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A (Release Please auto-generates from the conventional commit) ## Additional Notes **Accepted risks (the 2 residual CVEs).** Both originate solely from the EleutherAI `lm-evaluation-harness` under the **opt-in `[benchmark]` extra**, which Headroom invokes as a subprocess (never imports): - `sqlitedict` CVE-2024-35515 (High) — pickle deserialization; package abandoned (last release 2021), **no upstream fix exists**. - `nltk` CVE-2026-54293 (High) — path traversal in `nltk.data.load()`; affects ≤3.9.4 (current latest), **no patched release**. Neither is in `[all]`, the published wheel, or the container. They are documented in `sbom/README.md` and will be picked up automatically once upstream ships fixes. **Release/CHANGELOG:** N/A items above are because this is a dependency/security PR with no Python source changes; CHANGELOG is Release-Please-managed via the conventional commit message.
2026-06-27 15:28:12 -07:00
"foldhash",
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
"serde",
"serde_core",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
]
[[package]]
name = "hashbrown"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.17.1"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a"
fix(deps): remediate dependency CVEs and publish SBOM (#1509) ## Description Supply-chain hardening: takes the **shipped** dependency surface from **26 known CVEs to 0**. `pip install headroom-ai[all]` now resolves with no known vulnerabilities (verified with Anchore syft + grype). Also publishes a checked-in SBOM package (`sbom/`) so any user — especially pilots running their own security review — can verify what's inside and that we track it. This addresses the Dependabot alerts on `main` (9 high / 4 moderate / 7 low at time of writing). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made **Rust** - `pyo3` 0.24 → 0.29 (GHSA-36hh-v3qg-5jq4 High, GHSA-chgr-c6px-7xpp Med). Migrated `Python::allow_threads` → `Python::detach` (10 sites) and added `from_py_object` to the `Clone`-deriving `#[pyclass]` types (both required by the 0.25+ API). - `pyo3-log` 0.12 → 0.13; `lru` 0.12 → 0.18 (GHSA-rhfx-m35p-ff5j). **Python** - `torch` → 2.12.1, `mem0ai` → 2.x. - Floor-pinned transitive CVE deps via `[tool.uv] constraint-dependencies`: `pygments>=2.20.0`, `pydantic-settings>=2.14.2`, `gitpython>=3.1.50`, `langsmith>=0.9.0`. - **Removed `benchmark` from the `[all]` aggregate** so the default install is CVE-free. `lm-eval` is invoked as an external subprocess (`python -m lm_eval`) and never imported, so it is not a true runtime dep — it remains available via the opt-in `[benchmark]` extra. See [Accepted Risks](#additional-notes). **npm (build/test tooling — never shipped in the wheel/container/published SDK)** - `esbuild` override `>=0.28.1` in `sdk/typescript` + `plugins/openclaw` (GHSA-g7r4-m6w7-qqqr). - `docs/`: `@anthropic-ai/sdk` → `^0.106.0` (GHSA-p7fg-763f-g4gf), `postcss` override to force Next.js's bundled copy ≥8.5.10 (GHSA-qx2v-qp2m-jg93); regenerated a stale `bun.lock` that carried a **Critical** vitest/vite. **CI** - Pinned `pypa/gh-action-pypi-publish` `@release/v1` → `@v1.13.0` (GHSA-vxmw-7h4f-hqxh) in `release.yml` + `publish.yml`. **SBOM** - New `sbom/` directory: CycloneDX 1.7 + SPDX 2.3 SBOMs, grype scan evidence, 330-package license inventory, and a regeneration guide. ## Testing - [ ] Unit tests pass (`pytest`) — N/A, no Python source changed (deps/config only) - [x] Linting passes — `cargo fmt --check` + `cargo clippy` clean on the changed crate; 0 `.py` files changed so `ruff`/`mypy` scope is unaffected - [x] Type checking passes — `cargo check --workspace` (0 errors) - [ ] New tests added — N/A (dependency bumps; covered by existing suites) - [x] Manual testing performed — see Real Behavior Proof ### Test Output ```text # headroom-ai[all] product surface — the number that matters $ grype sbom:sbom/headroom-sbom-all-extra.cdx.json No vulnerabilities found # full repo scan (universal lock incl. opt-in [benchmark] + dev) $ grype sbom:sbom/headroom-sbom.cdx.json NAME INSTALLED TYPE VULNERABILITY SEVERITY sqlitedict 2.1.0 python GHSA-g4r7-86gm-pgqc High # [benchmark]-only, unpatchable, accepted nltk 3.9.4 python GHSA-p4gq-832x-fm9v High # [benchmark]-only, unpatchable, accepted # pyo3 0.29 migration — extension builds + imports + runs $ cargo check --workspace Finished `dev` profile [unoptimized + debuginfo] target(s) $ maturin develop && python -c "from headroom._core import DiffCompressor, SmartCrusher; ..." extension OK — detach + from_py_object paths exercised # lru 0.18 — eviction path $ cargo test -p headroom-proxy --lib drift 14 passed, 213 filtered out # per-ecosystem npm audits $ (cd sdk/typescript && npm audit) -> found 0 vulnerabilities $ (cd plugins/openclaw && npm audit) -> found 0 vulnerabilities $ (cd docs && npm audit && bun audit) -> found 0 vulnerabilities / No vulnerabilities found ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0, arm64), Python 3.12 `.venv`, Rust 1.95 toolchain, syft 1.46.0, grype 0.115.0, bun 1.3.14, maturin 1.13.3. - Exact command / steps: (1) `uv export --extra all --no-dev --no-emit-project | syft → grype` for the product surface; (2) `cargo check --workspace` + `maturin develop` + extension import/compress smoke test; (3) `cargo test -p headroom-proxy --lib drift`; (4) `cargo fmt --check` + `cargo clippy -p headroom-py`; (5) `npm audit` in sdk/openclaw/docs + `bun audit` in docs. - Observed result: `headroom-ai[all]` resolution scans clean — "No vulnerabilities found" (179 pkgs); full/prod SBOM shows only the 2 documented accepted CVEs; pyo3 0.29 extension imports and runs (detach + from_py_object paths exercised); drift tests 14/14 pass; cargo fmt + clippy clean; all npm/bun audits report 0. - Not tested: full `pytest` suite (no Python source changed); release-profile wheel build (used dev-profile `maturin develop` for the import proof — the extension is semantically identical). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (`sbom/README.md`) - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective — N/A (dependency bumps; existing suites + scans cover it) - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A (Release Please auto-generates from the conventional commit) ## Additional Notes **Accepted risks (the 2 residual CVEs).** Both originate solely from the EleutherAI `lm-evaluation-harness` under the **opt-in `[benchmark]` extra**, which Headroom invokes as a subprocess (never imports): - `sqlitedict` CVE-2024-35515 (High) — pickle deserialization; package abandoned (last release 2021), **no upstream fix exists**. - `nltk` CVE-2026-54293 (High) — path traversal in `nltk.data.load()`; affects ≤3.9.4 (current latest), **no patched release**. Neither is in `[all]`, the published wheel, or the container. They are documented in `sbom/README.md` and will be picked up automatically once upstream ships fixes. **Release/CHANGELOG:** N/A items above are because this is a dependency/security PR with no Python source changes; CHANGELOG is Release-Please-managed via the conventional commit message.
2026-06-27 15:28:12 -07:00
dependencies = [
"allocator-api2",
"equivalent",
"foldhash",
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
fix: B7 — CCR hardening: persistent backends + always-on tool P2-25, P2-26: CCR (Compress-Cache-Retrieve) used an in-memory store that fragmented across uvicorn workers and was wiped on restart, and the `headroom_retrieve` tool was registered/unregistered per-request based on whether the latest body happened to contain compression markers — every flip busted the prompt cache. Both are sticky side-channels: once a session has done CCR, the tool list bytes and the retrieval store must stay stable. This PR fixes both. Rust: * Split `ccr.rs` into `ccr/` with `backends/` submodule (`in_memory.rs`, `sqlite.rs`, `redis.rs` cfg-gated). * `SqliteCcrStore` (production default): WAL mode, prepared upsert, lazy TTL purge on read, persistent across worker restarts and shareable across workers on the same host via SQLite file locking. * `RedisCcrStore` (cfg-gated behind `feature = "redis"`): SETEX with startup PING smoke-test, no key-prefix collision risk, no sticky session required at the LB. * `CcrBackendConfig::{InMemory, Sqlite, Redis}` + `from_config(...)` factory — every init failure surfaces (no silent fallback per `feedback_no_silent_fallbacks.md`). * `ccr::compute_key` (BLAKE3 → first 24 hex chars) and `ccr::marker_for("HASH") -> "<<ccr:HASH>>"` centralize the hash + marker format; one definition for the live-zone dispatcher and the Python regex (`headroom/ccr/tool_injection.py:211`). * `compress_anthropic_live_zone_with_ccr` accepts `Option<&dyn CcrStore>`. When wired, every accepted compression puts the original bytes into the backend and appends `<<ccr:HASH>>` to the compressed string. The token-validation gate runs on the marker-augmented string so the `compressed_tokens >= original_tokens` rejection stays honest. Python: * `SessionCcrTracker` + `apply_session_sticky_ccr_tool` mirror the PR-A7 `SessionToolTracker` / `apply_session_sticky_memory_tools` pattern: once a session has done CCR, every subsequent request injects the recorded golden tool-definition bytes. Tool list bytes are byte-stable across turns (snapshot test pins them). * `headroom/ccr/tool_injection.py::inject_tool_definition` accepts a new `session_has_done_ccr` kwarg per the PR-B7 spec change at line 302-328. The legacy per-request path stays intact for callers that don't yet thread a session id (e.g. Google handler). * Anthropic + OpenAI handlers route their CCR tool-list updates through `apply_session_sticky_ccr_tool`, keyed off the existing `session_tracker_store.compute_session_id(...)` plumbing. Backend selection model: `CcrBackendConfig::Sqlite { path }` is the production default — single host, persistent, multi-worker safe with sticky session. `CcrBackendConfig::Redis { url }` is the multi-host scale-out option — no stickiness needed. `InMemory` is for tests and single-worker dev only. RUST_DEV.md "Multi-worker deployment — CCR fragmentation" rewritten around this matrix. Tests: * `crates/headroom-core/tests/ccr_backends.rs` — 7 tests covering SQLite round-trip, TTL purge, proxy-restart survival, cross-backend byte-equal keys, `from_config` paths, and the no-redis-feature loud-failure check (+ 2 redis tests gated behind the feature). * `crates/headroom-core/tests/live_zone_ccr.rs` — confirms `<<ccr:HASH>>` marker injection, store population, and no-marker-when-no-store invariants end-to-end. * `tests/test_ccr_tool_always_on.py` — 12 tests pinning the always-on behaviour, session/provider isolation, LRU bound, no- session-id fallback, and (per-acceptance-criterion) the byte-stable tool-definition snapshot. Per-PR-B7 plan: REALIGNMENT/04-phase-B-live-zone.md.
2026-05-02 16:49:08 -07:00
[[package]]
name = "hashlink"
version = "0.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6ba4ff7128dee98c7dc9794b6a411377e1404dba1c97deb8d1a55297bd25d8af"
dependencies = [
"hashbrown 0.14.5",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "headroom-core"
version = "0.1.0"
dependencies = [
feat(rust): signals trait module + KeywordDetector (Phase 3e.1) Establish `crates/headroom-core/src/signals/` as a top-level module holding cross-cutting detection traits. Phase 3e.1 ports `error_detection.py` to a `LineImportanceDetector` trait + a `Tiered<T>` combinator + a single concrete `KeywordDetector` impl backed by aho-corasick. Three traits at three granularities are sketched (line / blob / item); only line-importance is implemented today. Two bug fixes from the Python source bake into both the Rust impl and the Python regex shim: 1. `ERROR_KEYWORDS` listed `timeout|abort|denied|rejected` but `ERROR_PATTERN` regex omitted them. Lines like `"Connection timeout"` were silently neutral despite the keyword being canonical. Both surfaces now flag them. 2. `SECURITY_KEYWORDS` carried `token`, which false-positived on every reference to LLM tokens (`input_tokens`, `tokens_saved`, ...) in our own product. Dropped from the security set. The Python `error_detection.py` shim now reflects keyword data out of Rust via `keyword_registry_snapshot()` and recompiles the legacy `re.Pattern` objects on the fly. Existing callers (text_compressor, search_compressor, intelligent_context) continue to import the same names with no source changes; caller migration to the trait API happens in their own port PRs. The trait architecture is the seam where a future ML detector slots in without touching `KeywordDetector` or any caller. The canonical extension is documented in `signals/README.md` as a classifier head on the existing `bge-small-en-v1.5` embedder loaded by `relevance::EmbeddingScorer` -- 384-dim -> 4-class softmax, ~1.5 KB head, ~1 ms inference, no extra model file. Two alternatives (distilled tinyBERT in ONNX, logistic regression on lexical features) are kept open in case BGE-head underfits. Per the no-silent-fallbacks rule: only `KeywordDetector` lands as a concrete impl. No NoOp, no MockDetector, no stub-ML -- those will arrive with their real implementations. Phase 3g (Compression Pipeline Formalization, issue #315) is queued as the cross-cutting follow-up that will make lossless-then-lossy- then-CCR ordering an explicit, observable architecture rather than implicit per-compressor logic. Trait shapes there will reuse the signals primitive landed in this PR.
2026-04-29 15:55:13 -07:00
"aho-corasick",
fix: B7 — CCR hardening: persistent backends + always-on tool P2-25, P2-26: CCR (Compress-Cache-Retrieve) used an in-memory store that fragmented across uvicorn workers and was wiped on restart, and the `headroom_retrieve` tool was registered/unregistered per-request based on whether the latest body happened to contain compression markers — every flip busted the prompt cache. Both are sticky side-channels: once a session has done CCR, the tool list bytes and the retrieval store must stay stable. This PR fixes both. Rust: * Split `ccr.rs` into `ccr/` with `backends/` submodule (`in_memory.rs`, `sqlite.rs`, `redis.rs` cfg-gated). * `SqliteCcrStore` (production default): WAL mode, prepared upsert, lazy TTL purge on read, persistent across worker restarts and shareable across workers on the same host via SQLite file locking. * `RedisCcrStore` (cfg-gated behind `feature = "redis"`): SETEX with startup PING smoke-test, no key-prefix collision risk, no sticky session required at the LB. * `CcrBackendConfig::{InMemory, Sqlite, Redis}` + `from_config(...)` factory — every init failure surfaces (no silent fallback per `feedback_no_silent_fallbacks.md`). * `ccr::compute_key` (BLAKE3 → first 24 hex chars) and `ccr::marker_for("HASH") -> "<<ccr:HASH>>"` centralize the hash + marker format; one definition for the live-zone dispatcher and the Python regex (`headroom/ccr/tool_injection.py:211`). * `compress_anthropic_live_zone_with_ccr` accepts `Option<&dyn CcrStore>`. When wired, every accepted compression puts the original bytes into the backend and appends `<<ccr:HASH>>` to the compressed string. The token-validation gate runs on the marker-augmented string so the `compressed_tokens >= original_tokens` rejection stays honest. Python: * `SessionCcrTracker` + `apply_session_sticky_ccr_tool` mirror the PR-A7 `SessionToolTracker` / `apply_session_sticky_memory_tools` pattern: once a session has done CCR, every subsequent request injects the recorded golden tool-definition bytes. Tool list bytes are byte-stable across turns (snapshot test pins them). * `headroom/ccr/tool_injection.py::inject_tool_definition` accepts a new `session_has_done_ccr` kwarg per the PR-B7 spec change at line 302-328. The legacy per-request path stays intact for callers that don't yet thread a session id (e.g. Google handler). * Anthropic + OpenAI handlers route their CCR tool-list updates through `apply_session_sticky_ccr_tool`, keyed off the existing `session_tracker_store.compute_session_id(...)` plumbing. Backend selection model: `CcrBackendConfig::Sqlite { path }` is the production default — single host, persistent, multi-worker safe with sticky session. `CcrBackendConfig::Redis { url }` is the multi-host scale-out option — no stickiness needed. `InMemory` is for tests and single-worker dev only. RUST_DEV.md "Multi-worker deployment — CCR fragmentation" rewritten around this matrix. Tests: * `crates/headroom-core/tests/ccr_backends.rs` — 7 tests covering SQLite round-trip, TTL purge, proxy-restart survival, cross-backend byte-equal keys, `from_config` paths, and the no-redis-feature loud-failure check (+ 2 redis tests gated behind the feature). * `crates/headroom-core/tests/live_zone_ccr.rs` — confirms `<<ccr:HASH>>` marker injection, store population, and no-marker-when-no-store invariants end-to-end. * `tests/test_ccr_tool_always_on.py` — 12 tests pinning the always-on behaviour, session/provider isolation, LRU bound, no- session-id fallback, and (per-acceptance-criterion) the byte-stable tool-definition snapshot. Per-PR-B7 plan: REALIGNMENT/04-phase-B-live-zone.md.
2026-05-02 16:49:08 -07:00
"blake3",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"bytes",
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
"criterion",
perf(rust): tier-1 multi-worker wins — GIL release, sharded CCR store, single-serialize CCR write Three orthogonal hot-path fixes targeting concurrent-request throughput. Each is independently bench-measured below; the proxy hot path benefits from all three at once. == 1. PyO3 GIL release on heavy compute == PyO3 methods (crush, smart_crush_content, crush_array_json, compact_document_json, compress, compress_with_stats) used to hold the GIL across the entire Rust call. Result: a 100ms compress() blocked EVERY other Python thread for 100ms — multi-worker uvicorn deployments serialized through SmartCrusher. Wrap each compute call in `py.allow_threads(|| ...)`. Inputs (`&str` from Python) are copied to owned `String` first because PyO3 ties them to the GIL hold. PyDict construction stays on the GIL side. Measured: 4 Python threads each running 20 crushes: before (GIL held): ~3.3s wall (serialized — equivalent to 4×0.83s) after (allow_threads): 826ms wall (4.01x speedup, perfect parallel) == 2. CcrStore: Mutex<HashMap> -> DashMap-backed sharded == Single Mutex was the dominant bottleneck under multi-worker load — every put/get serialized through one lock. Replace with DashMap (sharded concurrent map, lock-free reads within a shard) plus a separate small Mutex<VecDeque> for FIFO insertion-order eviction. Reads of distinct keys never contend; writes only contend during the brief order-queue push or capacity-sweep. A/B bench (200 mixed put/get ops × N threads, in benches/ccr_store.rs): Threads | DashMap Legacy Mutex Speedup ------------------------------------------- 1 | 63 µs 71 µs 1.13x 2 | 98 µs 194 µs 2.0x 4 | 178 µs 707 µs 4.0x 8 | 342 µs 1267 µs 3.7x Legacy degrades ~linearly with thread count; DashMap stays near-flat per-thread. Real multi-worker scaling. == 3. Single-serialize the lossy CCR payload == The lossy `crush_array` path used to serialize the full array TWICE: once in `hash_array_for_ccr` (allocates `Value::Array(items.to_vec())`, deep-clones every Value subtree, then serializes), and a second time in the store-write site. For a 50-item dict array that's ~MB of allocator pressure per crushed array. Introduce `canonical_array_json` (serializes `&[Value]` directly — same bytes as `Value::Array(items.to_vec())` but no wrapper allocation + no tree clone), call it ONCE per lossy path, then both hash and store from those same bytes. Hash-format stable — all 17 parity fixtures match byte-for-byte. == Tests == - 8 ccr.rs unit tests including a new concurrent-stress test (8 threads × 200 puts/gets, every key readable afterwards) - 14 ccr_roundtrip integration tests stay green - parity-run smart_crusher: 17/17 fixtures match - 479 lib + 14 integration + 185 Python tests all pass - New benches/ccr_store.rs runs the A/B and is committed for regression visibility == Dependencies added == - dashmap v6 (mature, widely-used in tokio/linkerd ecosystem)
2026-04-27 20:44:55 -07:00
"dashmap",
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
"fastembed",
feat(rust): smart_crusher universal crushers — string, number, object Three crushers from headroom/transforms/smart_crusher.py ported. Each takes a SmartCrusherConfig + bias and returns (crushed_items, strategy_string). All schema-preserving — output is items/values from the original; no generated text. What's in: 1. compute_k_split (smart_crusher.py:2693) Wraps adaptive_sizer::compute_optimal_k. Splits k_total into first/last/importance via config.first_fraction / last_fraction. Uses f64::round_ties_even() (Rust 1.77+) to match Python's banker's-rounding round() — important for off-by-one parity on .5-edged k computations. 2. crush_string_array (smart_crusher.py:2727) Adaptive K via Kneedle. Mandatory-keep: error-keyword strings + length-anomaly strings (>variance_threshold σ from mean length). Boundary-keep: first K_first + last K_last. Stride-based diverse fill with content-dedup. Output preserves original array order (BTreeSet iteration). Strategy includes dedup= and errors= counts when nonzero. 3. crush_number_array (smart_crusher.py:2810) — CARRIES BUG #1 Statistics-driven (mean/median/stdev/p25/p75). Outliers flagged at variance_threshold σ. Change-points via window-mean comparison (config.preserve_change_points + n>10 gates). Strategy string embeds full stats summary via format_g (Python's :.4g approximation). BUG #1 — percentile off-by-one — ported AS-IS: sorted_finite[len/4] / sorted_finite[3*len/4]. Cosmetic (strategy-string only). Test bug1_percentile_off_by_one_documented pins the buggy index choice; commit 7 fixes both languages and regenerates fixtures. 4. crush_object (smart_crusher.py:3015) Token-budget gate (config.min_tokens_to_crush=200). Three passthrough exits: n<=8, total tokens too low, k_total>=n. Always keeps: error-keyword values + small values (<=12 tokens via len/4 + len/4 + 2 heuristic). Boundary keys + stride fill with Python's recompute-each-iter cap (mirrored faithfully — slower but parity-true). Output preserves key insertion order via serde_json/preserve_order's IndexMap. Supporting helpers in stats_math.rs: - median(values) — Python statistics.median (mean-of-middles for even, total_cmp sort for NaN determinism). - format_g(x) — approximate Python f"{x:.4g}" (4 sig figs, scientific outside [-4, 4) exponent range, trailing-zero strip, explicit-sign 2-digit exponent). Pinned by 5 fixed-output tests. Field iteration order: key/object iteration uses BTreeMap-sorted (in analyzer) and IndexMap-insertion-order (in serde_json::Map for crush_object). The Python sorted-key fix scheduled for commit 7 also covers crush_object's iteration paths. Net: 266 unit tests passing in headroom-core, clippy clean (MSRV 1.80), parity harness intact (4/4 diff_compressor). Next commit: planning + execution layer (_create_plan, _execute_plan, plan-builder methods) with BUG #4 fix (k-split overshoot).
2026-04-26 18:02:23 -07:00
"flate2",
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
"hf-hub 0.4.3",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
chore(rust): port Magika detection (Stage 3d PR3 — Tier 1) Adds Google's `magika` ONNX-backed content classifier as the first tier of the new Stage-3d ContentRouter detection arch (`magika` → `unidiff-rs` → `PlainText` fall-through; no regex tier on the Rust side). What lands: - New module `crates/headroom-core/src/transforms/magika_detector.rs`: - `magika_detect(content: &str) -> Result<ContentType, _>` - `OnceLock<Mutex<Result<Session, _>>>` singleton: model loads once per process; init failure is recorded once and cheaply replayed (no retry — rust-side `feedback_no_silent_fallbacks`). - `map_magika_label(&str) -> ContentType`: explicit match arms against magika's 200+ labels, mapped onto Headroom's existing `ContentType` enum so the dispatch (PR5) stays enum-stable. Unmapped labels passthrough to `PlainText` rather than misroute. - 16 unit tests: empty fast-path, JSON / Python / Rust / JS / diff / markdown / plain prose / HTML / YAML / shell / SQL, singleton-reuse smoke, default-passthrough for unmapped labels, pure-table-lookup sanity. What does NOT land here (per PR scope): - No PyO3 surface yet — PR3 is detector-only. - No router rewiring — the existing regex `content_detector` still drives `ContentRouter` until PR5 flips the dispatch. - No `unidiff-rs` Tier-2 — that's PR4. The `magika` crate brings `ndarray` + `ort` (already in our dep tree via `fastembed`); adding it shares the ONNX Runtime singleton rather than pulling a second ML stack. `make ci-precheck` green.
2026-04-28 22:36:28 -07:00
"magika",
feat(rust): diff_compressor port — byte-equal parity + sidecar stats Stage 3a: first real transform port. Faithful Rust port of `headroom.transforms.diff_compressor` with byte-equal parity against all 20 recorded fixtures. # Algorithm (matching Python) 1. Hand-rolled unified-diff parser (state machine over `diff --git`, `index`, `--- a/`, `+++ b/`, `@@`, mode/binary/rename markers, +/- / space lines, "other" lines like `\ No newline at end of file`). 2. File cap (`max_files=20`): when fired, sort by total changes (most first) and keep top N. 3. Per-file hunk cap (`max_hunks_per_file=10`): keep first + last + top relevance-scored middle, then resort by hunk-header start line to restore appearance order. 4. Relevance scoring: change-density base + user-query word overlap + priority patterns (ERROR / IMPORTANCE / SECURITY regexes — matches `error_detection.PRIORITY_PATTERNS_DIFF`). 5. Per-hunk context trim: keep `max_context_lines=2` lines either side of each `+`/`-` line. 6. CCR cache_key: `md5(original)[:24]` (matches `compression_store.CompressionStore.store`). Emitted only when compression saved >20% of lines. Parity result: `[diff_compressor ] total=20 matched=20 skipped=0 diffed=0`. # Information preservation hardening Three pass-through paths inherited from Python that we keep deliberate (would lose info if we changed them): - Below `min_lines_for_ccr` (50): return input unchanged. - No diff sections parsed: return input unchanged. - Below 20% compression savings: emit compressed output but no CCR marker (the original is the cheaper representation anyway). Plus a parity-bound subtlety: `compressed_line_count` is captured BEFORE the CCR retrieval marker is appended, both for the marker text (`compressed to N`) and the result field. The output string therefore ends up with one more line than the field reports — by design, matching Python exactly. An off-by-one bug from recounting after appending the CCR marker was caught and pinned by a synthetic 8-file diff test. # Observability — the Rust escape hatch Python's `DiffCompressionResult` has thin observability: input/output line counts, additions/deletions, hunks_kept/removed, files_affected, cache_key. The Rust port adds a sidecar `DiffCompressorStats` struct with metrics Python doesn't emit: - `files_dropped: Vec<String>` — names (old → new path) of files silently discarded by the `max_files` cap. Python loses these. - `hunks_dropped_per_file: BTreeMap<String, usize>` — per-file hunk drops, stable iteration via `BTreeMap`. - `context_lines_input` / `context_lines_kept` / `context_lines_trimmed` — directly proxies info loss from the context trim. - `largest_hunk_kept_lines` / `largest_hunk_dropped_lines` — outlier detection (a single huge dropped hunk is much worse than many small). - `parse_warnings: Vec<String>` — surfaces malformed input rather than dropping silently. - `processing_duration_us` — latency budget. - `cache_key_emitted` + `ccr_skipped_reason: Option<String>` — explicit signal for "we chose not to emit CCR and this is why". A `tracing::info!(target: "diff_compressor", ...)` event is emitted on every call, carrying these fields for OTel scraping in prod. The sidecar struct is returned alongside via `compress_with_stats`; the parity-only `compress` API discards it. # Module layout - `crates/headroom-core/src/transforms/mod.rs` — namespace, doc comment with the guiding principle ("information preservation > aggressive compression") so future ports inherit the philosophy. - `crates/headroom-core/src/transforms/diff_compressor.rs` — full port (parser, scorer, hunk selector, context trimmer, formatter, CCR layer, stats, tracing). # Dependencies added to headroom-core - `md-5 = "0.10"` — for the CCR cache_key (matches Python MD5[:24]). - `regex = "1"` — was a transitive dep via tokenizers; now a direct dependency for the hunk-header parser and priority patterns. # Tests 6 unit tests covering pass-through paths, MD5 hex truncation, the Python `split("\n")` line-count semantics, sidecar stats emission, and a synthetic 8-file diff that locks the byte-equal behavior found in the parity fixtures.
2026-04-25 15:44:25 -07:00
"md-5",
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
"proptest",
fix(rust): reformat/offload pipeline + log templates + diff noise (Phase 3g rework) Replaces PR1's lossless/lossy split with ReformatTransform (pack denser, no info lost) and OffloadTransform (drop bytes, CCR-stash original via required cache_key). With CCR every transform is information-preserving end-to-end, so the lossless/lossy distinction misnamed the architecture. OffloadTransform carries a cheap, structural estimate_bloat() method scoped to its domain — generic byte-redundancy heuristics miss domain semantics. The orchestrator runs reformat phase + per-offload bloat estimation in parallel via rayon::join + par_iter, then runs offload iff bloat clears threshold OR reformat underwhelmed. Transforms shipped: REFORMATS (lossless): - JsonMinifier: serde_json round-trip whitespace stripping. - LogTemplate: Drain-inspired order-preserving template miner. Collapses consecutive runs of same-template lines into [Template Tn: ...] (Nx) + variant table. Win comes from emitting the constant-token prefix once instead of N times. Lossless: every original line reconstructible from template + variants. OFFLOADS (drop bytes, stash original via CCR): - LogOffload: wraps existing LogCompressor; bloat = repetition x uniqueness_weight + dilution x priority_dilution_weight. - DiffOffload: wraps existing DiffCompressor; bloat = context-to- change ratio. Bug-fix-on-port — persists original under the cache_key the parity-bound DiffCompressor mints (closes a leak). - DiffNoise: drops lockfile hunks (Cargo.lock, package-lock.json, yarn.lock, etc., suffix list configurable in TOML) and whitespace-only hunks. Stashes original via CCR for retrieval. Search offload exists but is not in default re-exports — modern agents (Claude Code, Codex) use scoped rg/grep, the marginal value didn't justify default registration. Reach via the explicit module path if opting in. JSON Offload is intentionally absent from this PR — already lives at SmartCrusher; Phase 3g PR3 wraps it in the OffloadTransform contract. Thresholds and weights live in config/pipeline.toml, embedded via include_str!; PipelineConfig::from_toml_str loads runtime overrides. 98 new pipeline tests; full headroom-core suite (714) and workspace tests green; cargo fmt clean. No regex, per project convention.
2026-04-30 11:02:29 -07:00
"rayon",
fix: B7 — CCR hardening: persistent backends + always-on tool P2-25, P2-26: CCR (Compress-Cache-Retrieve) used an in-memory store that fragmented across uvicorn workers and was wiped on restart, and the `headroom_retrieve` tool was registered/unregistered per-request based on whether the latest body happened to contain compression markers — every flip busted the prompt cache. Both are sticky side-channels: once a session has done CCR, the tool list bytes and the retrieval store must stay stable. This PR fixes both. Rust: * Split `ccr.rs` into `ccr/` with `backends/` submodule (`in_memory.rs`, `sqlite.rs`, `redis.rs` cfg-gated). * `SqliteCcrStore` (production default): WAL mode, prepared upsert, lazy TTL purge on read, persistent across worker restarts and shareable across workers on the same host via SQLite file locking. * `RedisCcrStore` (cfg-gated behind `feature = "redis"`): SETEX with startup PING smoke-test, no key-prefix collision risk, no sticky session required at the LB. * `CcrBackendConfig::{InMemory, Sqlite, Redis}` + `from_config(...)` factory — every init failure surfaces (no silent fallback per `feedback_no_silent_fallbacks.md`). * `ccr::compute_key` (BLAKE3 → first 24 hex chars) and `ccr::marker_for("HASH") -> "<<ccr:HASH>>"` centralize the hash + marker format; one definition for the live-zone dispatcher and the Python regex (`headroom/ccr/tool_injection.py:211`). * `compress_anthropic_live_zone_with_ccr` accepts `Option<&dyn CcrStore>`. When wired, every accepted compression puts the original bytes into the backend and appends `<<ccr:HASH>>` to the compressed string. The token-validation gate runs on the marker-augmented string so the `compressed_tokens >= original_tokens` rejection stays honest. Python: * `SessionCcrTracker` + `apply_session_sticky_ccr_tool` mirror the PR-A7 `SessionToolTracker` / `apply_session_sticky_memory_tools` pattern: once a session has done CCR, every subsequent request injects the recorded golden tool-definition bytes. Tool list bytes are byte-stable across turns (snapshot test pins them). * `headroom/ccr/tool_injection.py::inject_tool_definition` accepts a new `session_has_done_ccr` kwarg per the PR-B7 spec change at line 302-328. The legacy per-request path stays intact for callers that don't yet thread a session id (e.g. Google handler). * Anthropic + OpenAI handlers route their CCR tool-list updates through `apply_session_sticky_ccr_tool`, keyed off the existing `session_tracker_store.compute_session_id(...)` plumbing. Backend selection model: `CcrBackendConfig::Sqlite { path }` is the production default — single host, persistent, multi-worker safe with sticky session. `CcrBackendConfig::Redis { url }` is the multi-host scale-out option — no stickiness needed. `InMemory` is for tests and single-worker dev only. RUST_DEV.md "Multi-worker deployment — CCR fragmentation" rewritten around this matrix. Tests: * `crates/headroom-core/tests/ccr_backends.rs` — 7 tests covering SQLite round-trip, TTL purge, proxy-restart survival, cross-backend byte-equal keys, `from_config` paths, and the no-redis-feature loud-failure check (+ 2 redis tests gated behind the feature). * `crates/headroom-core/tests/live_zone_ccr.rs` — confirms `<<ccr:HASH>>` marker injection, store population, and no-marker-when-no-store invariants end-to-end. * `tests/test_ccr_tool_always_on.py` — 12 tests pinning the always-on behaviour, session/provider isolation, LRU bound, no- session-id fallback, and (per-acceptance-criterion) the byte-stable tool-definition snapshot. Per-PR-B7 plan: REALIGNMENT/04-phase-B-live-zone.md.
2026-05-02 16:49:08 -07:00
"redis",
feat(rust): diff_compressor port — byte-equal parity + sidecar stats Stage 3a: first real transform port. Faithful Rust port of `headroom.transforms.diff_compressor` with byte-equal parity against all 20 recorded fixtures. # Algorithm (matching Python) 1. Hand-rolled unified-diff parser (state machine over `diff --git`, `index`, `--- a/`, `+++ b/`, `@@`, mode/binary/rename markers, +/- / space lines, "other" lines like `\ No newline at end of file`). 2. File cap (`max_files=20`): when fired, sort by total changes (most first) and keep top N. 3. Per-file hunk cap (`max_hunks_per_file=10`): keep first + last + top relevance-scored middle, then resort by hunk-header start line to restore appearance order. 4. Relevance scoring: change-density base + user-query word overlap + priority patterns (ERROR / IMPORTANCE / SECURITY regexes — matches `error_detection.PRIORITY_PATTERNS_DIFF`). 5. Per-hunk context trim: keep `max_context_lines=2` lines either side of each `+`/`-` line. 6. CCR cache_key: `md5(original)[:24]` (matches `compression_store.CompressionStore.store`). Emitted only when compression saved >20% of lines. Parity result: `[diff_compressor ] total=20 matched=20 skipped=0 diffed=0`. # Information preservation hardening Three pass-through paths inherited from Python that we keep deliberate (would lose info if we changed them): - Below `min_lines_for_ccr` (50): return input unchanged. - No diff sections parsed: return input unchanged. - Below 20% compression savings: emit compressed output but no CCR marker (the original is the cheaper representation anyway). Plus a parity-bound subtlety: `compressed_line_count` is captured BEFORE the CCR retrieval marker is appended, both for the marker text (`compressed to N`) and the result field. The output string therefore ends up with one more line than the field reports — by design, matching Python exactly. An off-by-one bug from recounting after appending the CCR marker was caught and pinned by a synthetic 8-file diff test. # Observability — the Rust escape hatch Python's `DiffCompressionResult` has thin observability: input/output line counts, additions/deletions, hunks_kept/removed, files_affected, cache_key. The Rust port adds a sidecar `DiffCompressorStats` struct with metrics Python doesn't emit: - `files_dropped: Vec<String>` — names (old → new path) of files silently discarded by the `max_files` cap. Python loses these. - `hunks_dropped_per_file: BTreeMap<String, usize>` — per-file hunk drops, stable iteration via `BTreeMap`. - `context_lines_input` / `context_lines_kept` / `context_lines_trimmed` — directly proxies info loss from the context trim. - `largest_hunk_kept_lines` / `largest_hunk_dropped_lines` — outlier detection (a single huge dropped hunk is much worse than many small). - `parse_warnings: Vec<String>` — surfaces malformed input rather than dropping silently. - `processing_duration_us` — latency budget. - `cache_key_emitted` + `ccr_skipped_reason: Option<String>` — explicit signal for "we chose not to emit CCR and this is why". A `tracing::info!(target: "diff_compressor", ...)` event is emitted on every call, carrying these fields for OTel scraping in prod. The sidecar struct is returned alongside via `compress_with_stats`; the parity-only `compress` API discards it. # Module layout - `crates/headroom-core/src/transforms/mod.rs` — namespace, doc comment with the guiding principle ("information preservation > aggressive compression") so future ports inherit the philosophy. - `crates/headroom-core/src/transforms/diff_compressor.rs` — full port (parser, scorer, hunk selector, context trimmer, formatter, CCR layer, stats, tracing). # Dependencies added to headroom-core - `md-5 = "0.10"` — for the CCR cache_key (matches Python MD5[:24]). - `regex = "1"` — was a transitive dep via tokenizers; now a direct dependency for the hunk-header parser and priority patterns. # Tests 6 unit tests covering pass-through paths, MD5 hex truncation, the Python `split("\n")` line-count semantics, sidecar stats emission, and a synthetic 8-file diff that locks the byte-equal behavior found in the parity fixtures.
2026-04-25 15:44:25 -07:00
"regex",
fix: B7 — CCR hardening: persistent backends + always-on tool P2-25, P2-26: CCR (Compress-Cache-Retrieve) used an in-memory store that fragmented across uvicorn workers and was wiped on restart, and the `headroom_retrieve` tool was registered/unregistered per-request based on whether the latest body happened to contain compression markers — every flip busted the prompt cache. Both are sticky side-channels: once a session has done CCR, the tool list bytes and the retrieval store must stay stable. This PR fixes both. Rust: * Split `ccr.rs` into `ccr/` with `backends/` submodule (`in_memory.rs`, `sqlite.rs`, `redis.rs` cfg-gated). * `SqliteCcrStore` (production default): WAL mode, prepared upsert, lazy TTL purge on read, persistent across worker restarts and shareable across workers on the same host via SQLite file locking. * `RedisCcrStore` (cfg-gated behind `feature = "redis"`): SETEX with startup PING smoke-test, no key-prefix collision risk, no sticky session required at the LB. * `CcrBackendConfig::{InMemory, Sqlite, Redis}` + `from_config(...)` factory — every init failure surfaces (no silent fallback per `feedback_no_silent_fallbacks.md`). * `ccr::compute_key` (BLAKE3 → first 24 hex chars) and `ccr::marker_for("HASH") -> "<<ccr:HASH>>"` centralize the hash + marker format; one definition for the live-zone dispatcher and the Python regex (`headroom/ccr/tool_injection.py:211`). * `compress_anthropic_live_zone_with_ccr` accepts `Option<&dyn CcrStore>`. When wired, every accepted compression puts the original bytes into the backend and appends `<<ccr:HASH>>` to the compressed string. The token-validation gate runs on the marker-augmented string so the `compressed_tokens >= original_tokens` rejection stays honest. Python: * `SessionCcrTracker` + `apply_session_sticky_ccr_tool` mirror the PR-A7 `SessionToolTracker` / `apply_session_sticky_memory_tools` pattern: once a session has done CCR, every subsequent request injects the recorded golden tool-definition bytes. Tool list bytes are byte-stable across turns (snapshot test pins them). * `headroom/ccr/tool_injection.py::inject_tool_definition` accepts a new `session_has_done_ccr` kwarg per the PR-B7 spec change at line 302-328. The legacy per-request path stays intact for callers that don't yet thread a session id (e.g. Google handler). * Anthropic + OpenAI handlers route their CCR tool-list updates through `apply_session_sticky_ccr_tool`, keyed off the existing `session_tracker_store.compute_session_id(...)` plumbing. Backend selection model: `CcrBackendConfig::Sqlite { path }` is the production default — single host, persistent, multi-worker safe with sticky session. `CcrBackendConfig::Redis { url }` is the multi-host scale-out option — no stickiness needed. `InMemory` is for tests and single-worker dev only. RUST_DEV.md "Multi-worker deployment — CCR fragmentation" rewritten around this matrix. Tests: * `crates/headroom-core/tests/ccr_backends.rs` — 7 tests covering SQLite round-trip, TTL purge, proxy-restart survival, cross-backend byte-equal keys, `from_config` paths, and the no-redis-feature loud-failure check (+ 2 redis tests gated behind the feature). * `crates/headroom-core/tests/live_zone_ccr.rs` — confirms `<<ccr:HASH>>` marker injection, store population, and no-marker-when-no-store invariants end-to-end. * `tests/test_ccr_tool_always_on.py` — 12 tests pinning the always-on behaviour, session/provider isolation, LRU bound, no- session-id fallback, and (per-acceptance-criterion) the byte-stable tool-definition snapshot. Per-PR-B7 plan: REALIGNMENT/04-phase-B-live-zone.md.
2026-05-02 16:49:08 -07:00
"rusqlite",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"serde",
"serde_json",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"sha2 0.10.9",
fix: B7 — CCR hardening: persistent backends + always-on tool P2-25, P2-26: CCR (Compress-Cache-Retrieve) used an in-memory store that fragmented across uvicorn workers and was wiped on restart, and the `headroom_retrieve` tool was registered/unregistered per-request based on whether the latest body happened to contain compression markers — every flip busted the prompt cache. Both are sticky side-channels: once a session has done CCR, the tool list bytes and the retrieval store must stay stable. This PR fixes both. Rust: * Split `ccr.rs` into `ccr/` with `backends/` submodule (`in_memory.rs`, `sqlite.rs`, `redis.rs` cfg-gated). * `SqliteCcrStore` (production default): WAL mode, prepared upsert, lazy TTL purge on read, persistent across worker restarts and shareable across workers on the same host via SQLite file locking. * `RedisCcrStore` (cfg-gated behind `feature = "redis"`): SETEX with startup PING smoke-test, no key-prefix collision risk, no sticky session required at the LB. * `CcrBackendConfig::{InMemory, Sqlite, Redis}` + `from_config(...)` factory — every init failure surfaces (no silent fallback per `feedback_no_silent_fallbacks.md`). * `ccr::compute_key` (BLAKE3 → first 24 hex chars) and `ccr::marker_for("HASH") -> "<<ccr:HASH>>"` centralize the hash + marker format; one definition for the live-zone dispatcher and the Python regex (`headroom/ccr/tool_injection.py:211`). * `compress_anthropic_live_zone_with_ccr` accepts `Option<&dyn CcrStore>`. When wired, every accepted compression puts the original bytes into the backend and appends `<<ccr:HASH>>` to the compressed string. The token-validation gate runs on the marker-augmented string so the `compressed_tokens >= original_tokens` rejection stays honest. Python: * `SessionCcrTracker` + `apply_session_sticky_ccr_tool` mirror the PR-A7 `SessionToolTracker` / `apply_session_sticky_memory_tools` pattern: once a session has done CCR, every subsequent request injects the recorded golden tool-definition bytes. Tool list bytes are byte-stable across turns (snapshot test pins them). * `headroom/ccr/tool_injection.py::inject_tool_definition` accepts a new `session_has_done_ccr` kwarg per the PR-B7 spec change at line 302-328. The legacy per-request path stays intact for callers that don't yet thread a session id (e.g. Google handler). * Anthropic + OpenAI handlers route their CCR tool-list updates through `apply_session_sticky_ccr_tool`, keyed off the existing `session_tracker_store.compute_session_id(...)` plumbing. Backend selection model: `CcrBackendConfig::Sqlite { path }` is the production default — single host, persistent, multi-worker safe with sticky session. `CcrBackendConfig::Redis { url }` is the multi-host scale-out option — no stickiness needed. `InMemory` is for tests and single-worker dev only. RUST_DEV.md "Multi-worker deployment — CCR fragmentation" rewritten around this matrix. Tests: * `crates/headroom-core/tests/ccr_backends.rs` — 7 tests covering SQLite round-trip, TTL purge, proxy-restart survival, cross-backend byte-equal keys, `from_config` paths, and the no-redis-feature loud-failure check (+ 2 redis tests gated behind the feature). * `crates/headroom-core/tests/live_zone_ccr.rs` — confirms `<<ccr:HASH>>` marker injection, store population, and no-marker-when-no-store invariants end-to-end. * `tests/test_ccr_tool_always_on.py` — 12 tests pinning the always-on behaviour, session/provider isolation, LRU bound, no- session-id fallback, and (per-acceptance-criterion) the byte-stable tool-definition snapshot. Per-PR-B7 plan: REALIGNMENT/04-phase-B-live-zone.md.
2026-05-02 16:49:08 -07:00
"tempfile",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"thiserror 1.0.69",
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
"tiktoken-rs",
fix(rust): audit cleanup — DiffCompressor CCR leak, CCR TOCTOU race, clippy debt, dep dedup Closes findings from the post-Phase-3g audit. Five surgical fixes plus telemetry-discoverability docs. PyO3 0.22 → 0.24 security upgrade is its own PR (issue #335). 1. DiffCompressor cache_key persistence (production bug) --------------------------------------------------------- Pre-fix: `RustDiffCompressor.compress()` minted a `cache_key`, embedded `[... hash=abc123]` in the wire marker, and returned without storing the original anywhere. Python ContentRouter then returned the compressed text with a dangling marker — every retrieval tool call from the LLM 404'd. Sibling compressors (LogCompressor, SearchCompressor) already had the right pattern: Rust mints the key, Python's `_persist_to_python_ccr` writes the original to the production `CompressionStore`. DiffCompressor was the asymmetric one. Fix: - Rust: add `DiffCompressor::compress_with_store(content, context, Option<&dyn CcrStore>)` mirroring siblings. Calls `store.put` when a key is minted; legacy `compress()` and `compress_with_stats()` delegate with `None` for parity. - Python: add `_persist_to_python_ccr` helper to `headroom/transforms/diff_compressor.py.compress()` mirroring `log_compressor.py` and `search_compressor.py`. - Pipeline `DiffOffload`: switch to `compress_with_store(Some(store))` and drop the post-hoc double-store hack that papered over this bug at the orchestrator boundary. 2. CCR store TOCTOU race in `get()` ----------------------------------- `InMemoryCcrStore::get()` checked TTL under a read lock, dropped the lock, then called `remove()`. Between drop and remove a concurrent `put()` of the same hash with fresh data could land — and our `remove` would then wipe that fresh entry. Under multi-worker proxy load this manifested as "I just stored it; why is it gone?" Fix: use `DashMap::remove_if`. Predicate runs under the shard write lock so check-and-remove is atomic. New regression test exercises a tight contention loop between writer and reader on the same key. 3. Pre-existing clippy debt in smart_crusher -------------------------------------------- - 3× `field_reassign_with_default` in `crusher.rs` test setup — switch to struct-update syntax `Config { field: x, ..Default }`. - `hash_array_for_ccr` was `#[cfg(test)]` but unused; deleted with a comment so a future test can reintroduce it as a one-liner. `cargo clippy --workspace --all-targets -- -D warnings` is now clean across the whole workspace; previous CI patches that allowed these warnings can be removed in a follow-up. 4. Tokenizers dependency dedup ------------------------------ `tokenizers 0.21` (direct dep) + `tokenizers 0.22` (transitive via fastembed) compiled twice into the binary. Bumped direct dep to `0.22` to align; API is compatible (verified by full tokenizer test suite). Saves compile time + binary bloat. 5. Telemetry-discoverability doc (no new code) ---------------------------------------------- The audit recommended a per-transform invocation counter to inform the next Python → Rust port. Discovered the infrastructure already exists at `/stats`: - `compressions_by_strategy` — invocation count per strategy - `pipeline_timing` — count + avg/max ms per transform name - `tokens_saved_by_strategy` — savings attribution Added a section to `RUST_DEV.md` showing the `curl + jq` recipes to read this data, with example output highlighting how to spot zero-invocation deferral candidates (e.g. `code_compressor`). Verification: workspace tests 734 + 14 + 5 + 4 + 6 + 5 + 2 + 2 + 3 + 4 + 2 + 2 + 1 = all green; cargo fmt clean; cargo clippy --all-targets clean; Python tests 185 pass; commitlint clean.
2026-04-30 20:54:22 -07:00
"tokenizers",
fix(rust): reformat/offload pipeline + log templates + diff noise (Phase 3g rework) Replaces PR1's lossless/lossy split with ReformatTransform (pack denser, no info lost) and OffloadTransform (drop bytes, CCR-stash original via required cache_key). With CCR every transform is information-preserving end-to-end, so the lossless/lossy distinction misnamed the architecture. OffloadTransform carries a cheap, structural estimate_bloat() method scoped to its domain — generic byte-redundancy heuristics miss domain semantics. The orchestrator runs reformat phase + per-offload bloat estimation in parallel via rayon::join + par_iter, then runs offload iff bloat clears threshold OR reformat underwhelmed. Transforms shipped: REFORMATS (lossless): - JsonMinifier: serde_json round-trip whitespace stripping. - LogTemplate: Drain-inspired order-preserving template miner. Collapses consecutive runs of same-template lines into [Template Tn: ...] (Nx) + variant table. Win comes from emitting the constant-token prefix once instead of N times. Lossless: every original line reconstructible from template + variants. OFFLOADS (drop bytes, stash original via CCR): - LogOffload: wraps existing LogCompressor; bloat = repetition x uniqueness_weight + dilution x priority_dilution_weight. - DiffOffload: wraps existing DiffCompressor; bloat = context-to- change ratio. Bug-fix-on-port — persists original under the cache_key the parity-bound DiffCompressor mints (closes a leak). - DiffNoise: drops lockfile hunks (Cargo.lock, package-lock.json, yarn.lock, etc., suffix list configurable in TOML) and whitespace-only hunks. Stashes original via CCR for retrieval. Search offload exists but is not in default re-exports — modern agents (Claude Code, Codex) use scoped rg/grep, the marginal value didn't justify default registration. Reach via the explicit module path if opting in. JSON Offload is intentionally absent from this PR — already lives at SmartCrusher; Phase 3g PR3 wraps it in the OffloadTransform contract. Thresholds and weights live in config/pipeline.toml, embedded via include_str!; PipelineConfig::from_toml_str loads runtime overrides. 98 new pipeline tests; full headroom-core suite (714) and workspace tests green; cargo fmt clean. No regex, per project convention.
2026-04-30 11:02:29 -07:00
"toml",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"tracing",
chore(rust): port unidiff Tier 2 diff detector (Stage 3d PR4) Adds the second tier of the Stage-3d ContentRouter detection arch. Magika (PR3) is a probabilistic ML classifier — short, prose-prefixed, or "looks like code because the lines are code" diffs can slip past it into PlainText. PR4 catches those by running the [`unidiff`] parser as a deterministic oracle: anything that parses to ≥1 PatchedFile with ≥1 hunk is a diff. What lands: - `crates/headroom-core/src/transforms/unidiff_detector.rs`: - `is_diff(content) -> bool`: predicate. - `detect_diff(content) -> Option<ContentType>`: typed wrapper for the router (PR5) to chain after Magika. - Empty input shortcuts to false without invoking the parser. - "Found zero hunks" is treated as **not** a diff — `unidiff::PatchSet ::parse` returns Ok(()) on plain text (just finds zero files); we explicitly require non-empty patch + non-empty hunk to avoid silently routing prose through the diff compressor. - 14 unit tests: standard git diff, naked hunk without git header, multi-file, added/removed-only files, JSON/HTML/YAML/source/prose negatives, "almost looks like a diff" prose with @@/--- in passing, truncated-diff canary. Known gaps (deliberately punted to PR5+): - Combined-merge headers (`@@@ ... @@@`) — `unidiff`'s hunk regex is for plain `@@`. Rare in proxy traffic; PR5 router can fall back to the regex content_detector if needed. - Pathological CRLF-stripped inputs — `input.lines()` strips `\r` only when paired with `\n`. Acceptable. What does NOT land here (per PR scope): - No PyO3 surface — module-only. - No router rewiring — the existing regex `content_detector` still drives `ContentRouter`. PR5 chains magika → unidiff → PlainText. The `unidiff` crate brings `regex` (already in tree) and `encoding_rs` (default features) — small dep impact. `make ci-precheck` green.
2026-04-28 23:00:11 -07:00
"unidiff",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
]
[[package]]
name = "headroom-parity"
version = "0.1.0"
dependencies = [
"anyhow",
"clap",
"headroom-core",
"serde",
"serde_json",
"thiserror 1.0.69",
]
[[package]]
name = "headroom-proxy"
version = "0.1.0"
dependencies = [
fix(proxy): PR-D4 native Vertex publisher path + ADC bearer auth Adds a Rust-native Vertex AI publisher route ahead of the LiteLLM Python converter (which dropped `thinking`, `redacted_thinking`, `document`, `image`, `server_tool_use`, `mcp_tool_use` block kinds — the P4-37 / P4-38 bug). After this PR the Vertex `:rawPredict` and `:streamRawPredict` calls survive byte-equal upstream and benefit from the live-zone Anthropic dispatcher (PR-B-series) running over the body — same behaviour as `/v1/messages`. New module `crates/headroom-proxy/src/vertex/`: - `mod.rs` — single dispatch handler at the `/v1beta1/.../models/:model_action` route. Splits the trailing `:<verb>` segment with `str::rsplit_once(':')` (no regex) and flips an `attach_sse_tee` flag to dispatch to the streaming or non-streaming arm. Both verbs share one axum route shape because matchit can't distinguish two patterns that overlap on a parameter. - `envelope.rs` — `VertexEnvelope` parser. Confirms `anthropic_version` present + `model` field absent (the two fingerprints of the Vertex envelope vs `/v1/messages`). - `adc.rs` — `TokenSource` trait + `GcpAdcTokenSource` (production, `gcp_auth` 0.12) + `StaticTokenSource` (tests). Caches tokens with a 60s refresh-ahead-of-expiry window. Emits structured `event = "vertex_adc_token_refreshed"` per refresh. - `raw_predict.rs` — POST handler + shared `forward_vertex_request`. Buffers body, parses envelope, runs live-zone Anthropic compression, fetches ADC bearer, attaches `Authorization: Bearer <token>` (overwrites client-supplied Authorization header), forwards. SSE telemetry tee for the streaming verb reuses PR-C1's `AnthropicStreamState` directly (Vertex streams plain SSE, unlike Bedrock's binary EventStream). - `stream_raw_predict.rs` — module-level docs + alias to the shared dispatcher (the streaming-vs-non-streaming difference is one boolean flag inside the shared forwarder). Modifications: - `proxy.rs::build_app` — registers the single Vertex route. - `proxy.rs::AppState` — new `vertex_token_source: Arc<dyn TokenSource>` field. Production constructs `GcpAdcTokenSource` lazily (no GCP call until first `bearer()`); tests inject `StaticTokenSource` via the new `AppState::with_token_source` helper. - `config.rs` — adds `--vertex-region` / `HEADROOM_PROXY_VERTEX_REGION` (default `us-central1`, observability tag only — the upstream URL is `--upstream`) and `--vertex-adc-scope` / `HEADROOM_PROXY_VERTEX_ADC_SCOPE` (default `cloud-platform`). - `Cargo.toml` (workspace + proxy) — adds `gcp_auth = "0.12"` and `async-trait = "0.1"`. - `tests/common/mod.rs` — `start_proxy_with_state` accepts both config + state customizers; `install_static_token_source` helper for tests. `crates/headroom-proxy/tests/integration_vertex_raw_predict.rs` — all five tests pass: 1. `native_envelope_round_trip_byte_equal` — Vertex-shape body (with `anthropic_version`, no `model`) round-trips SHA-256 byte-equal upstream. 2. `adc_bearer_token_signed_correctly` — `Authorization: Bearer <static-test-token>` reaches upstream verbatim and OVERWRITES a client-supplied Authorization header. 3. `thinking_block_preserved` — request with `thinking` (incl. signature) + `redacted_thinking` (incl. opaque `data`) blocks round-trips byte-equal even with `LiveZone` compression mode enabled. This is the P4-37 / P4-38 teeth. 4. `stream_raw_predict_sse_handled` — `:streamRawPredict` proxies an Anthropic SSE response (full `message_start` → `content_block_delta` → `message_stop` sequence) back to the client without corruption; SSE content-type preserved end-to-end; bearer attached. 5. (bonus, no-silent-fallback contract) `adc_failure_returns_5xx_no_silent_forward` — when the token source returns `Err`, the proxy returns 5xx and never reaches upstream. Verifies the `event = "vertex_adc_fetch_failed"` error path. Workspace: `cargo test --workspace` green; `cargo clippy --workspace -- -D warnings` clean; `make ci-precheck` passes. - No silent fallbacks: ADC failure → structured 5xx, never an unauthenticated forward. - No hardcodes: every knob (region, ADC scope, upstream URL) is CLI-flag + env-var configurable. - No regexes: axum path parameters + `str::rsplit_once` only. - Comprehensive structured logs: `event` field on every decision point — `vertex_envelope_parsed`, `vertex_envelope_invalid`, `vertex_compression_skipped`, `vertex_compression_applied`, `vertex_adc_token_refreshed`, `vertex_adc_fetch_failed`, `vertex_streaming_pipeline_active`, `vertex_sse_stream_closed`, `vertex_forwarded`, `vertex_unknown_verb`, etc. - Performant: no body clone; ADC token cached + refreshed ahead-of-expiry, not fetched per request. - Comprehensive tests: realistic Anthropic block content (signature payload, redacted_thinking opaque blob) in `thinking_block_preserved`. The local `gcloud auth application-default print-access-token` returns no credentials, so manual validation against a real Vertex endpoint is not possible in this PR. Follow-up: the user runs `gcloud auth application-default login` once and exercises a live Vertex request — should be a no-code-change check. PR-D1 (Bedrock native) is running concurrently and will land its own envelope module at `crates/headroom-proxy/src/bedrock/envelope.rs`. The two envelope modules are intentionally siblings (not a shared trait) — the shapes differ (Bedrock has a different `anthropic_version` value, no `model` field, AWS SigV4 instead of GCP ADC), and a premature shared abstraction would obscure the provider-specific contracts. Whichever PR merges second rebases without conflict. Retires P4-38 (and the Vertex parts of P4-39); marketplace BYOC pitch (per project memory) gets one more native provider.
2026-05-03 16:22:07 -07:00
"async-trait",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"aws-config",
"aws-credential-types",
"aws-sigv4",
"aws-smithy-runtime-api",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"axum",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"bytes",
"bytesize",
"clap",
fix(proxy): PR-D2 Bedrock streaming via binary EventStream Add the Phase D PR-D2 streaming counterpart to PR-D1's native Bedrock InvokeModel route. Bedrock's `/model/{id}/invoke-with-response-stream` returns `application/vnd.amazon.eventstream` — a binary, length-prefixed, CRC32-checksummed framing format. This PR adds an incremental parser, an SSE translator, and the streaming POST handler. Components: - `bedrock/eventstream.rs` — stateful incremental EventStream parser. Validates prelude + message CRC32 (configurable via `--bedrock-validate-eventstream-crc`, default on). Returns structured `ParseError` on every malformed-bytes path; never panics. Supports all 10 AWS header value types; bytes-typed values surfaced via `HeaderValue::Bytes`, strings via `HeaderValue::String`. - `bedrock/eventstream_to_sse.rs` — translator. Picks output mode per `Accept` header: `application/vnd.amazon.eventstream` → byte-equal passthrough; everything else (default) → SSE translation. Each `chunk` payload becomes a canonical Anthropic `event: <type>\ndata: <json>\n\n` SSE frame so existing `AnthropicStreamState` telemetry runs unchanged. - `bedrock/invoke_streaming.rs` — POST handler. Reuses D1's `BedrockEnvelope`, live-zone compression, SigV4 signing. Tees translated SSE frames into `AnthropicStreamState` via the same bounded-mpsc tee pattern as `/v1/messages` — byte path never blocks on parser readiness. Config: - New `--bedrock-validate-eventstream-crc` / env `HEADROOM_PROXY_BEDROCK_VALIDATE_EVENTSTREAM_CRC` flag, default on. Disabling logs a warn at app-build time. Routing: - `proxy.rs::build_app` mounts `POST /model/:model_id/invoke-with-response-stream` only when `enable_bedrock_native` is on (matches D1). Failure modes (all loud; no silent fallbacks): - CRC mismatch → `event=bedrock_eventstream_crc_mismatch` warn, closes the stream with an SSE error frame. - Parse error → `event=bedrock_eventstream_parse_failed` warn + SSE error frame. - `:message-type == exception` → `event=bedrock_eventstream_upstream_exception` warn + SSE error frame. - Unknown `:event-type` → `event=bedrock_eventstream_unknown_event_type` warn, skipped. - Missing creds / SigV4 fail → 5xx, identical to D1. Tests added (12 total): - 4 parser unit-style integration: byte-equal round trip, drip-feed one-byte-at-a-time, CRC mismatch surfaces structured error, validation-off accepts corrupt. - 3 end-to-end: `eventstream_translated_to_sse`, `usage_extracted_from_translated_stream`, `client_can_choose_eventstream_or_sse`. - 2 property tests via `proptest`: random bytes never panic the parser (1024 cases each: bulk + drip-feed). - 3 trivial smoke tests in unit modules (`eventstream::tests::*`, `eventstream_to_sse::tests::*`). Manual cloud validation: - Not exercised — running `aws bedrock-runtime invoke-model-with- response-stream` against the proxy in the sandbox would require AWS API access this environment does not have. The wiremock- served binary EventStream + property tests cover the parser semantics and CRC validation rigorously. Stacked on PR-D1 (#364). Will be rebased onto main once D1 lands.
2026-05-03 16:48:28 -07:00
"crc32fast",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"futures",
"futures-util",
fix(proxy): PR-D4 native Vertex publisher path + ADC bearer auth Adds a Rust-native Vertex AI publisher route ahead of the LiteLLM Python converter (which dropped `thinking`, `redacted_thinking`, `document`, `image`, `server_tool_use`, `mcp_tool_use` block kinds — the P4-37 / P4-38 bug). After this PR the Vertex `:rawPredict` and `:streamRawPredict` calls survive byte-equal upstream and benefit from the live-zone Anthropic dispatcher (PR-B-series) running over the body — same behaviour as `/v1/messages`. New module `crates/headroom-proxy/src/vertex/`: - `mod.rs` — single dispatch handler at the `/v1beta1/.../models/:model_action` route. Splits the trailing `:<verb>` segment with `str::rsplit_once(':')` (no regex) and flips an `attach_sse_tee` flag to dispatch to the streaming or non-streaming arm. Both verbs share one axum route shape because matchit can't distinguish two patterns that overlap on a parameter. - `envelope.rs` — `VertexEnvelope` parser. Confirms `anthropic_version` present + `model` field absent (the two fingerprints of the Vertex envelope vs `/v1/messages`). - `adc.rs` — `TokenSource` trait + `GcpAdcTokenSource` (production, `gcp_auth` 0.12) + `StaticTokenSource` (tests). Caches tokens with a 60s refresh-ahead-of-expiry window. Emits structured `event = "vertex_adc_token_refreshed"` per refresh. - `raw_predict.rs` — POST handler + shared `forward_vertex_request`. Buffers body, parses envelope, runs live-zone Anthropic compression, fetches ADC bearer, attaches `Authorization: Bearer <token>` (overwrites client-supplied Authorization header), forwards. SSE telemetry tee for the streaming verb reuses PR-C1's `AnthropicStreamState` directly (Vertex streams plain SSE, unlike Bedrock's binary EventStream). - `stream_raw_predict.rs` — module-level docs + alias to the shared dispatcher (the streaming-vs-non-streaming difference is one boolean flag inside the shared forwarder). Modifications: - `proxy.rs::build_app` — registers the single Vertex route. - `proxy.rs::AppState` — new `vertex_token_source: Arc<dyn TokenSource>` field. Production constructs `GcpAdcTokenSource` lazily (no GCP call until first `bearer()`); tests inject `StaticTokenSource` via the new `AppState::with_token_source` helper. - `config.rs` — adds `--vertex-region` / `HEADROOM_PROXY_VERTEX_REGION` (default `us-central1`, observability tag only — the upstream URL is `--upstream`) and `--vertex-adc-scope` / `HEADROOM_PROXY_VERTEX_ADC_SCOPE` (default `cloud-platform`). - `Cargo.toml` (workspace + proxy) — adds `gcp_auth = "0.12"` and `async-trait = "0.1"`. - `tests/common/mod.rs` — `start_proxy_with_state` accepts both config + state customizers; `install_static_token_source` helper for tests. `crates/headroom-proxy/tests/integration_vertex_raw_predict.rs` — all five tests pass: 1. `native_envelope_round_trip_byte_equal` — Vertex-shape body (with `anthropic_version`, no `model`) round-trips SHA-256 byte-equal upstream. 2. `adc_bearer_token_signed_correctly` — `Authorization: Bearer <static-test-token>` reaches upstream verbatim and OVERWRITES a client-supplied Authorization header. 3. `thinking_block_preserved` — request with `thinking` (incl. signature) + `redacted_thinking` (incl. opaque `data`) blocks round-trips byte-equal even with `LiveZone` compression mode enabled. This is the P4-37 / P4-38 teeth. 4. `stream_raw_predict_sse_handled` — `:streamRawPredict` proxies an Anthropic SSE response (full `message_start` → `content_block_delta` → `message_stop` sequence) back to the client without corruption; SSE content-type preserved end-to-end; bearer attached. 5. (bonus, no-silent-fallback contract) `adc_failure_returns_5xx_no_silent_forward` — when the token source returns `Err`, the proxy returns 5xx and never reaches upstream. Verifies the `event = "vertex_adc_fetch_failed"` error path. Workspace: `cargo test --workspace` green; `cargo clippy --workspace -- -D warnings` clean; `make ci-precheck` passes. - No silent fallbacks: ADC failure → structured 5xx, never an unauthenticated forward. - No hardcodes: every knob (region, ADC scope, upstream URL) is CLI-flag + env-var configurable. - No regexes: axum path parameters + `str::rsplit_once` only. - Comprehensive structured logs: `event` field on every decision point — `vertex_envelope_parsed`, `vertex_envelope_invalid`, `vertex_compression_skipped`, `vertex_compression_applied`, `vertex_adc_token_refreshed`, `vertex_adc_fetch_failed`, `vertex_streaming_pipeline_active`, `vertex_sse_stream_closed`, `vertex_forwarded`, `vertex_unknown_verb`, etc. - Performant: no body clone; ADC token cached + refreshed ahead-of-expiry, not fetched per request. - Comprehensive tests: realistic Anthropic block content (signature payload, redacted_thinking opaque blob) in `thinking_block_preserved`. The local `gcloud auth application-default print-access-token` returns no credentials, so manual validation against a real Vertex endpoint is not possible in this PR. Follow-up: the user runs `gcloud auth application-default login` once and exercises a live Vertex request — should be a no-code-change check. PR-D1 (Bedrock native) is running concurrently and will land its own envelope module at `crates/headroom-proxy/src/bedrock/envelope.rs`. The two envelope modules are intentionally siblings (not a shared trait) — the shapes differ (Bedrock has a different `anthropic_version` value, no `model` field, AWS SigV4 instead of GCP ADC), and a premature shared abstraction would obscure the provider-specific contracts. Whichever PR merges second rebases without conflict. Retires P4-38 (and the Vertex parts of P4-39); marketplace BYOC pitch (per project memory) gets one more native provider.
2026-05-03 16:22:07 -07:00
"gcp_auth",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"headroom-core",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"http-body-util",
"humantime",
"hyper",
"hyper-util",
fix: PR-E6 cache-bust drift detector telemetry (Phase E) Per-session SHA-256 fingerprint of the cache hot zone (system / tools / first 3 messages) with structured-log emission on drift. Detector is read-only: never mutates request bytes, preserves the Phase A passthrough invariant. Surfaces invisible cache busts (system prompt edited mid-session, tools reshuffled, early message changed) without rewriting them. * crates/headroom-proxy/src/cache_stabilization/drift_detector.rs: StructuralHash (system, tools, early_messages digests), compute_structural_hash, observe_drift, derive_session_key, DriftState (LRU bounded to 1000 sessions in production). * Session keys derive from Authorization / x-api-key / client IP / (IP, user-agent). Bearer tokens and API keys are SHA-256 hashed before they ever reach the log line; the raw secret is never logged. * Wired into forward_http after the body is buffered, before the compression dispatcher runs. Skips paths whose wire shape is not Anthropic / OpenAI Chat / OpenAI Responses. * AppState gains drift_state: DriftState. Bedrock unit-test literal-construction sites updated. * 14 unit tests + 1 integration test covering first-request, no-drift, per-dimension drift, multi-dim drift, LRU eviction, non-mutation invariant, and bearer-token-never-logged. Adds lru = "0.12" and promotes sha2 = "0.10" to a normal dependency on headroom-proxy.
2026-05-04 12:42:32 -07:00
"lru",
fix: PR-E1 tool array deterministic sort (Phase E) Sort `tools[]` alphabetically by name on the way out so cache hits no longer depend on the customer-side iteration order (commonly hash- randomized via `set()` / `dict`). Mutates request bytes only when: 1. Auth mode is PAYG (`headroom_core::auth_mode::classify`). 2. No tool already carries a `cache_control` marker (reordering would shift cache scope and silently void customer intent). Every gate skip emits a structured `e1_skipped` event with `reason = auth_mode | marker_present` so dashboards can see policy adoption. Wired into all three live-zone walkers — Anthropic `/v1/messages`, OpenAI `/v1/chat/completions`, OpenAI `/v1/responses` — plus the Bedrock invoke + invoke-streaming entry points. Each passes `auth_mode` (already pre-classified by Phase F PR-F1 middleware) into the dispatcher so the gate evaluates without re-classifying. Sort key uses `tool["name"]` (Anthropic) or `tool["function"]["name"]` (OpenAI). Unnamed tools (rare; malformed inputs only) fall back to MD5 of canonical-JSON serialization for a stable in-process key — collision odds are astronomically small and `Vec::sort_by` is stable. Tests: unit tests for sort + marker detection + idempotency + the permutation property; integration tests boot the real proxy in front of a wiremock upstream and assert PAYG -> sorted, OAuth/Subscription/ marker -> byte-equal passthrough (SHA-256).
2026-05-04 15:27:54 -07:00
"md-5",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"pin-project-lite",
fix(proxy): PR-D3 Bedrock observability + auth-mode integration Phase D close. Adds the operator-facing observability surface that PRs D1 (native invoke) and D2 (streaming EventStream) deferred, and wires the Phase F PR-F1 auth-mode classifier into the Bedrock route so downstream cache/compression policy gates have something to read. Changes ------- * New `bedrock::auth_mode_layer` middleware. Classifies every inbound Bedrock request via F1's `classify`, coerces the result to `AuthMode::OAuth` per the Bedrock policy matrix (SigV4 IAM is OAuth-equivalent), and stores the resolved value in `request.extensions()` so PR-F2/F3 can read it without re-classifying. Mismatches are logged at WARN with `event=bedrock_auth_mode_unexpected` — no silent coercion. * New `observability` module with three Prometheus families: - `bedrock_invoke_count_total{model, region, auth_mode}` (counter) - `bedrock_invoke_latency_seconds{model, region}` (histogram) - `bedrock_eventstream_message_count_total{model, region, event_type}` (counter) Registered lazily via `OnceLock` so per-request work is just `inc_with_label_values` / `observe`. Latency observed via an RAII `LatencyGuard` so every error path is instrumented; a future regression that adds a new return path can't drop the observation. * New `GET /metrics` endpoint serves the registry in Prometheus text format. Mounted unconditionally — no feature flag gate — so scrape works regardless of which provider routes are mounted. * Bedrock invoke + invoke-streaming handlers now extract `Extension<AuthMode>`, log it in their entry breadcrumbs (`event=bedrock_invoke_received`, `event=bedrock_invoke_streaming_received`), and pass `model`/`region` into `translate_stream` so per-message metrics carry the right labels. * Operator docs at `docs/bedrock.md`: AWS credential chain, region/endpoint config, supported model IDs (`anthropic.*` literal-match — no regexes), compression behaviour, sample PromQL queries, structured-log correlation, rollback path. Tests added (6, all green) -------------------------- Auth-mode (`integration_bedrock_authmode.rs`): 1. `bedrock_classified_as_oauth` — empty headers → OAuth in extensions. 2. `oauth_policy_passthrough_prefer` — body byte-equal upstream; no auto cache_control / prompt_cache_key injected. Metrics (`integration_bedrock_metrics.rs`): 3. `metrics_increment_per_invoke` — 3 invokes → counter=3 with correct labels. 4. `metrics_observe_latency` — 1 invoke → histogram count=1, sum>0. 5. `eventstream_metrics_per_message_type` — 5 chunks → counter=5 with `event_type=chunk`. 6. `metrics_endpoint_serves_scrape` — `/metrics` returns 200, `text/plain`, all three metric families' HELP/TYPE lines present. Each metrics test owns a unique (model, region) tuple so the global `prometheus` registry — shared across parallel tests in the same binary — gives each test isolated label rows. Without isolation, parallel tests cross-contaminate counters. Constraints honoured -------------------- * No silent fallbacks — auth-mode coercion is logged at WARN. * No hardcodes — region from `--bedrock-region`, model from axum path parameter. * No regexes — vendor prefix is literal `anthropic.`. * Comprehensive structured logs — every metric increment paired with `tracing::debug!` carrying the same labels for incident correlation. * Performant — `OnceLock`-cached descriptors, RAII guard, total D3 overhead well under 1us per request. * Cardinality bounded — labels driven by config + bounded enums, never by user-controlled bytes. Live cloud validation deferred ------------------------------ The wiremock-backed integration tests are the canonical correctness gate for D3. A real Bedrock smoke test requires `bedrock:InvokeModel` permissions in the developer's AWS account and is documented in `docs/bedrock.md` — both D1 and D2 hit sandbox permission issues trying this path; D3 follows the same convention. Stacked on ---------- PR #364 (D1 native invoke), PR #365 (D2 streaming EventStream), PR #366 (F1 classifier helper). Merge those first; this PR will be rebased onto main once they land.
2026-05-03 17:57:00 -07:00
"prometheus",
fix: C1 — byte-level SSE parser + state machines Foundation of Phase C. Delivers: * Byte-level SSE framing (bytes::Bytes / BytesMut) with UTF-8 decoded only at \n\n event boundaries — no per-chunk decode, no errors=ignore data loss across TCP reads. * Three provider state machines: - Anthropic: blocks keyed by index, all delta types (text/thinking/input_json/citations/signature) preserved byte-equal. - OpenAI Chat: ToolCallState concatenation, refusal field, include_usage final chunk handling. - OpenAI Responses: items keyed by id (not position) for out-of-order completion; full event coverage. * State machine runs in parallel with byte-passthrough via a tokio::spawn task fed by a bounded mpsc — clients see raw bytes immediately; telemetry populates without blocking. Retires P1-8, P1-9, P1-14, P1-15, P1-17, P4-48 in the Rust path (Python A8 hotfix preserved as fallback until Phase H). Per-PR-C1 plan: REALIGNMENT/05-phase-C-rust-proxy.md.
2026-05-02 21:12:08 -07:00
"proptest",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"reqwest",
"serde",
"serde_json",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"sha2 0.10.9",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"thiserror 1.0.69",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"tokio",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"tokio-stream",
"tokio-tungstenite",
docs(rust): lockfile + RUST_DEV.md for proxy CLI Cargo.lock: pick up tokio-util added in the WS half-close fix. RUST_DEV.md: document how to run headroom-proxy in passthrough mode (listen + upstream flags, e2e test gate, env vars).
2026-04-25 12:49:36 -07:00
"tokio-util",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"tower",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"tower-http",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"tracing",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"tracing-subscriber",
"url",
"uuid",
"wiremock",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
]
[[package]]
name = "headroom-py"
version = "0.1.0"
dependencies = [
fix: ship glibc 2.38 compat shim + wheel symbol audit (closes #355) Issue #355: published headroom_ai-*-manylinux_2_28_*.whl fails to import on Ubuntu 22.04, Debian 11/12, Conda envs with libc < 2.38: 'ImportError: undefined symbol: __isoc23_strtoll'. Root cause: ORT prebuilt artifacts (downloaded via fastembed's ort-download-binaries-rustls-tls feature) are compiled with gcc-14.2.1 on a glibc-2.38+ host and reference __isoc23_strtoll. Our manylinux build host has glibc 2.38 so the link succeeds; end users with older glibc don't. Two-part fix: (1) crates/headroom-py/glibc_compat.c provides weak-alias definitions for __isoc23_strtol/strtoll/strtoul/strtoull delegating to the older strtol* family, compiled by build.rs on Linux/glibc only. The dynamic linker prefers glibc's strong symbol when present (>= 2.38) and falls back to ours when not (< 2.38). (2) scripts/audit_wheel_glibc_symbols.py is a release.yml gate that runs objdump -T on every Linux wheel and rejects any UND symbol whose required glibc version exceeds the wheel's manylinux floor. Validated: the audit correctly rejects the actually-broken v0.20.26 wheel with a precise diagnostic. The shim itself is a tiny static link with zero runtime cost. Regression tests in tests/test_release_workflows.py pin the shim's load-bearing pieces (.c file, build.rs trigger, [build-dependencies] cc dep) and the audit invocation in release.yml. Future drift fails at PR time, not in the next release. This is the same bug class as PR #371 (rustls-everywhere). Each instance gets fixed; the audit gate now catches the *class* — any future static linkage that introduces a post-floor symbol gets blocked before publish.
2026-05-04 21:31:19 -07:00
"cc",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"headroom-core",
"pyo3",
Pin ORT dylib on Windows; init Python logging (#1010) ## Description On Windows, headroom's Rust core resolves `onnxruntime.dll` at runtime via `ort-load-dynamic`. Without an explicit `ORT_DYLIB_PATH`, the bare DLL search can land on `C:\Windows\System32\onnxruntime.dll`, the Windows ML OS component, and `Session::new()` can deadlock instead of returning an error. Since a hang is not an `Err`, the tiered fallback cannot engage until the proxy-level timeout fires. This PR pins `ORT_DYLIB_PATH` to the pip-installed `onnxruntime` DLL at import time, and wires Rust `tracing` events into Python logging so the proxy log surfaces these failures when they occur. Closes #928 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `headroom/_ort.py` with a Windows-only, idempotent `ensure_ort_dylib_pinned()` resolver that respects an existing `ORT_DYLIB_PATH`. - Call the pin from `headroom/__init__.py` before importing `_core` consumers. - Log the effective ORT dylib path from the content router startup path on Windows. - Enable Rust tracing-to-log compatibility and initialize `pyo3-log` in the `_core` module. - Add timeout diagnostics in the Magika detector with the effective `ORT_DYLIB_PATH`. - Document `ORT_DYLIB_PATH` and `HEADROOM_MAGIKA_INIT_TIMEOUT_SECS`. - Add unit coverage for the resolver behavior. ## Testing - [x] Unit tests pass (`python -m pytest tests/test_transforms/test_ort_dylib.py -q`) - [x] Linting passes (`ruff check headroom/_ort.py headroom/__init__.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py`) - [x] Formatting passes (`ruff format --check headroom/_ort.py headroom/__init__.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_transforms/test_ort_dylib.py -q 7 passed in 0.19s $ ruff check headroom/_ort.py headroom/__init__.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py All checks passed! $ ruff format --check headroom/_ort.py headroom/__init__.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py 4 files already formatted $ cargo check -p headroom-py cargo: The term 'cargo' is not recognized as a name of a cmdlet, function, script file, or executable program. ``` ## Real Behavior Proof - Environment: Windows 11 24H2, Python 3.13, RTX 4080 - Exact command / steps: `python -c "import headroom; from headroom._core import detect_content_type as d; print(d(open('headroom/compress.py').read()).content_type)"` - Observed result: `source_code` in 301ms, clean exit, `Magika: ENABLED` in proxy log - Not tested: macOS/Linux manual runtime behavior; `_ort.py` is a no-op outside Windows, and CI covers cross-platform build/test behavior. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable (N/A: repo uses release-please) ## Additional Notes The branch was rebased onto current `main` and the commit subject was updated to satisfy commitlint. Local Rust verification could not be run on this Windows machine because `cargo` is not installed; GitHub CI should be treated as the Rust build verification for the `pyo3-log` dependency and workspace lockfile changes. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-23 14:46:24 +02:00
"pyo3-log",
chore(rust): SmartCrusher CCR storage layer + roundtrip verification CcrStore trait + InMemoryCcrStore (1000 entries, 5-min TTL, FIFO eviction, idempotent re-store) live at the crate root. SmartCrusher's lossy crush_array path now actually stashes the full original [items] canonical-JSON into the configured store keyed by the same ccr_hash it embeds in the prompt marker -- closing the no-data-loss contract that was previously hash-only. PyO3 surface: - crusher.crush_array_json(items_json) -> dict with ccr_hash + kept items - crusher.ccr_get(hash) -> Optional[str] for retrieval - crusher.ccr_len() -> int for telemetry Python shim passes both through. Default constructors enable the store (matches Python's CCR-enabled default); without_compaction() also gets it because CCR is a contract, not an opt-in extra. Tests proving compress -> store -> retrieve -> reconstruct: - 7 unit tests in ccr.rs (put/get/eviction/expiry) - 9 Rust integration tests (crates/headroom-core/tests/ccr_roundtrip.rs) - 10 Python tests including 4 explicit before/after element-equality assertions through both the native PyO3 surface and the Python shim Plugin manifest versions auto-bumped by the sync-plugin-versions pre-commit hook (unrelated to CCR but co-resident in the working tree).
2026-04-27 19:36:14 -07:00
"serde_json",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
]
[[package]]
name = "heck"
version = "0.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "hermit-abi"
version = "0.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fc0fef456e4baa96da950455cd02c081ca953b141298e41db3fc7e36b1da849c"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
[[package]]
name = "hex"
version = "0.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70"
feat(rust): HfTokenizer::from_pretrained — HuggingFace Hub auto-download Stage 2.1: closes the loop on the HuggingFace tokenizer story. Stage 2 shipped `HfTokenizer::from_bytes`/`from_file`, which required callers to manage their own tokenizer.json files. This adds the third constructor: let t = HfTokenizer::from_pretrained("CohereForAI/c4ai-command-r-v01")?; register_hf("command-", t); `from_pretrained` is a thin wrapper around the `hf-hub` crate's blocking `ureq` API. First call downloads `tokenizer.json` to `~/.cache/huggingface/ hub` (or `$HF_HOME` if set); subsequent calls reuse the on-disk cache. Uses the `main` revision; gated repos (Llama, Mistral) require `HF_TOKEN` in env or `~/.cache/huggingface/token`. Also adds `try_register_hf(prefix, repo)` as the obvious one-liner for proxy startup code: let _ = try_register_hf("command-", "CohereForAI/c4ai-command-r-v01"); let _ = try_register_hf("mistral-", "mistralai/Mistral-7B-v0.1"); Each call is independent — a download failure for one model (e.g. gated without a token) does not affect others. `HfTokenizerError` gains a new `Hub` variant so callers can distinguish "couldn't fetch" from "fetched but malformed" — relevant when deciding whether to retry, surface to the user, or fall back to the estimator. Why blocking, not async: `from_pretrained` is called once at startup. A sync API works from `main()`, from a `OnceLock` initializer, or from `tokio::task::spawn_blocking` if a tokio caller needs it later. The async hf-hub backend would force callers to await at startup, which doesn't fit the `register_hf` registry pattern. Why rustls, not native-tls: keeps the binary statically linkable for AWS deploys (no system OpenSSL dependency). Tests: a network-dependent integration test (`#[ignore]`d in CI; hits HF for `gpt2`, ~1.4 MB) verifies the real download + load + count path. A non-network negative test verifies that an invalid repo name surfaces as `HfTokenizerError::Hub`, not a panic. 44 unit tests + 5 proptests + 1 doctest pass; parity stays 40/40 byte-equal.
2026-04-25 15:05:09 -07:00
[[package]]
name = "hf-hub"
version = "0.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "629d8f3bbeda9d148036d6b0de0a3ab947abd08ce90626327fc3547a49d59d97"
dependencies = [
"dirs",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
"indicatif 0.17.11",
feat(rust): HfTokenizer::from_pretrained — HuggingFace Hub auto-download Stage 2.1: closes the loop on the HuggingFace tokenizer story. Stage 2 shipped `HfTokenizer::from_bytes`/`from_file`, which required callers to manage their own tokenizer.json files. This adds the third constructor: let t = HfTokenizer::from_pretrained("CohereForAI/c4ai-command-r-v01")?; register_hf("command-", t); `from_pretrained` is a thin wrapper around the `hf-hub` crate's blocking `ureq` API. First call downloads `tokenizer.json` to `~/.cache/huggingface/ hub` (or `$HF_HOME` if set); subsequent calls reuse the on-disk cache. Uses the `main` revision; gated repos (Llama, Mistral) require `HF_TOKEN` in env or `~/.cache/huggingface/token`. Also adds `try_register_hf(prefix, repo)` as the obvious one-liner for proxy startup code: let _ = try_register_hf("command-", "CohereForAI/c4ai-command-r-v01"); let _ = try_register_hf("mistral-", "mistralai/Mistral-7B-v0.1"); Each call is independent — a download failure for one model (e.g. gated without a token) does not affect others. `HfTokenizerError` gains a new `Hub` variant so callers can distinguish "couldn't fetch" from "fetched but malformed" — relevant when deciding whether to retry, surface to the user, or fall back to the estimator. Why blocking, not async: `from_pretrained` is called once at startup. A sync API works from `main()`, from a `OnceLock` initializer, or from `tokio::task::spawn_blocking` if a tokio caller needs it later. The async hf-hub backend would force callers to await at startup, which doesn't fit the `register_hf` registry pattern. Why rustls, not native-tls: keeps the binary statically linkable for AWS deploys (no system OpenSSL dependency). Tests: a network-dependent integration test (`#[ignore]`d in CI; hits HF for `gpt2`, ~1.4 MB) verifies the real download + load + count path. A non-network negative test verifies that an invalid repo name surfaces as `HfTokenizerError::Hub`, not a panic. 44 unit tests + 5 proptests + 1 doctest pass; parity stays 40/40 byte-equal.
2026-04-25 15:05:09 -07:00
"libc",
"log",
"rand 0.9.4",
"reqwest",
"serde",
"serde_json",
"thiserror 2.0.18",
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
"ureq 2.12.1",
feat(rust): HfTokenizer::from_pretrained — HuggingFace Hub auto-download Stage 2.1: closes the loop on the HuggingFace tokenizer story. Stage 2 shipped `HfTokenizer::from_bytes`/`from_file`, which required callers to manage their own tokenizer.json files. This adds the third constructor: let t = HfTokenizer::from_pretrained("CohereForAI/c4ai-command-r-v01")?; register_hf("command-", t); `from_pretrained` is a thin wrapper around the `hf-hub` crate's blocking `ureq` API. First call downloads `tokenizer.json` to `~/.cache/huggingface/ hub` (or `$HF_HOME` if set); subsequent calls reuse the on-disk cache. Uses the `main` revision; gated repos (Llama, Mistral) require `HF_TOKEN` in env or `~/.cache/huggingface/token`. Also adds `try_register_hf(prefix, repo)` as the obvious one-liner for proxy startup code: let _ = try_register_hf("command-", "CohereForAI/c4ai-command-r-v01"); let _ = try_register_hf("mistral-", "mistralai/Mistral-7B-v0.1"); Each call is independent — a download failure for one model (e.g. gated without a token) does not affect others. `HfTokenizerError` gains a new `Hub` variant so callers can distinguish "couldn't fetch" from "fetched but malformed" — relevant when deciding whether to retry, surface to the user, or fall back to the estimator. Why blocking, not async: `from_pretrained` is called once at startup. A sync API works from `main()`, from a `OnceLock` initializer, or from `tokio::task::spawn_blocking` if a tokio caller needs it later. The async hf-hub backend would force callers to await at startup, which doesn't fit the `register_hf` registry pattern. Why rustls, not native-tls: keeps the binary statically linkable for AWS deploys (no system OpenSSL dependency). Tests: a network-dependent integration test (`#[ignore]`d in CI; hits HF for `gpt2`, ~1.4 MB) verifies the real download + load + count path. A non-network negative test verifies that an invalid repo name surfaces as `HfTokenizerError::Hub`, not a panic. 44 unit tests + 5 proptests + 1 doctest pass; parity stays 40/40 byte-equal.
2026-04-25 15:05:09 -07:00
"windows-sys 0.60.2",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "hf-hub"
version = "0.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "aef3982638978efa195ff11b305f51f1f22f4f0a6cabee7af79b383ebee6a213"
dependencies = [
"dirs",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
"indicatif 0.18.4",
"libc",
"log",
"rand 0.9.4",
"reqwest",
"serde",
"serde_json",
"thiserror 2.0.18",
"ureq 3.3.0",
"windows-sys 0.61.2",
]
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
[[package]]
name = "hmac"
version = "0.13.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6303bc9732ae41b04cb554b844a762b4115a61bfaa81e3e83050991eeb56863f"
dependencies = [
"digest 0.11.3",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "hmac-sha256"
version = "1.1.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ec9d92d097f4749b64e8cc33d924d9f40a2d4eb91402b458014b781f5733d60f"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
[[package]]
name = "http"
version = "0.2.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "601cbb57e577e2f5ef5be8e7b83f0f63994f25aa94d673e54a92d5c516d101f1"
dependencies = [
"bytes",
"fnv",
"itoa",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "http"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.4.2"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "6970f50e31d6fc17d3fa27329444bfa74e196cf62e95052a3f6fee181dba6425"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"bytes",
"itoa",
]
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
[[package]]
name = "http-body"
version = "0.4.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7ceab25649e9960c0311ea418d17bee82c0dcec1bd053b5f9a66e265a693bed2"
dependencies = [
"bytes",
"http 0.2.12",
"pin-project-lite",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "http-body"
version = "1.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1efedce1fb8e6913f23e0c92de8e62cd5b772a67e7b3946df930a62566c93184"
dependencies = [
"bytes",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
]
[[package]]
name = "http-body-util"
version = "0.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b021d93e26becf5dc7e1b75b1bed1fd93124b374ceb73f43d4d4eafec896a64a"
dependencies = [
"bytes",
"futures-core",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"http-body 1.0.1",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"pin-project-lite",
]
[[package]]
name = "httparse"
version = "1.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87"
[[package]]
name = "httpdate"
version = "1.0.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "humantime"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "135b12329e5e3ce057a9f972339ea52bc954fe1e9358ef27f95e89716fbc5424"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
[[package]]
name = "hybrid-array"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.4.12"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "9155a582abd142abc056962c29e3ce5ff2ad5469f4246b537ed42c5deba857da"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
dependencies = [
"typenum",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "hyper"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.10.1"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "55281c53a1894c864990125767da440a4e630446785086f52523b20033b74498"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"atomic-waker",
"bytes",
"futures-channel",
"futures-core",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"h2",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"http-body 1.0.1",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"httparse",
"httpdate",
"itoa",
"pin-project-lite",
"smallvec",
"tokio",
"want",
]
[[package]]
name = "hyper-rustls"
version = "0.27.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "33ca68d021ef39cf6463ab54c1d0f5daf03377b70561305bb89a8f83aab66e0f"
dependencies = [
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"hyper",
"hyper-util",
"rustls",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"rustls-native-certs",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"tokio",
"tokio-rustls",
"tower-service",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"webpki-roots 1.0.8",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
]
[[package]]
name = "hyper-util"
version = "0.1.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0"
dependencies = [
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
"base64 0.22.1",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"bytes",
"futures-channel",
"futures-util",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"http-body 1.0.1",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"hyper",
"ipnet",
"libc",
"percent-encoding",
"pin-project-lite",
"socket2",
"tokio",
"tower-service",
"tracing",
]
fix(proxy): PR-D4 native Vertex publisher path + ADC bearer auth Adds a Rust-native Vertex AI publisher route ahead of the LiteLLM Python converter (which dropped `thinking`, `redacted_thinking`, `document`, `image`, `server_tool_use`, `mcp_tool_use` block kinds — the P4-37 / P4-38 bug). After this PR the Vertex `:rawPredict` and `:streamRawPredict` calls survive byte-equal upstream and benefit from the live-zone Anthropic dispatcher (PR-B-series) running over the body — same behaviour as `/v1/messages`. New module `crates/headroom-proxy/src/vertex/`: - `mod.rs` — single dispatch handler at the `/v1beta1/.../models/:model_action` route. Splits the trailing `:<verb>` segment with `str::rsplit_once(':')` (no regex) and flips an `attach_sse_tee` flag to dispatch to the streaming or non-streaming arm. Both verbs share one axum route shape because matchit can't distinguish two patterns that overlap on a parameter. - `envelope.rs` — `VertexEnvelope` parser. Confirms `anthropic_version` present + `model` field absent (the two fingerprints of the Vertex envelope vs `/v1/messages`). - `adc.rs` — `TokenSource` trait + `GcpAdcTokenSource` (production, `gcp_auth` 0.12) + `StaticTokenSource` (tests). Caches tokens with a 60s refresh-ahead-of-expiry window. Emits structured `event = "vertex_adc_token_refreshed"` per refresh. - `raw_predict.rs` — POST handler + shared `forward_vertex_request`. Buffers body, parses envelope, runs live-zone Anthropic compression, fetches ADC bearer, attaches `Authorization: Bearer <token>` (overwrites client-supplied Authorization header), forwards. SSE telemetry tee for the streaming verb reuses PR-C1's `AnthropicStreamState` directly (Vertex streams plain SSE, unlike Bedrock's binary EventStream). - `stream_raw_predict.rs` — module-level docs + alias to the shared dispatcher (the streaming-vs-non-streaming difference is one boolean flag inside the shared forwarder). Modifications: - `proxy.rs::build_app` — registers the single Vertex route. - `proxy.rs::AppState` — new `vertex_token_source: Arc<dyn TokenSource>` field. Production constructs `GcpAdcTokenSource` lazily (no GCP call until first `bearer()`); tests inject `StaticTokenSource` via the new `AppState::with_token_source` helper. - `config.rs` — adds `--vertex-region` / `HEADROOM_PROXY_VERTEX_REGION` (default `us-central1`, observability tag only — the upstream URL is `--upstream`) and `--vertex-adc-scope` / `HEADROOM_PROXY_VERTEX_ADC_SCOPE` (default `cloud-platform`). - `Cargo.toml` (workspace + proxy) — adds `gcp_auth = "0.12"` and `async-trait = "0.1"`. - `tests/common/mod.rs` — `start_proxy_with_state` accepts both config + state customizers; `install_static_token_source` helper for tests. `crates/headroom-proxy/tests/integration_vertex_raw_predict.rs` — all five tests pass: 1. `native_envelope_round_trip_byte_equal` — Vertex-shape body (with `anthropic_version`, no `model`) round-trips SHA-256 byte-equal upstream. 2. `adc_bearer_token_signed_correctly` — `Authorization: Bearer <static-test-token>` reaches upstream verbatim and OVERWRITES a client-supplied Authorization header. 3. `thinking_block_preserved` — request with `thinking` (incl. signature) + `redacted_thinking` (incl. opaque `data`) blocks round-trips byte-equal even with `LiveZone` compression mode enabled. This is the P4-37 / P4-38 teeth. 4. `stream_raw_predict_sse_handled` — `:streamRawPredict` proxies an Anthropic SSE response (full `message_start` → `content_block_delta` → `message_stop` sequence) back to the client without corruption; SSE content-type preserved end-to-end; bearer attached. 5. (bonus, no-silent-fallback contract) `adc_failure_returns_5xx_no_silent_forward` — when the token source returns `Err`, the proxy returns 5xx and never reaches upstream. Verifies the `event = "vertex_adc_fetch_failed"` error path. Workspace: `cargo test --workspace` green; `cargo clippy --workspace -- -D warnings` clean; `make ci-precheck` passes. - No silent fallbacks: ADC failure → structured 5xx, never an unauthenticated forward. - No hardcodes: every knob (region, ADC scope, upstream URL) is CLI-flag + env-var configurable. - No regexes: axum path parameters + `str::rsplit_once` only. - Comprehensive structured logs: `event` field on every decision point — `vertex_envelope_parsed`, `vertex_envelope_invalid`, `vertex_compression_skipped`, `vertex_compression_applied`, `vertex_adc_token_refreshed`, `vertex_adc_fetch_failed`, `vertex_streaming_pipeline_active`, `vertex_sse_stream_closed`, `vertex_forwarded`, `vertex_unknown_verb`, etc. - Performant: no body clone; ADC token cached + refreshed ahead-of-expiry, not fetched per request. - Comprehensive tests: realistic Anthropic block content (signature payload, redacted_thinking opaque blob) in `thinking_block_preserved`. The local `gcloud auth application-default print-access-token` returns no credentials, so manual validation against a real Vertex endpoint is not possible in this PR. Follow-up: the user runs `gcloud auth application-default login` once and exercises a live Vertex request — should be a no-code-change check. PR-D1 (Bedrock native) is running concurrently and will land its own envelope module at `crates/headroom-proxy/src/bedrock/envelope.rs`. The two envelope modules are intentionally siblings (not a shared trait) — the shapes differ (Bedrock has a different `anthropic_version` value, no `model` field, AWS SigV4 instead of GCP ADC), and a premature shared abstraction would obscure the provider-specific contracts. Whichever PR merges second rebases without conflict. Retires P4-38 (and the Vertex parts of P4-39); marketplace BYOC pitch (per project memory) gets one more native provider.
2026-05-03 16:22:07 -07:00
[[package]]
name = "iana-time-zone"
version = "0.1.65"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470"
dependencies = [
"android_system_properties",
"core-foundation-sys",
"iana-time-zone-haiku",
"js-sys",
"log",
"wasm-bindgen",
"windows-core",
]
[[package]]
name = "iana-time-zone-haiku"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f"
dependencies = [
"cc",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "icu_collections"
version = "2.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2984d1cd16c883d7935b9e07e44071dca8d917fd52ecc02c04d5fa0b5a3f191c"
dependencies = [
"displaydoc",
"potential_utf",
"utf8_iter",
"yoke",
"zerofrom",
"zerovec",
]
[[package]]
name = "icu_locale_core"
version = "2.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92219b62b3e2b4d88ac5119f8904c10f8f61bf7e95b640d25ba3075e6cac2c29"
dependencies = [
"displaydoc",
"litemap",
"tinystr",
"writeable",
"zerovec",
]
[[package]]
name = "icu_normalizer"
version = "2.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c56e5ee99d6e3d33bd91c5d85458b6005a22140021cc324cea84dd0e72cff3b4"
dependencies = [
"icu_collections",
"icu_normalizer_data",
"icu_properties",
"icu_provider",
"smallvec",
"zerovec",
]
[[package]]
name = "icu_normalizer_data"
version = "2.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "da3be0ae77ea334f4da67c12f149704f19f81d1adf7c51cf482943e84a2bad38"
[[package]]
name = "icu_properties"
version = "2.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bee3b67d0ea5c2cca5003417989af8996f8604e34fb9ddf96208a033901e70de"
dependencies = [
"icu_collections",
"icu_locale_core",
"icu_properties_data",
"icu_provider",
"zerotrie",
"zerovec",
]
[[package]]
name = "icu_properties_data"
version = "2.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8e2bbb201e0c04f7b4b3e14382af113e17ba4f63e2c9d2ee626b720cbce54a14"
[[package]]
name = "icu_provider"
version = "2.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "139c4cf31c8b5f33d7e199446eff9c1e02decfc2f0eec2c8d71f65befa45b421"
dependencies = [
"displaydoc",
"icu_locale_core",
"writeable",
"yoke",
"zerofrom",
"zerotrie",
"zerovec",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "ident_case"
version = "1.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b9e0384b61958566e926dc50660321d12159025e767c18e043daf26b70104c39"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "idna"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3b0875f23caa03898994f6ddc501886a45c7d3d62d04d2d90788d47be1b1e4de"
dependencies = [
"idna_adapter",
"smallvec",
"utf8_iter",
]
[[package]]
name = "idna_adapter"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.2.2"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "cb68373c0d6620ef8105e855e7745e18b0d00d3bdb07fb532e434244cdb9a714"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"icu_normalizer",
"icu_properties",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "image"
version = "0.25.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "85ab80394333c02fe689eaf900ab500fbd0c2213da414687ebf995a65d5a6104"
dependencies = [
"bytemuck",
"byteorder-lite",
"color_quant",
"exr",
"gif",
"image-webp",
"moxcms",
"num-traits",
"png",
"qoi",
"ravif",
"rayon",
"rgb",
"tiff",
"zune-core",
"zune-jpeg",
]
[[package]]
name = "image-webp"
version = "0.2.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "525e9ff3e1a4be2fbea1fdf0e98686a6d98b4d8f937e1bf7402245af1909e8c3"
dependencies = [
"byteorder-lite",
"quick-error 2.0.1",
]
[[package]]
name = "imgref"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.12.2"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "89194689a993ab15268672e99e7b0e19da2da3268ac682e8f02d29d4d1434cd7"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "indexmap"
version = "2.14.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9"
dependencies = [
"equivalent",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"hashbrown 0.17.1",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "indicatif"
version = "0.17.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "183b3088984b400f4cfac3620d5e076c84da5364016b4f49473de574b2586235"
dependencies = [
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
"console 0.15.11",
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
"number_prefix",
"portable-atomic",
"unicode-width",
"web-time",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "indicatif"
version = "0.18.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "25470f23803092da7d239834776d653104d551bc4d7eacaf31e6837854b8e9eb"
dependencies = [
"console 0.16.3",
"portable-atomic",
"unicode-width",
"unit-prefix",
"web-time",
]
[[package]]
name = "interpolate_name"
version = "0.2.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c34819042dc3d3971c46c2190835914dfbe0c3c13f61449b2997f4e9722dfa60"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "ipnet"
version = "2.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d98f6fed1fde3f8c21bc40a1abb88dd75e67924f9cffc3ef95607bad8017f8e2"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "is-terminal"
version = "0.4.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3640c1c38b8e4e43584d8df18be5fc6b0aa314ce6ebf51b53313d4306cca8e46"
dependencies = [
"hermit-abi",
"libc",
"windows-sys 0.61.2",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "is_terminal_polyfill"
version = "1.70.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "itertools"
version = "0.10.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b0fd2260e829bddf4cb6ea802289de2f86d6a7a690192fbe91b3f46e0f2c8473"
dependencies = [
"either",
]
fix: B7 — CCR hardening: persistent backends + always-on tool P2-25, P2-26: CCR (Compress-Cache-Retrieve) used an in-memory store that fragmented across uvicorn workers and was wiped on restart, and the `headroom_retrieve` tool was registered/unregistered per-request based on whether the latest body happened to contain compression markers — every flip busted the prompt cache. Both are sticky side-channels: once a session has done CCR, the tool list bytes and the retrieval store must stay stable. This PR fixes both. Rust: * Split `ccr.rs` into `ccr/` with `backends/` submodule (`in_memory.rs`, `sqlite.rs`, `redis.rs` cfg-gated). * `SqliteCcrStore` (production default): WAL mode, prepared upsert, lazy TTL purge on read, persistent across worker restarts and shareable across workers on the same host via SQLite file locking. * `RedisCcrStore` (cfg-gated behind `feature = "redis"`): SETEX with startup PING smoke-test, no key-prefix collision risk, no sticky session required at the LB. * `CcrBackendConfig::{InMemory, Sqlite, Redis}` + `from_config(...)` factory — every init failure surfaces (no silent fallback per `feedback_no_silent_fallbacks.md`). * `ccr::compute_key` (BLAKE3 → first 24 hex chars) and `ccr::marker_for("HASH") -> "<<ccr:HASH>>"` centralize the hash + marker format; one definition for the live-zone dispatcher and the Python regex (`headroom/ccr/tool_injection.py:211`). * `compress_anthropic_live_zone_with_ccr` accepts `Option<&dyn CcrStore>`. When wired, every accepted compression puts the original bytes into the backend and appends `<<ccr:HASH>>` to the compressed string. The token-validation gate runs on the marker-augmented string so the `compressed_tokens >= original_tokens` rejection stays honest. Python: * `SessionCcrTracker` + `apply_session_sticky_ccr_tool` mirror the PR-A7 `SessionToolTracker` / `apply_session_sticky_memory_tools` pattern: once a session has done CCR, every subsequent request injects the recorded golden tool-definition bytes. Tool list bytes are byte-stable across turns (snapshot test pins them). * `headroom/ccr/tool_injection.py::inject_tool_definition` accepts a new `session_has_done_ccr` kwarg per the PR-B7 spec change at line 302-328. The legacy per-request path stays intact for callers that don't yet thread a session id (e.g. Google handler). * Anthropic + OpenAI handlers route their CCR tool-list updates through `apply_session_sticky_ccr_tool`, keyed off the existing `session_tracker_store.compute_session_id(...)` plumbing. Backend selection model: `CcrBackendConfig::Sqlite { path }` is the production default — single host, persistent, multi-worker safe with sticky session. `CcrBackendConfig::Redis { url }` is the multi-host scale-out option — no stickiness needed. `InMemory` is for tests and single-worker dev only. RUST_DEV.md "Multi-worker deployment — CCR fragmentation" rewritten around this matrix. Tests: * `crates/headroom-core/tests/ccr_backends.rs` — 7 tests covering SQLite round-trip, TTL purge, proxy-restart survival, cross-backend byte-equal keys, `from_config` paths, and the no-redis-feature loud-failure check (+ 2 redis tests gated behind the feature). * `crates/headroom-core/tests/live_zone_ccr.rs` — confirms `<<ccr:HASH>>` marker injection, store population, and no-marker-when-no-store invariants end-to-end. * `tests/test_ccr_tool_always_on.py` — 12 tests pinning the always-on behaviour, session/provider isolation, LRU bound, no- session-id fallback, and (per-acceptance-criterion) the byte-stable tool-definition snapshot. Per-PR-B7 plan: REALIGNMENT/04-phase-B-live-zone.md.
2026-05-02 16:49:08 -07:00
[[package]]
name = "itertools"
version = "0.13.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "413ee7dfc52ee1a4949ceeb7dbc8a33f2d6c088194d9f922fb8318faf1f01186"
dependencies = [
"either",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "itertools"
version = "0.14.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285"
dependencies = [
"either",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "itoa"
version = "1.0.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "jobserver"
version = "0.1.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9afb3de4395d6b3e67a780b6de64b51c978ecf11cb9a462c66be7d4ca9039d33"
dependencies = [
"getrandom 0.3.4",
"libc",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "js-sys"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.3.102"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "03d04c30968dffe80775bd4d7fb676131cd04a1fb46d2686dbffbaec2d9dfd31"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"cfg-if",
"futures-util",
"wasm-bindgen",
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "lazy_static"
version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "lebe"
version = "0.5.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7a79a3332a6609480d7d0c9eab957bca6b455b91bb84e66d19f5ff66294b85b8"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "libc"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.2.186"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "libfuzzer-sys"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.4.13"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "a9fd2f41a1cba099f79a0b6b6c35656cf7c03351a7bae8ff0f28f25270f929d2"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
dependencies = [
"arbitrary",
"cc",
]
Fix Windows ORT builds and Docker signing retries
2026-05-10 20:59:28 -07:00
[[package]]
name = "libloading"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "754ca22de805bb5744484a5b151a9e1a8e837d5dc232c2d7d8c2e3492edc8b60"
dependencies = [
"cfg-if",
"windows-link",
]
feat(rust): HfTokenizer::from_pretrained — HuggingFace Hub auto-download Stage 2.1: closes the loop on the HuggingFace tokenizer story. Stage 2 shipped `HfTokenizer::from_bytes`/`from_file`, which required callers to manage their own tokenizer.json files. This adds the third constructor: let t = HfTokenizer::from_pretrained("CohereForAI/c4ai-command-r-v01")?; register_hf("command-", t); `from_pretrained` is a thin wrapper around the `hf-hub` crate's blocking `ureq` API. First call downloads `tokenizer.json` to `~/.cache/huggingface/ hub` (or `$HF_HOME` if set); subsequent calls reuse the on-disk cache. Uses the `main` revision; gated repos (Llama, Mistral) require `HF_TOKEN` in env or `~/.cache/huggingface/token`. Also adds `try_register_hf(prefix, repo)` as the obvious one-liner for proxy startup code: let _ = try_register_hf("command-", "CohereForAI/c4ai-command-r-v01"); let _ = try_register_hf("mistral-", "mistralai/Mistral-7B-v0.1"); Each call is independent — a download failure for one model (e.g. gated without a token) does not affect others. `HfTokenizerError` gains a new `Hub` variant so callers can distinguish "couldn't fetch" from "fetched but malformed" — relevant when deciding whether to retry, surface to the user, or fall back to the estimator. Why blocking, not async: `from_pretrained` is called once at startup. A sync API works from `main()`, from a `OnceLock` initializer, or from `tokio::task::spawn_blocking` if a tokio caller needs it later. The async hf-hub backend would force callers to await at startup, which doesn't fit the `register_hf` registry pattern. Why rustls, not native-tls: keeps the binary statically linkable for AWS deploys (no system OpenSSL dependency). Tests: a network-dependent integration test (`#[ignore]`d in CI; hits HF for `gpt2`, ~1.4 MB) verifies the real download + load + count path. A non-network negative test verifies that an invalid repo name surfaces as `HfTokenizerError::Hub`, not a panic. 44 unit tests + 5 proptests + 1 doctest pass; parity stays 40/40 byte-equal.
2026-04-25 15:05:09 -07:00
[[package]]
name = "libredox"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.1.17"
feat(rust): HfTokenizer::from_pretrained — HuggingFace Hub auto-download Stage 2.1: closes the loop on the HuggingFace tokenizer story. Stage 2 shipped `HfTokenizer::from_bytes`/`from_file`, which required callers to manage their own tokenizer.json files. This adds the third constructor: let t = HfTokenizer::from_pretrained("CohereForAI/c4ai-command-r-v01")?; register_hf("command-", t); `from_pretrained` is a thin wrapper around the `hf-hub` crate's blocking `ureq` API. First call downloads `tokenizer.json` to `~/.cache/huggingface/ hub` (or `$HF_HOME` if set); subsequent calls reuse the on-disk cache. Uses the `main` revision; gated repos (Llama, Mistral) require `HF_TOKEN` in env or `~/.cache/huggingface/token`. Also adds `try_register_hf(prefix, repo)` as the obvious one-liner for proxy startup code: let _ = try_register_hf("command-", "CohereForAI/c4ai-command-r-v01"); let _ = try_register_hf("mistral-", "mistralai/Mistral-7B-v0.1"); Each call is independent — a download failure for one model (e.g. gated without a token) does not affect others. `HfTokenizerError` gains a new `Hub` variant so callers can distinguish "couldn't fetch" from "fetched but malformed" — relevant when deciding whether to retry, surface to the user, or fall back to the estimator. Why blocking, not async: `from_pretrained` is called once at startup. A sync API works from `main()`, from a `OnceLock` initializer, or from `tokio::task::spawn_blocking` if a tokio caller needs it later. The async hf-hub backend would force callers to await at startup, which doesn't fit the `register_hf` registry pattern. Why rustls, not native-tls: keeps the binary statically linkable for AWS deploys (no system OpenSSL dependency). Tests: a network-dependent integration test (`#[ignore]`d in CI; hits HF for `gpt2`, ~1.4 MB) verifies the real download + load + count path. A non-network negative test verifies that an invalid repo name surfaces as `HfTokenizerError::Hub`, not a panic. 44 unit tests + 5 proptests + 1 doctest pass; parity stays 40/40 byte-equal.
2026-04-25 15:05:09 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "f02ab6bace2054fb888a3c16f990117b579d14a3088e472d63c6011fa185c9d3"
feat(rust): HfTokenizer::from_pretrained — HuggingFace Hub auto-download Stage 2.1: closes the loop on the HuggingFace tokenizer story. Stage 2 shipped `HfTokenizer::from_bytes`/`from_file`, which required callers to manage their own tokenizer.json files. This adds the third constructor: let t = HfTokenizer::from_pretrained("CohereForAI/c4ai-command-r-v01")?; register_hf("command-", t); `from_pretrained` is a thin wrapper around the `hf-hub` crate's blocking `ureq` API. First call downloads `tokenizer.json` to `~/.cache/huggingface/ hub` (or `$HF_HOME` if set); subsequent calls reuse the on-disk cache. Uses the `main` revision; gated repos (Llama, Mistral) require `HF_TOKEN` in env or `~/.cache/huggingface/token`. Also adds `try_register_hf(prefix, repo)` as the obvious one-liner for proxy startup code: let _ = try_register_hf("command-", "CohereForAI/c4ai-command-r-v01"); let _ = try_register_hf("mistral-", "mistralai/Mistral-7B-v0.1"); Each call is independent — a download failure for one model (e.g. gated without a token) does not affect others. `HfTokenizerError` gains a new `Hub` variant so callers can distinguish "couldn't fetch" from "fetched but malformed" — relevant when deciding whether to retry, surface to the user, or fall back to the estimator. Why blocking, not async: `from_pretrained` is called once at startup. A sync API works from `main()`, from a `OnceLock` initializer, or from `tokio::task::spawn_blocking` if a tokio caller needs it later. The async hf-hub backend would force callers to await at startup, which doesn't fit the `register_hf` registry pattern. Why rustls, not native-tls: keeps the binary statically linkable for AWS deploys (no system OpenSSL dependency). Tests: a network-dependent integration test (`#[ignore]`d in CI; hits HF for `gpt2`, ~1.4 MB) verifies the real download + load + count path. A non-network negative test verifies that an invalid repo name surfaces as `HfTokenizerError::Hub`, not a panic. 44 unit tests + 5 proptests + 1 doctest pass; parity stays 40/40 byte-equal.
2026-04-25 15:05:09 -07:00
dependencies = [
"libc",
]
fix: B7 — CCR hardening: persistent backends + always-on tool P2-25, P2-26: CCR (Compress-Cache-Retrieve) used an in-memory store that fragmented across uvicorn workers and was wiped on restart, and the `headroom_retrieve` tool was registered/unregistered per-request based on whether the latest body happened to contain compression markers — every flip busted the prompt cache. Both are sticky side-channels: once a session has done CCR, the tool list bytes and the retrieval store must stay stable. This PR fixes both. Rust: * Split `ccr.rs` into `ccr/` with `backends/` submodule (`in_memory.rs`, `sqlite.rs`, `redis.rs` cfg-gated). * `SqliteCcrStore` (production default): WAL mode, prepared upsert, lazy TTL purge on read, persistent across worker restarts and shareable across workers on the same host via SQLite file locking. * `RedisCcrStore` (cfg-gated behind `feature = "redis"`): SETEX with startup PING smoke-test, no key-prefix collision risk, no sticky session required at the LB. * `CcrBackendConfig::{InMemory, Sqlite, Redis}` + `from_config(...)` factory — every init failure surfaces (no silent fallback per `feedback_no_silent_fallbacks.md`). * `ccr::compute_key` (BLAKE3 → first 24 hex chars) and `ccr::marker_for("HASH") -> "<<ccr:HASH>>"` centralize the hash + marker format; one definition for the live-zone dispatcher and the Python regex (`headroom/ccr/tool_injection.py:211`). * `compress_anthropic_live_zone_with_ccr` accepts `Option<&dyn CcrStore>`. When wired, every accepted compression puts the original bytes into the backend and appends `<<ccr:HASH>>` to the compressed string. The token-validation gate runs on the marker-augmented string so the `compressed_tokens >= original_tokens` rejection stays honest. Python: * `SessionCcrTracker` + `apply_session_sticky_ccr_tool` mirror the PR-A7 `SessionToolTracker` / `apply_session_sticky_memory_tools` pattern: once a session has done CCR, every subsequent request injects the recorded golden tool-definition bytes. Tool list bytes are byte-stable across turns (snapshot test pins them). * `headroom/ccr/tool_injection.py::inject_tool_definition` accepts a new `session_has_done_ccr` kwarg per the PR-B7 spec change at line 302-328. The legacy per-request path stays intact for callers that don't yet thread a session id (e.g. Google handler). * Anthropic + OpenAI handlers route their CCR tool-list updates through `apply_session_sticky_ccr_tool`, keyed off the existing `session_tracker_store.compute_session_id(...)` plumbing. Backend selection model: `CcrBackendConfig::Sqlite { path }` is the production default — single host, persistent, multi-worker safe with sticky session. `CcrBackendConfig::Redis { url }` is the multi-host scale-out option — no stickiness needed. `InMemory` is for tests and single-worker dev only. RUST_DEV.md "Multi-worker deployment — CCR fragmentation" rewritten around this matrix. Tests: * `crates/headroom-core/tests/ccr_backends.rs` — 7 tests covering SQLite round-trip, TTL purge, proxy-restart survival, cross-backend byte-equal keys, `from_config` paths, and the no-redis-feature loud-failure check (+ 2 redis tests gated behind the feature). * `crates/headroom-core/tests/live_zone_ccr.rs` — confirms `<<ccr:HASH>>` marker injection, store population, and no-marker-when-no-store invariants end-to-end. * `tests/test_ccr_tool_always_on.py` — 12 tests pinning the always-on behaviour, session/provider isolation, LRU bound, no- session-id fallback, and (per-acceptance-criterion) the byte-stable tool-definition snapshot. Per-PR-B7 plan: REALIGNMENT/04-phase-B-live-zone.md.
2026-05-02 16:49:08 -07:00
[[package]]
name = "libsqlite3-sys"
version = "0.30.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2e99fb7a497b1e3339bc746195567ed8d3e24945ecd636e3619d20b9de9e9149"
dependencies = [
"cc",
"pkg-config",
"vcpkg",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "linux-raw-sys"
version = "0.12.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "litemap"
version = "0.8.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92daf443525c4cce67b150400bc2316076100ce0b3686209eb8cf3c31612e6f0"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "litrs"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "11d3d7f243d5c5a8b9bb5d6dd2b1602c0cb0b9db1621bafc7ed66e35ff9fe092"
perf(rust): tier-1 multi-worker wins — GIL release, sharded CCR store, single-serialize CCR write Three orthogonal hot-path fixes targeting concurrent-request throughput. Each is independently bench-measured below; the proxy hot path benefits from all three at once. == 1. PyO3 GIL release on heavy compute == PyO3 methods (crush, smart_crush_content, crush_array_json, compact_document_json, compress, compress_with_stats) used to hold the GIL across the entire Rust call. Result: a 100ms compress() blocked EVERY other Python thread for 100ms — multi-worker uvicorn deployments serialized through SmartCrusher. Wrap each compute call in `py.allow_threads(|| ...)`. Inputs (`&str` from Python) are copied to owned `String` first because PyO3 ties them to the GIL hold. PyDict construction stays on the GIL side. Measured: 4 Python threads each running 20 crushes: before (GIL held): ~3.3s wall (serialized — equivalent to 4×0.83s) after (allow_threads): 826ms wall (4.01x speedup, perfect parallel) == 2. CcrStore: Mutex<HashMap> -> DashMap-backed sharded == Single Mutex was the dominant bottleneck under multi-worker load — every put/get serialized through one lock. Replace with DashMap (sharded concurrent map, lock-free reads within a shard) plus a separate small Mutex<VecDeque> for FIFO insertion-order eviction. Reads of distinct keys never contend; writes only contend during the brief order-queue push or capacity-sweep. A/B bench (200 mixed put/get ops × N threads, in benches/ccr_store.rs): Threads | DashMap Legacy Mutex Speedup ------------------------------------------- 1 | 63 µs 71 µs 1.13x 2 | 98 µs 194 µs 2.0x 4 | 178 µs 707 µs 4.0x 8 | 342 µs 1267 µs 3.7x Legacy degrades ~linearly with thread count; DashMap stays near-flat per-thread. Real multi-worker scaling. == 3. Single-serialize the lossy CCR payload == The lossy `crush_array` path used to serialize the full array TWICE: once in `hash_array_for_ccr` (allocates `Value::Array(items.to_vec())`, deep-clones every Value subtree, then serializes), and a second time in the store-write site. For a 50-item dict array that's ~MB of allocator pressure per crushed array. Introduce `canonical_array_json` (serializes `&[Value]` directly — same bytes as `Value::Array(items.to_vec())` but no wrapper allocation + no tree clone), call it ONCE per lossy path, then both hash and store from those same bytes. Hash-format stable — all 17 parity fixtures match byte-for-byte. == Tests == - 8 ccr.rs unit tests including a new concurrent-stress test (8 threads × 200 puts/gets, every key readable afterwards) - 14 ccr_roundtrip integration tests stay green - parity-run smart_crusher: 17/17 fixtures match - 479 lib + 14 integration + 185 Python tests all pass - New benches/ccr_store.rs runs the A/B and is committed for regression visibility == Dependencies added == - dashmap v6 (mature, widely-used in tokio/linkerd ecosystem)
2026-04-27 20:44:55 -07:00
[[package]]
name = "lock_api"
version = "0.4.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965"
dependencies = [
"scopeguard",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "log"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.4.33"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "loop9"
version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0fae87c125b03c1d2c0150c90365d7d6bcc53fb73a9acaef207d2d065860f062"
dependencies = [
"imgref",
]
fix: PR-E6 cache-bust drift detector telemetry (Phase E) Per-session SHA-256 fingerprint of the cache hot zone (system / tools / first 3 messages) with structured-log emission on drift. Detector is read-only: never mutates request bytes, preserves the Phase A passthrough invariant. Surfaces invisible cache busts (system prompt edited mid-session, tools reshuffled, early message changed) without rewriting them. * crates/headroom-proxy/src/cache_stabilization/drift_detector.rs: StructuralHash (system, tools, early_messages digests), compute_structural_hash, observe_drift, derive_session_key, DriftState (LRU bounded to 1000 sessions in production). * Session keys derive from Authorization / x-api-key / client IP / (IP, user-agent). Bearer tokens and API keys are SHA-256 hashed before they ever reach the log line; the raw secret is never logged. * Wired into forward_http after the body is buffered, before the compression dispatcher runs. Skips paths whose wire shape is not Anthropic / OpenAI Chat / OpenAI Responses. * AppState gains drift_state: DriftState. Bedrock unit-test literal-construction sites updated. * 14 unit tests + 1 integration test covering first-request, no-drift, per-dimension drift, multi-dim drift, LRU eviction, non-mutation invariant, and bearer-token-never-logged. Adds lru = "0.12" and promotes sha2 = "0.10" to a normal dependency on headroom-proxy.
2026-05-04 12:42:32 -07:00
[[package]]
name = "lru"
fix(deps): remediate dependency CVEs and publish SBOM (#1509) ## Description Supply-chain hardening: takes the **shipped** dependency surface from **26 known CVEs to 0**. `pip install headroom-ai[all]` now resolves with no known vulnerabilities (verified with Anchore syft + grype). Also publishes a checked-in SBOM package (`sbom/`) so any user — especially pilots running their own security review — can verify what's inside and that we track it. This addresses the Dependabot alerts on `main` (9 high / 4 moderate / 7 low at time of writing). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made **Rust** - `pyo3` 0.24 → 0.29 (GHSA-36hh-v3qg-5jq4 High, GHSA-chgr-c6px-7xpp Med). Migrated `Python::allow_threads` → `Python::detach` (10 sites) and added `from_py_object` to the `Clone`-deriving `#[pyclass]` types (both required by the 0.25+ API). - `pyo3-log` 0.12 → 0.13; `lru` 0.12 → 0.18 (GHSA-rhfx-m35p-ff5j). **Python** - `torch` → 2.12.1, `mem0ai` → 2.x. - Floor-pinned transitive CVE deps via `[tool.uv] constraint-dependencies`: `pygments>=2.20.0`, `pydantic-settings>=2.14.2`, `gitpython>=3.1.50`, `langsmith>=0.9.0`. - **Removed `benchmark` from the `[all]` aggregate** so the default install is CVE-free. `lm-eval` is invoked as an external subprocess (`python -m lm_eval`) and never imported, so it is not a true runtime dep — it remains available via the opt-in `[benchmark]` extra. See [Accepted Risks](#additional-notes). **npm (build/test tooling — never shipped in the wheel/container/published SDK)** - `esbuild` override `>=0.28.1` in `sdk/typescript` + `plugins/openclaw` (GHSA-g7r4-m6w7-qqqr). - `docs/`: `@anthropic-ai/sdk` → `^0.106.0` (GHSA-p7fg-763f-g4gf), `postcss` override to force Next.js's bundled copy ≥8.5.10 (GHSA-qx2v-qp2m-jg93); regenerated a stale `bun.lock` that carried a **Critical** vitest/vite. **CI** - Pinned `pypa/gh-action-pypi-publish` `@release/v1` → `@v1.13.0` (GHSA-vxmw-7h4f-hqxh) in `release.yml` + `publish.yml`. **SBOM** - New `sbom/` directory: CycloneDX 1.7 + SPDX 2.3 SBOMs, grype scan evidence, 330-package license inventory, and a regeneration guide. ## Testing - [ ] Unit tests pass (`pytest`) — N/A, no Python source changed (deps/config only) - [x] Linting passes — `cargo fmt --check` + `cargo clippy` clean on the changed crate; 0 `.py` files changed so `ruff`/`mypy` scope is unaffected - [x] Type checking passes — `cargo check --workspace` (0 errors) - [ ] New tests added — N/A (dependency bumps; covered by existing suites) - [x] Manual testing performed — see Real Behavior Proof ### Test Output ```text # headroom-ai[all] product surface — the number that matters $ grype sbom:sbom/headroom-sbom-all-extra.cdx.json No vulnerabilities found # full repo scan (universal lock incl. opt-in [benchmark] + dev) $ grype sbom:sbom/headroom-sbom.cdx.json NAME INSTALLED TYPE VULNERABILITY SEVERITY sqlitedict 2.1.0 python GHSA-g4r7-86gm-pgqc High # [benchmark]-only, unpatchable, accepted nltk 3.9.4 python GHSA-p4gq-832x-fm9v High # [benchmark]-only, unpatchable, accepted # pyo3 0.29 migration — extension builds + imports + runs $ cargo check --workspace Finished `dev` profile [unoptimized + debuginfo] target(s) $ maturin develop && python -c "from headroom._core import DiffCompressor, SmartCrusher; ..." extension OK — detach + from_py_object paths exercised # lru 0.18 — eviction path $ cargo test -p headroom-proxy --lib drift 14 passed, 213 filtered out # per-ecosystem npm audits $ (cd sdk/typescript && npm audit) -> found 0 vulnerabilities $ (cd plugins/openclaw && npm audit) -> found 0 vulnerabilities $ (cd docs && npm audit && bun audit) -> found 0 vulnerabilities / No vulnerabilities found ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0, arm64), Python 3.12 `.venv`, Rust 1.95 toolchain, syft 1.46.0, grype 0.115.0, bun 1.3.14, maturin 1.13.3. - Exact command / steps: (1) `uv export --extra all --no-dev --no-emit-project | syft → grype` for the product surface; (2) `cargo check --workspace` + `maturin develop` + extension import/compress smoke test; (3) `cargo test -p headroom-proxy --lib drift`; (4) `cargo fmt --check` + `cargo clippy -p headroom-py`; (5) `npm audit` in sdk/openclaw/docs + `bun audit` in docs. - Observed result: `headroom-ai[all]` resolution scans clean — "No vulnerabilities found" (179 pkgs); full/prod SBOM shows only the 2 documented accepted CVEs; pyo3 0.29 extension imports and runs (detach + from_py_object paths exercised); drift tests 14/14 pass; cargo fmt + clippy clean; all npm/bun audits report 0. - Not tested: full `pytest` suite (no Python source changed); release-profile wheel build (used dev-profile `maturin develop` for the import proof — the extension is semantically identical). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (`sbom/README.md`) - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective — N/A (dependency bumps; existing suites + scans cover it) - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A (Release Please auto-generates from the conventional commit) ## Additional Notes **Accepted risks (the 2 residual CVEs).** Both originate solely from the EleutherAI `lm-evaluation-harness` under the **opt-in `[benchmark]` extra**, which Headroom invokes as a subprocess (never imports): - `sqlitedict` CVE-2024-35515 (High) — pickle deserialization; package abandoned (last release 2021), **no upstream fix exists**. - `nltk` CVE-2026-54293 (High) — path traversal in `nltk.data.load()`; affects ≤3.9.4 (current latest), **no patched release**. Neither is in `[all]`, the published wheel, or the container. They are documented in `sbom/README.md` and will be picked up automatically once upstream ships fixes. **Release/CHANGELOG:** N/A items above are because this is a dependency/security PR with no Python source changes; CHANGELOG is Release-Please-managed via the conventional commit message.
2026-06-27 15:28:12 -07:00
version = "0.18.0"
fix: PR-E6 cache-bust drift detector telemetry (Phase E) Per-session SHA-256 fingerprint of the cache hot zone (system / tools / first 3 messages) with structured-log emission on drift. Detector is read-only: never mutates request bytes, preserves the Phase A passthrough invariant. Surfaces invisible cache busts (system prompt edited mid-session, tools reshuffled, early message changed) without rewriting them. * crates/headroom-proxy/src/cache_stabilization/drift_detector.rs: StructuralHash (system, tools, early_messages digests), compute_structural_hash, observe_drift, derive_session_key, DriftState (LRU bounded to 1000 sessions in production). * Session keys derive from Authorization / x-api-key / client IP / (IP, user-agent). Bearer tokens and API keys are SHA-256 hashed before they ever reach the log line; the raw secret is never logged. * Wired into forward_http after the body is buffered, before the compression dispatcher runs. Skips paths whose wire shape is not Anthropic / OpenAI Chat / OpenAI Responses. * AppState gains drift_state: DriftState. Bedrock unit-test literal-construction sites updated. * 14 unit tests + 1 integration test covering first-request, no-drift, per-dimension drift, multi-dim drift, LRU eviction, non-mutation invariant, and bearer-token-never-logged. Adds lru = "0.12" and promotes sha2 = "0.10" to a normal dependency on headroom-proxy.
2026-05-04 12:42:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
fix(deps): remediate dependency CVEs and publish SBOM (#1509) ## Description Supply-chain hardening: takes the **shipped** dependency surface from **26 known CVEs to 0**. `pip install headroom-ai[all]` now resolves with no known vulnerabilities (verified with Anchore syft + grype). Also publishes a checked-in SBOM package (`sbom/`) so any user — especially pilots running their own security review — can verify what's inside and that we track it. This addresses the Dependabot alerts on `main` (9 high / 4 moderate / 7 low at time of writing). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made **Rust** - `pyo3` 0.24 → 0.29 (GHSA-36hh-v3qg-5jq4 High, GHSA-chgr-c6px-7xpp Med). Migrated `Python::allow_threads` → `Python::detach` (10 sites) and added `from_py_object` to the `Clone`-deriving `#[pyclass]` types (both required by the 0.25+ API). - `pyo3-log` 0.12 → 0.13; `lru` 0.12 → 0.18 (GHSA-rhfx-m35p-ff5j). **Python** - `torch` → 2.12.1, `mem0ai` → 2.x. - Floor-pinned transitive CVE deps via `[tool.uv] constraint-dependencies`: `pygments>=2.20.0`, `pydantic-settings>=2.14.2`, `gitpython>=3.1.50`, `langsmith>=0.9.0`. - **Removed `benchmark` from the `[all]` aggregate** so the default install is CVE-free. `lm-eval` is invoked as an external subprocess (`python -m lm_eval`) and never imported, so it is not a true runtime dep — it remains available via the opt-in `[benchmark]` extra. See [Accepted Risks](#additional-notes). **npm (build/test tooling — never shipped in the wheel/container/published SDK)** - `esbuild` override `>=0.28.1` in `sdk/typescript` + `plugins/openclaw` (GHSA-g7r4-m6w7-qqqr). - `docs/`: `@anthropic-ai/sdk` → `^0.106.0` (GHSA-p7fg-763f-g4gf), `postcss` override to force Next.js's bundled copy ≥8.5.10 (GHSA-qx2v-qp2m-jg93); regenerated a stale `bun.lock` that carried a **Critical** vitest/vite. **CI** - Pinned `pypa/gh-action-pypi-publish` `@release/v1` → `@v1.13.0` (GHSA-vxmw-7h4f-hqxh) in `release.yml` + `publish.yml`. **SBOM** - New `sbom/` directory: CycloneDX 1.7 + SPDX 2.3 SBOMs, grype scan evidence, 330-package license inventory, and a regeneration guide. ## Testing - [ ] Unit tests pass (`pytest`) — N/A, no Python source changed (deps/config only) - [x] Linting passes — `cargo fmt --check` + `cargo clippy` clean on the changed crate; 0 `.py` files changed so `ruff`/`mypy` scope is unaffected - [x] Type checking passes — `cargo check --workspace` (0 errors) - [ ] New tests added — N/A (dependency bumps; covered by existing suites) - [x] Manual testing performed — see Real Behavior Proof ### Test Output ```text # headroom-ai[all] product surface — the number that matters $ grype sbom:sbom/headroom-sbom-all-extra.cdx.json No vulnerabilities found # full repo scan (universal lock incl. opt-in [benchmark] + dev) $ grype sbom:sbom/headroom-sbom.cdx.json NAME INSTALLED TYPE VULNERABILITY SEVERITY sqlitedict 2.1.0 python GHSA-g4r7-86gm-pgqc High # [benchmark]-only, unpatchable, accepted nltk 3.9.4 python GHSA-p4gq-832x-fm9v High # [benchmark]-only, unpatchable, accepted # pyo3 0.29 migration — extension builds + imports + runs $ cargo check --workspace Finished `dev` profile [unoptimized + debuginfo] target(s) $ maturin develop && python -c "from headroom._core import DiffCompressor, SmartCrusher; ..." extension OK — detach + from_py_object paths exercised # lru 0.18 — eviction path $ cargo test -p headroom-proxy --lib drift 14 passed, 213 filtered out # per-ecosystem npm audits $ (cd sdk/typescript && npm audit) -> found 0 vulnerabilities $ (cd plugins/openclaw && npm audit) -> found 0 vulnerabilities $ (cd docs && npm audit && bun audit) -> found 0 vulnerabilities / No vulnerabilities found ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0, arm64), Python 3.12 `.venv`, Rust 1.95 toolchain, syft 1.46.0, grype 0.115.0, bun 1.3.14, maturin 1.13.3. - Exact command / steps: (1) `uv export --extra all --no-dev --no-emit-project | syft → grype` for the product surface; (2) `cargo check --workspace` + `maturin develop` + extension import/compress smoke test; (3) `cargo test -p headroom-proxy --lib drift`; (4) `cargo fmt --check` + `cargo clippy -p headroom-py`; (5) `npm audit` in sdk/openclaw/docs + `bun audit` in docs. - Observed result: `headroom-ai[all]` resolution scans clean — "No vulnerabilities found" (179 pkgs); full/prod SBOM shows only the 2 documented accepted CVEs; pyo3 0.29 extension imports and runs (detach + from_py_object paths exercised); drift tests 14/14 pass; cargo fmt + clippy clean; all npm/bun audits report 0. - Not tested: full `pytest` suite (no Python source changed); release-profile wheel build (used dev-profile `maturin develop` for the import proof — the extension is semantically identical). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (`sbom/README.md`) - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective — N/A (dependency bumps; existing suites + scans cover it) - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A (Release Please auto-generates from the conventional commit) ## Additional Notes **Accepted risks (the 2 residual CVEs).** Both originate solely from the EleutherAI `lm-evaluation-harness` under the **opt-in `[benchmark]` extra**, which Headroom invokes as a subprocess (never imports): - `sqlitedict` CVE-2024-35515 (High) — pickle deserialization; package abandoned (last release 2021), **no upstream fix exists**. - `nltk` CVE-2026-54293 (High) — path traversal in `nltk.data.load()`; affects ≤3.9.4 (current latest), **no patched release**. Neither is in `[all]`, the published wheel, or the container. They are documented in `sbom/README.md` and will be picked up automatically once upstream ships fixes. **Release/CHANGELOG:** N/A items above are because this is a dependency/security PR with no Python source changes; CHANGELOG is Release-Please-managed via the conventional commit message.
2026-06-27 15:28:12 -07:00
checksum = "8a860605968fce16869fd239cf4237a82f3ac470723415db603b0e8b6c8d4fb9"
fix: PR-E6 cache-bust drift detector telemetry (Phase E) Per-session SHA-256 fingerprint of the cache hot zone (system / tools / first 3 messages) with structured-log emission on drift. Detector is read-only: never mutates request bytes, preserves the Phase A passthrough invariant. Surfaces invisible cache busts (system prompt edited mid-session, tools reshuffled, early message changed) without rewriting them. * crates/headroom-proxy/src/cache_stabilization/drift_detector.rs: StructuralHash (system, tools, early_messages digests), compute_structural_hash, observe_drift, derive_session_key, DriftState (LRU bounded to 1000 sessions in production). * Session keys derive from Authorization / x-api-key / client IP / (IP, user-agent). Bearer tokens and API keys are SHA-256 hashed before they ever reach the log line; the raw secret is never logged. * Wired into forward_http after the body is buffered, before the compression dispatcher runs. Skips paths whose wire shape is not Anthropic / OpenAI Chat / OpenAI Responses. * AppState gains drift_state: DriftState. Bedrock unit-test literal-construction sites updated. * 14 unit tests + 1 integration test covering first-request, no-drift, per-dimension drift, multi-dim drift, LRU eviction, non-mutation invariant, and bearer-token-never-logged. Adds lru = "0.12" and promotes sha2 = "0.10" to a normal dependency on headroom-proxy.
2026-05-04 12:42:32 -07:00
dependencies = [
fix(deps): remediate dependency CVEs and publish SBOM (#1509) ## Description Supply-chain hardening: takes the **shipped** dependency surface from **26 known CVEs to 0**. `pip install headroom-ai[all]` now resolves with no known vulnerabilities (verified with Anchore syft + grype). Also publishes a checked-in SBOM package (`sbom/`) so any user — especially pilots running their own security review — can verify what's inside and that we track it. This addresses the Dependabot alerts on `main` (9 high / 4 moderate / 7 low at time of writing). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made **Rust** - `pyo3` 0.24 → 0.29 (GHSA-36hh-v3qg-5jq4 High, GHSA-chgr-c6px-7xpp Med). Migrated `Python::allow_threads` → `Python::detach` (10 sites) and added `from_py_object` to the `Clone`-deriving `#[pyclass]` types (both required by the 0.25+ API). - `pyo3-log` 0.12 → 0.13; `lru` 0.12 → 0.18 (GHSA-rhfx-m35p-ff5j). **Python** - `torch` → 2.12.1, `mem0ai` → 2.x. - Floor-pinned transitive CVE deps via `[tool.uv] constraint-dependencies`: `pygments>=2.20.0`, `pydantic-settings>=2.14.2`, `gitpython>=3.1.50`, `langsmith>=0.9.0`. - **Removed `benchmark` from the `[all]` aggregate** so the default install is CVE-free. `lm-eval` is invoked as an external subprocess (`python -m lm_eval`) and never imported, so it is not a true runtime dep — it remains available via the opt-in `[benchmark]` extra. See [Accepted Risks](#additional-notes). **npm (build/test tooling — never shipped in the wheel/container/published SDK)** - `esbuild` override `>=0.28.1` in `sdk/typescript` + `plugins/openclaw` (GHSA-g7r4-m6w7-qqqr). - `docs/`: `@anthropic-ai/sdk` → `^0.106.0` (GHSA-p7fg-763f-g4gf), `postcss` override to force Next.js's bundled copy ≥8.5.10 (GHSA-qx2v-qp2m-jg93); regenerated a stale `bun.lock` that carried a **Critical** vitest/vite. **CI** - Pinned `pypa/gh-action-pypi-publish` `@release/v1` → `@v1.13.0` (GHSA-vxmw-7h4f-hqxh) in `release.yml` + `publish.yml`. **SBOM** - New `sbom/` directory: CycloneDX 1.7 + SPDX 2.3 SBOMs, grype scan evidence, 330-package license inventory, and a regeneration guide. ## Testing - [ ] Unit tests pass (`pytest`) — N/A, no Python source changed (deps/config only) - [x] Linting passes — `cargo fmt --check` + `cargo clippy` clean on the changed crate; 0 `.py` files changed so `ruff`/`mypy` scope is unaffected - [x] Type checking passes — `cargo check --workspace` (0 errors) - [ ] New tests added — N/A (dependency bumps; covered by existing suites) - [x] Manual testing performed — see Real Behavior Proof ### Test Output ```text # headroom-ai[all] product surface — the number that matters $ grype sbom:sbom/headroom-sbom-all-extra.cdx.json No vulnerabilities found # full repo scan (universal lock incl. opt-in [benchmark] + dev) $ grype sbom:sbom/headroom-sbom.cdx.json NAME INSTALLED TYPE VULNERABILITY SEVERITY sqlitedict 2.1.0 python GHSA-g4r7-86gm-pgqc High # [benchmark]-only, unpatchable, accepted nltk 3.9.4 python GHSA-p4gq-832x-fm9v High # [benchmark]-only, unpatchable, accepted # pyo3 0.29 migration — extension builds + imports + runs $ cargo check --workspace Finished `dev` profile [unoptimized + debuginfo] target(s) $ maturin develop && python -c "from headroom._core import DiffCompressor, SmartCrusher; ..." extension OK — detach + from_py_object paths exercised # lru 0.18 — eviction path $ cargo test -p headroom-proxy --lib drift 14 passed, 213 filtered out # per-ecosystem npm audits $ (cd sdk/typescript && npm audit) -> found 0 vulnerabilities $ (cd plugins/openclaw && npm audit) -> found 0 vulnerabilities $ (cd docs && npm audit && bun audit) -> found 0 vulnerabilities / No vulnerabilities found ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0, arm64), Python 3.12 `.venv`, Rust 1.95 toolchain, syft 1.46.0, grype 0.115.0, bun 1.3.14, maturin 1.13.3. - Exact command / steps: (1) `uv export --extra all --no-dev --no-emit-project | syft → grype` for the product surface; (2) `cargo check --workspace` + `maturin develop` + extension import/compress smoke test; (3) `cargo test -p headroom-proxy --lib drift`; (4) `cargo fmt --check` + `cargo clippy -p headroom-py`; (5) `npm audit` in sdk/openclaw/docs + `bun audit` in docs. - Observed result: `headroom-ai[all]` resolution scans clean — "No vulnerabilities found" (179 pkgs); full/prod SBOM shows only the 2 documented accepted CVEs; pyo3 0.29 extension imports and runs (detach + from_py_object paths exercised); drift tests 14/14 pass; cargo fmt + clippy clean; all npm/bun audits report 0. - Not tested: full `pytest` suite (no Python source changed); release-profile wheel build (used dev-profile `maturin develop` for the import proof — the extension is semantically identical). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (`sbom/README.md`) - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective — N/A (dependency bumps; existing suites + scans cover it) - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A (Release Please auto-generates from the conventional commit) ## Additional Notes **Accepted risks (the 2 residual CVEs).** Both originate solely from the EleutherAI `lm-evaluation-harness` under the **opt-in `[benchmark]` extra**, which Headroom invokes as a subprocess (never imports): - `sqlitedict` CVE-2024-35515 (High) — pickle deserialization; package abandoned (last release 2021), **no upstream fix exists**. - `nltk` CVE-2026-54293 (High) — path traversal in `nltk.data.load()`; affects ≤3.9.4 (current latest), **no patched release**. Neither is in `[all]`, the published wheel, or the container. They are documented in `sbom/README.md` and will be picked up automatically once upstream ships fixes. **Release/CHANGELOG:** N/A items above are because this is a dependency/security PR with no Python source changes; CHANGELOG is Release-Please-managed via the conventional commit message.
2026-06-27 15:28:12 -07:00
"hashbrown 0.17.1",
fix: PR-E6 cache-bust drift detector telemetry (Phase E) Per-session SHA-256 fingerprint of the cache hot zone (system / tools / first 3 messages) with structured-log emission on drift. Detector is read-only: never mutates request bytes, preserves the Phase A passthrough invariant. Surfaces invisible cache busts (system prompt edited mid-session, tools reshuffled, early message changed) without rewriting them. * crates/headroom-proxy/src/cache_stabilization/drift_detector.rs: StructuralHash (system, tools, early_messages digests), compute_structural_hash, observe_drift, derive_session_key, DriftState (LRU bounded to 1000 sessions in production). * Session keys derive from Authorization / x-api-key / client IP / (IP, user-agent). Bearer tokens and API keys are SHA-256 hashed before they ever reach the log line; the raw secret is never logged. * Wired into forward_http after the body is buffered, before the compression dispatcher runs. Skips paths whose wire shape is not Anthropic / OpenAI Chat / OpenAI Responses. * AppState gains drift_state: DriftState. Bedrock unit-test literal-construction sites updated. * 14 unit tests + 1 integration test covering first-request, no-drift, per-dimension drift, multi-dim drift, LRU eviction, non-mutation invariant, and bearer-token-never-logged. Adds lru = "0.12" and promotes sha2 = "0.10" to a normal dependency on headroom-proxy.
2026-05-04 12:42:32 -07:00
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "lru-slab"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "lzma-rust2"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.15.8"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "e20f57f9918e5bd7bc58c22cdd70a6afc7375d4dd9683af5f2b34bd3d2bba619"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "macro_rules_attribute"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "65049d7923698040cd0b1ddcced9b0eb14dd22c5f86ae59c3740eab64a676520"
dependencies = [
"macro_rules_attribute-proc_macro",
"paste",
]
[[package]]
name = "macro_rules_attribute-proc_macro"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "670fdfda89751bc4a84ac13eaa63e205cf0fd22b4c9a5fbfa085b63c1f1d3a30"
chore(rust): port Magika detection (Stage 3d PR3 — Tier 1) Adds Google's `magika` ONNX-backed content classifier as the first tier of the new Stage-3d ContentRouter detection arch (`magika` → `unidiff-rs` → `PlainText` fall-through; no regex tier on the Rust side). What lands: - New module `crates/headroom-core/src/transforms/magika_detector.rs`: - `magika_detect(content: &str) -> Result<ContentType, _>` - `OnceLock<Mutex<Result<Session, _>>>` singleton: model loads once per process; init failure is recorded once and cheaply replayed (no retry — rust-side `feedback_no_silent_fallbacks`). - `map_magika_label(&str) -> ContentType`: explicit match arms against magika's 200+ labels, mapped onto Headroom's existing `ContentType` enum so the dispatch (PR5) stays enum-stable. Unmapped labels passthrough to `PlainText` rather than misroute. - 16 unit tests: empty fast-path, JSON / Python / Rust / JS / diff / markdown / plain prose / HTML / YAML / shell / SQL, singleton-reuse smoke, default-passthrough for unmapped labels, pure-table-lookup sanity. What does NOT land here (per PR scope): - No PyO3 surface yet — PR3 is detector-only. - No router rewiring — the existing regex `content_detector` still drives `ContentRouter` until PR5 flips the dispatch. - No `unidiff-rs` Tier-2 — that's PR4. The `magika` crate brings `ndarray` + `ort` (already in our dep tree via `fastembed`); adding it shares the ONNX Runtime singleton rather than pulling a second ML stack. `make ci-precheck` green.
2026-04-28 22:36:28 -07:00
[[package]]
name = "magika"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3aee5ecdbd182547ca3dfcd74c5bcd7f8c57384ad03cb79ef6e3bdf8d56abcdf"
dependencies = [
"ndarray",
"ort",
"thiserror 1.0.69",
"tokio",
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "matchers"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d1525a2a28c7f4fa0fc98bb91ae755d1e2d1505079e05539e35bc876b5d65ae9"
dependencies = [
"regex-automata",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "matchit"
version = "0.7.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0e7465ac9959cc2b1404e8e2367b43684a6d13790fe23056cc8c6c5a6b7bcb94"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "matrixmultiply"
version = "0.3.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a06de3016e9fae57a36fd14dba131fccf49f74b40b7fbdb472f96e361ec71a08"
dependencies = [
"autocfg",
"rawpointer",
]
[[package]]
name = "maybe-rayon"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8ea1f30cedd69f0a2954655f7188c6a834246d2bcf1e315e2ac40c4b24dc9519"
dependencies = [
"cfg-if",
"rayon",
]
feat(rust): diff_compressor port — byte-equal parity + sidecar stats Stage 3a: first real transform port. Faithful Rust port of `headroom.transforms.diff_compressor` with byte-equal parity against all 20 recorded fixtures. # Algorithm (matching Python) 1. Hand-rolled unified-diff parser (state machine over `diff --git`, `index`, `--- a/`, `+++ b/`, `@@`, mode/binary/rename markers, +/- / space lines, "other" lines like `\ No newline at end of file`). 2. File cap (`max_files=20`): when fired, sort by total changes (most first) and keep top N. 3. Per-file hunk cap (`max_hunks_per_file=10`): keep first + last + top relevance-scored middle, then resort by hunk-header start line to restore appearance order. 4. Relevance scoring: change-density base + user-query word overlap + priority patterns (ERROR / IMPORTANCE / SECURITY regexes — matches `error_detection.PRIORITY_PATTERNS_DIFF`). 5. Per-hunk context trim: keep `max_context_lines=2` lines either side of each `+`/`-` line. 6. CCR cache_key: `md5(original)[:24]` (matches `compression_store.CompressionStore.store`). Emitted only when compression saved >20% of lines. Parity result: `[diff_compressor ] total=20 matched=20 skipped=0 diffed=0`. # Information preservation hardening Three pass-through paths inherited from Python that we keep deliberate (would lose info if we changed them): - Below `min_lines_for_ccr` (50): return input unchanged. - No diff sections parsed: return input unchanged. - Below 20% compression savings: emit compressed output but no CCR marker (the original is the cheaper representation anyway). Plus a parity-bound subtlety: `compressed_line_count` is captured BEFORE the CCR retrieval marker is appended, both for the marker text (`compressed to N`) and the result field. The output string therefore ends up with one more line than the field reports — by design, matching Python exactly. An off-by-one bug from recounting after appending the CCR marker was caught and pinned by a synthetic 8-file diff test. # Observability — the Rust escape hatch Python's `DiffCompressionResult` has thin observability: input/output line counts, additions/deletions, hunks_kept/removed, files_affected, cache_key. The Rust port adds a sidecar `DiffCompressorStats` struct with metrics Python doesn't emit: - `files_dropped: Vec<String>` — names (old → new path) of files silently discarded by the `max_files` cap. Python loses these. - `hunks_dropped_per_file: BTreeMap<String, usize>` — per-file hunk drops, stable iteration via `BTreeMap`. - `context_lines_input` / `context_lines_kept` / `context_lines_trimmed` — directly proxies info loss from the context trim. - `largest_hunk_kept_lines` / `largest_hunk_dropped_lines` — outlier detection (a single huge dropped hunk is much worse than many small). - `parse_warnings: Vec<String>` — surfaces malformed input rather than dropping silently. - `processing_duration_us` — latency budget. - `cache_key_emitted` + `ccr_skipped_reason: Option<String>` — explicit signal for "we chose not to emit CCR and this is why". A `tracing::info!(target: "diff_compressor", ...)` event is emitted on every call, carrying these fields for OTel scraping in prod. The sidecar struct is returned alongside via `compress_with_stats`; the parity-only `compress` API discards it. # Module layout - `crates/headroom-core/src/transforms/mod.rs` — namespace, doc comment with the guiding principle ("information preservation > aggressive compression") so future ports inherit the philosophy. - `crates/headroom-core/src/transforms/diff_compressor.rs` — full port (parser, scorer, hunk selector, context trimmer, formatter, CCR layer, stats, tracing). # Dependencies added to headroom-core - `md-5 = "0.10"` — for the CCR cache_key (matches Python MD5[:24]). - `regex = "1"` — was a transitive dep via tokenizers; now a direct dependency for the hunk-header parser and priority patterns. # Tests 6 unit tests covering pass-through paths, MD5 hex truncation, the Python `split("\n")` line-count semantics, sidecar stats emission, and a synthetic 8-file diff that locks the byte-equal behavior found in the parity fixtures.
2026-04-25 15:44:25 -07:00
[[package]]
name = "md-5"
version = "0.10.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d89e7ee0cfbedfc4da3340218492196241d89eefb6dab27de5df917a6d2e78cf"
dependencies = [
"cfg-if",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"digest 0.10.7",
feat(rust): diff_compressor port — byte-equal parity + sidecar stats Stage 3a: first real transform port. Faithful Rust port of `headroom.transforms.diff_compressor` with byte-equal parity against all 20 recorded fixtures. # Algorithm (matching Python) 1. Hand-rolled unified-diff parser (state machine over `diff --git`, `index`, `--- a/`, `+++ b/`, `@@`, mode/binary/rename markers, +/- / space lines, "other" lines like `\ No newline at end of file`). 2. File cap (`max_files=20`): when fired, sort by total changes (most first) and keep top N. 3. Per-file hunk cap (`max_hunks_per_file=10`): keep first + last + top relevance-scored middle, then resort by hunk-header start line to restore appearance order. 4. Relevance scoring: change-density base + user-query word overlap + priority patterns (ERROR / IMPORTANCE / SECURITY regexes — matches `error_detection.PRIORITY_PATTERNS_DIFF`). 5. Per-hunk context trim: keep `max_context_lines=2` lines either side of each `+`/`-` line. 6. CCR cache_key: `md5(original)[:24]` (matches `compression_store.CompressionStore.store`). Emitted only when compression saved >20% of lines. Parity result: `[diff_compressor ] total=20 matched=20 skipped=0 diffed=0`. # Information preservation hardening Three pass-through paths inherited from Python that we keep deliberate (would lose info if we changed them): - Below `min_lines_for_ccr` (50): return input unchanged. - No diff sections parsed: return input unchanged. - Below 20% compression savings: emit compressed output but no CCR marker (the original is the cheaper representation anyway). Plus a parity-bound subtlety: `compressed_line_count` is captured BEFORE the CCR retrieval marker is appended, both for the marker text (`compressed to N`) and the result field. The output string therefore ends up with one more line than the field reports — by design, matching Python exactly. An off-by-one bug from recounting after appending the CCR marker was caught and pinned by a synthetic 8-file diff test. # Observability — the Rust escape hatch Python's `DiffCompressionResult` has thin observability: input/output line counts, additions/deletions, hunks_kept/removed, files_affected, cache_key. The Rust port adds a sidecar `DiffCompressorStats` struct with metrics Python doesn't emit: - `files_dropped: Vec<String>` — names (old → new path) of files silently discarded by the `max_files` cap. Python loses these. - `hunks_dropped_per_file: BTreeMap<String, usize>` — per-file hunk drops, stable iteration via `BTreeMap`. - `context_lines_input` / `context_lines_kept` / `context_lines_trimmed` — directly proxies info loss from the context trim. - `largest_hunk_kept_lines` / `largest_hunk_dropped_lines` — outlier detection (a single huge dropped hunk is much worse than many small). - `parse_warnings: Vec<String>` — surfaces malformed input rather than dropping silently. - `processing_duration_us` — latency budget. - `cache_key_emitted` + `ccr_skipped_reason: Option<String>` — explicit signal for "we chose not to emit CCR and this is why". A `tracing::info!(target: "diff_compressor", ...)` event is emitted on every call, carrying these fields for OTel scraping in prod. The sidecar struct is returned alongside via `compress_with_stats`; the parity-only `compress` API discards it. # Module layout - `crates/headroom-core/src/transforms/mod.rs` — namespace, doc comment with the guiding principle ("information preservation > aggressive compression") so future ports inherit the philosophy. - `crates/headroom-core/src/transforms/diff_compressor.rs` — full port (parser, scorer, hunk selector, context trimmer, formatter, CCR layer, stats, tracing). # Dependencies added to headroom-core - `md-5 = "0.10"` — for the CCR cache_key (matches Python MD5[:24]). - `regex = "1"` — was a transitive dep via tokenizers; now a direct dependency for the hunk-header parser and priority patterns. # Tests 6 unit tests covering pass-through paths, MD5 hex truncation, the Python `split("\n")` line-count semantics, sidecar stats emission, and a synthetic 8-file diff that locks the byte-equal behavior found in the parity fixtures.
2026-04-25 15:44:25 -07:00
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "memchr"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "2.8.2"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "mime"
version = "0.3.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "minimal-lexical"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a"
feat(rust): HfTokenizer::from_pretrained — HuggingFace Hub auto-download Stage 2.1: closes the loop on the HuggingFace tokenizer story. Stage 2 shipped `HfTokenizer::from_bytes`/`from_file`, which required callers to manage their own tokenizer.json files. This adds the third constructor: let t = HfTokenizer::from_pretrained("CohereForAI/c4ai-command-r-v01")?; register_hf("command-", t); `from_pretrained` is a thin wrapper around the `hf-hub` crate's blocking `ureq` API. First call downloads `tokenizer.json` to `~/.cache/huggingface/ hub` (or `$HF_HOME` if set); subsequent calls reuse the on-disk cache. Uses the `main` revision; gated repos (Llama, Mistral) require `HF_TOKEN` in env or `~/.cache/huggingface/token`. Also adds `try_register_hf(prefix, repo)` as the obvious one-liner for proxy startup code: let _ = try_register_hf("command-", "CohereForAI/c4ai-command-r-v01"); let _ = try_register_hf("mistral-", "mistralai/Mistral-7B-v0.1"); Each call is independent — a download failure for one model (e.g. gated without a token) does not affect others. `HfTokenizerError` gains a new `Hub` variant so callers can distinguish "couldn't fetch" from "fetched but malformed" — relevant when deciding whether to retry, surface to the user, or fall back to the estimator. Why blocking, not async: `from_pretrained` is called once at startup. A sync API works from `main()`, from a `OnceLock` initializer, or from `tokio::task::spawn_blocking` if a tokio caller needs it later. The async hf-hub backend would force callers to await at startup, which doesn't fit the `register_hf` registry pattern. Why rustls, not native-tls: keeps the binary statically linkable for AWS deploys (no system OpenSSL dependency). Tests: a network-dependent integration test (`#[ignore]`d in CI; hits HF for `gpt2`, ~1.4 MB) verifies the real download + load + count path. A non-network negative test verifies that an invalid repo name surfaces as `HfTokenizerError::Hub`, not a panic. 44 unit tests + 5 proptests + 1 doctest pass; parity stays 40/40 byte-equal.
2026-04-25 15:05:09 -07:00
[[package]]
name = "miniz_oxide"
version = "0.8.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316"
dependencies = [
"adler2",
"simd-adler32",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "mio"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.2.1"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "02bd0af71c67b473010cbbc60715ee815645a4dc942899111f494b4b737d6fda"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"libc",
"wasi",
"windows-sys 0.61.2",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "monostate"
version = "0.1.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3341a273f6c9d5bef1908f17b7267bbab0e95c9bf69a0d4dcf8e9e1b2c76ef67"
dependencies = [
"monostate-impl",
"serde",
"serde_core",
]
[[package]]
name = "monostate-impl"
version = "0.1.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e4db6d5580af57bf992f59068d4ea26fd518574ff48d7639b255a36f9de6e7e9"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "moxcms"
version = "0.8.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bb85c154ba489f01b25c0d36ae69a87e4a1c73a72631fc6c0eb6dde34a73e44b"
dependencies = [
"num-traits",
"pxfm",
]
[[package]]
name = "ndarray"
version = "0.17.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "520080814a7a6b4a6e9070823bb24b4531daac8c4627e08ba5de8c5ef2f2752d"
dependencies = [
"matrixmultiply",
"num-complex",
"num-integer",
"num-traits",
"portable-atomic",
"portable-atomic-util",
"rawpointer",
]
[[package]]
name = "new_debug_unreachable"
version = "1.0.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "650eef8c711430f1a879fdd01d4745a7deea475becfb90269c06775983bbf086"
[[package]]
name = "no_std_io2"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.9.4"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "418abd1b6d34fbf6cae440dc874771b0525a604428704c76e48b29a5e67b8003"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
dependencies = [
"memchr",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "nom"
version = "7.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a"
dependencies = [
"memchr",
"minimal-lexical",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "nom"
version = "8.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "df9761775871bdef83bee530e60050f7e54b1105350d6884eb0fb4f46c2f9405"
dependencies = [
"memchr",
]
[[package]]
name = "noop_proc_macro"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0676bb32a98c1a483ce53e500a81ad9c3d5b3f7c920c28c24e9cb0980d0b5bc8"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "nu-ansi-term"
version = "0.50.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5"
dependencies = [
"windows-sys 0.61.2",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "num-bigint"
version = "0.4.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a5e44f723f1133c9deac646763579fdb3ac745e418f2a7af9cd0c431da1f20b9"
dependencies = [
"num-integer",
"num-traits",
]
[[package]]
name = "num-complex"
version = "0.4.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "73f88a1307638156682bada9d7604135552957b7818057dcef22705b4d509495"
dependencies = [
"num-traits",
]
[[package]]
name = "num-conv"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.2.2"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "521739c6d2bac4aa25192232afe6841231376b2b26d4d9fae5ecf8ca5772e441"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "num-derive"
version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ed3955f1a9c7c0c15e092f9c887db08b1fc683305fdf6eb6684f22555355e202"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "num-integer"
version = "0.1.46"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7969661fd2958a5cb096e56c8e1ad0444ac2bbcd0061bd28660485a44879858f"
dependencies = [
"num-traits",
]
[[package]]
name = "num-rational"
version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f83d14da390562dca69fc84082e73e548e1ad308d24accdedd2720017cb37824"
dependencies = [
"num-bigint",
"num-integer",
"num-traits",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "num-traits"
version = "0.2.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841"
dependencies = [
"autocfg",
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "num_cpus"
version = "1.17.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "91df4bbde75afed763b708b7eee1e8e7651e02d97f6d5dd763e89367e957b23b"
dependencies = [
"hermit-abi",
"libc",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "number_prefix"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "830b246a0e5f20af87141b25c173cd1b609bd7779a4617d6ec582abaf90870f3"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "once_cell"
version = "1.21.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50"
[[package]]
name = "once_cell_polyfill"
version = "1.70.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "onig"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "6.5.3"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "0cc3cbf698f9438986c11a880c90a6d04b9de27575afd28bbf45b154b6c709e2"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
dependencies = [
"bitflags",
"libc",
"once_cell",
"onig_sys",
]
[[package]]
name = "onig_sys"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "69.9.3"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "1e68317604e77e53b85896388e1a803c1d21b74c899ec9e5e1112db90735edd7"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
dependencies = [
"cc",
"pkg-config",
]
[[package]]
name = "oorandom"
version = "11.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d6790f58c7ff633d8771f42965289203411a5e5c68388703c06e14f24770b41e"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "openssl-probe"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe"
feat(rust): HfTokenizer::from_pretrained — HuggingFace Hub auto-download Stage 2.1: closes the loop on the HuggingFace tokenizer story. Stage 2 shipped `HfTokenizer::from_bytes`/`from_file`, which required callers to manage their own tokenizer.json files. This adds the third constructor: let t = HfTokenizer::from_pretrained("CohereForAI/c4ai-command-r-v01")?; register_hf("command-", t); `from_pretrained` is a thin wrapper around the `hf-hub` crate's blocking `ureq` API. First call downloads `tokenizer.json` to `~/.cache/huggingface/ hub` (or `$HF_HOME` if set); subsequent calls reuse the on-disk cache. Uses the `main` revision; gated repos (Llama, Mistral) require `HF_TOKEN` in env or `~/.cache/huggingface/token`. Also adds `try_register_hf(prefix, repo)` as the obvious one-liner for proxy startup code: let _ = try_register_hf("command-", "CohereForAI/c4ai-command-r-v01"); let _ = try_register_hf("mistral-", "mistralai/Mistral-7B-v0.1"); Each call is independent — a download failure for one model (e.g. gated without a token) does not affect others. `HfTokenizerError` gains a new `Hub` variant so callers can distinguish "couldn't fetch" from "fetched but malformed" — relevant when deciding whether to retry, surface to the user, or fall back to the estimator. Why blocking, not async: `from_pretrained` is called once at startup. A sync API works from `main()`, from a `OnceLock` initializer, or from `tokio::task::spawn_blocking` if a tokio caller needs it later. The async hf-hub backend would force callers to await at startup, which doesn't fit the `register_hf` registry pattern. Why rustls, not native-tls: keeps the binary statically linkable for AWS deploys (no system OpenSSL dependency). Tests: a network-dependent integration test (`#[ignore]`d in CI; hits HF for `gpt2`, ~1.4 MB) verifies the real download + load + count path. A non-network negative test verifies that an invalid repo name surfaces as `HfTokenizerError::Hub`, not a panic. 44 unit tests + 5 proptests + 1 doctest pass; parity stays 40/40 byte-equal.
2026-04-25 15:05:09 -07:00
[[package]]
name = "option-ext"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "04744f49eae99ab78e0d5c0b603ab218f515ea8cfe5a456d7629ad883a3b6e7d"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "ort"
version = "2.0.0-rc.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d7de3af33d24a745ffb8fab904b13478438d1cd52868e6f17735ef6e1f8bf133"
dependencies = [
Fix Windows ORT builds and Docker signing retries
2026-05-10 20:59:28 -07:00
"libloading",
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
"ndarray",
"ort-sys",
"smallvec",
"tracing",
"ureq 3.3.0",
]
[[package]]
name = "ort-sys"
version = "2.0.0-rc.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d7b497d21a8b6fbb4b5a544f8fadb77e801a09ae0add9e411d31c6f89e3c1e90"
dependencies = [
"hmac-sha256",
"lzma-rust2",
"ureq 3.3.0",
]
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
[[package]]
name = "outref"
version = "0.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1a80800c0488c3a21695ea981a54918fbb37abf04f4d0720c453632255e2ff0e"
fix(proxy): PR-D3 Bedrock observability + auth-mode integration Phase D close. Adds the operator-facing observability surface that PRs D1 (native invoke) and D2 (streaming EventStream) deferred, and wires the Phase F PR-F1 auth-mode classifier into the Bedrock route so downstream cache/compression policy gates have something to read. Changes ------- * New `bedrock::auth_mode_layer` middleware. Classifies every inbound Bedrock request via F1's `classify`, coerces the result to `AuthMode::OAuth` per the Bedrock policy matrix (SigV4 IAM is OAuth-equivalent), and stores the resolved value in `request.extensions()` so PR-F2/F3 can read it without re-classifying. Mismatches are logged at WARN with `event=bedrock_auth_mode_unexpected` — no silent coercion. * New `observability` module with three Prometheus families: - `bedrock_invoke_count_total{model, region, auth_mode}` (counter) - `bedrock_invoke_latency_seconds{model, region}` (histogram) - `bedrock_eventstream_message_count_total{model, region, event_type}` (counter) Registered lazily via `OnceLock` so per-request work is just `inc_with_label_values` / `observe`. Latency observed via an RAII `LatencyGuard` so every error path is instrumented; a future regression that adds a new return path can't drop the observation. * New `GET /metrics` endpoint serves the registry in Prometheus text format. Mounted unconditionally — no feature flag gate — so scrape works regardless of which provider routes are mounted. * Bedrock invoke + invoke-streaming handlers now extract `Extension<AuthMode>`, log it in their entry breadcrumbs (`event=bedrock_invoke_received`, `event=bedrock_invoke_streaming_received`), and pass `model`/`region` into `translate_stream` so per-message metrics carry the right labels. * Operator docs at `docs/bedrock.md`: AWS credential chain, region/endpoint config, supported model IDs (`anthropic.*` literal-match — no regexes), compression behaviour, sample PromQL queries, structured-log correlation, rollback path. Tests added (6, all green) -------------------------- Auth-mode (`integration_bedrock_authmode.rs`): 1. `bedrock_classified_as_oauth` — empty headers → OAuth in extensions. 2. `oauth_policy_passthrough_prefer` — body byte-equal upstream; no auto cache_control / prompt_cache_key injected. Metrics (`integration_bedrock_metrics.rs`): 3. `metrics_increment_per_invoke` — 3 invokes → counter=3 with correct labels. 4. `metrics_observe_latency` — 1 invoke → histogram count=1, sum>0. 5. `eventstream_metrics_per_message_type` — 5 chunks → counter=5 with `event_type=chunk`. 6. `metrics_endpoint_serves_scrape` — `/metrics` returns 200, `text/plain`, all three metric families' HELP/TYPE lines present. Each metrics test owns a unique (model, region) tuple so the global `prometheus` registry — shared across parallel tests in the same binary — gives each test isolated label rows. Without isolation, parallel tests cross-contaminate counters. Constraints honoured -------------------- * No silent fallbacks — auth-mode coercion is logged at WARN. * No hardcodes — region from `--bedrock-region`, model from axum path parameter. * No regexes — vendor prefix is literal `anthropic.`. * Comprehensive structured logs — every metric increment paired with `tracing::debug!` carrying the same labels for incident correlation. * Performant — `OnceLock`-cached descriptors, RAII guard, total D3 overhead well under 1us per request. * Cardinality bounded — labels driven by config + bounded enums, never by user-controlled bytes. Live cloud validation deferred ------------------------------ The wiremock-backed integration tests are the canonical correctness gate for D3. A real Bedrock smoke test requires `bedrock:InvokeModel` permissions in the developer's AWS account and is documented in `docs/bedrock.md` — both D1 and D2 hit sandbox permission issues trying this path; D3 follows the same convention. Stacked on ---------- PR #364 (D1 native invoke), PR #365 (D2 streaming EventStream), PR #366 (F1 classifier helper). Merge those first; this PR will be rebased onto main once they land.
2026-05-03 17:57:00 -07:00
[[package]]
name = "parking_lot"
version = "0.12.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "93857453250e3077bd71ff98b6a65ea6621a19bb0f559a85248955ac12c45a1a"
dependencies = [
"lock_api",
"parking_lot_core",
]
perf(rust): tier-1 multi-worker wins — GIL release, sharded CCR store, single-serialize CCR write Three orthogonal hot-path fixes targeting concurrent-request throughput. Each is independently bench-measured below; the proxy hot path benefits from all three at once. == 1. PyO3 GIL release on heavy compute == PyO3 methods (crush, smart_crush_content, crush_array_json, compact_document_json, compress, compress_with_stats) used to hold the GIL across the entire Rust call. Result: a 100ms compress() blocked EVERY other Python thread for 100ms — multi-worker uvicorn deployments serialized through SmartCrusher. Wrap each compute call in `py.allow_threads(|| ...)`. Inputs (`&str` from Python) are copied to owned `String` first because PyO3 ties them to the GIL hold. PyDict construction stays on the GIL side. Measured: 4 Python threads each running 20 crushes: before (GIL held): ~3.3s wall (serialized — equivalent to 4×0.83s) after (allow_threads): 826ms wall (4.01x speedup, perfect parallel) == 2. CcrStore: Mutex<HashMap> -> DashMap-backed sharded == Single Mutex was the dominant bottleneck under multi-worker load — every put/get serialized through one lock. Replace with DashMap (sharded concurrent map, lock-free reads within a shard) plus a separate small Mutex<VecDeque> for FIFO insertion-order eviction. Reads of distinct keys never contend; writes only contend during the brief order-queue push or capacity-sweep. A/B bench (200 mixed put/get ops × N threads, in benches/ccr_store.rs): Threads | DashMap Legacy Mutex Speedup ------------------------------------------- 1 | 63 µs 71 µs 1.13x 2 | 98 µs 194 µs 2.0x 4 | 178 µs 707 µs 4.0x 8 | 342 µs 1267 µs 3.7x Legacy degrades ~linearly with thread count; DashMap stays near-flat per-thread. Real multi-worker scaling. == 3. Single-serialize the lossy CCR payload == The lossy `crush_array` path used to serialize the full array TWICE: once in `hash_array_for_ccr` (allocates `Value::Array(items.to_vec())`, deep-clones every Value subtree, then serializes), and a second time in the store-write site. For a 50-item dict array that's ~MB of allocator pressure per crushed array. Introduce `canonical_array_json` (serializes `&[Value]` directly — same bytes as `Value::Array(items.to_vec())` but no wrapper allocation + no tree clone), call it ONCE per lossy path, then both hash and store from those same bytes. Hash-format stable — all 17 parity fixtures match byte-for-byte. == Tests == - 8 ccr.rs unit tests including a new concurrent-stress test (8 threads × 200 puts/gets, every key readable afterwards) - 14 ccr_roundtrip integration tests stay green - parity-run smart_crusher: 17/17 fixtures match - 479 lib + 14 integration + 185 Python tests all pass - New benches/ccr_store.rs runs the A/B and is committed for regression visibility == Dependencies added == - dashmap v6 (mature, widely-used in tokio/linkerd ecosystem)
2026-04-27 20:44:55 -07:00
[[package]]
name = "parking_lot_core"
version = "0.9.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2621685985a2ebf1c516881c026032ac7deafcda1a2c9b7850dc81e3dfcb64c1"
dependencies = [
"cfg-if",
"libc",
"redox_syscall",
"smallvec",
"windows-link",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "paste"
version = "1.0.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "pastey"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "35fb2e5f958ec131621fdd531e9fc186ed768cbe395337403ae56c17a74c68ec"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "percent-encoding"
version = "2.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220"
fix(proxy): PR-D4 native Vertex publisher path + ADC bearer auth Adds a Rust-native Vertex AI publisher route ahead of the LiteLLM Python converter (which dropped `thinking`, `redacted_thinking`, `document`, `image`, `server_tool_use`, `mcp_tool_use` block kinds — the P4-37 / P4-38 bug). After this PR the Vertex `:rawPredict` and `:streamRawPredict` calls survive byte-equal upstream and benefit from the live-zone Anthropic dispatcher (PR-B-series) running over the body — same behaviour as `/v1/messages`. New module `crates/headroom-proxy/src/vertex/`: - `mod.rs` — single dispatch handler at the `/v1beta1/.../models/:model_action` route. Splits the trailing `:<verb>` segment with `str::rsplit_once(':')` (no regex) and flips an `attach_sse_tee` flag to dispatch to the streaming or non-streaming arm. Both verbs share one axum route shape because matchit can't distinguish two patterns that overlap on a parameter. - `envelope.rs` — `VertexEnvelope` parser. Confirms `anthropic_version` present + `model` field absent (the two fingerprints of the Vertex envelope vs `/v1/messages`). - `adc.rs` — `TokenSource` trait + `GcpAdcTokenSource` (production, `gcp_auth` 0.12) + `StaticTokenSource` (tests). Caches tokens with a 60s refresh-ahead-of-expiry window. Emits structured `event = "vertex_adc_token_refreshed"` per refresh. - `raw_predict.rs` — POST handler + shared `forward_vertex_request`. Buffers body, parses envelope, runs live-zone Anthropic compression, fetches ADC bearer, attaches `Authorization: Bearer <token>` (overwrites client-supplied Authorization header), forwards. SSE telemetry tee for the streaming verb reuses PR-C1's `AnthropicStreamState` directly (Vertex streams plain SSE, unlike Bedrock's binary EventStream). - `stream_raw_predict.rs` — module-level docs + alias to the shared dispatcher (the streaming-vs-non-streaming difference is one boolean flag inside the shared forwarder). Modifications: - `proxy.rs::build_app` — registers the single Vertex route. - `proxy.rs::AppState` — new `vertex_token_source: Arc<dyn TokenSource>` field. Production constructs `GcpAdcTokenSource` lazily (no GCP call until first `bearer()`); tests inject `StaticTokenSource` via the new `AppState::with_token_source` helper. - `config.rs` — adds `--vertex-region` / `HEADROOM_PROXY_VERTEX_REGION` (default `us-central1`, observability tag only — the upstream URL is `--upstream`) and `--vertex-adc-scope` / `HEADROOM_PROXY_VERTEX_ADC_SCOPE` (default `cloud-platform`). - `Cargo.toml` (workspace + proxy) — adds `gcp_auth = "0.12"` and `async-trait = "0.1"`. - `tests/common/mod.rs` — `start_proxy_with_state` accepts both config + state customizers; `install_static_token_source` helper for tests. `crates/headroom-proxy/tests/integration_vertex_raw_predict.rs` — all five tests pass: 1. `native_envelope_round_trip_byte_equal` — Vertex-shape body (with `anthropic_version`, no `model`) round-trips SHA-256 byte-equal upstream. 2. `adc_bearer_token_signed_correctly` — `Authorization: Bearer <static-test-token>` reaches upstream verbatim and OVERWRITES a client-supplied Authorization header. 3. `thinking_block_preserved` — request with `thinking` (incl. signature) + `redacted_thinking` (incl. opaque `data`) blocks round-trips byte-equal even with `LiveZone` compression mode enabled. This is the P4-37 / P4-38 teeth. 4. `stream_raw_predict_sse_handled` — `:streamRawPredict` proxies an Anthropic SSE response (full `message_start` → `content_block_delta` → `message_stop` sequence) back to the client without corruption; SSE content-type preserved end-to-end; bearer attached. 5. (bonus, no-silent-fallback contract) `adc_failure_returns_5xx_no_silent_forward` — when the token source returns `Err`, the proxy returns 5xx and never reaches upstream. Verifies the `event = "vertex_adc_fetch_failed"` error path. Workspace: `cargo test --workspace` green; `cargo clippy --workspace -- -D warnings` clean; `make ci-precheck` passes. - No silent fallbacks: ADC failure → structured 5xx, never an unauthenticated forward. - No hardcodes: every knob (region, ADC scope, upstream URL) is CLI-flag + env-var configurable. - No regexes: axum path parameters + `str::rsplit_once` only. - Comprehensive structured logs: `event` field on every decision point — `vertex_envelope_parsed`, `vertex_envelope_invalid`, `vertex_compression_skipped`, `vertex_compression_applied`, `vertex_adc_token_refreshed`, `vertex_adc_fetch_failed`, `vertex_streaming_pipeline_active`, `vertex_sse_stream_closed`, `vertex_forwarded`, `vertex_unknown_verb`, etc. - Performant: no body clone; ADC token cached + refreshed ahead-of-expiry, not fetched per request. - Comprehensive tests: realistic Anthropic block content (signature payload, redacted_thinking opaque blob) in `thinking_block_preserved`. The local `gcloud auth application-default print-access-token` returns no credentials, so manual validation against a real Vertex endpoint is not possible in this PR. Follow-up: the user runs `gcloud auth application-default login` once and exercises a live Vertex request — should be a no-code-change check. PR-D1 (Bedrock native) is running concurrently and will land its own envelope module at `crates/headroom-proxy/src/bedrock/envelope.rs`. The two envelope modules are intentionally siblings (not a shared trait) — the shapes differ (Bedrock has a different `anthropic_version` value, no `model` field, AWS SigV4 instead of GCP ADC), and a premature shared abstraction would obscure the provider-specific contracts. Whichever PR merges second rebases without conflict. Retires P4-38 (and the Vertex parts of P4-39); marketplace BYOC pitch (per project memory) gets one more native provider.
2026-05-03 16:22:07 -07:00
[[package]]
name = "pin-project"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.1.13"
fix(proxy): PR-D4 native Vertex publisher path + ADC bearer auth Adds a Rust-native Vertex AI publisher route ahead of the LiteLLM Python converter (which dropped `thinking`, `redacted_thinking`, `document`, `image`, `server_tool_use`, `mcp_tool_use` block kinds — the P4-37 / P4-38 bug). After this PR the Vertex `:rawPredict` and `:streamRawPredict` calls survive byte-equal upstream and benefit from the live-zone Anthropic dispatcher (PR-B-series) running over the body — same behaviour as `/v1/messages`. New module `crates/headroom-proxy/src/vertex/`: - `mod.rs` — single dispatch handler at the `/v1beta1/.../models/:model_action` route. Splits the trailing `:<verb>` segment with `str::rsplit_once(':')` (no regex) and flips an `attach_sse_tee` flag to dispatch to the streaming or non-streaming arm. Both verbs share one axum route shape because matchit can't distinguish two patterns that overlap on a parameter. - `envelope.rs` — `VertexEnvelope` parser. Confirms `anthropic_version` present + `model` field absent (the two fingerprints of the Vertex envelope vs `/v1/messages`). - `adc.rs` — `TokenSource` trait + `GcpAdcTokenSource` (production, `gcp_auth` 0.12) + `StaticTokenSource` (tests). Caches tokens with a 60s refresh-ahead-of-expiry window. Emits structured `event = "vertex_adc_token_refreshed"` per refresh. - `raw_predict.rs` — POST handler + shared `forward_vertex_request`. Buffers body, parses envelope, runs live-zone Anthropic compression, fetches ADC bearer, attaches `Authorization: Bearer <token>` (overwrites client-supplied Authorization header), forwards. SSE telemetry tee for the streaming verb reuses PR-C1's `AnthropicStreamState` directly (Vertex streams plain SSE, unlike Bedrock's binary EventStream). - `stream_raw_predict.rs` — module-level docs + alias to the shared dispatcher (the streaming-vs-non-streaming difference is one boolean flag inside the shared forwarder). Modifications: - `proxy.rs::build_app` — registers the single Vertex route. - `proxy.rs::AppState` — new `vertex_token_source: Arc<dyn TokenSource>` field. Production constructs `GcpAdcTokenSource` lazily (no GCP call until first `bearer()`); tests inject `StaticTokenSource` via the new `AppState::with_token_source` helper. - `config.rs` — adds `--vertex-region` / `HEADROOM_PROXY_VERTEX_REGION` (default `us-central1`, observability tag only — the upstream URL is `--upstream`) and `--vertex-adc-scope` / `HEADROOM_PROXY_VERTEX_ADC_SCOPE` (default `cloud-platform`). - `Cargo.toml` (workspace + proxy) — adds `gcp_auth = "0.12"` and `async-trait = "0.1"`. - `tests/common/mod.rs` — `start_proxy_with_state` accepts both config + state customizers; `install_static_token_source` helper for tests. `crates/headroom-proxy/tests/integration_vertex_raw_predict.rs` — all five tests pass: 1. `native_envelope_round_trip_byte_equal` — Vertex-shape body (with `anthropic_version`, no `model`) round-trips SHA-256 byte-equal upstream. 2. `adc_bearer_token_signed_correctly` — `Authorization: Bearer <static-test-token>` reaches upstream verbatim and OVERWRITES a client-supplied Authorization header. 3. `thinking_block_preserved` — request with `thinking` (incl. signature) + `redacted_thinking` (incl. opaque `data`) blocks round-trips byte-equal even with `LiveZone` compression mode enabled. This is the P4-37 / P4-38 teeth. 4. `stream_raw_predict_sse_handled` — `:streamRawPredict` proxies an Anthropic SSE response (full `message_start` → `content_block_delta` → `message_stop` sequence) back to the client without corruption; SSE content-type preserved end-to-end; bearer attached. 5. (bonus, no-silent-fallback contract) `adc_failure_returns_5xx_no_silent_forward` — when the token source returns `Err`, the proxy returns 5xx and never reaches upstream. Verifies the `event = "vertex_adc_fetch_failed"` error path. Workspace: `cargo test --workspace` green; `cargo clippy --workspace -- -D warnings` clean; `make ci-precheck` passes. - No silent fallbacks: ADC failure → structured 5xx, never an unauthenticated forward. - No hardcodes: every knob (region, ADC scope, upstream URL) is CLI-flag + env-var configurable. - No regexes: axum path parameters + `str::rsplit_once` only. - Comprehensive structured logs: `event` field on every decision point — `vertex_envelope_parsed`, `vertex_envelope_invalid`, `vertex_compression_skipped`, `vertex_compression_applied`, `vertex_adc_token_refreshed`, `vertex_adc_fetch_failed`, `vertex_streaming_pipeline_active`, `vertex_sse_stream_closed`, `vertex_forwarded`, `vertex_unknown_verb`, etc. - Performant: no body clone; ADC token cached + refreshed ahead-of-expiry, not fetched per request. - Comprehensive tests: realistic Anthropic block content (signature payload, redacted_thinking opaque blob) in `thinking_block_preserved`. The local `gcloud auth application-default print-access-token` returns no credentials, so manual validation against a real Vertex endpoint is not possible in this PR. Follow-up: the user runs `gcloud auth application-default login` once and exercises a live Vertex request — should be a no-code-change check. PR-D1 (Bedrock native) is running concurrently and will land its own envelope module at `crates/headroom-proxy/src/bedrock/envelope.rs`. The two envelope modules are intentionally siblings (not a shared trait) — the shapes differ (Bedrock has a different `anthropic_version` value, no `model` field, AWS SigV4 instead of GCP ADC), and a premature shared abstraction would obscure the provider-specific contracts. Whichever PR merges second rebases without conflict. Retires P4-38 (and the Vertex parts of P4-39); marketplace BYOC pitch (per project memory) gets one more native provider.
2026-05-03 16:22:07 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "2466b2336ed02bcdca6b294417127b90ec92038d1d5c4fbeac971a922e0e0924"
fix(proxy): PR-D4 native Vertex publisher path + ADC bearer auth Adds a Rust-native Vertex AI publisher route ahead of the LiteLLM Python converter (which dropped `thinking`, `redacted_thinking`, `document`, `image`, `server_tool_use`, `mcp_tool_use` block kinds — the P4-37 / P4-38 bug). After this PR the Vertex `:rawPredict` and `:streamRawPredict` calls survive byte-equal upstream and benefit from the live-zone Anthropic dispatcher (PR-B-series) running over the body — same behaviour as `/v1/messages`. New module `crates/headroom-proxy/src/vertex/`: - `mod.rs` — single dispatch handler at the `/v1beta1/.../models/:model_action` route. Splits the trailing `:<verb>` segment with `str::rsplit_once(':')` (no regex) and flips an `attach_sse_tee` flag to dispatch to the streaming or non-streaming arm. Both verbs share one axum route shape because matchit can't distinguish two patterns that overlap on a parameter. - `envelope.rs` — `VertexEnvelope` parser. Confirms `anthropic_version` present + `model` field absent (the two fingerprints of the Vertex envelope vs `/v1/messages`). - `adc.rs` — `TokenSource` trait + `GcpAdcTokenSource` (production, `gcp_auth` 0.12) + `StaticTokenSource` (tests). Caches tokens with a 60s refresh-ahead-of-expiry window. Emits structured `event = "vertex_adc_token_refreshed"` per refresh. - `raw_predict.rs` — POST handler + shared `forward_vertex_request`. Buffers body, parses envelope, runs live-zone Anthropic compression, fetches ADC bearer, attaches `Authorization: Bearer <token>` (overwrites client-supplied Authorization header), forwards. SSE telemetry tee for the streaming verb reuses PR-C1's `AnthropicStreamState` directly (Vertex streams plain SSE, unlike Bedrock's binary EventStream). - `stream_raw_predict.rs` — module-level docs + alias to the shared dispatcher (the streaming-vs-non-streaming difference is one boolean flag inside the shared forwarder). Modifications: - `proxy.rs::build_app` — registers the single Vertex route. - `proxy.rs::AppState` — new `vertex_token_source: Arc<dyn TokenSource>` field. Production constructs `GcpAdcTokenSource` lazily (no GCP call until first `bearer()`); tests inject `StaticTokenSource` via the new `AppState::with_token_source` helper. - `config.rs` — adds `--vertex-region` / `HEADROOM_PROXY_VERTEX_REGION` (default `us-central1`, observability tag only — the upstream URL is `--upstream`) and `--vertex-adc-scope` / `HEADROOM_PROXY_VERTEX_ADC_SCOPE` (default `cloud-platform`). - `Cargo.toml` (workspace + proxy) — adds `gcp_auth = "0.12"` and `async-trait = "0.1"`. - `tests/common/mod.rs` — `start_proxy_with_state` accepts both config + state customizers; `install_static_token_source` helper for tests. `crates/headroom-proxy/tests/integration_vertex_raw_predict.rs` — all five tests pass: 1. `native_envelope_round_trip_byte_equal` — Vertex-shape body (with `anthropic_version`, no `model`) round-trips SHA-256 byte-equal upstream. 2. `adc_bearer_token_signed_correctly` — `Authorization: Bearer <static-test-token>` reaches upstream verbatim and OVERWRITES a client-supplied Authorization header. 3. `thinking_block_preserved` — request with `thinking` (incl. signature) + `redacted_thinking` (incl. opaque `data`) blocks round-trips byte-equal even with `LiveZone` compression mode enabled. This is the P4-37 / P4-38 teeth. 4. `stream_raw_predict_sse_handled` — `:streamRawPredict` proxies an Anthropic SSE response (full `message_start` → `content_block_delta` → `message_stop` sequence) back to the client without corruption; SSE content-type preserved end-to-end; bearer attached. 5. (bonus, no-silent-fallback contract) `adc_failure_returns_5xx_no_silent_forward` — when the token source returns `Err`, the proxy returns 5xx and never reaches upstream. Verifies the `event = "vertex_adc_fetch_failed"` error path. Workspace: `cargo test --workspace` green; `cargo clippy --workspace -- -D warnings` clean; `make ci-precheck` passes. - No silent fallbacks: ADC failure → structured 5xx, never an unauthenticated forward. - No hardcodes: every knob (region, ADC scope, upstream URL) is CLI-flag + env-var configurable. - No regexes: axum path parameters + `str::rsplit_once` only. - Comprehensive structured logs: `event` field on every decision point — `vertex_envelope_parsed`, `vertex_envelope_invalid`, `vertex_compression_skipped`, `vertex_compression_applied`, `vertex_adc_token_refreshed`, `vertex_adc_fetch_failed`, `vertex_streaming_pipeline_active`, `vertex_sse_stream_closed`, `vertex_forwarded`, `vertex_unknown_verb`, etc. - Performant: no body clone; ADC token cached + refreshed ahead-of-expiry, not fetched per request. - Comprehensive tests: realistic Anthropic block content (signature payload, redacted_thinking opaque blob) in `thinking_block_preserved`. The local `gcloud auth application-default print-access-token` returns no credentials, so manual validation against a real Vertex endpoint is not possible in this PR. Follow-up: the user runs `gcloud auth application-default login` once and exercises a live Vertex request — should be a no-code-change check. PR-D1 (Bedrock native) is running concurrently and will land its own envelope module at `crates/headroom-proxy/src/bedrock/envelope.rs`. The two envelope modules are intentionally siblings (not a shared trait) — the shapes differ (Bedrock has a different `anthropic_version` value, no `model` field, AWS SigV4 instead of GCP ADC), and a premature shared abstraction would obscure the provider-specific contracts. Whichever PR merges second rebases without conflict. Retires P4-38 (and the Vertex parts of P4-39); marketplace BYOC pitch (per project memory) gets one more native provider.
2026-05-03 16:22:07 -07:00
dependencies = [
"pin-project-internal",
]
[[package]]
name = "pin-project-internal"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.1.13"
fix(proxy): PR-D4 native Vertex publisher path + ADC bearer auth Adds a Rust-native Vertex AI publisher route ahead of the LiteLLM Python converter (which dropped `thinking`, `redacted_thinking`, `document`, `image`, `server_tool_use`, `mcp_tool_use` block kinds — the P4-37 / P4-38 bug). After this PR the Vertex `:rawPredict` and `:streamRawPredict` calls survive byte-equal upstream and benefit from the live-zone Anthropic dispatcher (PR-B-series) running over the body — same behaviour as `/v1/messages`. New module `crates/headroom-proxy/src/vertex/`: - `mod.rs` — single dispatch handler at the `/v1beta1/.../models/:model_action` route. Splits the trailing `:<verb>` segment with `str::rsplit_once(':')` (no regex) and flips an `attach_sse_tee` flag to dispatch to the streaming or non-streaming arm. Both verbs share one axum route shape because matchit can't distinguish two patterns that overlap on a parameter. - `envelope.rs` — `VertexEnvelope` parser. Confirms `anthropic_version` present + `model` field absent (the two fingerprints of the Vertex envelope vs `/v1/messages`). - `adc.rs` — `TokenSource` trait + `GcpAdcTokenSource` (production, `gcp_auth` 0.12) + `StaticTokenSource` (tests). Caches tokens with a 60s refresh-ahead-of-expiry window. Emits structured `event = "vertex_adc_token_refreshed"` per refresh. - `raw_predict.rs` — POST handler + shared `forward_vertex_request`. Buffers body, parses envelope, runs live-zone Anthropic compression, fetches ADC bearer, attaches `Authorization: Bearer <token>` (overwrites client-supplied Authorization header), forwards. SSE telemetry tee for the streaming verb reuses PR-C1's `AnthropicStreamState` directly (Vertex streams plain SSE, unlike Bedrock's binary EventStream). - `stream_raw_predict.rs` — module-level docs + alias to the shared dispatcher (the streaming-vs-non-streaming difference is one boolean flag inside the shared forwarder). Modifications: - `proxy.rs::build_app` — registers the single Vertex route. - `proxy.rs::AppState` — new `vertex_token_source: Arc<dyn TokenSource>` field. Production constructs `GcpAdcTokenSource` lazily (no GCP call until first `bearer()`); tests inject `StaticTokenSource` via the new `AppState::with_token_source` helper. - `config.rs` — adds `--vertex-region` / `HEADROOM_PROXY_VERTEX_REGION` (default `us-central1`, observability tag only — the upstream URL is `--upstream`) and `--vertex-adc-scope` / `HEADROOM_PROXY_VERTEX_ADC_SCOPE` (default `cloud-platform`). - `Cargo.toml` (workspace + proxy) — adds `gcp_auth = "0.12"` and `async-trait = "0.1"`. - `tests/common/mod.rs` — `start_proxy_with_state` accepts both config + state customizers; `install_static_token_source` helper for tests. `crates/headroom-proxy/tests/integration_vertex_raw_predict.rs` — all five tests pass: 1. `native_envelope_round_trip_byte_equal` — Vertex-shape body (with `anthropic_version`, no `model`) round-trips SHA-256 byte-equal upstream. 2. `adc_bearer_token_signed_correctly` — `Authorization: Bearer <static-test-token>` reaches upstream verbatim and OVERWRITES a client-supplied Authorization header. 3. `thinking_block_preserved` — request with `thinking` (incl. signature) + `redacted_thinking` (incl. opaque `data`) blocks round-trips byte-equal even with `LiveZone` compression mode enabled. This is the P4-37 / P4-38 teeth. 4. `stream_raw_predict_sse_handled` — `:streamRawPredict` proxies an Anthropic SSE response (full `message_start` → `content_block_delta` → `message_stop` sequence) back to the client without corruption; SSE content-type preserved end-to-end; bearer attached. 5. (bonus, no-silent-fallback contract) `adc_failure_returns_5xx_no_silent_forward` — when the token source returns `Err`, the proxy returns 5xx and never reaches upstream. Verifies the `event = "vertex_adc_fetch_failed"` error path. Workspace: `cargo test --workspace` green; `cargo clippy --workspace -- -D warnings` clean; `make ci-precheck` passes. - No silent fallbacks: ADC failure → structured 5xx, never an unauthenticated forward. - No hardcodes: every knob (region, ADC scope, upstream URL) is CLI-flag + env-var configurable. - No regexes: axum path parameters + `str::rsplit_once` only. - Comprehensive structured logs: `event` field on every decision point — `vertex_envelope_parsed`, `vertex_envelope_invalid`, `vertex_compression_skipped`, `vertex_compression_applied`, `vertex_adc_token_refreshed`, `vertex_adc_fetch_failed`, `vertex_streaming_pipeline_active`, `vertex_sse_stream_closed`, `vertex_forwarded`, `vertex_unknown_verb`, etc. - Performant: no body clone; ADC token cached + refreshed ahead-of-expiry, not fetched per request. - Comprehensive tests: realistic Anthropic block content (signature payload, redacted_thinking opaque blob) in `thinking_block_preserved`. The local `gcloud auth application-default print-access-token` returns no credentials, so manual validation against a real Vertex endpoint is not possible in this PR. Follow-up: the user runs `gcloud auth application-default login` once and exercises a live Vertex request — should be a no-code-change check. PR-D1 (Bedrock native) is running concurrently and will land its own envelope module at `crates/headroom-proxy/src/bedrock/envelope.rs`. The two envelope modules are intentionally siblings (not a shared trait) — the shapes differ (Bedrock has a different `anthropic_version` value, no `model` field, AWS SigV4 instead of GCP ADC), and a premature shared abstraction would obscure the provider-specific contracts. Whichever PR merges second rebases without conflict. Retires P4-38 (and the Vertex parts of P4-39); marketplace BYOC pitch (per project memory) gets one more native provider.
2026-05-03 16:22:07 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "c96395f0a926bc13b1c17622aaddda1ecb55d49c8f1bf9777e4d877800a43f8b"
fix(proxy): PR-D4 native Vertex publisher path + ADC bearer auth Adds a Rust-native Vertex AI publisher route ahead of the LiteLLM Python converter (which dropped `thinking`, `redacted_thinking`, `document`, `image`, `server_tool_use`, `mcp_tool_use` block kinds — the P4-37 / P4-38 bug). After this PR the Vertex `:rawPredict` and `:streamRawPredict` calls survive byte-equal upstream and benefit from the live-zone Anthropic dispatcher (PR-B-series) running over the body — same behaviour as `/v1/messages`. New module `crates/headroom-proxy/src/vertex/`: - `mod.rs` — single dispatch handler at the `/v1beta1/.../models/:model_action` route. Splits the trailing `:<verb>` segment with `str::rsplit_once(':')` (no regex) and flips an `attach_sse_tee` flag to dispatch to the streaming or non-streaming arm. Both verbs share one axum route shape because matchit can't distinguish two patterns that overlap on a parameter. - `envelope.rs` — `VertexEnvelope` parser. Confirms `anthropic_version` present + `model` field absent (the two fingerprints of the Vertex envelope vs `/v1/messages`). - `adc.rs` — `TokenSource` trait + `GcpAdcTokenSource` (production, `gcp_auth` 0.12) + `StaticTokenSource` (tests). Caches tokens with a 60s refresh-ahead-of-expiry window. Emits structured `event = "vertex_adc_token_refreshed"` per refresh. - `raw_predict.rs` — POST handler + shared `forward_vertex_request`. Buffers body, parses envelope, runs live-zone Anthropic compression, fetches ADC bearer, attaches `Authorization: Bearer <token>` (overwrites client-supplied Authorization header), forwards. SSE telemetry tee for the streaming verb reuses PR-C1's `AnthropicStreamState` directly (Vertex streams plain SSE, unlike Bedrock's binary EventStream). - `stream_raw_predict.rs` — module-level docs + alias to the shared dispatcher (the streaming-vs-non-streaming difference is one boolean flag inside the shared forwarder). Modifications: - `proxy.rs::build_app` — registers the single Vertex route. - `proxy.rs::AppState` — new `vertex_token_source: Arc<dyn TokenSource>` field. Production constructs `GcpAdcTokenSource` lazily (no GCP call until first `bearer()`); tests inject `StaticTokenSource` via the new `AppState::with_token_source` helper. - `config.rs` — adds `--vertex-region` / `HEADROOM_PROXY_VERTEX_REGION` (default `us-central1`, observability tag only — the upstream URL is `--upstream`) and `--vertex-adc-scope` / `HEADROOM_PROXY_VERTEX_ADC_SCOPE` (default `cloud-platform`). - `Cargo.toml` (workspace + proxy) — adds `gcp_auth = "0.12"` and `async-trait = "0.1"`. - `tests/common/mod.rs` — `start_proxy_with_state` accepts both config + state customizers; `install_static_token_source` helper for tests. `crates/headroom-proxy/tests/integration_vertex_raw_predict.rs` — all five tests pass: 1. `native_envelope_round_trip_byte_equal` — Vertex-shape body (with `anthropic_version`, no `model`) round-trips SHA-256 byte-equal upstream. 2. `adc_bearer_token_signed_correctly` — `Authorization: Bearer <static-test-token>` reaches upstream verbatim and OVERWRITES a client-supplied Authorization header. 3. `thinking_block_preserved` — request with `thinking` (incl. signature) + `redacted_thinking` (incl. opaque `data`) blocks round-trips byte-equal even with `LiveZone` compression mode enabled. This is the P4-37 / P4-38 teeth. 4. `stream_raw_predict_sse_handled` — `:streamRawPredict` proxies an Anthropic SSE response (full `message_start` → `content_block_delta` → `message_stop` sequence) back to the client without corruption; SSE content-type preserved end-to-end; bearer attached. 5. (bonus, no-silent-fallback contract) `adc_failure_returns_5xx_no_silent_forward` — when the token source returns `Err`, the proxy returns 5xx and never reaches upstream. Verifies the `event = "vertex_adc_fetch_failed"` error path. Workspace: `cargo test --workspace` green; `cargo clippy --workspace -- -D warnings` clean; `make ci-precheck` passes. - No silent fallbacks: ADC failure → structured 5xx, never an unauthenticated forward. - No hardcodes: every knob (region, ADC scope, upstream URL) is CLI-flag + env-var configurable. - No regexes: axum path parameters + `str::rsplit_once` only. - Comprehensive structured logs: `event` field on every decision point — `vertex_envelope_parsed`, `vertex_envelope_invalid`, `vertex_compression_skipped`, `vertex_compression_applied`, `vertex_adc_token_refreshed`, `vertex_adc_fetch_failed`, `vertex_streaming_pipeline_active`, `vertex_sse_stream_closed`, `vertex_forwarded`, `vertex_unknown_verb`, etc. - Performant: no body clone; ADC token cached + refreshed ahead-of-expiry, not fetched per request. - Comprehensive tests: realistic Anthropic block content (signature payload, redacted_thinking opaque blob) in `thinking_block_preserved`. The local `gcloud auth application-default print-access-token` returns no credentials, so manual validation against a real Vertex endpoint is not possible in this PR. Follow-up: the user runs `gcloud auth application-default login` once and exercises a live Vertex request — should be a no-code-change check. PR-D1 (Bedrock native) is running concurrently and will land its own envelope module at `crates/headroom-proxy/src/bedrock/envelope.rs`. The two envelope modules are intentionally siblings (not a shared trait) — the shapes differ (Bedrock has a different `anthropic_version` value, no `model` field, AWS SigV4 instead of GCP ADC), and a premature shared abstraction would obscure the provider-specific contracts. Whichever PR merges second rebases without conflict. Retires P4-38 (and the Vertex parts of P4-39); marketplace BYOC pitch (per project memory) gets one more native provider.
2026-05-03 16:22:07 -07:00
dependencies = [
"proc-macro2",
"quote",
"syn",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "pin-project-lite"
version = "0.2.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
[[package]]
name = "pin-utils"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "pkg-config"
version = "0.3.33"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e"
[[package]]
name = "plotters"
version = "0.3.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5aeb6f403d7a4911efb1e33402027fc44f29b5bf6def3effcc22d7bb75f2b747"
dependencies = [
"num-traits",
"plotters-backend",
"plotters-svg",
"wasm-bindgen",
"web-sys",
]
[[package]]
name = "plotters-backend"
version = "0.3.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "df42e13c12958a16b3f7f4386b9ab1f3e7933914ecea48da7139435263a4172a"
[[package]]
name = "plotters-svg"
version = "0.3.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "51bae2ac328883f7acdfea3d66a7c35751187f870bc81f94563733a154d7a670"
dependencies = [
"plotters-backend",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "png"
version = "0.18.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "60769b8b31b2a9f263dae2776c37b1b28ae246943cf719eb6946a1db05128a61"
dependencies = [
"bitflags",
"crc32fast",
"fdeflate",
"flate2",
"miniz_oxide",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "portable-atomic"
version = "1.13.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "portable-atomic-util"
version = "0.2.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618"
dependencies = [
"portable-atomic",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "potential_utf"
version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0103b1cef7ec0cf76490e969665504990193874ea05c85ff9bab8b911d0a0564"
dependencies = [
"zerovec",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "powerfmt"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "ppv-lite86"
version = "0.2.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9"
dependencies = [
"zerocopy",
]
[[package]]
name = "proc-macro2"
version = "1.0.106"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934"
dependencies = [
"unicode-ident",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "profiling"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.0.18"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "3d595e54a326bc53c1c197b32d295e14b169e3cfeaa8dc82b529f947fba6bcf5"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
dependencies = [
"profiling-procmacros",
]
[[package]]
name = "profiling-procmacros"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.0.18"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "4488a4a36b9a4ba6b9334a32a39971f77c1436ec82c38707bce707699cc3bbcb"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
dependencies = [
"quote",
"syn",
]
fix(proxy): PR-D3 Bedrock observability + auth-mode integration Phase D close. Adds the operator-facing observability surface that PRs D1 (native invoke) and D2 (streaming EventStream) deferred, and wires the Phase F PR-F1 auth-mode classifier into the Bedrock route so downstream cache/compression policy gates have something to read. Changes ------- * New `bedrock::auth_mode_layer` middleware. Classifies every inbound Bedrock request via F1's `classify`, coerces the result to `AuthMode::OAuth` per the Bedrock policy matrix (SigV4 IAM is OAuth-equivalent), and stores the resolved value in `request.extensions()` so PR-F2/F3 can read it without re-classifying. Mismatches are logged at WARN with `event=bedrock_auth_mode_unexpected` — no silent coercion. * New `observability` module with three Prometheus families: - `bedrock_invoke_count_total{model, region, auth_mode}` (counter) - `bedrock_invoke_latency_seconds{model, region}` (histogram) - `bedrock_eventstream_message_count_total{model, region, event_type}` (counter) Registered lazily via `OnceLock` so per-request work is just `inc_with_label_values` / `observe`. Latency observed via an RAII `LatencyGuard` so every error path is instrumented; a future regression that adds a new return path can't drop the observation. * New `GET /metrics` endpoint serves the registry in Prometheus text format. Mounted unconditionally — no feature flag gate — so scrape works regardless of which provider routes are mounted. * Bedrock invoke + invoke-streaming handlers now extract `Extension<AuthMode>`, log it in their entry breadcrumbs (`event=bedrock_invoke_received`, `event=bedrock_invoke_streaming_received`), and pass `model`/`region` into `translate_stream` so per-message metrics carry the right labels. * Operator docs at `docs/bedrock.md`: AWS credential chain, region/endpoint config, supported model IDs (`anthropic.*` literal-match — no regexes), compression behaviour, sample PromQL queries, structured-log correlation, rollback path. Tests added (6, all green) -------------------------- Auth-mode (`integration_bedrock_authmode.rs`): 1. `bedrock_classified_as_oauth` — empty headers → OAuth in extensions. 2. `oauth_policy_passthrough_prefer` — body byte-equal upstream; no auto cache_control / prompt_cache_key injected. Metrics (`integration_bedrock_metrics.rs`): 3. `metrics_increment_per_invoke` — 3 invokes → counter=3 with correct labels. 4. `metrics_observe_latency` — 1 invoke → histogram count=1, sum>0. 5. `eventstream_metrics_per_message_type` — 5 chunks → counter=5 with `event_type=chunk`. 6. `metrics_endpoint_serves_scrape` — `/metrics` returns 200, `text/plain`, all three metric families' HELP/TYPE lines present. Each metrics test owns a unique (model, region) tuple so the global `prometheus` registry — shared across parallel tests in the same binary — gives each test isolated label rows. Without isolation, parallel tests cross-contaminate counters. Constraints honoured -------------------- * No silent fallbacks — auth-mode coercion is logged at WARN. * No hardcodes — region from `--bedrock-region`, model from axum path parameter. * No regexes — vendor prefix is literal `anthropic.`. * Comprehensive structured logs — every metric increment paired with `tracing::debug!` carrying the same labels for incident correlation. * Performant — `OnceLock`-cached descriptors, RAII guard, total D3 overhead well under 1us per request. * Cardinality bounded — labels driven by config + bounded enums, never by user-controlled bytes. Live cloud validation deferred ------------------------------ The wiremock-backed integration tests are the canonical correctness gate for D3. A real Bedrock smoke test requires `bedrock:InvokeModel` permissions in the developer's AWS account and is documented in `docs/bedrock.md` — both D1 and D2 hit sandbox permission issues trying this path; D3 follows the same convention. Stacked on ---------- PR #364 (D1 native invoke), PR #365 (D2 streaming EventStream), PR #366 (F1 classifier helper). Merge those first; this PR will be rebased onto main once they land.
2026-05-03 17:57:00 -07:00
[[package]]
name = "prometheus"
version = "0.13.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3d33c28a30771f7f96db69893f78b857f7450d7e0237e9c8fc6427a81bae7ed1"
dependencies = [
"cfg-if",
"fnv",
"lazy_static",
"memchr",
"parking_lot",
"thiserror 1.0.69",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "proptest"
version = "1.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4b45fcc2344c680f5025fe57779faef368840d0bd1f42f216291f0dc4ace4744"
dependencies = [
"bit-set",
"bit-vec",
"bitflags",
"num-traits",
"rand 0.9.4",
"rand_chacha 0.9.0",
"rand_xorshift",
"regex-syntax",
"rusty-fork",
"tempfile",
"unarray",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "pxfm"
version = "0.1.29"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e0c5ccf5294c6ccd63a74f1565028353830a9c2f5eb0c682c355c471726a6e3f"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "pyo3"
fix(deps): remediate dependency CVEs and publish SBOM (#1509) ## Description Supply-chain hardening: takes the **shipped** dependency surface from **26 known CVEs to 0**. `pip install headroom-ai[all]` now resolves with no known vulnerabilities (verified with Anchore syft + grype). Also publishes a checked-in SBOM package (`sbom/`) so any user — especially pilots running their own security review — can verify what's inside and that we track it. This addresses the Dependabot alerts on `main` (9 high / 4 moderate / 7 low at time of writing). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made **Rust** - `pyo3` 0.24 → 0.29 (GHSA-36hh-v3qg-5jq4 High, GHSA-chgr-c6px-7xpp Med). Migrated `Python::allow_threads` → `Python::detach` (10 sites) and added `from_py_object` to the `Clone`-deriving `#[pyclass]` types (both required by the 0.25+ API). - `pyo3-log` 0.12 → 0.13; `lru` 0.12 → 0.18 (GHSA-rhfx-m35p-ff5j). **Python** - `torch` → 2.12.1, `mem0ai` → 2.x. - Floor-pinned transitive CVE deps via `[tool.uv] constraint-dependencies`: `pygments>=2.20.0`, `pydantic-settings>=2.14.2`, `gitpython>=3.1.50`, `langsmith>=0.9.0`. - **Removed `benchmark` from the `[all]` aggregate** so the default install is CVE-free. `lm-eval` is invoked as an external subprocess (`python -m lm_eval`) and never imported, so it is not a true runtime dep — it remains available via the opt-in `[benchmark]` extra. See [Accepted Risks](#additional-notes). **npm (build/test tooling — never shipped in the wheel/container/published SDK)** - `esbuild` override `>=0.28.1` in `sdk/typescript` + `plugins/openclaw` (GHSA-g7r4-m6w7-qqqr). - `docs/`: `@anthropic-ai/sdk` → `^0.106.0` (GHSA-p7fg-763f-g4gf), `postcss` override to force Next.js's bundled copy ≥8.5.10 (GHSA-qx2v-qp2m-jg93); regenerated a stale `bun.lock` that carried a **Critical** vitest/vite. **CI** - Pinned `pypa/gh-action-pypi-publish` `@release/v1` → `@v1.13.0` (GHSA-vxmw-7h4f-hqxh) in `release.yml` + `publish.yml`. **SBOM** - New `sbom/` directory: CycloneDX 1.7 + SPDX 2.3 SBOMs, grype scan evidence, 330-package license inventory, and a regeneration guide. ## Testing - [ ] Unit tests pass (`pytest`) — N/A, no Python source changed (deps/config only) - [x] Linting passes — `cargo fmt --check` + `cargo clippy` clean on the changed crate; 0 `.py` files changed so `ruff`/`mypy` scope is unaffected - [x] Type checking passes — `cargo check --workspace` (0 errors) - [ ] New tests added — N/A (dependency bumps; covered by existing suites) - [x] Manual testing performed — see Real Behavior Proof ### Test Output ```text # headroom-ai[all] product surface — the number that matters $ grype sbom:sbom/headroom-sbom-all-extra.cdx.json No vulnerabilities found # full repo scan (universal lock incl. opt-in [benchmark] + dev) $ grype sbom:sbom/headroom-sbom.cdx.json NAME INSTALLED TYPE VULNERABILITY SEVERITY sqlitedict 2.1.0 python GHSA-g4r7-86gm-pgqc High # [benchmark]-only, unpatchable, accepted nltk 3.9.4 python GHSA-p4gq-832x-fm9v High # [benchmark]-only, unpatchable, accepted # pyo3 0.29 migration — extension builds + imports + runs $ cargo check --workspace Finished `dev` profile [unoptimized + debuginfo] target(s) $ maturin develop && python -c "from headroom._core import DiffCompressor, SmartCrusher; ..." extension OK — detach + from_py_object paths exercised # lru 0.18 — eviction path $ cargo test -p headroom-proxy --lib drift 14 passed, 213 filtered out # per-ecosystem npm audits $ (cd sdk/typescript && npm audit) -> found 0 vulnerabilities $ (cd plugins/openclaw && npm audit) -> found 0 vulnerabilities $ (cd docs && npm audit && bun audit) -> found 0 vulnerabilities / No vulnerabilities found ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0, arm64), Python 3.12 `.venv`, Rust 1.95 toolchain, syft 1.46.0, grype 0.115.0, bun 1.3.14, maturin 1.13.3. - Exact command / steps: (1) `uv export --extra all --no-dev --no-emit-project | syft → grype` for the product surface; (2) `cargo check --workspace` + `maturin develop` + extension import/compress smoke test; (3) `cargo test -p headroom-proxy --lib drift`; (4) `cargo fmt --check` + `cargo clippy -p headroom-py`; (5) `npm audit` in sdk/openclaw/docs + `bun audit` in docs. - Observed result: `headroom-ai[all]` resolution scans clean — "No vulnerabilities found" (179 pkgs); full/prod SBOM shows only the 2 documented accepted CVEs; pyo3 0.29 extension imports and runs (detach + from_py_object paths exercised); drift tests 14/14 pass; cargo fmt + clippy clean; all npm/bun audits report 0. - Not tested: full `pytest` suite (no Python source changed); release-profile wheel build (used dev-profile `maturin develop` for the import proof — the extension is semantically identical). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (`sbom/README.md`) - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective — N/A (dependency bumps; existing suites + scans cover it) - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A (Release Please auto-generates from the conventional commit) ## Additional Notes **Accepted risks (the 2 residual CVEs).** Both originate solely from the EleutherAI `lm-evaluation-harness` under the **opt-in `[benchmark]` extra**, which Headroom invokes as a subprocess (never imports): - `sqlitedict` CVE-2024-35515 (High) — pickle deserialization; package abandoned (last release 2021), **no upstream fix exists**. - `nltk` CVE-2026-54293 (High) — path traversal in `nltk.data.load()`; affects ≤3.9.4 (current latest), **no patched release**. Neither is in `[all]`, the published wheel, or the container. They are documented in `sbom/README.md` and will be picked up automatically once upstream ships fixes. **Release/CHANGELOG:** N/A items above are because this is a dependency/security PR with no Python source changes; CHANGELOG is Release-Please-managed via the conventional commit message.
2026-06-27 15:28:12 -07:00
version = "0.29.0"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
fix(deps): remediate dependency CVEs and publish SBOM (#1509) ## Description Supply-chain hardening: takes the **shipped** dependency surface from **26 known CVEs to 0**. `pip install headroom-ai[all]` now resolves with no known vulnerabilities (verified with Anchore syft + grype). Also publishes a checked-in SBOM package (`sbom/`) so any user — especially pilots running their own security review — can verify what's inside and that we track it. This addresses the Dependabot alerts on `main` (9 high / 4 moderate / 7 low at time of writing). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made **Rust** - `pyo3` 0.24 → 0.29 (GHSA-36hh-v3qg-5jq4 High, GHSA-chgr-c6px-7xpp Med). Migrated `Python::allow_threads` → `Python::detach` (10 sites) and added `from_py_object` to the `Clone`-deriving `#[pyclass]` types (both required by the 0.25+ API). - `pyo3-log` 0.12 → 0.13; `lru` 0.12 → 0.18 (GHSA-rhfx-m35p-ff5j). **Python** - `torch` → 2.12.1, `mem0ai` → 2.x. - Floor-pinned transitive CVE deps via `[tool.uv] constraint-dependencies`: `pygments>=2.20.0`, `pydantic-settings>=2.14.2`, `gitpython>=3.1.50`, `langsmith>=0.9.0`. - **Removed `benchmark` from the `[all]` aggregate** so the default install is CVE-free. `lm-eval` is invoked as an external subprocess (`python -m lm_eval`) and never imported, so it is not a true runtime dep — it remains available via the opt-in `[benchmark]` extra. See [Accepted Risks](#additional-notes). **npm (build/test tooling — never shipped in the wheel/container/published SDK)** - `esbuild` override `>=0.28.1` in `sdk/typescript` + `plugins/openclaw` (GHSA-g7r4-m6w7-qqqr). - `docs/`: `@anthropic-ai/sdk` → `^0.106.0` (GHSA-p7fg-763f-g4gf), `postcss` override to force Next.js's bundled copy ≥8.5.10 (GHSA-qx2v-qp2m-jg93); regenerated a stale `bun.lock` that carried a **Critical** vitest/vite. **CI** - Pinned `pypa/gh-action-pypi-publish` `@release/v1` → `@v1.13.0` (GHSA-vxmw-7h4f-hqxh) in `release.yml` + `publish.yml`. **SBOM** - New `sbom/` directory: CycloneDX 1.7 + SPDX 2.3 SBOMs, grype scan evidence, 330-package license inventory, and a regeneration guide. ## Testing - [ ] Unit tests pass (`pytest`) — N/A, no Python source changed (deps/config only) - [x] Linting passes — `cargo fmt --check` + `cargo clippy` clean on the changed crate; 0 `.py` files changed so `ruff`/`mypy` scope is unaffected - [x] Type checking passes — `cargo check --workspace` (0 errors) - [ ] New tests added — N/A (dependency bumps; covered by existing suites) - [x] Manual testing performed — see Real Behavior Proof ### Test Output ```text # headroom-ai[all] product surface — the number that matters $ grype sbom:sbom/headroom-sbom-all-extra.cdx.json No vulnerabilities found # full repo scan (universal lock incl. opt-in [benchmark] + dev) $ grype sbom:sbom/headroom-sbom.cdx.json NAME INSTALLED TYPE VULNERABILITY SEVERITY sqlitedict 2.1.0 python GHSA-g4r7-86gm-pgqc High # [benchmark]-only, unpatchable, accepted nltk 3.9.4 python GHSA-p4gq-832x-fm9v High # [benchmark]-only, unpatchable, accepted # pyo3 0.29 migration — extension builds + imports + runs $ cargo check --workspace Finished `dev` profile [unoptimized + debuginfo] target(s) $ maturin develop && python -c "from headroom._core import DiffCompressor, SmartCrusher; ..." extension OK — detach + from_py_object paths exercised # lru 0.18 — eviction path $ cargo test -p headroom-proxy --lib drift 14 passed, 213 filtered out # per-ecosystem npm audits $ (cd sdk/typescript && npm audit) -> found 0 vulnerabilities $ (cd plugins/openclaw && npm audit) -> found 0 vulnerabilities $ (cd docs && npm audit && bun audit) -> found 0 vulnerabilities / No vulnerabilities found ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0, arm64), Python 3.12 `.venv`, Rust 1.95 toolchain, syft 1.46.0, grype 0.115.0, bun 1.3.14, maturin 1.13.3. - Exact command / steps: (1) `uv export --extra all --no-dev --no-emit-project | syft → grype` for the product surface; (2) `cargo check --workspace` + `maturin develop` + extension import/compress smoke test; (3) `cargo test -p headroom-proxy --lib drift`; (4) `cargo fmt --check` + `cargo clippy -p headroom-py`; (5) `npm audit` in sdk/openclaw/docs + `bun audit` in docs. - Observed result: `headroom-ai[all]` resolution scans clean — "No vulnerabilities found" (179 pkgs); full/prod SBOM shows only the 2 documented accepted CVEs; pyo3 0.29 extension imports and runs (detach + from_py_object paths exercised); drift tests 14/14 pass; cargo fmt + clippy clean; all npm/bun audits report 0. - Not tested: full `pytest` suite (no Python source changed); release-profile wheel build (used dev-profile `maturin develop` for the import proof — the extension is semantically identical). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (`sbom/README.md`) - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective — N/A (dependency bumps; existing suites + scans cover it) - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A (Release Please auto-generates from the conventional commit) ## Additional Notes **Accepted risks (the 2 residual CVEs).** Both originate solely from the EleutherAI `lm-evaluation-harness` under the **opt-in `[benchmark]` extra**, which Headroom invokes as a subprocess (never imports): - `sqlitedict` CVE-2024-35515 (High) — pickle deserialization; package abandoned (last release 2021), **no upstream fix exists**. - `nltk` CVE-2026-54293 (High) — path traversal in `nltk.data.load()`; affects ≤3.9.4 (current latest), **no patched release**. Neither is in `[all]`, the published wheel, or the container. They are documented in `sbom/README.md` and will be picked up automatically once upstream ships fixes. **Release/CHANGELOG:** N/A items above are because this is a dependency/security PR with no Python source changes; CHANGELOG is Release-Please-managed via the conventional commit message.
2026-06-27 15:28:12 -07:00
checksum = "cd274650b21d4bfc26a0a47587962c1edb425f69287324355cd040c3ea66071c"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"libc",
"once_cell",
"portable-atomic",
"pyo3-build-config",
"pyo3-ffi",
"pyo3-macros",
]
[[package]]
name = "pyo3-build-config"
fix(deps): remediate dependency CVEs and publish SBOM (#1509) ## Description Supply-chain hardening: takes the **shipped** dependency surface from **26 known CVEs to 0**. `pip install headroom-ai[all]` now resolves with no known vulnerabilities (verified with Anchore syft + grype). Also publishes a checked-in SBOM package (`sbom/`) so any user — especially pilots running their own security review — can verify what's inside and that we track it. This addresses the Dependabot alerts on `main` (9 high / 4 moderate / 7 low at time of writing). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made **Rust** - `pyo3` 0.24 → 0.29 (GHSA-36hh-v3qg-5jq4 High, GHSA-chgr-c6px-7xpp Med). Migrated `Python::allow_threads` → `Python::detach` (10 sites) and added `from_py_object` to the `Clone`-deriving `#[pyclass]` types (both required by the 0.25+ API). - `pyo3-log` 0.12 → 0.13; `lru` 0.12 → 0.18 (GHSA-rhfx-m35p-ff5j). **Python** - `torch` → 2.12.1, `mem0ai` → 2.x. - Floor-pinned transitive CVE deps via `[tool.uv] constraint-dependencies`: `pygments>=2.20.0`, `pydantic-settings>=2.14.2`, `gitpython>=3.1.50`, `langsmith>=0.9.0`. - **Removed `benchmark` from the `[all]` aggregate** so the default install is CVE-free. `lm-eval` is invoked as an external subprocess (`python -m lm_eval`) and never imported, so it is not a true runtime dep — it remains available via the opt-in `[benchmark]` extra. See [Accepted Risks](#additional-notes). **npm (build/test tooling — never shipped in the wheel/container/published SDK)** - `esbuild` override `>=0.28.1` in `sdk/typescript` + `plugins/openclaw` (GHSA-g7r4-m6w7-qqqr). - `docs/`: `@anthropic-ai/sdk` → `^0.106.0` (GHSA-p7fg-763f-g4gf), `postcss` override to force Next.js's bundled copy ≥8.5.10 (GHSA-qx2v-qp2m-jg93); regenerated a stale `bun.lock` that carried a **Critical** vitest/vite. **CI** - Pinned `pypa/gh-action-pypi-publish` `@release/v1` → `@v1.13.0` (GHSA-vxmw-7h4f-hqxh) in `release.yml` + `publish.yml`. **SBOM** - New `sbom/` directory: CycloneDX 1.7 + SPDX 2.3 SBOMs, grype scan evidence, 330-package license inventory, and a regeneration guide. ## Testing - [ ] Unit tests pass (`pytest`) — N/A, no Python source changed (deps/config only) - [x] Linting passes — `cargo fmt --check` + `cargo clippy` clean on the changed crate; 0 `.py` files changed so `ruff`/`mypy` scope is unaffected - [x] Type checking passes — `cargo check --workspace` (0 errors) - [ ] New tests added — N/A (dependency bumps; covered by existing suites) - [x] Manual testing performed — see Real Behavior Proof ### Test Output ```text # headroom-ai[all] product surface — the number that matters $ grype sbom:sbom/headroom-sbom-all-extra.cdx.json No vulnerabilities found # full repo scan (universal lock incl. opt-in [benchmark] + dev) $ grype sbom:sbom/headroom-sbom.cdx.json NAME INSTALLED TYPE VULNERABILITY SEVERITY sqlitedict 2.1.0 python GHSA-g4r7-86gm-pgqc High # [benchmark]-only, unpatchable, accepted nltk 3.9.4 python GHSA-p4gq-832x-fm9v High # [benchmark]-only, unpatchable, accepted # pyo3 0.29 migration — extension builds + imports + runs $ cargo check --workspace Finished `dev` profile [unoptimized + debuginfo] target(s) $ maturin develop && python -c "from headroom._core import DiffCompressor, SmartCrusher; ..." extension OK — detach + from_py_object paths exercised # lru 0.18 — eviction path $ cargo test -p headroom-proxy --lib drift 14 passed, 213 filtered out # per-ecosystem npm audits $ (cd sdk/typescript && npm audit) -> found 0 vulnerabilities $ (cd plugins/openclaw && npm audit) -> found 0 vulnerabilities $ (cd docs && npm audit && bun audit) -> found 0 vulnerabilities / No vulnerabilities found ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0, arm64), Python 3.12 `.venv`, Rust 1.95 toolchain, syft 1.46.0, grype 0.115.0, bun 1.3.14, maturin 1.13.3. - Exact command / steps: (1) `uv export --extra all --no-dev --no-emit-project | syft → grype` for the product surface; (2) `cargo check --workspace` + `maturin develop` + extension import/compress smoke test; (3) `cargo test -p headroom-proxy --lib drift`; (4) `cargo fmt --check` + `cargo clippy -p headroom-py`; (5) `npm audit` in sdk/openclaw/docs + `bun audit` in docs. - Observed result: `headroom-ai[all]` resolution scans clean — "No vulnerabilities found" (179 pkgs); full/prod SBOM shows only the 2 documented accepted CVEs; pyo3 0.29 extension imports and runs (detach + from_py_object paths exercised); drift tests 14/14 pass; cargo fmt + clippy clean; all npm/bun audits report 0. - Not tested: full `pytest` suite (no Python source changed); release-profile wheel build (used dev-profile `maturin develop` for the import proof — the extension is semantically identical). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (`sbom/README.md`) - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective — N/A (dependency bumps; existing suites + scans cover it) - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A (Release Please auto-generates from the conventional commit) ## Additional Notes **Accepted risks (the 2 residual CVEs).** Both originate solely from the EleutherAI `lm-evaluation-harness` under the **opt-in `[benchmark]` extra**, which Headroom invokes as a subprocess (never imports): - `sqlitedict` CVE-2024-35515 (High) — pickle deserialization; package abandoned (last release 2021), **no upstream fix exists**. - `nltk` CVE-2026-54293 (High) — path traversal in `nltk.data.load()`; affects ≤3.9.4 (current latest), **no patched release**. Neither is in `[all]`, the published wheel, or the container. They are documented in `sbom/README.md` and will be picked up automatically once upstream ships fixes. **Release/CHANGELOG:** N/A items above are because this is a dependency/security PR with no Python source changes; CHANGELOG is Release-Please-managed via the conventional commit message.
2026-06-27 15:28:12 -07:00
version = "0.29.0"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
fix(deps): remediate dependency CVEs and publish SBOM (#1509) ## Description Supply-chain hardening: takes the **shipped** dependency surface from **26 known CVEs to 0**. `pip install headroom-ai[all]` now resolves with no known vulnerabilities (verified with Anchore syft + grype). Also publishes a checked-in SBOM package (`sbom/`) so any user — especially pilots running their own security review — can verify what's inside and that we track it. This addresses the Dependabot alerts on `main` (9 high / 4 moderate / 7 low at time of writing). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made **Rust** - `pyo3` 0.24 → 0.29 (GHSA-36hh-v3qg-5jq4 High, GHSA-chgr-c6px-7xpp Med). Migrated `Python::allow_threads` → `Python::detach` (10 sites) and added `from_py_object` to the `Clone`-deriving `#[pyclass]` types (both required by the 0.25+ API). - `pyo3-log` 0.12 → 0.13; `lru` 0.12 → 0.18 (GHSA-rhfx-m35p-ff5j). **Python** - `torch` → 2.12.1, `mem0ai` → 2.x. - Floor-pinned transitive CVE deps via `[tool.uv] constraint-dependencies`: `pygments>=2.20.0`, `pydantic-settings>=2.14.2`, `gitpython>=3.1.50`, `langsmith>=0.9.0`. - **Removed `benchmark` from the `[all]` aggregate** so the default install is CVE-free. `lm-eval` is invoked as an external subprocess (`python -m lm_eval`) and never imported, so it is not a true runtime dep — it remains available via the opt-in `[benchmark]` extra. See [Accepted Risks](#additional-notes). **npm (build/test tooling — never shipped in the wheel/container/published SDK)** - `esbuild` override `>=0.28.1` in `sdk/typescript` + `plugins/openclaw` (GHSA-g7r4-m6w7-qqqr). - `docs/`: `@anthropic-ai/sdk` → `^0.106.0` (GHSA-p7fg-763f-g4gf), `postcss` override to force Next.js's bundled copy ≥8.5.10 (GHSA-qx2v-qp2m-jg93); regenerated a stale `bun.lock` that carried a **Critical** vitest/vite. **CI** - Pinned `pypa/gh-action-pypi-publish` `@release/v1` → `@v1.13.0` (GHSA-vxmw-7h4f-hqxh) in `release.yml` + `publish.yml`. **SBOM** - New `sbom/` directory: CycloneDX 1.7 + SPDX 2.3 SBOMs, grype scan evidence, 330-package license inventory, and a regeneration guide. ## Testing - [ ] Unit tests pass (`pytest`) — N/A, no Python source changed (deps/config only) - [x] Linting passes — `cargo fmt --check` + `cargo clippy` clean on the changed crate; 0 `.py` files changed so `ruff`/`mypy` scope is unaffected - [x] Type checking passes — `cargo check --workspace` (0 errors) - [ ] New tests added — N/A (dependency bumps; covered by existing suites) - [x] Manual testing performed — see Real Behavior Proof ### Test Output ```text # headroom-ai[all] product surface — the number that matters $ grype sbom:sbom/headroom-sbom-all-extra.cdx.json No vulnerabilities found # full repo scan (universal lock incl. opt-in [benchmark] + dev) $ grype sbom:sbom/headroom-sbom.cdx.json NAME INSTALLED TYPE VULNERABILITY SEVERITY sqlitedict 2.1.0 python GHSA-g4r7-86gm-pgqc High # [benchmark]-only, unpatchable, accepted nltk 3.9.4 python GHSA-p4gq-832x-fm9v High # [benchmark]-only, unpatchable, accepted # pyo3 0.29 migration — extension builds + imports + runs $ cargo check --workspace Finished `dev` profile [unoptimized + debuginfo] target(s) $ maturin develop && python -c "from headroom._core import DiffCompressor, SmartCrusher; ..." extension OK — detach + from_py_object paths exercised # lru 0.18 — eviction path $ cargo test -p headroom-proxy --lib drift 14 passed, 213 filtered out # per-ecosystem npm audits $ (cd sdk/typescript && npm audit) -> found 0 vulnerabilities $ (cd plugins/openclaw && npm audit) -> found 0 vulnerabilities $ (cd docs && npm audit && bun audit) -> found 0 vulnerabilities / No vulnerabilities found ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0, arm64), Python 3.12 `.venv`, Rust 1.95 toolchain, syft 1.46.0, grype 0.115.0, bun 1.3.14, maturin 1.13.3. - Exact command / steps: (1) `uv export --extra all --no-dev --no-emit-project | syft → grype` for the product surface; (2) `cargo check --workspace` + `maturin develop` + extension import/compress smoke test; (3) `cargo test -p headroom-proxy --lib drift`; (4) `cargo fmt --check` + `cargo clippy -p headroom-py`; (5) `npm audit` in sdk/openclaw/docs + `bun audit` in docs. - Observed result: `headroom-ai[all]` resolution scans clean — "No vulnerabilities found" (179 pkgs); full/prod SBOM shows only the 2 documented accepted CVEs; pyo3 0.29 extension imports and runs (detach + from_py_object paths exercised); drift tests 14/14 pass; cargo fmt + clippy clean; all npm/bun audits report 0. - Not tested: full `pytest` suite (no Python source changed); release-profile wheel build (used dev-profile `maturin develop` for the import proof — the extension is semantically identical). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (`sbom/README.md`) - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective — N/A (dependency bumps; existing suites + scans cover it) - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A (Release Please auto-generates from the conventional commit) ## Additional Notes **Accepted risks (the 2 residual CVEs).** Both originate solely from the EleutherAI `lm-evaluation-harness` under the **opt-in `[benchmark]` extra**, which Headroom invokes as a subprocess (never imports): - `sqlitedict` CVE-2024-35515 (High) — pickle deserialization; package abandoned (last release 2021), **no upstream fix exists**. - `nltk` CVE-2026-54293 (High) — path traversal in `nltk.data.load()`; affects ≤3.9.4 (current latest), **no patched release**. Neither is in `[all]`, the published wheel, or the container. They are documented in `sbom/README.md` and will be picked up automatically once upstream ships fixes. **Release/CHANGELOG:** N/A items above are because this is a dependency/security PR with no Python source changes; CHANGELOG is Release-Please-managed via the conventional commit message.
2026-06-27 15:28:12 -07:00
checksum = "c5e2a7d2f0d013342f295c048ad19237add5154a55b1c5a254c0ec93d4109078"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"target-lexicon",
]
[[package]]
name = "pyo3-ffi"
fix(deps): remediate dependency CVEs and publish SBOM (#1509) ## Description Supply-chain hardening: takes the **shipped** dependency surface from **26 known CVEs to 0**. `pip install headroom-ai[all]` now resolves with no known vulnerabilities (verified with Anchore syft + grype). Also publishes a checked-in SBOM package (`sbom/`) so any user — especially pilots running their own security review — can verify what's inside and that we track it. This addresses the Dependabot alerts on `main` (9 high / 4 moderate / 7 low at time of writing). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made **Rust** - `pyo3` 0.24 → 0.29 (GHSA-36hh-v3qg-5jq4 High, GHSA-chgr-c6px-7xpp Med). Migrated `Python::allow_threads` → `Python::detach` (10 sites) and added `from_py_object` to the `Clone`-deriving `#[pyclass]` types (both required by the 0.25+ API). - `pyo3-log` 0.12 → 0.13; `lru` 0.12 → 0.18 (GHSA-rhfx-m35p-ff5j). **Python** - `torch` → 2.12.1, `mem0ai` → 2.x. - Floor-pinned transitive CVE deps via `[tool.uv] constraint-dependencies`: `pygments>=2.20.0`, `pydantic-settings>=2.14.2`, `gitpython>=3.1.50`, `langsmith>=0.9.0`. - **Removed `benchmark` from the `[all]` aggregate** so the default install is CVE-free. `lm-eval` is invoked as an external subprocess (`python -m lm_eval`) and never imported, so it is not a true runtime dep — it remains available via the opt-in `[benchmark]` extra. See [Accepted Risks](#additional-notes). **npm (build/test tooling — never shipped in the wheel/container/published SDK)** - `esbuild` override `>=0.28.1` in `sdk/typescript` + `plugins/openclaw` (GHSA-g7r4-m6w7-qqqr). - `docs/`: `@anthropic-ai/sdk` → `^0.106.0` (GHSA-p7fg-763f-g4gf), `postcss` override to force Next.js's bundled copy ≥8.5.10 (GHSA-qx2v-qp2m-jg93); regenerated a stale `bun.lock` that carried a **Critical** vitest/vite. **CI** - Pinned `pypa/gh-action-pypi-publish` `@release/v1` → `@v1.13.0` (GHSA-vxmw-7h4f-hqxh) in `release.yml` + `publish.yml`. **SBOM** - New `sbom/` directory: CycloneDX 1.7 + SPDX 2.3 SBOMs, grype scan evidence, 330-package license inventory, and a regeneration guide. ## Testing - [ ] Unit tests pass (`pytest`) — N/A, no Python source changed (deps/config only) - [x] Linting passes — `cargo fmt --check` + `cargo clippy` clean on the changed crate; 0 `.py` files changed so `ruff`/`mypy` scope is unaffected - [x] Type checking passes — `cargo check --workspace` (0 errors) - [ ] New tests added — N/A (dependency bumps; covered by existing suites) - [x] Manual testing performed — see Real Behavior Proof ### Test Output ```text # headroom-ai[all] product surface — the number that matters $ grype sbom:sbom/headroom-sbom-all-extra.cdx.json No vulnerabilities found # full repo scan (universal lock incl. opt-in [benchmark] + dev) $ grype sbom:sbom/headroom-sbom.cdx.json NAME INSTALLED TYPE VULNERABILITY SEVERITY sqlitedict 2.1.0 python GHSA-g4r7-86gm-pgqc High # [benchmark]-only, unpatchable, accepted nltk 3.9.4 python GHSA-p4gq-832x-fm9v High # [benchmark]-only, unpatchable, accepted # pyo3 0.29 migration — extension builds + imports + runs $ cargo check --workspace Finished `dev` profile [unoptimized + debuginfo] target(s) $ maturin develop && python -c "from headroom._core import DiffCompressor, SmartCrusher; ..." extension OK — detach + from_py_object paths exercised # lru 0.18 — eviction path $ cargo test -p headroom-proxy --lib drift 14 passed, 213 filtered out # per-ecosystem npm audits $ (cd sdk/typescript && npm audit) -> found 0 vulnerabilities $ (cd plugins/openclaw && npm audit) -> found 0 vulnerabilities $ (cd docs && npm audit && bun audit) -> found 0 vulnerabilities / No vulnerabilities found ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0, arm64), Python 3.12 `.venv`, Rust 1.95 toolchain, syft 1.46.0, grype 0.115.0, bun 1.3.14, maturin 1.13.3. - Exact command / steps: (1) `uv export --extra all --no-dev --no-emit-project | syft → grype` for the product surface; (2) `cargo check --workspace` + `maturin develop` + extension import/compress smoke test; (3) `cargo test -p headroom-proxy --lib drift`; (4) `cargo fmt --check` + `cargo clippy -p headroom-py`; (5) `npm audit` in sdk/openclaw/docs + `bun audit` in docs. - Observed result: `headroom-ai[all]` resolution scans clean — "No vulnerabilities found" (179 pkgs); full/prod SBOM shows only the 2 documented accepted CVEs; pyo3 0.29 extension imports and runs (detach + from_py_object paths exercised); drift tests 14/14 pass; cargo fmt + clippy clean; all npm/bun audits report 0. - Not tested: full `pytest` suite (no Python source changed); release-profile wheel build (used dev-profile `maturin develop` for the import proof — the extension is semantically identical). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (`sbom/README.md`) - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective — N/A (dependency bumps; existing suites + scans cover it) - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A (Release Please auto-generates from the conventional commit) ## Additional Notes **Accepted risks (the 2 residual CVEs).** Both originate solely from the EleutherAI `lm-evaluation-harness` under the **opt-in `[benchmark]` extra**, which Headroom invokes as a subprocess (never imports): - `sqlitedict` CVE-2024-35515 (High) — pickle deserialization; package abandoned (last release 2021), **no upstream fix exists**. - `nltk` CVE-2026-54293 (High) — path traversal in `nltk.data.load()`; affects ≤3.9.4 (current latest), **no patched release**. Neither is in `[all]`, the published wheel, or the container. They are documented in `sbom/README.md` and will be picked up automatically once upstream ships fixes. **Release/CHANGELOG:** N/A items above are because this is a dependency/security PR with no Python source changes; CHANGELOG is Release-Please-managed via the conventional commit message.
2026-06-27 15:28:12 -07:00
version = "0.29.0"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
fix(deps): remediate dependency CVEs and publish SBOM (#1509) ## Description Supply-chain hardening: takes the **shipped** dependency surface from **26 known CVEs to 0**. `pip install headroom-ai[all]` now resolves with no known vulnerabilities (verified with Anchore syft + grype). Also publishes a checked-in SBOM package (`sbom/`) so any user — especially pilots running their own security review — can verify what's inside and that we track it. This addresses the Dependabot alerts on `main` (9 high / 4 moderate / 7 low at time of writing). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made **Rust** - `pyo3` 0.24 → 0.29 (GHSA-36hh-v3qg-5jq4 High, GHSA-chgr-c6px-7xpp Med). Migrated `Python::allow_threads` → `Python::detach` (10 sites) and added `from_py_object` to the `Clone`-deriving `#[pyclass]` types (both required by the 0.25+ API). - `pyo3-log` 0.12 → 0.13; `lru` 0.12 → 0.18 (GHSA-rhfx-m35p-ff5j). **Python** - `torch` → 2.12.1, `mem0ai` → 2.x. - Floor-pinned transitive CVE deps via `[tool.uv] constraint-dependencies`: `pygments>=2.20.0`, `pydantic-settings>=2.14.2`, `gitpython>=3.1.50`, `langsmith>=0.9.0`. - **Removed `benchmark` from the `[all]` aggregate** so the default install is CVE-free. `lm-eval` is invoked as an external subprocess (`python -m lm_eval`) and never imported, so it is not a true runtime dep — it remains available via the opt-in `[benchmark]` extra. See [Accepted Risks](#additional-notes). **npm (build/test tooling — never shipped in the wheel/container/published SDK)** - `esbuild` override `>=0.28.1` in `sdk/typescript` + `plugins/openclaw` (GHSA-g7r4-m6w7-qqqr). - `docs/`: `@anthropic-ai/sdk` → `^0.106.0` (GHSA-p7fg-763f-g4gf), `postcss` override to force Next.js's bundled copy ≥8.5.10 (GHSA-qx2v-qp2m-jg93); regenerated a stale `bun.lock` that carried a **Critical** vitest/vite. **CI** - Pinned `pypa/gh-action-pypi-publish` `@release/v1` → `@v1.13.0` (GHSA-vxmw-7h4f-hqxh) in `release.yml` + `publish.yml`. **SBOM** - New `sbom/` directory: CycloneDX 1.7 + SPDX 2.3 SBOMs, grype scan evidence, 330-package license inventory, and a regeneration guide. ## Testing - [ ] Unit tests pass (`pytest`) — N/A, no Python source changed (deps/config only) - [x] Linting passes — `cargo fmt --check` + `cargo clippy` clean on the changed crate; 0 `.py` files changed so `ruff`/`mypy` scope is unaffected - [x] Type checking passes — `cargo check --workspace` (0 errors) - [ ] New tests added — N/A (dependency bumps; covered by existing suites) - [x] Manual testing performed — see Real Behavior Proof ### Test Output ```text # headroom-ai[all] product surface — the number that matters $ grype sbom:sbom/headroom-sbom-all-extra.cdx.json No vulnerabilities found # full repo scan (universal lock incl. opt-in [benchmark] + dev) $ grype sbom:sbom/headroom-sbom.cdx.json NAME INSTALLED TYPE VULNERABILITY SEVERITY sqlitedict 2.1.0 python GHSA-g4r7-86gm-pgqc High # [benchmark]-only, unpatchable, accepted nltk 3.9.4 python GHSA-p4gq-832x-fm9v High # [benchmark]-only, unpatchable, accepted # pyo3 0.29 migration — extension builds + imports + runs $ cargo check --workspace Finished `dev` profile [unoptimized + debuginfo] target(s) $ maturin develop && python -c "from headroom._core import DiffCompressor, SmartCrusher; ..." extension OK — detach + from_py_object paths exercised # lru 0.18 — eviction path $ cargo test -p headroom-proxy --lib drift 14 passed, 213 filtered out # per-ecosystem npm audits $ (cd sdk/typescript && npm audit) -> found 0 vulnerabilities $ (cd plugins/openclaw && npm audit) -> found 0 vulnerabilities $ (cd docs && npm audit && bun audit) -> found 0 vulnerabilities / No vulnerabilities found ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0, arm64), Python 3.12 `.venv`, Rust 1.95 toolchain, syft 1.46.0, grype 0.115.0, bun 1.3.14, maturin 1.13.3. - Exact command / steps: (1) `uv export --extra all --no-dev --no-emit-project | syft → grype` for the product surface; (2) `cargo check --workspace` + `maturin develop` + extension import/compress smoke test; (3) `cargo test -p headroom-proxy --lib drift`; (4) `cargo fmt --check` + `cargo clippy -p headroom-py`; (5) `npm audit` in sdk/openclaw/docs + `bun audit` in docs. - Observed result: `headroom-ai[all]` resolution scans clean — "No vulnerabilities found" (179 pkgs); full/prod SBOM shows only the 2 documented accepted CVEs; pyo3 0.29 extension imports and runs (detach + from_py_object paths exercised); drift tests 14/14 pass; cargo fmt + clippy clean; all npm/bun audits report 0. - Not tested: full `pytest` suite (no Python source changed); release-profile wheel build (used dev-profile `maturin develop` for the import proof — the extension is semantically identical). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (`sbom/README.md`) - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective — N/A (dependency bumps; existing suites + scans cover it) - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A (Release Please auto-generates from the conventional commit) ## Additional Notes **Accepted risks (the 2 residual CVEs).** Both originate solely from the EleutherAI `lm-evaluation-harness` under the **opt-in `[benchmark]` extra**, which Headroom invokes as a subprocess (never imports): - `sqlitedict` CVE-2024-35515 (High) — pickle deserialization; package abandoned (last release 2021), **no upstream fix exists**. - `nltk` CVE-2026-54293 (High) — path traversal in `nltk.data.load()`; affects ≤3.9.4 (current latest), **no patched release**. Neither is in `[all]`, the published wheel, or the container. They are documented in `sbom/README.md` and will be picked up automatically once upstream ships fixes. **Release/CHANGELOG:** N/A items above are because this is a dependency/security PR with no Python source changes; CHANGELOG is Release-Please-managed via the conventional commit message.
2026-06-27 15:28:12 -07:00
checksum = "ca85c467da1bbc8d866eea5deff9cf29ea5f7785054a17da36e65bda9c05845b"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"libc",
"pyo3-build-config",
]
Pin ORT dylib on Windows; init Python logging (#1010) ## Description On Windows, headroom's Rust core resolves `onnxruntime.dll` at runtime via `ort-load-dynamic`. Without an explicit `ORT_DYLIB_PATH`, the bare DLL search can land on `C:\Windows\System32\onnxruntime.dll`, the Windows ML OS component, and `Session::new()` can deadlock instead of returning an error. Since a hang is not an `Err`, the tiered fallback cannot engage until the proxy-level timeout fires. This PR pins `ORT_DYLIB_PATH` to the pip-installed `onnxruntime` DLL at import time, and wires Rust `tracing` events into Python logging so the proxy log surfaces these failures when they occur. Closes #928 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `headroom/_ort.py` with a Windows-only, idempotent `ensure_ort_dylib_pinned()` resolver that respects an existing `ORT_DYLIB_PATH`. - Call the pin from `headroom/__init__.py` before importing `_core` consumers. - Log the effective ORT dylib path from the content router startup path on Windows. - Enable Rust tracing-to-log compatibility and initialize `pyo3-log` in the `_core` module. - Add timeout diagnostics in the Magika detector with the effective `ORT_DYLIB_PATH`. - Document `ORT_DYLIB_PATH` and `HEADROOM_MAGIKA_INIT_TIMEOUT_SECS`. - Add unit coverage for the resolver behavior. ## Testing - [x] Unit tests pass (`python -m pytest tests/test_transforms/test_ort_dylib.py -q`) - [x] Linting passes (`ruff check headroom/_ort.py headroom/__init__.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py`) - [x] Formatting passes (`ruff format --check headroom/_ort.py headroom/__init__.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_transforms/test_ort_dylib.py -q 7 passed in 0.19s $ ruff check headroom/_ort.py headroom/__init__.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py All checks passed! $ ruff format --check headroom/_ort.py headroom/__init__.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py 4 files already formatted $ cargo check -p headroom-py cargo: The term 'cargo' is not recognized as a name of a cmdlet, function, script file, or executable program. ``` ## Real Behavior Proof - Environment: Windows 11 24H2, Python 3.13, RTX 4080 - Exact command / steps: `python -c "import headroom; from headroom._core import detect_content_type as d; print(d(open('headroom/compress.py').read()).content_type)"` - Observed result: `source_code` in 301ms, clean exit, `Magika: ENABLED` in proxy log - Not tested: macOS/Linux manual runtime behavior; `_ort.py` is a no-op outside Windows, and CI covers cross-platform build/test behavior. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable (N/A: repo uses release-please) ## Additional Notes The branch was rebased onto current `main` and the commit subject was updated to satisfy commitlint. Local Rust verification could not be run on this Windows machine because `cargo` is not installed; GitHub CI should be treated as the Rust build verification for the `pyo3-log` dependency and workspace lockfile changes. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-23 14:46:24 +02:00
[[package]]
name = "pyo3-log"
fix(deps): remediate dependency CVEs and publish SBOM (#1509) ## Description Supply-chain hardening: takes the **shipped** dependency surface from **26 known CVEs to 0**. `pip install headroom-ai[all]` now resolves with no known vulnerabilities (verified with Anchore syft + grype). Also publishes a checked-in SBOM package (`sbom/`) so any user — especially pilots running their own security review — can verify what's inside and that we track it. This addresses the Dependabot alerts on `main` (9 high / 4 moderate / 7 low at time of writing). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made **Rust** - `pyo3` 0.24 → 0.29 (GHSA-36hh-v3qg-5jq4 High, GHSA-chgr-c6px-7xpp Med). Migrated `Python::allow_threads` → `Python::detach` (10 sites) and added `from_py_object` to the `Clone`-deriving `#[pyclass]` types (both required by the 0.25+ API). - `pyo3-log` 0.12 → 0.13; `lru` 0.12 → 0.18 (GHSA-rhfx-m35p-ff5j). **Python** - `torch` → 2.12.1, `mem0ai` → 2.x. - Floor-pinned transitive CVE deps via `[tool.uv] constraint-dependencies`: `pygments>=2.20.0`, `pydantic-settings>=2.14.2`, `gitpython>=3.1.50`, `langsmith>=0.9.0`. - **Removed `benchmark` from the `[all]` aggregate** so the default install is CVE-free. `lm-eval` is invoked as an external subprocess (`python -m lm_eval`) and never imported, so it is not a true runtime dep — it remains available via the opt-in `[benchmark]` extra. See [Accepted Risks](#additional-notes). **npm (build/test tooling — never shipped in the wheel/container/published SDK)** - `esbuild` override `>=0.28.1` in `sdk/typescript` + `plugins/openclaw` (GHSA-g7r4-m6w7-qqqr). - `docs/`: `@anthropic-ai/sdk` → `^0.106.0` (GHSA-p7fg-763f-g4gf), `postcss` override to force Next.js's bundled copy ≥8.5.10 (GHSA-qx2v-qp2m-jg93); regenerated a stale `bun.lock` that carried a **Critical** vitest/vite. **CI** - Pinned `pypa/gh-action-pypi-publish` `@release/v1` → `@v1.13.0` (GHSA-vxmw-7h4f-hqxh) in `release.yml` + `publish.yml`. **SBOM** - New `sbom/` directory: CycloneDX 1.7 + SPDX 2.3 SBOMs, grype scan evidence, 330-package license inventory, and a regeneration guide. ## Testing - [ ] Unit tests pass (`pytest`) — N/A, no Python source changed (deps/config only) - [x] Linting passes — `cargo fmt --check` + `cargo clippy` clean on the changed crate; 0 `.py` files changed so `ruff`/`mypy` scope is unaffected - [x] Type checking passes — `cargo check --workspace` (0 errors) - [ ] New tests added — N/A (dependency bumps; covered by existing suites) - [x] Manual testing performed — see Real Behavior Proof ### Test Output ```text # headroom-ai[all] product surface — the number that matters $ grype sbom:sbom/headroom-sbom-all-extra.cdx.json No vulnerabilities found # full repo scan (universal lock incl. opt-in [benchmark] + dev) $ grype sbom:sbom/headroom-sbom.cdx.json NAME INSTALLED TYPE VULNERABILITY SEVERITY sqlitedict 2.1.0 python GHSA-g4r7-86gm-pgqc High # [benchmark]-only, unpatchable, accepted nltk 3.9.4 python GHSA-p4gq-832x-fm9v High # [benchmark]-only, unpatchable, accepted # pyo3 0.29 migration — extension builds + imports + runs $ cargo check --workspace Finished `dev` profile [unoptimized + debuginfo] target(s) $ maturin develop && python -c "from headroom._core import DiffCompressor, SmartCrusher; ..." extension OK — detach + from_py_object paths exercised # lru 0.18 — eviction path $ cargo test -p headroom-proxy --lib drift 14 passed, 213 filtered out # per-ecosystem npm audits $ (cd sdk/typescript && npm audit) -> found 0 vulnerabilities $ (cd plugins/openclaw && npm audit) -> found 0 vulnerabilities $ (cd docs && npm audit && bun audit) -> found 0 vulnerabilities / No vulnerabilities found ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0, arm64), Python 3.12 `.venv`, Rust 1.95 toolchain, syft 1.46.0, grype 0.115.0, bun 1.3.14, maturin 1.13.3. - Exact command / steps: (1) `uv export --extra all --no-dev --no-emit-project | syft → grype` for the product surface; (2) `cargo check --workspace` + `maturin develop` + extension import/compress smoke test; (3) `cargo test -p headroom-proxy --lib drift`; (4) `cargo fmt --check` + `cargo clippy -p headroom-py`; (5) `npm audit` in sdk/openclaw/docs + `bun audit` in docs. - Observed result: `headroom-ai[all]` resolution scans clean — "No vulnerabilities found" (179 pkgs); full/prod SBOM shows only the 2 documented accepted CVEs; pyo3 0.29 extension imports and runs (detach + from_py_object paths exercised); drift tests 14/14 pass; cargo fmt + clippy clean; all npm/bun audits report 0. - Not tested: full `pytest` suite (no Python source changed); release-profile wheel build (used dev-profile `maturin develop` for the import proof — the extension is semantically identical). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (`sbom/README.md`) - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective — N/A (dependency bumps; existing suites + scans cover it) - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A (Release Please auto-generates from the conventional commit) ## Additional Notes **Accepted risks (the 2 residual CVEs).** Both originate solely from the EleutherAI `lm-evaluation-harness` under the **opt-in `[benchmark]` extra**, which Headroom invokes as a subprocess (never imports): - `sqlitedict` CVE-2024-35515 (High) — pickle deserialization; package abandoned (last release 2021), **no upstream fix exists**. - `nltk` CVE-2026-54293 (High) — path traversal in `nltk.data.load()`; affects ≤3.9.4 (current latest), **no patched release**. Neither is in `[all]`, the published wheel, or the container. They are documented in `sbom/README.md` and will be picked up automatically once upstream ships fixes. **Release/CHANGELOG:** N/A items above are because this is a dependency/security PR with no Python source changes; CHANGELOG is Release-Please-managed via the conventional commit message.
2026-06-27 15:28:12 -07:00
version = "0.13.4"
Pin ORT dylib on Windows; init Python logging (#1010) ## Description On Windows, headroom's Rust core resolves `onnxruntime.dll` at runtime via `ort-load-dynamic`. Without an explicit `ORT_DYLIB_PATH`, the bare DLL search can land on `C:\Windows\System32\onnxruntime.dll`, the Windows ML OS component, and `Session::new()` can deadlock instead of returning an error. Since a hang is not an `Err`, the tiered fallback cannot engage until the proxy-level timeout fires. This PR pins `ORT_DYLIB_PATH` to the pip-installed `onnxruntime` DLL at import time, and wires Rust `tracing` events into Python logging so the proxy log surfaces these failures when they occur. Closes #928 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `headroom/_ort.py` with a Windows-only, idempotent `ensure_ort_dylib_pinned()` resolver that respects an existing `ORT_DYLIB_PATH`. - Call the pin from `headroom/__init__.py` before importing `_core` consumers. - Log the effective ORT dylib path from the content router startup path on Windows. - Enable Rust tracing-to-log compatibility and initialize `pyo3-log` in the `_core` module. - Add timeout diagnostics in the Magika detector with the effective `ORT_DYLIB_PATH`. - Document `ORT_DYLIB_PATH` and `HEADROOM_MAGIKA_INIT_TIMEOUT_SECS`. - Add unit coverage for the resolver behavior. ## Testing - [x] Unit tests pass (`python -m pytest tests/test_transforms/test_ort_dylib.py -q`) - [x] Linting passes (`ruff check headroom/_ort.py headroom/__init__.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py`) - [x] Formatting passes (`ruff format --check headroom/_ort.py headroom/__init__.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_transforms/test_ort_dylib.py -q 7 passed in 0.19s $ ruff check headroom/_ort.py headroom/__init__.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py All checks passed! $ ruff format --check headroom/_ort.py headroom/__init__.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py 4 files already formatted $ cargo check -p headroom-py cargo: The term 'cargo' is not recognized as a name of a cmdlet, function, script file, or executable program. ``` ## Real Behavior Proof - Environment: Windows 11 24H2, Python 3.13, RTX 4080 - Exact command / steps: `python -c "import headroom; from headroom._core import detect_content_type as d; print(d(open('headroom/compress.py').read()).content_type)"` - Observed result: `source_code` in 301ms, clean exit, `Magika: ENABLED` in proxy log - Not tested: macOS/Linux manual runtime behavior; `_ort.py` is a no-op outside Windows, and CI covers cross-platform build/test behavior. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable (N/A: repo uses release-please) ## Additional Notes The branch was rebased onto current `main` and the commit subject was updated to satisfy commitlint. Local Rust verification could not be run on this Windows machine because `cargo` is not installed; GitHub CI should be treated as the Rust build verification for the `pyo3-log` dependency and workspace lockfile changes. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-23 14:46:24 +02:00
source = "registry+https://github.com/rust-lang/crates.io-index"
fix(deps): remediate dependency CVEs and publish SBOM (#1509) ## Description Supply-chain hardening: takes the **shipped** dependency surface from **26 known CVEs to 0**. `pip install headroom-ai[all]` now resolves with no known vulnerabilities (verified with Anchore syft + grype). Also publishes a checked-in SBOM package (`sbom/`) so any user — especially pilots running their own security review — can verify what's inside and that we track it. This addresses the Dependabot alerts on `main` (9 high / 4 moderate / 7 low at time of writing). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made **Rust** - `pyo3` 0.24 → 0.29 (GHSA-36hh-v3qg-5jq4 High, GHSA-chgr-c6px-7xpp Med). Migrated `Python::allow_threads` → `Python::detach` (10 sites) and added `from_py_object` to the `Clone`-deriving `#[pyclass]` types (both required by the 0.25+ API). - `pyo3-log` 0.12 → 0.13; `lru` 0.12 → 0.18 (GHSA-rhfx-m35p-ff5j). **Python** - `torch` → 2.12.1, `mem0ai` → 2.x. - Floor-pinned transitive CVE deps via `[tool.uv] constraint-dependencies`: `pygments>=2.20.0`, `pydantic-settings>=2.14.2`, `gitpython>=3.1.50`, `langsmith>=0.9.0`. - **Removed `benchmark` from the `[all]` aggregate** so the default install is CVE-free. `lm-eval` is invoked as an external subprocess (`python -m lm_eval`) and never imported, so it is not a true runtime dep — it remains available via the opt-in `[benchmark]` extra. See [Accepted Risks](#additional-notes). **npm (build/test tooling — never shipped in the wheel/container/published SDK)** - `esbuild` override `>=0.28.1` in `sdk/typescript` + `plugins/openclaw` (GHSA-g7r4-m6w7-qqqr). - `docs/`: `@anthropic-ai/sdk` → `^0.106.0` (GHSA-p7fg-763f-g4gf), `postcss` override to force Next.js's bundled copy ≥8.5.10 (GHSA-qx2v-qp2m-jg93); regenerated a stale `bun.lock` that carried a **Critical** vitest/vite. **CI** - Pinned `pypa/gh-action-pypi-publish` `@release/v1` → `@v1.13.0` (GHSA-vxmw-7h4f-hqxh) in `release.yml` + `publish.yml`. **SBOM** - New `sbom/` directory: CycloneDX 1.7 + SPDX 2.3 SBOMs, grype scan evidence, 330-package license inventory, and a regeneration guide. ## Testing - [ ] Unit tests pass (`pytest`) — N/A, no Python source changed (deps/config only) - [x] Linting passes — `cargo fmt --check` + `cargo clippy` clean on the changed crate; 0 `.py` files changed so `ruff`/`mypy` scope is unaffected - [x] Type checking passes — `cargo check --workspace` (0 errors) - [ ] New tests added — N/A (dependency bumps; covered by existing suites) - [x] Manual testing performed — see Real Behavior Proof ### Test Output ```text # headroom-ai[all] product surface — the number that matters $ grype sbom:sbom/headroom-sbom-all-extra.cdx.json No vulnerabilities found # full repo scan (universal lock incl. opt-in [benchmark] + dev) $ grype sbom:sbom/headroom-sbom.cdx.json NAME INSTALLED TYPE VULNERABILITY SEVERITY sqlitedict 2.1.0 python GHSA-g4r7-86gm-pgqc High # [benchmark]-only, unpatchable, accepted nltk 3.9.4 python GHSA-p4gq-832x-fm9v High # [benchmark]-only, unpatchable, accepted # pyo3 0.29 migration — extension builds + imports + runs $ cargo check --workspace Finished `dev` profile [unoptimized + debuginfo] target(s) $ maturin develop && python -c "from headroom._core import DiffCompressor, SmartCrusher; ..." extension OK — detach + from_py_object paths exercised # lru 0.18 — eviction path $ cargo test -p headroom-proxy --lib drift 14 passed, 213 filtered out # per-ecosystem npm audits $ (cd sdk/typescript && npm audit) -> found 0 vulnerabilities $ (cd plugins/openclaw && npm audit) -> found 0 vulnerabilities $ (cd docs && npm audit && bun audit) -> found 0 vulnerabilities / No vulnerabilities found ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0, arm64), Python 3.12 `.venv`, Rust 1.95 toolchain, syft 1.46.0, grype 0.115.0, bun 1.3.14, maturin 1.13.3. - Exact command / steps: (1) `uv export --extra all --no-dev --no-emit-project | syft → grype` for the product surface; (2) `cargo check --workspace` + `maturin develop` + extension import/compress smoke test; (3) `cargo test -p headroom-proxy --lib drift`; (4) `cargo fmt --check` + `cargo clippy -p headroom-py`; (5) `npm audit` in sdk/openclaw/docs + `bun audit` in docs. - Observed result: `headroom-ai[all]` resolution scans clean — "No vulnerabilities found" (179 pkgs); full/prod SBOM shows only the 2 documented accepted CVEs; pyo3 0.29 extension imports and runs (detach + from_py_object paths exercised); drift tests 14/14 pass; cargo fmt + clippy clean; all npm/bun audits report 0. - Not tested: full `pytest` suite (no Python source changed); release-profile wheel build (used dev-profile `maturin develop` for the import proof — the extension is semantically identical). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (`sbom/README.md`) - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective — N/A (dependency bumps; existing suites + scans cover it) - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A (Release Please auto-generates from the conventional commit) ## Additional Notes **Accepted risks (the 2 residual CVEs).** Both originate solely from the EleutherAI `lm-evaluation-harness` under the **opt-in `[benchmark]` extra**, which Headroom invokes as a subprocess (never imports): - `sqlitedict` CVE-2024-35515 (High) — pickle deserialization; package abandoned (last release 2021), **no upstream fix exists**. - `nltk` CVE-2026-54293 (High) — path traversal in `nltk.data.load()`; affects ≤3.9.4 (current latest), **no patched release**. Neither is in `[all]`, the published wheel, or the container. They are documented in `sbom/README.md` and will be picked up automatically once upstream ships fixes. **Release/CHANGELOG:** N/A items above are because this is a dependency/security PR with no Python source changes; CHANGELOG is Release-Please-managed via the conventional commit message.
2026-06-27 15:28:12 -07:00
checksum = "f64083bd3a16a353d9d62335808e8e13d0552d2a2b83fdb084496192dcfa9fcd"
Pin ORT dylib on Windows; init Python logging (#1010) ## Description On Windows, headroom's Rust core resolves `onnxruntime.dll` at runtime via `ort-load-dynamic`. Without an explicit `ORT_DYLIB_PATH`, the bare DLL search can land on `C:\Windows\System32\onnxruntime.dll`, the Windows ML OS component, and `Session::new()` can deadlock instead of returning an error. Since a hang is not an `Err`, the tiered fallback cannot engage until the proxy-level timeout fires. This PR pins `ORT_DYLIB_PATH` to the pip-installed `onnxruntime` DLL at import time, and wires Rust `tracing` events into Python logging so the proxy log surfaces these failures when they occur. Closes #928 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `headroom/_ort.py` with a Windows-only, idempotent `ensure_ort_dylib_pinned()` resolver that respects an existing `ORT_DYLIB_PATH`. - Call the pin from `headroom/__init__.py` before importing `_core` consumers. - Log the effective ORT dylib path from the content router startup path on Windows. - Enable Rust tracing-to-log compatibility and initialize `pyo3-log` in the `_core` module. - Add timeout diagnostics in the Magika detector with the effective `ORT_DYLIB_PATH`. - Document `ORT_DYLIB_PATH` and `HEADROOM_MAGIKA_INIT_TIMEOUT_SECS`. - Add unit coverage for the resolver behavior. ## Testing - [x] Unit tests pass (`python -m pytest tests/test_transforms/test_ort_dylib.py -q`) - [x] Linting passes (`ruff check headroom/_ort.py headroom/__init__.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py`) - [x] Formatting passes (`ruff format --check headroom/_ort.py headroom/__init__.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_transforms/test_ort_dylib.py -q 7 passed in 0.19s $ ruff check headroom/_ort.py headroom/__init__.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py All checks passed! $ ruff format --check headroom/_ort.py headroom/__init__.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py 4 files already formatted $ cargo check -p headroom-py cargo: The term 'cargo' is not recognized as a name of a cmdlet, function, script file, or executable program. ``` ## Real Behavior Proof - Environment: Windows 11 24H2, Python 3.13, RTX 4080 - Exact command / steps: `python -c "import headroom; from headroom._core import detect_content_type as d; print(d(open('headroom/compress.py').read()).content_type)"` - Observed result: `source_code` in 301ms, clean exit, `Magika: ENABLED` in proxy log - Not tested: macOS/Linux manual runtime behavior; `_ort.py` is a no-op outside Windows, and CI covers cross-platform build/test behavior. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable (N/A: repo uses release-please) ## Additional Notes The branch was rebased onto current `main` and the commit subject was updated to satisfy commitlint. Local Rust verification could not be run on this Windows machine because `cargo` is not installed; GitHub CI should be treated as the Rust build verification for the `pyo3-log` dependency and workspace lockfile changes. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-23 14:46:24 +02:00
dependencies = [
"arc-swap",
"log",
"pyo3",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "pyo3-macros"
fix(deps): remediate dependency CVEs and publish SBOM (#1509) ## Description Supply-chain hardening: takes the **shipped** dependency surface from **26 known CVEs to 0**. `pip install headroom-ai[all]` now resolves with no known vulnerabilities (verified with Anchore syft + grype). Also publishes a checked-in SBOM package (`sbom/`) so any user — especially pilots running their own security review — can verify what's inside and that we track it. This addresses the Dependabot alerts on `main` (9 high / 4 moderate / 7 low at time of writing). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made **Rust** - `pyo3` 0.24 → 0.29 (GHSA-36hh-v3qg-5jq4 High, GHSA-chgr-c6px-7xpp Med). Migrated `Python::allow_threads` → `Python::detach` (10 sites) and added `from_py_object` to the `Clone`-deriving `#[pyclass]` types (both required by the 0.25+ API). - `pyo3-log` 0.12 → 0.13; `lru` 0.12 → 0.18 (GHSA-rhfx-m35p-ff5j). **Python** - `torch` → 2.12.1, `mem0ai` → 2.x. - Floor-pinned transitive CVE deps via `[tool.uv] constraint-dependencies`: `pygments>=2.20.0`, `pydantic-settings>=2.14.2`, `gitpython>=3.1.50`, `langsmith>=0.9.0`. - **Removed `benchmark` from the `[all]` aggregate** so the default install is CVE-free. `lm-eval` is invoked as an external subprocess (`python -m lm_eval`) and never imported, so it is not a true runtime dep — it remains available via the opt-in `[benchmark]` extra. See [Accepted Risks](#additional-notes). **npm (build/test tooling — never shipped in the wheel/container/published SDK)** - `esbuild` override `>=0.28.1` in `sdk/typescript` + `plugins/openclaw` (GHSA-g7r4-m6w7-qqqr). - `docs/`: `@anthropic-ai/sdk` → `^0.106.0` (GHSA-p7fg-763f-g4gf), `postcss` override to force Next.js's bundled copy ≥8.5.10 (GHSA-qx2v-qp2m-jg93); regenerated a stale `bun.lock` that carried a **Critical** vitest/vite. **CI** - Pinned `pypa/gh-action-pypi-publish` `@release/v1` → `@v1.13.0` (GHSA-vxmw-7h4f-hqxh) in `release.yml` + `publish.yml`. **SBOM** - New `sbom/` directory: CycloneDX 1.7 + SPDX 2.3 SBOMs, grype scan evidence, 330-package license inventory, and a regeneration guide. ## Testing - [ ] Unit tests pass (`pytest`) — N/A, no Python source changed (deps/config only) - [x] Linting passes — `cargo fmt --check` + `cargo clippy` clean on the changed crate; 0 `.py` files changed so `ruff`/`mypy` scope is unaffected - [x] Type checking passes — `cargo check --workspace` (0 errors) - [ ] New tests added — N/A (dependency bumps; covered by existing suites) - [x] Manual testing performed — see Real Behavior Proof ### Test Output ```text # headroom-ai[all] product surface — the number that matters $ grype sbom:sbom/headroom-sbom-all-extra.cdx.json No vulnerabilities found # full repo scan (universal lock incl. opt-in [benchmark] + dev) $ grype sbom:sbom/headroom-sbom.cdx.json NAME INSTALLED TYPE VULNERABILITY SEVERITY sqlitedict 2.1.0 python GHSA-g4r7-86gm-pgqc High # [benchmark]-only, unpatchable, accepted nltk 3.9.4 python GHSA-p4gq-832x-fm9v High # [benchmark]-only, unpatchable, accepted # pyo3 0.29 migration — extension builds + imports + runs $ cargo check --workspace Finished `dev` profile [unoptimized + debuginfo] target(s) $ maturin develop && python -c "from headroom._core import DiffCompressor, SmartCrusher; ..." extension OK — detach + from_py_object paths exercised # lru 0.18 — eviction path $ cargo test -p headroom-proxy --lib drift 14 passed, 213 filtered out # per-ecosystem npm audits $ (cd sdk/typescript && npm audit) -> found 0 vulnerabilities $ (cd plugins/openclaw && npm audit) -> found 0 vulnerabilities $ (cd docs && npm audit && bun audit) -> found 0 vulnerabilities / No vulnerabilities found ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0, arm64), Python 3.12 `.venv`, Rust 1.95 toolchain, syft 1.46.0, grype 0.115.0, bun 1.3.14, maturin 1.13.3. - Exact command / steps: (1) `uv export --extra all --no-dev --no-emit-project | syft → grype` for the product surface; (2) `cargo check --workspace` + `maturin develop` + extension import/compress smoke test; (3) `cargo test -p headroom-proxy --lib drift`; (4) `cargo fmt --check` + `cargo clippy -p headroom-py`; (5) `npm audit` in sdk/openclaw/docs + `bun audit` in docs. - Observed result: `headroom-ai[all]` resolution scans clean — "No vulnerabilities found" (179 pkgs); full/prod SBOM shows only the 2 documented accepted CVEs; pyo3 0.29 extension imports and runs (detach + from_py_object paths exercised); drift tests 14/14 pass; cargo fmt + clippy clean; all npm/bun audits report 0. - Not tested: full `pytest` suite (no Python source changed); release-profile wheel build (used dev-profile `maturin develop` for the import proof — the extension is semantically identical). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (`sbom/README.md`) - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective — N/A (dependency bumps; existing suites + scans cover it) - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A (Release Please auto-generates from the conventional commit) ## Additional Notes **Accepted risks (the 2 residual CVEs).** Both originate solely from the EleutherAI `lm-evaluation-harness` under the **opt-in `[benchmark]` extra**, which Headroom invokes as a subprocess (never imports): - `sqlitedict` CVE-2024-35515 (High) — pickle deserialization; package abandoned (last release 2021), **no upstream fix exists**. - `nltk` CVE-2026-54293 (High) — path traversal in `nltk.data.load()`; affects ≤3.9.4 (current latest), **no patched release**. Neither is in `[all]`, the published wheel, or the container. They are documented in `sbom/README.md` and will be picked up automatically once upstream ships fixes. **Release/CHANGELOG:** N/A items above are because this is a dependency/security PR with no Python source changes; CHANGELOG is Release-Please-managed via the conventional commit message.
2026-06-27 15:28:12 -07:00
version = "0.29.0"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
fix(deps): remediate dependency CVEs and publish SBOM (#1509) ## Description Supply-chain hardening: takes the **shipped** dependency surface from **26 known CVEs to 0**. `pip install headroom-ai[all]` now resolves with no known vulnerabilities (verified with Anchore syft + grype). Also publishes a checked-in SBOM package (`sbom/`) so any user — especially pilots running their own security review — can verify what's inside and that we track it. This addresses the Dependabot alerts on `main` (9 high / 4 moderate / 7 low at time of writing). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made **Rust** - `pyo3` 0.24 → 0.29 (GHSA-36hh-v3qg-5jq4 High, GHSA-chgr-c6px-7xpp Med). Migrated `Python::allow_threads` → `Python::detach` (10 sites) and added `from_py_object` to the `Clone`-deriving `#[pyclass]` types (both required by the 0.25+ API). - `pyo3-log` 0.12 → 0.13; `lru` 0.12 → 0.18 (GHSA-rhfx-m35p-ff5j). **Python** - `torch` → 2.12.1, `mem0ai` → 2.x. - Floor-pinned transitive CVE deps via `[tool.uv] constraint-dependencies`: `pygments>=2.20.0`, `pydantic-settings>=2.14.2`, `gitpython>=3.1.50`, `langsmith>=0.9.0`. - **Removed `benchmark` from the `[all]` aggregate** so the default install is CVE-free. `lm-eval` is invoked as an external subprocess (`python -m lm_eval`) and never imported, so it is not a true runtime dep — it remains available via the opt-in `[benchmark]` extra. See [Accepted Risks](#additional-notes). **npm (build/test tooling — never shipped in the wheel/container/published SDK)** - `esbuild` override `>=0.28.1` in `sdk/typescript` + `plugins/openclaw` (GHSA-g7r4-m6w7-qqqr). - `docs/`: `@anthropic-ai/sdk` → `^0.106.0` (GHSA-p7fg-763f-g4gf), `postcss` override to force Next.js's bundled copy ≥8.5.10 (GHSA-qx2v-qp2m-jg93); regenerated a stale `bun.lock` that carried a **Critical** vitest/vite. **CI** - Pinned `pypa/gh-action-pypi-publish` `@release/v1` → `@v1.13.0` (GHSA-vxmw-7h4f-hqxh) in `release.yml` + `publish.yml`. **SBOM** - New `sbom/` directory: CycloneDX 1.7 + SPDX 2.3 SBOMs, grype scan evidence, 330-package license inventory, and a regeneration guide. ## Testing - [ ] Unit tests pass (`pytest`) — N/A, no Python source changed (deps/config only) - [x] Linting passes — `cargo fmt --check` + `cargo clippy` clean on the changed crate; 0 `.py` files changed so `ruff`/`mypy` scope is unaffected - [x] Type checking passes — `cargo check --workspace` (0 errors) - [ ] New tests added — N/A (dependency bumps; covered by existing suites) - [x] Manual testing performed — see Real Behavior Proof ### Test Output ```text # headroom-ai[all] product surface — the number that matters $ grype sbom:sbom/headroom-sbom-all-extra.cdx.json No vulnerabilities found # full repo scan (universal lock incl. opt-in [benchmark] + dev) $ grype sbom:sbom/headroom-sbom.cdx.json NAME INSTALLED TYPE VULNERABILITY SEVERITY sqlitedict 2.1.0 python GHSA-g4r7-86gm-pgqc High # [benchmark]-only, unpatchable, accepted nltk 3.9.4 python GHSA-p4gq-832x-fm9v High # [benchmark]-only, unpatchable, accepted # pyo3 0.29 migration — extension builds + imports + runs $ cargo check --workspace Finished `dev` profile [unoptimized + debuginfo] target(s) $ maturin develop && python -c "from headroom._core import DiffCompressor, SmartCrusher; ..." extension OK — detach + from_py_object paths exercised # lru 0.18 — eviction path $ cargo test -p headroom-proxy --lib drift 14 passed, 213 filtered out # per-ecosystem npm audits $ (cd sdk/typescript && npm audit) -> found 0 vulnerabilities $ (cd plugins/openclaw && npm audit) -> found 0 vulnerabilities $ (cd docs && npm audit && bun audit) -> found 0 vulnerabilities / No vulnerabilities found ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0, arm64), Python 3.12 `.venv`, Rust 1.95 toolchain, syft 1.46.0, grype 0.115.0, bun 1.3.14, maturin 1.13.3. - Exact command / steps: (1) `uv export --extra all --no-dev --no-emit-project | syft → grype` for the product surface; (2) `cargo check --workspace` + `maturin develop` + extension import/compress smoke test; (3) `cargo test -p headroom-proxy --lib drift`; (4) `cargo fmt --check` + `cargo clippy -p headroom-py`; (5) `npm audit` in sdk/openclaw/docs + `bun audit` in docs. - Observed result: `headroom-ai[all]` resolution scans clean — "No vulnerabilities found" (179 pkgs); full/prod SBOM shows only the 2 documented accepted CVEs; pyo3 0.29 extension imports and runs (detach + from_py_object paths exercised); drift tests 14/14 pass; cargo fmt + clippy clean; all npm/bun audits report 0. - Not tested: full `pytest` suite (no Python source changed); release-profile wheel build (used dev-profile `maturin develop` for the import proof — the extension is semantically identical). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (`sbom/README.md`) - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective — N/A (dependency bumps; existing suites + scans cover it) - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A (Release Please auto-generates from the conventional commit) ## Additional Notes **Accepted risks (the 2 residual CVEs).** Both originate solely from the EleutherAI `lm-evaluation-harness` under the **opt-in `[benchmark]` extra**, which Headroom invokes as a subprocess (never imports): - `sqlitedict` CVE-2024-35515 (High) — pickle deserialization; package abandoned (last release 2021), **no upstream fix exists**. - `nltk` CVE-2026-54293 (High) — path traversal in `nltk.data.load()`; affects ≤3.9.4 (current latest), **no patched release**. Neither is in `[all]`, the published wheel, or the container. They are documented in `sbom/README.md` and will be picked up automatically once upstream ships fixes. **Release/CHANGELOG:** N/A items above are because this is a dependency/security PR with no Python source changes; CHANGELOG is Release-Please-managed via the conventional commit message.
2026-06-27 15:28:12 -07:00
checksum = "9ac53762fd065daa3194dd09337a38bd793a188100fd1a9304c4ab312d901771"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"proc-macro2",
"pyo3-macros-backend",
"quote",
"syn",
]
[[package]]
name = "pyo3-macros-backend"
fix(deps): remediate dependency CVEs and publish SBOM (#1509) ## Description Supply-chain hardening: takes the **shipped** dependency surface from **26 known CVEs to 0**. `pip install headroom-ai[all]` now resolves with no known vulnerabilities (verified with Anchore syft + grype). Also publishes a checked-in SBOM package (`sbom/`) so any user — especially pilots running their own security review — can verify what's inside and that we track it. This addresses the Dependabot alerts on `main` (9 high / 4 moderate / 7 low at time of writing). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made **Rust** - `pyo3` 0.24 → 0.29 (GHSA-36hh-v3qg-5jq4 High, GHSA-chgr-c6px-7xpp Med). Migrated `Python::allow_threads` → `Python::detach` (10 sites) and added `from_py_object` to the `Clone`-deriving `#[pyclass]` types (both required by the 0.25+ API). - `pyo3-log` 0.12 → 0.13; `lru` 0.12 → 0.18 (GHSA-rhfx-m35p-ff5j). **Python** - `torch` → 2.12.1, `mem0ai` → 2.x. - Floor-pinned transitive CVE deps via `[tool.uv] constraint-dependencies`: `pygments>=2.20.0`, `pydantic-settings>=2.14.2`, `gitpython>=3.1.50`, `langsmith>=0.9.0`. - **Removed `benchmark` from the `[all]` aggregate** so the default install is CVE-free. `lm-eval` is invoked as an external subprocess (`python -m lm_eval`) and never imported, so it is not a true runtime dep — it remains available via the opt-in `[benchmark]` extra. See [Accepted Risks](#additional-notes). **npm (build/test tooling — never shipped in the wheel/container/published SDK)** - `esbuild` override `>=0.28.1` in `sdk/typescript` + `plugins/openclaw` (GHSA-g7r4-m6w7-qqqr). - `docs/`: `@anthropic-ai/sdk` → `^0.106.0` (GHSA-p7fg-763f-g4gf), `postcss` override to force Next.js's bundled copy ≥8.5.10 (GHSA-qx2v-qp2m-jg93); regenerated a stale `bun.lock` that carried a **Critical** vitest/vite. **CI** - Pinned `pypa/gh-action-pypi-publish` `@release/v1` → `@v1.13.0` (GHSA-vxmw-7h4f-hqxh) in `release.yml` + `publish.yml`. **SBOM** - New `sbom/` directory: CycloneDX 1.7 + SPDX 2.3 SBOMs, grype scan evidence, 330-package license inventory, and a regeneration guide. ## Testing - [ ] Unit tests pass (`pytest`) — N/A, no Python source changed (deps/config only) - [x] Linting passes — `cargo fmt --check` + `cargo clippy` clean on the changed crate; 0 `.py` files changed so `ruff`/`mypy` scope is unaffected - [x] Type checking passes — `cargo check --workspace` (0 errors) - [ ] New tests added — N/A (dependency bumps; covered by existing suites) - [x] Manual testing performed — see Real Behavior Proof ### Test Output ```text # headroom-ai[all] product surface — the number that matters $ grype sbom:sbom/headroom-sbom-all-extra.cdx.json No vulnerabilities found # full repo scan (universal lock incl. opt-in [benchmark] + dev) $ grype sbom:sbom/headroom-sbom.cdx.json NAME INSTALLED TYPE VULNERABILITY SEVERITY sqlitedict 2.1.0 python GHSA-g4r7-86gm-pgqc High # [benchmark]-only, unpatchable, accepted nltk 3.9.4 python GHSA-p4gq-832x-fm9v High # [benchmark]-only, unpatchable, accepted # pyo3 0.29 migration — extension builds + imports + runs $ cargo check --workspace Finished `dev` profile [unoptimized + debuginfo] target(s) $ maturin develop && python -c "from headroom._core import DiffCompressor, SmartCrusher; ..." extension OK — detach + from_py_object paths exercised # lru 0.18 — eviction path $ cargo test -p headroom-proxy --lib drift 14 passed, 213 filtered out # per-ecosystem npm audits $ (cd sdk/typescript && npm audit) -> found 0 vulnerabilities $ (cd plugins/openclaw && npm audit) -> found 0 vulnerabilities $ (cd docs && npm audit && bun audit) -> found 0 vulnerabilities / No vulnerabilities found ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0, arm64), Python 3.12 `.venv`, Rust 1.95 toolchain, syft 1.46.0, grype 0.115.0, bun 1.3.14, maturin 1.13.3. - Exact command / steps: (1) `uv export --extra all --no-dev --no-emit-project | syft → grype` for the product surface; (2) `cargo check --workspace` + `maturin develop` + extension import/compress smoke test; (3) `cargo test -p headroom-proxy --lib drift`; (4) `cargo fmt --check` + `cargo clippy -p headroom-py`; (5) `npm audit` in sdk/openclaw/docs + `bun audit` in docs. - Observed result: `headroom-ai[all]` resolution scans clean — "No vulnerabilities found" (179 pkgs); full/prod SBOM shows only the 2 documented accepted CVEs; pyo3 0.29 extension imports and runs (detach + from_py_object paths exercised); drift tests 14/14 pass; cargo fmt + clippy clean; all npm/bun audits report 0. - Not tested: full `pytest` suite (no Python source changed); release-profile wheel build (used dev-profile `maturin develop` for the import proof — the extension is semantically identical). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (`sbom/README.md`) - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective — N/A (dependency bumps; existing suites + scans cover it) - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A (Release Please auto-generates from the conventional commit) ## Additional Notes **Accepted risks (the 2 residual CVEs).** Both originate solely from the EleutherAI `lm-evaluation-harness` under the **opt-in `[benchmark]` extra**, which Headroom invokes as a subprocess (never imports): - `sqlitedict` CVE-2024-35515 (High) — pickle deserialization; package abandoned (last release 2021), **no upstream fix exists**. - `nltk` CVE-2026-54293 (High) — path traversal in `nltk.data.load()`; affects ≤3.9.4 (current latest), **no patched release**. Neither is in `[all]`, the published wheel, or the container. They are documented in `sbom/README.md` and will be picked up automatically once upstream ships fixes. **Release/CHANGELOG:** N/A items above are because this is a dependency/security PR with no Python source changes; CHANGELOG is Release-Please-managed via the conventional commit message.
2026-06-27 15:28:12 -07:00
version = "0.29.0"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
fix(deps): remediate dependency CVEs and publish SBOM (#1509) ## Description Supply-chain hardening: takes the **shipped** dependency surface from **26 known CVEs to 0**. `pip install headroom-ai[all]` now resolves with no known vulnerabilities (verified with Anchore syft + grype). Also publishes a checked-in SBOM package (`sbom/`) so any user — especially pilots running their own security review — can verify what's inside and that we track it. This addresses the Dependabot alerts on `main` (9 high / 4 moderate / 7 low at time of writing). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made **Rust** - `pyo3` 0.24 → 0.29 (GHSA-36hh-v3qg-5jq4 High, GHSA-chgr-c6px-7xpp Med). Migrated `Python::allow_threads` → `Python::detach` (10 sites) and added `from_py_object` to the `Clone`-deriving `#[pyclass]` types (both required by the 0.25+ API). - `pyo3-log` 0.12 → 0.13; `lru` 0.12 → 0.18 (GHSA-rhfx-m35p-ff5j). **Python** - `torch` → 2.12.1, `mem0ai` → 2.x. - Floor-pinned transitive CVE deps via `[tool.uv] constraint-dependencies`: `pygments>=2.20.0`, `pydantic-settings>=2.14.2`, `gitpython>=3.1.50`, `langsmith>=0.9.0`. - **Removed `benchmark` from the `[all]` aggregate** so the default install is CVE-free. `lm-eval` is invoked as an external subprocess (`python -m lm_eval`) and never imported, so it is not a true runtime dep — it remains available via the opt-in `[benchmark]` extra. See [Accepted Risks](#additional-notes). **npm (build/test tooling — never shipped in the wheel/container/published SDK)** - `esbuild` override `>=0.28.1` in `sdk/typescript` + `plugins/openclaw` (GHSA-g7r4-m6w7-qqqr). - `docs/`: `@anthropic-ai/sdk` → `^0.106.0` (GHSA-p7fg-763f-g4gf), `postcss` override to force Next.js's bundled copy ≥8.5.10 (GHSA-qx2v-qp2m-jg93); regenerated a stale `bun.lock` that carried a **Critical** vitest/vite. **CI** - Pinned `pypa/gh-action-pypi-publish` `@release/v1` → `@v1.13.0` (GHSA-vxmw-7h4f-hqxh) in `release.yml` + `publish.yml`. **SBOM** - New `sbom/` directory: CycloneDX 1.7 + SPDX 2.3 SBOMs, grype scan evidence, 330-package license inventory, and a regeneration guide. ## Testing - [ ] Unit tests pass (`pytest`) — N/A, no Python source changed (deps/config only) - [x] Linting passes — `cargo fmt --check` + `cargo clippy` clean on the changed crate; 0 `.py` files changed so `ruff`/`mypy` scope is unaffected - [x] Type checking passes — `cargo check --workspace` (0 errors) - [ ] New tests added — N/A (dependency bumps; covered by existing suites) - [x] Manual testing performed — see Real Behavior Proof ### Test Output ```text # headroom-ai[all] product surface — the number that matters $ grype sbom:sbom/headroom-sbom-all-extra.cdx.json No vulnerabilities found # full repo scan (universal lock incl. opt-in [benchmark] + dev) $ grype sbom:sbom/headroom-sbom.cdx.json NAME INSTALLED TYPE VULNERABILITY SEVERITY sqlitedict 2.1.0 python GHSA-g4r7-86gm-pgqc High # [benchmark]-only, unpatchable, accepted nltk 3.9.4 python GHSA-p4gq-832x-fm9v High # [benchmark]-only, unpatchable, accepted # pyo3 0.29 migration — extension builds + imports + runs $ cargo check --workspace Finished `dev` profile [unoptimized + debuginfo] target(s) $ maturin develop && python -c "from headroom._core import DiffCompressor, SmartCrusher; ..." extension OK — detach + from_py_object paths exercised # lru 0.18 — eviction path $ cargo test -p headroom-proxy --lib drift 14 passed, 213 filtered out # per-ecosystem npm audits $ (cd sdk/typescript && npm audit) -> found 0 vulnerabilities $ (cd plugins/openclaw && npm audit) -> found 0 vulnerabilities $ (cd docs && npm audit && bun audit) -> found 0 vulnerabilities / No vulnerabilities found ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0, arm64), Python 3.12 `.venv`, Rust 1.95 toolchain, syft 1.46.0, grype 0.115.0, bun 1.3.14, maturin 1.13.3. - Exact command / steps: (1) `uv export --extra all --no-dev --no-emit-project | syft → grype` for the product surface; (2) `cargo check --workspace` + `maturin develop` + extension import/compress smoke test; (3) `cargo test -p headroom-proxy --lib drift`; (4) `cargo fmt --check` + `cargo clippy -p headroom-py`; (5) `npm audit` in sdk/openclaw/docs + `bun audit` in docs. - Observed result: `headroom-ai[all]` resolution scans clean — "No vulnerabilities found" (179 pkgs); full/prod SBOM shows only the 2 documented accepted CVEs; pyo3 0.29 extension imports and runs (detach + from_py_object paths exercised); drift tests 14/14 pass; cargo fmt + clippy clean; all npm/bun audits report 0. - Not tested: full `pytest` suite (no Python source changed); release-profile wheel build (used dev-profile `maturin develop` for the import proof — the extension is semantically identical). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (`sbom/README.md`) - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective — N/A (dependency bumps; existing suites + scans cover it) - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A (Release Please auto-generates from the conventional commit) ## Additional Notes **Accepted risks (the 2 residual CVEs).** Both originate solely from the EleutherAI `lm-evaluation-harness` under the **opt-in `[benchmark]` extra**, which Headroom invokes as a subprocess (never imports): - `sqlitedict` CVE-2024-35515 (High) — pickle deserialization; package abandoned (last release 2021), **no upstream fix exists**. - `nltk` CVE-2026-54293 (High) — path traversal in `nltk.data.load()`; affects ≤3.9.4 (current latest), **no patched release**. Neither is in `[all]`, the published wheel, or the container. They are documented in `sbom/README.md` and will be picked up automatically once upstream ships fixes. **Release/CHANGELOG:** N/A items above are because this is a dependency/security PR with no Python source changes; CHANGELOG is Release-Please-managed via the conventional commit message.
2026-06-27 15:28:12 -07:00
checksum = "4ca3a1557399783172dc5bf39cfca835157732532cba56b71d2292161e53b362"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"heck",
"proc-macro2",
"quote",
"syn",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "qoi"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7f6d64c71eb498fe9eae14ce4ec935c555749aef511cca85b5568910d6e48001"
dependencies = [
"bytemuck",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "quick-error"
version = "1.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a1d01941d82fa2ab50be1e79e6714289dd7cde78eba4c074bc5a4374f650dfe0"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "quick-error"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a993555f31e5a609f617c12db6250dedcac1b0a85076912c436e6fc9b2c8e6a3"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "quinn"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.11.11"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "0c1a41e437b6bbd489372cd4971de128e85c855f56c57f283d20ff016cf7c0a8"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"bytes",
"cfg_aliases",
"pin-project-lite",
"quinn-proto",
"quinn-udp",
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
"rustc-hash 2.1.2",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"rustls",
"socket2",
"thiserror 2.0.18",
"tokio",
"tracing",
"web-time",
]
[[package]]
name = "quinn-proto"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.11.15"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "4fcb935c5bec503c2f0e306bdd3e58bb9029dcb14fa8d9ac76e3a5256ac0763e"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"bytes",
"getrandom 0.3.4",
"lru-slab",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"rand 0.9.4",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"ring",
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
"rustc-hash 2.1.2",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"rustls",
"rustls-pki-types",
"slab",
"thiserror 2.0.18",
"tinyvec",
"tracing",
"web-time",
]
[[package]]
name = "quinn-udp"
version = "0.5.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "addec6a0dcad8a8d96a771f815f0eaf55f9d1805756410b39f5fa81332574cbd"
dependencies = [
"cfg_aliases",
"libc",
"once_cell",
"socket2",
"tracing",
"windows-sys 0.60.2",
]
[[package]]
name = "quote"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.0.46"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "dfbc457d0c7a0759a614551b11a6409e5951f6c7537be1f1b7682b9ae9230368"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"proc-macro2",
]
[[package]]
name = "r-efi"
version = "5.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "r-efi"
version = "6.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf"
[[package]]
name = "rand"
version = "0.8.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5ca0ecfa931c29007047d1bc58e623ab12e5590e8c7cc53200d5202b69266d8a"
dependencies = [
"libc",
"rand_chacha 0.3.1",
"rand_core 0.6.4",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "rand"
version = "0.9.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "44c5af06bb1b7d3216d91932aed5265164bf384dc89cd6ba05cf59a35f5f76ea"
dependencies = [
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"rand_chacha 0.9.0",
"rand_core 0.9.5",
]
[[package]]
name = "rand_chacha"
version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88"
dependencies = [
"ppv-lite86",
"rand_core 0.6.4",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
]
[[package]]
name = "rand_chacha"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb"
dependencies = [
"ppv-lite86",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"rand_core 0.9.5",
]
[[package]]
name = "rand_core"
version = "0.6.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c"
dependencies = [
"getrandom 0.2.17",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
]
[[package]]
name = "rand_core"
version = "0.9.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c"
dependencies = [
"getrandom 0.3.4",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "rand_xorshift"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "513962919efc330f829edb2535844d1b912b0fbe2ca165d613e4e8788bb05a5a"
dependencies = [
"rand_core 0.9.5",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "rav1e"
version = "0.8.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "43b6dd56e85d9483277cde964fd1bdb0428de4fec5ebba7540995639a21cb32b"
dependencies = [
"aligned-vec",
"arbitrary",
"arg_enum_proc_macro",
"arrayvec",
"av-scenechange",
"av1-grain",
"bitstream-io",
"built",
"cfg-if",
"interpolate_name",
"itertools 0.14.0",
"libc",
"libfuzzer-sys",
"log",
"maybe-rayon",
"new_debug_unreachable",
"noop_proc_macro",
"num-derive",
"num-traits",
"paste",
"profiling",
"rand 0.9.4",
"rand_chacha 0.9.0",
"simd_helpers",
"thiserror 2.0.18",
"v_frame",
"wasm-bindgen",
]
[[package]]
name = "ravif"
version = "0.13.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e52310197d971b0f5be7fe6b57530dcd27beb35c1b013f29d66c1ad73fbbcc45"
dependencies = [
"avif-serialize",
"imgref",
"loop9",
"quick-error 2.0.1",
"rav1e",
"rayon",
"rgb",
]
[[package]]
name = "rawpointer"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "60a357793950651c4ed0f3f52338f53b2f809f32d83a07f72909fa13e4c6c1e3"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "rayon"
version = "1.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fb39b166781f92d482534ef4b4b1b2568f42613b53e5b6c160e24cfbfa30926d"
dependencies = [
"either",
"rayon-core",
]
[[package]]
name = "rayon-cond"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2964d0cf57a3e7a06e8183d14a8b527195c706b7983549cd5462d5aa3747438f"
dependencies = [
"either",
"itertools 0.14.0",
"rayon",
]
[[package]]
name = "rayon-core"
version = "1.13.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "22e18b0f0062d30d4230b2e85ff77fdfe4326feb054b9783a3460d8435c8ab91"
dependencies = [
"crossbeam-deque",
"crossbeam-utils",
]
fix: B7 — CCR hardening: persistent backends + always-on tool P2-25, P2-26: CCR (Compress-Cache-Retrieve) used an in-memory store that fragmented across uvicorn workers and was wiped on restart, and the `headroom_retrieve` tool was registered/unregistered per-request based on whether the latest body happened to contain compression markers — every flip busted the prompt cache. Both are sticky side-channels: once a session has done CCR, the tool list bytes and the retrieval store must stay stable. This PR fixes both. Rust: * Split `ccr.rs` into `ccr/` with `backends/` submodule (`in_memory.rs`, `sqlite.rs`, `redis.rs` cfg-gated). * `SqliteCcrStore` (production default): WAL mode, prepared upsert, lazy TTL purge on read, persistent across worker restarts and shareable across workers on the same host via SQLite file locking. * `RedisCcrStore` (cfg-gated behind `feature = "redis"`): SETEX with startup PING smoke-test, no key-prefix collision risk, no sticky session required at the LB. * `CcrBackendConfig::{InMemory, Sqlite, Redis}` + `from_config(...)` factory — every init failure surfaces (no silent fallback per `feedback_no_silent_fallbacks.md`). * `ccr::compute_key` (BLAKE3 → first 24 hex chars) and `ccr::marker_for("HASH") -> "<<ccr:HASH>>"` centralize the hash + marker format; one definition for the live-zone dispatcher and the Python regex (`headroom/ccr/tool_injection.py:211`). * `compress_anthropic_live_zone_with_ccr` accepts `Option<&dyn CcrStore>`. When wired, every accepted compression puts the original bytes into the backend and appends `<<ccr:HASH>>` to the compressed string. The token-validation gate runs on the marker-augmented string so the `compressed_tokens >= original_tokens` rejection stays honest. Python: * `SessionCcrTracker` + `apply_session_sticky_ccr_tool` mirror the PR-A7 `SessionToolTracker` / `apply_session_sticky_memory_tools` pattern: once a session has done CCR, every subsequent request injects the recorded golden tool-definition bytes. Tool list bytes are byte-stable across turns (snapshot test pins them). * `headroom/ccr/tool_injection.py::inject_tool_definition` accepts a new `session_has_done_ccr` kwarg per the PR-B7 spec change at line 302-328. The legacy per-request path stays intact for callers that don't yet thread a session id (e.g. Google handler). * Anthropic + OpenAI handlers route their CCR tool-list updates through `apply_session_sticky_ccr_tool`, keyed off the existing `session_tracker_store.compute_session_id(...)` plumbing. Backend selection model: `CcrBackendConfig::Sqlite { path }` is the production default — single host, persistent, multi-worker safe with sticky session. `CcrBackendConfig::Redis { url }` is the multi-host scale-out option — no stickiness needed. `InMemory` is for tests and single-worker dev only. RUST_DEV.md "Multi-worker deployment — CCR fragmentation" rewritten around this matrix. Tests: * `crates/headroom-core/tests/ccr_backends.rs` — 7 tests covering SQLite round-trip, TTL purge, proxy-restart survival, cross-backend byte-equal keys, `from_config` paths, and the no-redis-feature loud-failure check (+ 2 redis tests gated behind the feature). * `crates/headroom-core/tests/live_zone_ccr.rs` — confirms `<<ccr:HASH>>` marker injection, store population, and no-marker-when-no-store invariants end-to-end. * `tests/test_ccr_tool_always_on.py` — 12 tests pinning the always-on behaviour, session/provider isolation, LRU bound, no- session-id fallback, and (per-acceptance-criterion) the byte-stable tool-definition snapshot. Per-PR-B7 plan: REALIGNMENT/04-phase-B-live-zone.md.
2026-05-02 16:49:08 -07:00
[[package]]
name = "redis"
version = "0.27.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09d8f99a4090c89cc489a94833c901ead69bfbf3877b4867d5482e321ee875bc"
dependencies = [
"arc-swap",
"combine",
"itertools 0.13.0",
"itoa",
"num-bigint",
"percent-encoding",
"ryu",
"url",
]
perf(rust): tier-1 multi-worker wins — GIL release, sharded CCR store, single-serialize CCR write Three orthogonal hot-path fixes targeting concurrent-request throughput. Each is independently bench-measured below; the proxy hot path benefits from all three at once. == 1. PyO3 GIL release on heavy compute == PyO3 methods (crush, smart_crush_content, crush_array_json, compact_document_json, compress, compress_with_stats) used to hold the GIL across the entire Rust call. Result: a 100ms compress() blocked EVERY other Python thread for 100ms — multi-worker uvicorn deployments serialized through SmartCrusher. Wrap each compute call in `py.allow_threads(|| ...)`. Inputs (`&str` from Python) are copied to owned `String` first because PyO3 ties them to the GIL hold. PyDict construction stays on the GIL side. Measured: 4 Python threads each running 20 crushes: before (GIL held): ~3.3s wall (serialized — equivalent to 4×0.83s) after (allow_threads): 826ms wall (4.01x speedup, perfect parallel) == 2. CcrStore: Mutex<HashMap> -> DashMap-backed sharded == Single Mutex was the dominant bottleneck under multi-worker load — every put/get serialized through one lock. Replace with DashMap (sharded concurrent map, lock-free reads within a shard) plus a separate small Mutex<VecDeque> for FIFO insertion-order eviction. Reads of distinct keys never contend; writes only contend during the brief order-queue push or capacity-sweep. A/B bench (200 mixed put/get ops × N threads, in benches/ccr_store.rs): Threads | DashMap Legacy Mutex Speedup ------------------------------------------- 1 | 63 µs 71 µs 1.13x 2 | 98 µs 194 µs 2.0x 4 | 178 µs 707 µs 4.0x 8 | 342 µs 1267 µs 3.7x Legacy degrades ~linearly with thread count; DashMap stays near-flat per-thread. Real multi-worker scaling. == 3. Single-serialize the lossy CCR payload == The lossy `crush_array` path used to serialize the full array TWICE: once in `hash_array_for_ccr` (allocates `Value::Array(items.to_vec())`, deep-clones every Value subtree, then serializes), and a second time in the store-write site. For a 50-item dict array that's ~MB of allocator pressure per crushed array. Introduce `canonical_array_json` (serializes `&[Value]` directly — same bytes as `Value::Array(items.to_vec())` but no wrapper allocation + no tree clone), call it ONCE per lossy path, then both hash and store from those same bytes. Hash-format stable — all 17 parity fixtures match byte-for-byte. == Tests == - 8 ccr.rs unit tests including a new concurrent-stress test (8 threads × 200 puts/gets, every key readable afterwards) - 14 ccr_roundtrip integration tests stay green - parity-run smart_crusher: 17/17 fixtures match - 479 lib + 14 integration + 185 Python tests all pass - New benches/ccr_store.rs runs the A/B and is committed for regression visibility == Dependencies added == - dashmap v6 (mature, widely-used in tokio/linkerd ecosystem)
2026-04-27 20:44:55 -07:00
[[package]]
name = "redox_syscall"
version = "0.5.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d"
dependencies = [
"bitflags",
]
feat(rust): HfTokenizer::from_pretrained — HuggingFace Hub auto-download Stage 2.1: closes the loop on the HuggingFace tokenizer story. Stage 2 shipped `HfTokenizer::from_bytes`/`from_file`, which required callers to manage their own tokenizer.json files. This adds the third constructor: let t = HfTokenizer::from_pretrained("CohereForAI/c4ai-command-r-v01")?; register_hf("command-", t); `from_pretrained` is a thin wrapper around the `hf-hub` crate's blocking `ureq` API. First call downloads `tokenizer.json` to `~/.cache/huggingface/ hub` (or `$HF_HOME` if set); subsequent calls reuse the on-disk cache. Uses the `main` revision; gated repos (Llama, Mistral) require `HF_TOKEN` in env or `~/.cache/huggingface/token`. Also adds `try_register_hf(prefix, repo)` as the obvious one-liner for proxy startup code: let _ = try_register_hf("command-", "CohereForAI/c4ai-command-r-v01"); let _ = try_register_hf("mistral-", "mistralai/Mistral-7B-v0.1"); Each call is independent — a download failure for one model (e.g. gated without a token) does not affect others. `HfTokenizerError` gains a new `Hub` variant so callers can distinguish "couldn't fetch" from "fetched but malformed" — relevant when deciding whether to retry, surface to the user, or fall back to the estimator. Why blocking, not async: `from_pretrained` is called once at startup. A sync API works from `main()`, from a `OnceLock` initializer, or from `tokio::task::spawn_blocking` if a tokio caller needs it later. The async hf-hub backend would force callers to await at startup, which doesn't fit the `register_hf` registry pattern. Why rustls, not native-tls: keeps the binary statically linkable for AWS deploys (no system OpenSSL dependency). Tests: a network-dependent integration test (`#[ignore]`d in CI; hits HF for `gpt2`, ~1.4 MB) verifies the real download + load + count path. A non-network negative test verifies that an invalid repo name surfaces as `HfTokenizerError::Hub`, not a panic. 44 unit tests + 5 proptests + 1 doctest pass; parity stays 40/40 byte-equal.
2026-04-25 15:05:09 -07:00
[[package]]
name = "redox_users"
version = "0.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a4e608c6638b9c18977b00b475ac1f28d14e84b27d8d42f70e0bf1e3dec127ac"
dependencies = [
"getrandom 0.2.17",
"libredox",
"thiserror 2.0.18",
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "regex"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.12.4"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "f1292b7759ae1cb9ec195452d1390a074f0cd8541ab7a5a8c31cd6db45d4a6ba"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
dependencies = [
"aho-corasick",
"memchr",
"regex-automata",
"regex-syntax",
]
[[package]]
name = "regex-automata"
version = "0.4.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6e1dd4122fc1595e8162618945476892eefca7b88c52820e74af6262213cae8f"
dependencies = [
"aho-corasick",
"memchr",
"regex-syntax",
]
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
[[package]]
name = "regex-lite"
version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cab834c73d247e67f4fae452806d17d3c7501756d98c8808d7c9c7aa7d18f973"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "regex-syntax"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.8.11"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "reqwest"
version = "0.12.28"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "eddd3ca559203180a307f12d114c268abf583f59b03cb906fd0b3ff8646c1147"
dependencies = [
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
"base64 0.22.1",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"bytes",
"futures-core",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"futures-util",
"h2",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"http-body 1.0.1",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"http-body-util",
"hyper",
"hyper-rustls",
"hyper-util",
"js-sys",
"log",
"percent-encoding",
"pin-project-lite",
"quinn",
"rustls",
"rustls-pki-types",
"serde",
"serde_json",
"serde_urlencoded",
"sync_wrapper",
"tokio",
"tokio-rustls",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"tokio-util",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"tower",
"tower-http",
"tower-service",
"url",
"wasm-bindgen",
"wasm-bindgen-futures",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"wasm-streams",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"web-sys",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"webpki-roots 1.0.8",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "rgb"
version = "0.8.53"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "47b34b781b31e5d73e9fbc8689c70551fd1ade9a19e3e28cfec8580a79290cc4"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "ring"
version = "0.17.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7"
dependencies = [
"cc",
"cfg-if",
"getrandom 0.2.17",
"libc",
"untrusted",
"windows-sys 0.52.0",
]
fix: B7 — CCR hardening: persistent backends + always-on tool P2-25, P2-26: CCR (Compress-Cache-Retrieve) used an in-memory store that fragmented across uvicorn workers and was wiped on restart, and the `headroom_retrieve` tool was registered/unregistered per-request based on whether the latest body happened to contain compression markers — every flip busted the prompt cache. Both are sticky side-channels: once a session has done CCR, the tool list bytes and the retrieval store must stay stable. This PR fixes both. Rust: * Split `ccr.rs` into `ccr/` with `backends/` submodule (`in_memory.rs`, `sqlite.rs`, `redis.rs` cfg-gated). * `SqliteCcrStore` (production default): WAL mode, prepared upsert, lazy TTL purge on read, persistent across worker restarts and shareable across workers on the same host via SQLite file locking. * `RedisCcrStore` (cfg-gated behind `feature = "redis"`): SETEX with startup PING smoke-test, no key-prefix collision risk, no sticky session required at the LB. * `CcrBackendConfig::{InMemory, Sqlite, Redis}` + `from_config(...)` factory — every init failure surfaces (no silent fallback per `feedback_no_silent_fallbacks.md`). * `ccr::compute_key` (BLAKE3 → first 24 hex chars) and `ccr::marker_for("HASH") -> "<<ccr:HASH>>"` centralize the hash + marker format; one definition for the live-zone dispatcher and the Python regex (`headroom/ccr/tool_injection.py:211`). * `compress_anthropic_live_zone_with_ccr` accepts `Option<&dyn CcrStore>`. When wired, every accepted compression puts the original bytes into the backend and appends `<<ccr:HASH>>` to the compressed string. The token-validation gate runs on the marker-augmented string so the `compressed_tokens >= original_tokens` rejection stays honest. Python: * `SessionCcrTracker` + `apply_session_sticky_ccr_tool` mirror the PR-A7 `SessionToolTracker` / `apply_session_sticky_memory_tools` pattern: once a session has done CCR, every subsequent request injects the recorded golden tool-definition bytes. Tool list bytes are byte-stable across turns (snapshot test pins them). * `headroom/ccr/tool_injection.py::inject_tool_definition` accepts a new `session_has_done_ccr` kwarg per the PR-B7 spec change at line 302-328. The legacy per-request path stays intact for callers that don't yet thread a session id (e.g. Google handler). * Anthropic + OpenAI handlers route their CCR tool-list updates through `apply_session_sticky_ccr_tool`, keyed off the existing `session_tracker_store.compute_session_id(...)` plumbing. Backend selection model: `CcrBackendConfig::Sqlite { path }` is the production default — single host, persistent, multi-worker safe with sticky session. `CcrBackendConfig::Redis { url }` is the multi-host scale-out option — no stickiness needed. `InMemory` is for tests and single-worker dev only. RUST_DEV.md "Multi-worker deployment — CCR fragmentation" rewritten around this matrix. Tests: * `crates/headroom-core/tests/ccr_backends.rs` — 7 tests covering SQLite round-trip, TTL purge, proxy-restart survival, cross-backend byte-equal keys, `from_config` paths, and the no-redis-feature loud-failure check (+ 2 redis tests gated behind the feature). * `crates/headroom-core/tests/live_zone_ccr.rs` — confirms `<<ccr:HASH>>` marker injection, store population, and no-marker-when-no-store invariants end-to-end. * `tests/test_ccr_tool_always_on.py` — 12 tests pinning the always-on behaviour, session/provider isolation, LRU bound, no- session-id fallback, and (per-acceptance-criterion) the byte-stable tool-definition snapshot. Per-PR-B7 plan: REALIGNMENT/04-phase-B-live-zone.md.
2026-05-02 16:49:08 -07:00
[[package]]
name = "rusqlite"
version = "0.32.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7753b721174eb8ff87a9a0e799e2d7bc3749323e773db92e0984debb00019d6e"
dependencies = [
"bitflags",
"fallible-iterator",
"fallible-streaming-iterator",
"hashlink",
"libsqlite3-sys",
"smallvec",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "rustc-hash"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "rustc-hash"
version = "2.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
[[package]]
name = "rustc_version"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92"
dependencies = [
"semver",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "rustix"
version = "1.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190"
dependencies = [
"bitflags",
"errno",
"libc",
"linux-raw-sys",
"windows-sys 0.61.2",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "rustls"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.23.41"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "6b92b125634d9b795e7beca796cc790df15a7fb38323bf3196fda83292d06b1f"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"aws-lc-rs",
feat(rust): HfTokenizer::from_pretrained — HuggingFace Hub auto-download Stage 2.1: closes the loop on the HuggingFace tokenizer story. Stage 2 shipped `HfTokenizer::from_bytes`/`from_file`, which required callers to manage their own tokenizer.json files. This adds the third constructor: let t = HfTokenizer::from_pretrained("CohereForAI/c4ai-command-r-v01")?; register_hf("command-", t); `from_pretrained` is a thin wrapper around the `hf-hub` crate's blocking `ureq` API. First call downloads `tokenizer.json` to `~/.cache/huggingface/ hub` (or `$HF_HOME` if set); subsequent calls reuse the on-disk cache. Uses the `main` revision; gated repos (Llama, Mistral) require `HF_TOKEN` in env or `~/.cache/huggingface/token`. Also adds `try_register_hf(prefix, repo)` as the obvious one-liner for proxy startup code: let _ = try_register_hf("command-", "CohereForAI/c4ai-command-r-v01"); let _ = try_register_hf("mistral-", "mistralai/Mistral-7B-v0.1"); Each call is independent — a download failure for one model (e.g. gated without a token) does not affect others. `HfTokenizerError` gains a new `Hub` variant so callers can distinguish "couldn't fetch" from "fetched but malformed" — relevant when deciding whether to retry, surface to the user, or fall back to the estimator. Why blocking, not async: `from_pretrained` is called once at startup. A sync API works from `main()`, from a `OnceLock` initializer, or from `tokio::task::spawn_blocking` if a tokio caller needs it later. The async hf-hub backend would force callers to await at startup, which doesn't fit the `register_hf` registry pattern. Why rustls, not native-tls: keeps the binary statically linkable for AWS deploys (no system OpenSSL dependency). Tests: a network-dependent integration test (`#[ignore]`d in CI; hits HF for `gpt2`, ~1.4 MB) verifies the real download + load + count path. A non-network negative test verifies that an invalid repo name surfaces as `HfTokenizerError::Hub`, not a panic. 44 unit tests + 5 proptests + 1 doctest pass; parity stays 40/40 byte-equal.
2026-04-25 15:05:09 -07:00
"log",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"once_cell",
"ring",
"rustls-pki-types",
"rustls-webpki",
"subtle",
"zeroize",
]
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
[[package]]
name = "rustls-native-certs"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.8.4"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "dab5152771c58876a2146916e53e35057e1a4dfa2b9df0f0305b07f611fdea4d"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
dependencies = [
"openssl-probe",
"rustls-pki-types",
"schannel",
"security-framework",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "rustls-pki-types"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.14.1"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "30a7197ae7eb376e574fe940d068c30fe0462554a3ddbe4eca7838e049c937a9"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"web-time",
"zeroize",
]
[[package]]
name = "rustls-webpki"
version = "0.103.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e"
dependencies = [
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"aws-lc-rs",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"ring",
"rustls-pki-types",
"untrusted",
]
[[package]]
name = "rustversion"
version = "1.0.22"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "rusty-fork"
version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cc6bf79ff24e648f6da1f8d1f011e9cac26491b619e6b9280f2b47f1774e6ee2"
dependencies = [
"fnv",
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
"quick-error 1.2.3",
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
"tempfile",
"wait-timeout",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "ryu"
version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "safetensors"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.8.0"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "79b079b829cb27a1c3c374341345ed2e8b2c0c839034522cee576c140bd7f846"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
dependencies = [
"hashbrown 0.16.1",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"libc",
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
"serde",
"serde_json",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"tempfile",
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "same-file"
version = "1.0.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502"
dependencies = [
"winapi-util",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "schannel"
version = "0.1.29"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "91c1b7e4904c873ef0710c1f407dde2e6287de2bebc1bbbf7d430bb7cbffd939"
dependencies = [
"windows-sys 0.61.2",
]
perf(rust): tier-1 multi-worker wins — GIL release, sharded CCR store, single-serialize CCR write Three orthogonal hot-path fixes targeting concurrent-request throughput. Each is independently bench-measured below; the proxy hot path benefits from all three at once. == 1. PyO3 GIL release on heavy compute == PyO3 methods (crush, smart_crush_content, crush_array_json, compact_document_json, compress, compress_with_stats) used to hold the GIL across the entire Rust call. Result: a 100ms compress() blocked EVERY other Python thread for 100ms — multi-worker uvicorn deployments serialized through SmartCrusher. Wrap each compute call in `py.allow_threads(|| ...)`. Inputs (`&str` from Python) are copied to owned `String` first because PyO3 ties them to the GIL hold. PyDict construction stays on the GIL side. Measured: 4 Python threads each running 20 crushes: before (GIL held): ~3.3s wall (serialized — equivalent to 4×0.83s) after (allow_threads): 826ms wall (4.01x speedup, perfect parallel) == 2. CcrStore: Mutex<HashMap> -> DashMap-backed sharded == Single Mutex was the dominant bottleneck under multi-worker load — every put/get serialized through one lock. Replace with DashMap (sharded concurrent map, lock-free reads within a shard) plus a separate small Mutex<VecDeque> for FIFO insertion-order eviction. Reads of distinct keys never contend; writes only contend during the brief order-queue push or capacity-sweep. A/B bench (200 mixed put/get ops × N threads, in benches/ccr_store.rs): Threads | DashMap Legacy Mutex Speedup ------------------------------------------- 1 | 63 µs 71 µs 1.13x 2 | 98 µs 194 µs 2.0x 4 | 178 µs 707 µs 4.0x 8 | 342 µs 1267 µs 3.7x Legacy degrades ~linearly with thread count; DashMap stays near-flat per-thread. Real multi-worker scaling. == 3. Single-serialize the lossy CCR payload == The lossy `crush_array` path used to serialize the full array TWICE: once in `hash_array_for_ccr` (allocates `Value::Array(items.to_vec())`, deep-clones every Value subtree, then serializes), and a second time in the store-write site. For a 50-item dict array that's ~MB of allocator pressure per crushed array. Introduce `canonical_array_json` (serializes `&[Value]` directly — same bytes as `Value::Array(items.to_vec())` but no wrapper allocation + no tree clone), call it ONCE per lossy path, then both hash and store from those same bytes. Hash-format stable — all 17 parity fixtures match byte-for-byte. == Tests == - 8 ccr.rs unit tests including a new concurrent-stress test (8 threads × 200 puts/gets, every key readable afterwards) - 14 ccr_roundtrip integration tests stay green - parity-run smart_crusher: 17/17 fixtures match - 479 lib + 14 integration + 185 Python tests all pass - New benches/ccr_store.rs runs the A/B and is committed for regression visibility == Dependencies added == - dashmap v6 (mature, widely-used in tokio/linkerd ecosystem)
2026-04-27 20:44:55 -07:00
[[package]]
name = "scopeguard"
version = "1.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "security-framework"
version = "3.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b7f4bc775c73d9a02cde8bf7b2ec4c9d12743edf609006c7facc23998404cd1d"
dependencies = [
"bitflags",
fix(ci): rustls-everywhere — eliminate openssl-sys from build tree # Root cause of the wheel-build cascade We have shipped 5 release-pipeline hot-fixes in 12 hours, each addressing a different symptom of the same architectural problem: 1. PR #363 — npm artifact downloads + tried `yum openssl-devel` 2. PR #367 — vendored OpenSSL in `headroom-proxy` + dropped Intel mac 3. PR #369 — Debian-cross perl install (`perl` not `libipc-cmd-perl`) 4. PR #370 — moved `openssl/vendored` from headroom-proxy to headroom-py 5. (this PR) — ELIMINATE OpenSSL entirely Each fix exposed a different missing system package or feature flag in a different build surface (manylinux x86_64 vs aarch64-cross-Debian vs macOS Intel vs e2e/wrap Dockerfile vs e2e/init Dockerfile vs main Dockerfile vs devcontainer). We were playing whack-a-mole because every Cargo dep change to the OpenSSL surface required matching system-package updates in 6+ different Dockerfiles and workflows, and the PR-level CI didn't exercise all of them. # Why this PR is the structural fix `fastembed` exposes clean rustls feature flags: - `hf-hub-rustls-tls` (replaces default `hf-hub-native-tls`) - `ort-download-binaries-rustls-tls` (replaces default `…native-tls`) By disabling fastembed's default features and enabling the rustls variants explicitly, we remove `native-tls` (and therefore `openssl-sys`, `openssl`, `openssl-src`, perl modules, OpenSSL build-time deps, vendored OpenSSL ~30s build cost) from the entire workspace dep tree. Verified locally: $ cargo tree -p headroom-py -i openssl-sys error: package ID specification `openssl-sys` did not match any packages $ cargo tree -p headroom-py -i native-tls error: package ID specification `native-tls` did not match any packages $ cargo build --release -p headroom-py Finished `release` profile [optimized] target(s) in 25.57s (Down from 1m+ with vendored OpenSSL.) # Cleanups enabled by this change - crates/headroom-py/Cargo.toml — dropped the `openssl/vendored` workaround from PR #370. - crates/headroom-proxy/Cargo.toml — same dep removed. - e2e/wrap/Dockerfile — dropped `yum install openssl-devel pkgconfig perl-IPC-Cmd`. Comment retained explaining why. - e2e/init/Dockerfile — same. - Dockerfile (main) — dropped `pkg-config libssl-dev` from apt-get. - .devcontainer/Dockerfile — dropped `pkg-config libssl-dev`. - .github/workflows/release.yml — removed the entire before-script-linux block (perl install probe + multi-package-manager dispatch + fail-loud assertion). No longer needed. # Regression gate Three new structural tests in tests/test_release_workflows.py: - test_no_openssl_sys_in_wheel_build_tree — runs `cargo tree -p <crate> -i openssl-sys` for headroom-py / headroom-proxy / headroom-core. If openssl-sys reappears (a future native-tls enabler creeping in via a new dep), this fails AT PR TIME with an actionable message. - test_no_native_tls_in_wheel_build_tree — same shape, native-tls is the proximate cause. - test_fastembed_uses_rustls_features — checks the Cargo.toml so a future "let me bump fastembed and forget the features" doesn't silently re-introduce OpenSSL. Plus two cleanup gates: - test_dockerfiles_no_longer_install_openssl_devel - test_release_yml_does_not_install_openssl_or_perl_for_wheels All 13 release-workflow tests pass. `make ci-precheck` PASSED. # What this teaches us about rollouts (per user's ultrathink ask) The 5-fix cascade exposed three meta-problems: 1. PR checks don't block merges. PR #370 had docker-init-e2e, docker-wrap-e2e, docker-native-e2e all FAILED yet got merged. Branch protection should require these checks. Operator action needed (cannot fix in code). 2. Local validation is misleading. `cargo build -p headroom-py` from the workspace root used the workspace lockfile and looked green; CI did fresh resolution against headroom-py's manifest alone where the feature wasn't enabled. Lesson: verify structural invariants with `cargo tree -e features` before trusting that a build "works." 3. 6+ build surfaces with independent system-dep state. Every Cargo change required matching updates in 6 places. The structural answer (this PR) is to NOT depend on system OpenSSL at all. Where structural fixes are not possible, the answer is a single shared scripts/install-rust-build-deps.sh — but with this PR there's nothing left to install.
2026-05-03 23:26:04 -07:00
"core-foundation",
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
"core-foundation-sys",
"libc",
"security-framework-sys",
]
[[package]]
name = "security-framework-sys"
version = "2.17.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6ce2691df843ecc5d231c0b14ece2acc3efb62c0a398c7e1d875f3983ce020e3"
dependencies = [
"core-foundation-sys",
"libc",
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "semver"
version = "1.0.28"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "serde"
version = "1.0.228"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e"
dependencies = [
"serde_core",
"serde_derive",
]
[[package]]
name = "serde_core"
version = "1.0.228"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad"
dependencies = [
"serde_derive",
]
[[package]]
name = "serde_derive"
version = "1.0.228"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "serde_json"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.0.150"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
fix(rust): smart_crusher scaffold review findings — hash truncation, int parse, python-repr matcher Code review (`/code-review` on commit `d219bee`) caught one critical bug, two important parity gaps, and a few quality nits. Fixed all of them; all 135 unit tests pass; diff_compressor parity harness unaffected (27/27 still matched). # Critical fix — `hash_field_name` truncation length Rust truncated SHA-256 to **16** hex chars; Python uses **8** (per `smart_crusher.py:177`: `hashlib.sha256(...).hexdigest()[:8]`). 16-char hashes would never collide with TOIN's 8-char `preserve_fields`, silently disabling the entire `use_feedback_hints` cache lookup path. Fix: `hex[..8]` instead of `hex[..16]`. Three pinning tests re-verified against actual Python reference output. Doc comment now warns explicitly that the length must match Python or TOIN lookups silently miss. # Important fix — `python_int_parse` mirrors Python's `int()` semantics `statistics.rs::detect_sequential_pattern` previously called `s.parse::<i64>()`. Python's `int()` differs in three ways that affect realistic payloads: - strips ASCII whitespace (Rust's `parse` rejects) - accepts leading `+` (Rust accepts; same) - accepts PEP 515 underscores like `"3_000"` (Rust rejects) A field with `[" 1 ", " 2 ", " 3 ", "4", "5"]` would parse all five in Python (sequential = True) but only one in Rust (`nums.len() < 5` → False). Silent parity break. Fix: new private `python_int_parse` helper that strips whitespace, handles underscore separators, and rejects edge cases Python rejects. Six new tests pin the behavior. # Important fix — `python_repr` for `item_matches_anchors` Python compares anchors via `anchor in str(item).lower()`. We were using `serde_json::to_string(&item).to_lowercase()`, which differs in three ways that affect substring matching: - quote chars (`'` vs `"`) - bool/null literals (`True`/`False`/`None` vs `true`/`false`/`null`) - spacing (`key: value, ...` vs `key:value,...`) Anchor `"none"` would match Python form but not JSON. Inverse for `"null"`. Real divergence. Fix: new private `python_repr` walks `serde_json::Value` and emits Python-equivalent form. Plus enable `serde_json/preserve_order` at workspace level so `Value::Object` preserves JSON parse order (matching Python `dict` since 3.7). # Suggestion fixes - Classifier comment for `[True, False, 1] -> MIXED_ARRAY` now walks both Python and Rust paths step by step. - `ArrayAnalysis::field_stats` doc notes the BTreeMap vs Python-dict order nuance for the analyzer port to resolve. - Added regression tests for "all unparseable strings", "single int among strings", fractional-step sequential, and the email-typo pattern. # Build / test - `cargo build -p headroom-core` clean. - `cargo clippy -p headroom-core -- -D warnings` clean. - 135 unit tests in `headroom-core`, all passing (was 55). - `cargo run -p headroom-parity run` — diff_compressor 27/27 still matched.
2026-04-26 17:01:46 -07:00
"indexmap",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"itoa",
"memchr",
"serde",
"serde_core",
"zmij",
]
[[package]]
name = "serde_path_to_error"
version = "0.1.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "10a9ff822e371bb5403e391ecd83e182e0e77ba7f6fe0160b795797109d1b457"
dependencies = [
"itoa",
"serde",
"serde_core",
]
fix(rust): reformat/offload pipeline + log templates + diff noise (Phase 3g rework) Replaces PR1's lossless/lossy split with ReformatTransform (pack denser, no info lost) and OffloadTransform (drop bytes, CCR-stash original via required cache_key). With CCR every transform is information-preserving end-to-end, so the lossless/lossy distinction misnamed the architecture. OffloadTransform carries a cheap, structural estimate_bloat() method scoped to its domain — generic byte-redundancy heuristics miss domain semantics. The orchestrator runs reformat phase + per-offload bloat estimation in parallel via rayon::join + par_iter, then runs offload iff bloat clears threshold OR reformat underwhelmed. Transforms shipped: REFORMATS (lossless): - JsonMinifier: serde_json round-trip whitespace stripping. - LogTemplate: Drain-inspired order-preserving template miner. Collapses consecutive runs of same-template lines into [Template Tn: ...] (Nx) + variant table. Win comes from emitting the constant-token prefix once instead of N times. Lossless: every original line reconstructible from template + variants. OFFLOADS (drop bytes, stash original via CCR): - LogOffload: wraps existing LogCompressor; bloat = repetition x uniqueness_weight + dilution x priority_dilution_weight. - DiffOffload: wraps existing DiffCompressor; bloat = context-to- change ratio. Bug-fix-on-port — persists original under the cache_key the parity-bound DiffCompressor mints (closes a leak). - DiffNoise: drops lockfile hunks (Cargo.lock, package-lock.json, yarn.lock, etc., suffix list configurable in TOML) and whitespace-only hunks. Stashes original via CCR for retrieval. Search offload exists but is not in default re-exports — modern agents (Claude Code, Codex) use scoped rg/grep, the marginal value didn't justify default registration. Reach via the explicit module path if opting in. JSON Offload is intentionally absent from this PR — already lives at SmartCrusher; Phase 3g PR3 wraps it in the OffloadTransform contract. Thresholds and weights live in config/pipeline.toml, embedded via include_str!; PipelineConfig::from_toml_str loads runtime overrides. 98 new pipeline tests; full headroom-core suite (714) and workspace tests green; cargo fmt clean. No regex, per project convention.
2026-04-30 11:02:29 -07:00
[[package]]
name = "serde_spanned"
version = "0.6.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bf41e0cfaf7226dca15e8197172c295a782857fcb97fad1808a166870dee75a3"
dependencies = [
"serde",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "serde_urlencoded"
version = "0.7.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d3491c14715ca2294c4d6a88f15e84739788c1d030eed8c110436aafdaa2f3fd"
dependencies = [
"form_urlencoded",
"itoa",
"ryu",
"serde",
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "sha1"
version = "0.10.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e3bf829a2d51ab4a5ddf1352d8470c140cadc8301b2ae1789db023f01cedd6ba"
dependencies = [
"cfg-if",
fix: B7 — CCR hardening: persistent backends + always-on tool P2-25, P2-26: CCR (Compress-Cache-Retrieve) used an in-memory store that fragmented across uvicorn workers and was wiped on restart, and the `headroom_retrieve` tool was registered/unregistered per-request based on whether the latest body happened to contain compression markers — every flip busted the prompt cache. Both are sticky side-channels: once a session has done CCR, the tool list bytes and the retrieval store must stay stable. This PR fixes both. Rust: * Split `ccr.rs` into `ccr/` with `backends/` submodule (`in_memory.rs`, `sqlite.rs`, `redis.rs` cfg-gated). * `SqliteCcrStore` (production default): WAL mode, prepared upsert, lazy TTL purge on read, persistent across worker restarts and shareable across workers on the same host via SQLite file locking. * `RedisCcrStore` (cfg-gated behind `feature = "redis"`): SETEX with startup PING smoke-test, no key-prefix collision risk, no sticky session required at the LB. * `CcrBackendConfig::{InMemory, Sqlite, Redis}` + `from_config(...)` factory — every init failure surfaces (no silent fallback per `feedback_no_silent_fallbacks.md`). * `ccr::compute_key` (BLAKE3 → first 24 hex chars) and `ccr::marker_for("HASH") -> "<<ccr:HASH>>"` centralize the hash + marker format; one definition for the live-zone dispatcher and the Python regex (`headroom/ccr/tool_injection.py:211`). * `compress_anthropic_live_zone_with_ccr` accepts `Option<&dyn CcrStore>`. When wired, every accepted compression puts the original bytes into the backend and appends `<<ccr:HASH>>` to the compressed string. The token-validation gate runs on the marker-augmented string so the `compressed_tokens >= original_tokens` rejection stays honest. Python: * `SessionCcrTracker` + `apply_session_sticky_ccr_tool` mirror the PR-A7 `SessionToolTracker` / `apply_session_sticky_memory_tools` pattern: once a session has done CCR, every subsequent request injects the recorded golden tool-definition bytes. Tool list bytes are byte-stable across turns (snapshot test pins them). * `headroom/ccr/tool_injection.py::inject_tool_definition` accepts a new `session_has_done_ccr` kwarg per the PR-B7 spec change at line 302-328. The legacy per-request path stays intact for callers that don't yet thread a session id (e.g. Google handler). * Anthropic + OpenAI handlers route their CCR tool-list updates through `apply_session_sticky_ccr_tool`, keyed off the existing `session_tracker_store.compute_session_id(...)` plumbing. Backend selection model: `CcrBackendConfig::Sqlite { path }` is the production default — single host, persistent, multi-worker safe with sticky session. `CcrBackendConfig::Redis { url }` is the multi-host scale-out option — no stickiness needed. `InMemory` is for tests and single-worker dev only. RUST_DEV.md "Multi-worker deployment — CCR fragmentation" rewritten around this matrix. Tests: * `crates/headroom-core/tests/ccr_backends.rs` — 7 tests covering SQLite round-trip, TTL purge, proxy-restart survival, cross-backend byte-equal keys, `from_config` paths, and the no-redis-feature loud-failure check (+ 2 redis tests gated behind the feature). * `crates/headroom-core/tests/live_zone_ccr.rs` — confirms `<<ccr:HASH>>` marker injection, store population, and no-marker-when-no-store invariants end-to-end. * `tests/test_ccr_tool_always_on.py` — 12 tests pinning the always-on behaviour, session/provider isolation, LRU bound, no- session-id fallback, and (per-acceptance-criterion) the byte-stable tool-definition snapshot. Per-PR-B7 plan: REALIGNMENT/04-phase-B-live-zone.md.
2026-05-02 16:49:08 -07:00
"cpufeatures 0.2.17",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"digest 0.10.7",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
]
feat(rust): scaffold smart_crusher module + foundational helpers Stage 3c.1 — like-for-like Rust port of `headroom/transforms/smart_crusher.py`. This commit lays the foundation: module layout, configuration, foundational data types, and the simpler helpers (classification, hashing, anchors, basic statistics). Subsequent commits add the analyzer, crushers, plan execution, and the orchestrator. # What's in this commit `crates/headroom-core/src/transforms/smart_crusher/`: - `mod.rs` — module entry, public re-exports, port narrative. - `classifier.rs` — `classify_array` / `ArrayType` (dict/string/number/ bool/nested/mixed/empty). Direct port of `_classify_array`. - `config.rs` — `SmartCrusherConfig` with defaults pinned to Python byte-for-byte. - `hashing.rs` — `hash_field_name` (SHA-256 truncated to 16 hex chars), matches `hashlib.sha256(name.encode()).hexdigest()[:16]` exactly. - `statistics.rs` — `is_uuid_format`, `calculate_string_entropy`, `detect_sequential_pattern` (with **BUG #2 fix** — see below). - `anchors.rs` — `extract_query_anchors`, `item_matches_anchors`. Five regex patterns ported via `std::sync::LazyLock`. - `types.rs` — `CompressionStrategy`, `FieldStats`, `CrushabilityAnalysis`, `ArrayAnalysis`, `CompressionPlan`, `CrushResult`. Field-by-field mirror of the Python @dataclasses so the PyO3 bridge in 3c.1b can reconstruct them without manual translators. # Bug #2 fixed in this commit (Python fix lands later in same PR) `smart_crusher.py:444-448` — `_detect_sequential_pattern` calls `int(string_value)` and silently strips zero-padding, so padded string IDs like `["001", "002", ..., "100"]` get misclassified as a sequential numeric pattern. Fix: track whether each parsed numeric value originated as a string. If EVERY parsed value was a string, refuse to flag as sequential. Mixed numeric+string fields still detect correctly because the unambiguous numerics dominate. Test: `bug2_zero_padded_strings_no_longer_misclassified`. # What's NOT in this commit (subsequent commits) - `SmartAnalyzer` — `analyze_array`, `_analyze_field`, `_detect_change_points`, `_detect_pattern`, `_detect_temporal_field`, `analyze_crushability`, `_select_strategy`, `_estimate_reduction`. - The five array crushers (`_crush_array`, `_crush_string_array`, `_crush_number_array`, `_crush_mixed_array`, `_crush_object`). - Planning (`_compute_k_split`, `_create_plan`, `_plan_*` family). - Orchestration (`_prioritize_indices`, `_deduplicate_indices_by_content`, `_fill_remaining_slots`). - `SmartCrusher` orchestrator class itself. - Parity harness fixtures. - The remaining 3 Python bug fixes (#1, #3, #4) — landed alongside the code paths they affect. # Build / test - `cargo build -p headroom-core` — clean. - `cargo clippy -p headroom-core -- -D warnings` — clean. - 55 new unit tests across the 6 new files, all passing. Architectural improvements (lossless-first, unified saliency score, structured CCR markers) are deferred to Stage 3c.2 — see design doc at `~/Desktop/SmartCrusher-Architecture-Improvements.md`.
2026-04-26 16:45:42 -07:00
[[package]]
name = "sha2"
version = "0.10.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283"
dependencies = [
"cfg-if",
fix: B7 — CCR hardening: persistent backends + always-on tool P2-25, P2-26: CCR (Compress-Cache-Retrieve) used an in-memory store that fragmented across uvicorn workers and was wiped on restart, and the `headroom_retrieve` tool was registered/unregistered per-request based on whether the latest body happened to contain compression markers — every flip busted the prompt cache. Both are sticky side-channels: once a session has done CCR, the tool list bytes and the retrieval store must stay stable. This PR fixes both. Rust: * Split `ccr.rs` into `ccr/` with `backends/` submodule (`in_memory.rs`, `sqlite.rs`, `redis.rs` cfg-gated). * `SqliteCcrStore` (production default): WAL mode, prepared upsert, lazy TTL purge on read, persistent across worker restarts and shareable across workers on the same host via SQLite file locking. * `RedisCcrStore` (cfg-gated behind `feature = "redis"`): SETEX with startup PING smoke-test, no key-prefix collision risk, no sticky session required at the LB. * `CcrBackendConfig::{InMemory, Sqlite, Redis}` + `from_config(...)` factory — every init failure surfaces (no silent fallback per `feedback_no_silent_fallbacks.md`). * `ccr::compute_key` (BLAKE3 → first 24 hex chars) and `ccr::marker_for("HASH") -> "<<ccr:HASH>>"` centralize the hash + marker format; one definition for the live-zone dispatcher and the Python regex (`headroom/ccr/tool_injection.py:211`). * `compress_anthropic_live_zone_with_ccr` accepts `Option<&dyn CcrStore>`. When wired, every accepted compression puts the original bytes into the backend and appends `<<ccr:HASH>>` to the compressed string. The token-validation gate runs on the marker-augmented string so the `compressed_tokens >= original_tokens` rejection stays honest. Python: * `SessionCcrTracker` + `apply_session_sticky_ccr_tool` mirror the PR-A7 `SessionToolTracker` / `apply_session_sticky_memory_tools` pattern: once a session has done CCR, every subsequent request injects the recorded golden tool-definition bytes. Tool list bytes are byte-stable across turns (snapshot test pins them). * `headroom/ccr/tool_injection.py::inject_tool_definition` accepts a new `session_has_done_ccr` kwarg per the PR-B7 spec change at line 302-328. The legacy per-request path stays intact for callers that don't yet thread a session id (e.g. Google handler). * Anthropic + OpenAI handlers route their CCR tool-list updates through `apply_session_sticky_ccr_tool`, keyed off the existing `session_tracker_store.compute_session_id(...)` plumbing. Backend selection model: `CcrBackendConfig::Sqlite { path }` is the production default — single host, persistent, multi-worker safe with sticky session. `CcrBackendConfig::Redis { url }` is the multi-host scale-out option — no stickiness needed. `InMemory` is for tests and single-worker dev only. RUST_DEV.md "Multi-worker deployment — CCR fragmentation" rewritten around this matrix. Tests: * `crates/headroom-core/tests/ccr_backends.rs` — 7 tests covering SQLite round-trip, TTL purge, proxy-restart survival, cross-backend byte-equal keys, `from_config` paths, and the no-redis-feature loud-failure check (+ 2 redis tests gated behind the feature). * `crates/headroom-core/tests/live_zone_ccr.rs` — confirms `<<ccr:HASH>>` marker injection, store population, and no-marker-when-no-store invariants end-to-end. * `tests/test_ccr_tool_always_on.py` — 12 tests pinning the always-on behaviour, session/provider isolation, LRU bound, no- session-id fallback, and (per-acceptance-criterion) the byte-stable tool-definition snapshot. Per-PR-B7 plan: REALIGNMENT/04-phase-B-live-zone.md.
2026-05-02 16:49:08 -07:00
"cpufeatures 0.2.17",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"digest 0.10.7",
]
[[package]]
name = "sha2"
version = "0.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4"
dependencies = [
"cfg-if",
"cpufeatures 0.3.0",
"digest 0.11.3",
feat(rust): scaffold smart_crusher module + foundational helpers Stage 3c.1 — like-for-like Rust port of `headroom/transforms/smart_crusher.py`. This commit lays the foundation: module layout, configuration, foundational data types, and the simpler helpers (classification, hashing, anchors, basic statistics). Subsequent commits add the analyzer, crushers, plan execution, and the orchestrator. # What's in this commit `crates/headroom-core/src/transforms/smart_crusher/`: - `mod.rs` — module entry, public re-exports, port narrative. - `classifier.rs` — `classify_array` / `ArrayType` (dict/string/number/ bool/nested/mixed/empty). Direct port of `_classify_array`. - `config.rs` — `SmartCrusherConfig` with defaults pinned to Python byte-for-byte. - `hashing.rs` — `hash_field_name` (SHA-256 truncated to 16 hex chars), matches `hashlib.sha256(name.encode()).hexdigest()[:16]` exactly. - `statistics.rs` — `is_uuid_format`, `calculate_string_entropy`, `detect_sequential_pattern` (with **BUG #2 fix** — see below). - `anchors.rs` — `extract_query_anchors`, `item_matches_anchors`. Five regex patterns ported via `std::sync::LazyLock`. - `types.rs` — `CompressionStrategy`, `FieldStats`, `CrushabilityAnalysis`, `ArrayAnalysis`, `CompressionPlan`, `CrushResult`. Field-by-field mirror of the Python @dataclasses so the PyO3 bridge in 3c.1b can reconstruct them without manual translators. # Bug #2 fixed in this commit (Python fix lands later in same PR) `smart_crusher.py:444-448` — `_detect_sequential_pattern` calls `int(string_value)` and silently strips zero-padding, so padded string IDs like `["001", "002", ..., "100"]` get misclassified as a sequential numeric pattern. Fix: track whether each parsed numeric value originated as a string. If EVERY parsed value was a string, refuse to flag as sequential. Mixed numeric+string fields still detect correctly because the unambiguous numerics dominate. Test: `bug2_zero_padded_strings_no_longer_misclassified`. # What's NOT in this commit (subsequent commits) - `SmartAnalyzer` — `analyze_array`, `_analyze_field`, `_detect_change_points`, `_detect_pattern`, `_detect_temporal_field`, `analyze_crushability`, `_select_strategy`, `_estimate_reduction`. - The five array crushers (`_crush_array`, `_crush_string_array`, `_crush_number_array`, `_crush_mixed_array`, `_crush_object`). - Planning (`_compute_k_split`, `_create_plan`, `_plan_*` family). - Orchestration (`_prioritize_indices`, `_deduplicate_indices_by_content`, `_fill_remaining_slots`). - `SmartCrusher` orchestrator class itself. - Parity harness fixtures. - The remaining 3 Python bug fixes (#1, #3, #4) — landed alongside the code paths they affect. # Build / test - `cargo build -p headroom-core` — clean. - `cargo clippy -p headroom-core -- -D warnings` — clean. - 55 new unit tests across the 6 new files, all passing. Architectural improvements (lossless-first, unified saliency score, structured CCR markers) are deferred to Stage 3c.2 — see design doc at `~/Desktop/SmartCrusher-Architecture-Improvements.md`.
2026-04-26 16:45:42 -07:00
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "sharded-slab"
version = "0.1.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f40ca3c46823713e0d4209592e8d6e826aa57e928f09752619fc696c499637f6"
dependencies = [
"lazy_static",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "shlex"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "2.0.1"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "signal-hook-registry"
version = "1.4.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c4db69cba1110affc0e9f7bcd48bbf87b3f4fc7c61fc9155afd4c469eb3d6c1b"
dependencies = [
"errno",
"libc",
]
feat(rust): HfTokenizer::from_pretrained — HuggingFace Hub auto-download Stage 2.1: closes the loop on the HuggingFace tokenizer story. Stage 2 shipped `HfTokenizer::from_bytes`/`from_file`, which required callers to manage their own tokenizer.json files. This adds the third constructor: let t = HfTokenizer::from_pretrained("CohereForAI/c4ai-command-r-v01")?; register_hf("command-", t); `from_pretrained` is a thin wrapper around the `hf-hub` crate's blocking `ureq` API. First call downloads `tokenizer.json` to `~/.cache/huggingface/ hub` (or `$HF_HOME` if set); subsequent calls reuse the on-disk cache. Uses the `main` revision; gated repos (Llama, Mistral) require `HF_TOKEN` in env or `~/.cache/huggingface/token`. Also adds `try_register_hf(prefix, repo)` as the obvious one-liner for proxy startup code: let _ = try_register_hf("command-", "CohereForAI/c4ai-command-r-v01"); let _ = try_register_hf("mistral-", "mistralai/Mistral-7B-v0.1"); Each call is independent — a download failure for one model (e.g. gated without a token) does not affect others. `HfTokenizerError` gains a new `Hub` variant so callers can distinguish "couldn't fetch" from "fetched but malformed" — relevant when deciding whether to retry, surface to the user, or fall back to the estimator. Why blocking, not async: `from_pretrained` is called once at startup. A sync API works from `main()`, from a `OnceLock` initializer, or from `tokio::task::spawn_blocking` if a tokio caller needs it later. The async hf-hub backend would force callers to await at startup, which doesn't fit the `register_hf` registry pattern. Why rustls, not native-tls: keeps the binary statically linkable for AWS deploys (no system OpenSSL dependency). Tests: a network-dependent integration test (`#[ignore]`d in CI; hits HF for `gpt2`, ~1.4 MB) verifies the real download + load + count path. A non-network negative test verifies that an invalid repo name surfaces as `HfTokenizerError::Hub`, not a panic. 44 unit tests + 5 proptests + 1 doctest pass; parity stays 40/40 byte-equal.
2026-04-25 15:05:09 -07:00
[[package]]
name = "simd-adler32"
version = "0.3.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "703d5c7ef118737c72f1af64ad2f6f8c5e1921f818cdcb97b8fe6fc69bf66214"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "simd_helpers"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "95890f873bec569a0362c235787f3aca6e1e887302ba4840839bcc6459c42da6"
dependencies = [
"quote",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "slab"
version = "0.4.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5"
[[package]]
name = "smallvec"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.15.2"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "socket2"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.6.4"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "52d1cfed4120b4d927bf7c0f86d2087a4a7d6027c906d9f9d525a80573b9be51"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"libc",
"windows-sys 0.61.2",
]
feat(rust): HfTokenizer::from_pretrained — HuggingFace Hub auto-download Stage 2.1: closes the loop on the HuggingFace tokenizer story. Stage 2 shipped `HfTokenizer::from_bytes`/`from_file`, which required callers to manage their own tokenizer.json files. This adds the third constructor: let t = HfTokenizer::from_pretrained("CohereForAI/c4ai-command-r-v01")?; register_hf("command-", t); `from_pretrained` is a thin wrapper around the `hf-hub` crate's blocking `ureq` API. First call downloads `tokenizer.json` to `~/.cache/huggingface/ hub` (or `$HF_HOME` if set); subsequent calls reuse the on-disk cache. Uses the `main` revision; gated repos (Llama, Mistral) require `HF_TOKEN` in env or `~/.cache/huggingface/token`. Also adds `try_register_hf(prefix, repo)` as the obvious one-liner for proxy startup code: let _ = try_register_hf("command-", "CohereForAI/c4ai-command-r-v01"); let _ = try_register_hf("mistral-", "mistralai/Mistral-7B-v0.1"); Each call is independent — a download failure for one model (e.g. gated without a token) does not affect others. `HfTokenizerError` gains a new `Hub` variant so callers can distinguish "couldn't fetch" from "fetched but malformed" — relevant when deciding whether to retry, surface to the user, or fall back to the estimator. Why blocking, not async: `from_pretrained` is called once at startup. A sync API works from `main()`, from a `OnceLock` initializer, or from `tokio::task::spawn_blocking` if a tokio caller needs it later. The async hf-hub backend would force callers to await at startup, which doesn't fit the `register_hf` registry pattern. Why rustls, not native-tls: keeps the binary statically linkable for AWS deploys (no system OpenSSL dependency). Tests: a network-dependent integration test (`#[ignore]`d in CI; hits HF for `gpt2`, ~1.4 MB) verifies the real download + load + count path. A non-network negative test verifies that an invalid repo name surfaces as `HfTokenizerError::Hub`, not a panic. 44 unit tests + 5 proptests + 1 doctest pass; parity stays 40/40 byte-equal.
2026-04-25 15:05:09 -07:00
[[package]]
name = "socks"
version = "0.3.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f0c3dbbd9ae980613c6dd8e28a9407b50509d3803b57624d5dfe8315218cd58b"
dependencies = [
"byteorder",
"libc",
"winapi",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "spm_precompiled"
version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5851699c4033c63636f7ea4cf7b7c1f1bf06d0cc03cfb42e711de5a5c46cf326"
dependencies = [
"base64 0.13.1",
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
"nom 7.1.3",
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
"serde",
"unicode-segmentation",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "stable_deref_trait"
version = "1.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "static_assertions"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a2eb9349b6444b326872e140eb1cf5e7c522154d69e7a0ffb0fb81c06b37543f"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "strsim"
version = "0.11.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f"
[[package]]
name = "subtle"
version = "2.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292"
[[package]]
name = "syn"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "2.0.118"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "1b9ae57f904213ebb649ce6895b8a66c66f0203b9319718f69a5612a065b1422"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "sync_wrapper"
version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263"
dependencies = [
"futures-core",
]
[[package]]
name = "synstructure"
version = "0.13.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "target-lexicon"
ci: bump pyo3 from 0.22.6 to 0.24.1 in the cargo group across 1 directory (#270) Bumps the cargo group with 1 update in the / directory: [pyo3](https://github.com/pyo3/pyo3). Updates `pyo3` from 0.22.6 to 0.24.1 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pyo3/pyo3/releases">pyo3's releases</a>.</em></p> <blockquote> <h2>PyO3 0.24.1</h2> <p>This release is a security fix for the <code>PyString::from_object</code> method, which passed <code>&amp;str</code> data to the Python C API without checking for a terminating nul byte. All historical PyO3 versions are affected, and we recommend you upgrade if you are using <code>PyString::from_object</code>. Thank you to <a href="https://github.com/vthib"><code>@​vthib</code></a> for the report and <a href="https://github.com/Dr-Emann"><code>@​Dr-Emann</code></a> for the fix. A RUSTSEC advisory will be published shortly.</p> <p>Aside from the security fix, this release contains a number of other non-breaking additions:</p> <ul> <li>An <code>abi3-py313</code> feature to support compiling with the Python 3.13 stable ABI.</li> <li><code>PyAnyMethods::getattr_opt</code> to get optional attributes without paying the cost of a Python exception when the attribute in question does not exist.</li> <li>Constructor for <code>PyInt::new</code>.</li> <li><code>with_critical_section2</code> for locking two objects at the same time on the free-threaded build.</li> <li>Fix for a PyO3 0.24.0 regression with <code>Option&lt;&amp;str&gt;</code> and <code>Option&lt;&amp;T&gt;</code> (where <code>T: PyClass</code>) function arguments no longer being permitted</li> </ul> <p>There are also a few other small bug fixes for edge cases, mostly related to compile errors from PyO3's macro code.</p> <p>Thank you to the following contributors for the improvements:</p> <p><a href="https://github.com/bschoenmaeckers"><code>@​bschoenmaeckers</code></a> <a href="https://github.com/davidhewitt"><code>@​davidhewitt</code></a> <a href="https://github.com/Dr-Emann"><code>@​Dr-Emann</code></a> <a href="https://github.com/emmagordon"><code>@​emmagordon</code></a> <a href="https://github.com/epontan"><code>@​epontan</code></a> <a href="https://github.com/Icxolu"><code>@​Icxolu</code></a> <a href="https://github.com/IvanIsCoding"><code>@​IvanIsCoding</code></a> <a href="https://github.com/jelmer"><code>@​jelmer</code></a> <a href="https://github.com/jonaspleyer"><code>@​jonaspleyer</code></a> <a href="https://github.com/ngoldbaum"><code>@​ngoldbaum</code></a> <a href="https://github.com/Owen-CH-Leung"><code>@​Owen-CH-Leung</code></a> <a href="https://github.com/Tpt"><code>@​Tpt</code></a> <a href="https://github.com/Trolldemorted"><code>@​Trolldemorted</code></a> <a href="https://github.com/XuehaiPan"><code>@​XuehaiPan</code></a></p> <h2>PyO3 0.24.0</h2> <p>This release is an incremental improvement of refinements and optimizations following the new APIs established in PyO3's last few releases.</p> <p>Support for <code>jiff</code> datetime conversions have been added, and also UUID conversions.</p> <p>The <code>FromPyObject</code> derive macro has gained new <code>#[pyo3(default = ...)]</code> and <code>#[pyo3(rename_all = ...)]</code> options, and the <code>IntoPyObject</code> derive macro has gained a new <code>#[pyo3(into_py_with = ...)]</code> option.</p> <p>PyO3 will now pass positional arguments to Python functions using the &quot;vectorcall&quot; protocol in many cases, which should be an optimization over the previous behaviour (of creating a Python tuple of positional arguments).</p> <p>Many methods on iterators of Python collections have been optimized.</p> <p>There are also many other incremental improvements, bug fixes and smaller features.</p> <p>Thank you to everyone who contributed code, documentation, design ideas, bug reports, and feedback. The following contributors' commits are included in this release:</p> <p><a href="https://github.com/0x676e67"><code>@​0x676e67</code></a> <a href="https://github.com/alex"><code>@​alex</code></a> <a href="https://github.com/arielb1"><code>@​arielb1</code></a> <a href="https://github.com/bschoenmaeckers"><code>@​bschoenmaeckers</code></a> <a href="https://github.com/davidhewitt"><code>@​davidhewitt</code></a></p> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/PyO3/pyo3/blob/main/CHANGELOG.md">pyo3's changelog</a>.</em></p> <blockquote> <h2>[0.24.1] - 2025-03-31</h2> <h3>Added</h3> <ul> <li>Add <code>abi3-py313</code> feature. <a href="https://redirect.github.com/PyO3/pyo3/pull/4969">#4969</a></li> <li>Add <code>PyAnyMethods::getattr_opt</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4978">#4978</a></li> <li>Add <code>PyInt::new</code> constructor for all supported number types (i32, u32, i64, u64, isize, usize). <a href="https://redirect.github.com/PyO3/pyo3/pull/4984">#4984</a></li> <li>Add <code>pyo3::sync::with_critical_section2</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4992">#4992</a></li> <li>Implement <code>PyCallArgs</code> for <code>Borrowed&lt;'_, 'py, PyTuple&gt;</code>, <code>&amp;Bound&lt;'py, PyTuple&gt;</code>, and <code>&amp;Py&lt;PyTuple&gt;</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/5013">#5013</a></li> </ul> <h3>Fixed</h3> <ul> <li>Fix <code>is_type_of</code> for native types not using same specialized check as <code>is_type_of_bound</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4981">#4981</a></li> <li>Fix <code>Probe</code> class naming issue with <code>#[pymethods]</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4988">#4988</a></li> <li>Fix compile failure with required <code>#[pyfunction]</code> arguments taking <code>Option&lt;&amp;str&gt;</code> and <code>Option&lt;&amp;T&gt;</code> (for <code>#[pyclass]</code> types). <a href="https://redirect.github.com/PyO3/pyo3/pull/5002">#5002</a></li> <li>Fix <code>PyString::from_object</code> causing of bounds reads with <code>encoding</code> and <code>errors</code> parameters which are not nul-terminated. <a href="https://redirect.github.com/PyO3/pyo3/pull/5008">#5008</a></li> <li>Fix compile error when additional options follow after <code>crate</code> for <code>#[pyfunction]</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/5015">#5015</a></li> </ul> <h2>[0.24.0] - 2025-03-09</h2> <h3>Packaging</h3> <ul> <li>Add supported CPython/PyPy versions to cargo package metadata. <a href="https://redirect.github.com/PyO3/pyo3/pull/4756">#4756</a></li> <li>Bump <code>target-lexicon</code> dependency to 0.13. <a href="https://redirect.github.com/PyO3/pyo3/pull/4822">#4822</a></li> <li>Add optional <code>jiff</code> dependency to add conversions for <code>jiff</code> datetime types. <a href="https://redirect.github.com/PyO3/pyo3/pull/4823">#4823</a></li> <li>Add optional <code>uuid</code> dependency to add conversions for <code>uuid::Uuid</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4864">#4864</a></li> <li>Bump minimum supported <code>inventory</code> version to 0.3.5. <a href="https://redirect.github.com/PyO3/pyo3/pull/4954">#4954</a></li> </ul> <h3>Added</h3> <ul> <li>Add <code>PyIterator::send</code> method to allow sending values into a python generator. <a href="https://redirect.github.com/PyO3/pyo3/pull/4746">#4746</a></li> <li>Add <code>PyCallArgs</code> trait for passing arguments into the Python calling protocol. This enabled using a faster calling convention for certain types, improving performance. <a href="https://redirect.github.com/PyO3/pyo3/pull/4768">#4768</a></li> <li>Add <code>#[pyo3(default = ...']</code> option for <code>#[derive(FromPyObject)]</code> to set a default value for extracted fields of named structs. <a href="https://redirect.github.com/PyO3/pyo3/pull/4829">#4829</a></li> <li>Add <code>#[pyo3(into_py_with = ...)]</code> option for <code>#[derive(IntoPyObject, IntoPyObjectRef)]</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4850">#4850</a></li> <li>Add FFI definitions <code>PyThreadState_GetFrame</code> and <code>PyFrame_GetBack</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4866">#4866</a></li> <li>Optimize <code>last</code> for <code>BoundListIterator</code>, <code>BoundTupleIterator</code> and <code>BorrowedTupleIterator</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4878">#4878</a></li> <li>Optimize <code>Iterator::count()</code> for <code>PyDict</code>, <code>PyList</code>, <code>PyTuple</code> &amp; <code>PySet</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4878">#4878</a></li> <li>Optimize <code>nth</code>, <code>nth_back</code>, <code>advance_by</code> and <code>advance_back_by</code> for <code>BoundTupleIterator</code> <a href="https://redirect.github.com/PyO3/pyo3/pull/4897">#4897</a></li> <li>Add support for <code>types.GenericAlias</code> as <code>pyo3::types::PyGenericAlias</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4917">#4917</a></li> <li>Add <code>MutextExt</code> trait to help avoid deadlocks with the GIL while locking a <code>std::sync::Mutex</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4934">#4934</a></li> <li>Add <code>#[pyo3(rename_all = &quot;...&quot;)]</code> option for <code>#[derive(FromPyObject)]</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4941">#4941</a></li> </ul> <h3>Changed</h3> <ul> <li>Optimize <code>nth</code>, <code>nth_back</code>, <code>advance_by</code> and <code>advance_back_by</code> for <code>BoundListIterator</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4810">#4810</a></li> <li>Use <code>DerefToPyAny</code> in blanket implementations of <code>From&lt;Py&lt;T&gt;&gt;</code> and <code>From&lt;Bound&lt;'py, T&gt;&gt;</code> for <code>PyObject</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4593">#4593</a></li> <li>Map <code>io::ErrorKind::IsADirectory</code>/<code>NotADirectory</code> to the corresponding Python exception on Rust 1.83+. <a href="https://redirect.github.com/PyO3/pyo3/pull/4747">#4747</a></li> <li><code>PyAnyMethods::call</code> and friends now require <code>PyCallArgs</code> for their positional arguments. <a href="https://redirect.github.com/PyO3/pyo3/pull/4768">#4768</a></li> <li>Expose FFI definitions for <code>PyObject_Vectorcall(Method)</code> on the stable abi on 3.12+. <a href="https://redirect.github.com/PyO3/pyo3/pull/4853">#4853</a></li> <li><code>#[pyo3(from_py_with = ...)]</code> now take a path rather than a string literal <a href="https://redirect.github.com/PyO3/pyo3/pull/4860">#4860</a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/PyO3/pyo3/commit/a213b368bd5bf859c2acb655bfed029e17c3b447"><code>a213b36</code></a> release: 0.24.1 (<a href="https://redirect.github.com/pyo3/pyo3/issues/5021">#5021</a>)</li> <li><a href="https://github.com/PyO3/pyo3/commit/d85a02d9b11f7c057e3627a0393d5d9b876dbc0a"><code>d85a02d</code></a> split <code>PyFunctionArgument</code> to specialize <code>Option</code> (<a href="https://redirect.github.com/pyo3/pyo3/issues/5002">#5002</a>)</li> <li><a href="https://github.com/PyO3/pyo3/commit/c37a50a7a33e145f6bb87f40cb89cf85f9e5fac7"><code>c37a50a</code></a> Add example of more complex exceptions (<a href="https://redirect.github.com/pyo3/pyo3/issues/5014">#5014</a>)</li> <li><a href="https://github.com/PyO3/pyo3/commit/dcacb9bbbc8c130238bd88480fc53074e445b4fc"><code>dcacb9b</code></a> Simplify PyFunctionArgument impl on &amp;Bound&lt;T&gt; (<a href="https://redirect.github.com/pyo3/pyo3/issues/5018">#5018</a>)</li> <li><a href="https://github.com/PyO3/pyo3/commit/03c31c5c7affdd8805957b5944bd8ca05d1bdec8"><code>03c31c5</code></a> fix <code>#[pyfunction]</code> option parsing (<a href="https://redirect.github.com/pyo3/pyo3/issues/5015">#5015</a>)</li> <li><a href="https://github.com/PyO3/pyo3/commit/0f49eb14b0358a8fe85c5930db84c5c404f97dd7"><code>0f49eb1</code></a> docs: Remove examples with outdated PyO3 and unmaintained projects (<a href="https://redirect.github.com/pyo3/pyo3/issues/4952">#4952</a>)</li> <li><a href="https://github.com/PyO3/pyo3/commit/1b00b0d27f1b49d4b4237bc616d99016b06c1bd8"><code>1b00b0d</code></a> implement <code>PyCallArgs</code> for borrowed types (<a href="https://redirect.github.com/pyo3/pyo3/issues/5013">#5013</a>)</li> <li><a href="https://github.com/PyO3/pyo3/commit/5caaa371dce8fe8a93c64d7a465c3c2c80ce6e2f"><code>5caaa37</code></a> fix: convert to cstrings in PyString::from_object (<a href="https://redirect.github.com/pyo3/pyo3/issues/5008">#5008</a>)</li> <li><a href="https://github.com/PyO3/pyo3/commit/4aca459fd30441fa006c3eb388c812047f5465ce"><code>4aca459</code></a> docs: guide - add link to tables and traits (<a href="https://redirect.github.com/pyo3/pyo3/issues/5001">#5001</a>)</li> <li><a href="https://github.com/PyO3/pyo3/commit/0452c0ee5299a1af42f9d966ba3d136a79edb15d"><code>0452c0e</code></a> replace quansight-labs/setup-python with actions/setup-python (<a href="https://redirect.github.com/pyo3/pyo3/issues/5007">#5007</a>)</li> <li>Additional commits viewable in <a href="https://github.com/pyo3/pyo3/compare/v0.22.6...v0.24.1">compare view</a></li> </ul> </details> <br /> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-06-10 23:01:33 -05:00
version = "0.13.5"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
ci: bump pyo3 from 0.22.6 to 0.24.1 in the cargo group across 1 directory (#270) Bumps the cargo group with 1 update in the / directory: [pyo3](https://github.com/pyo3/pyo3). Updates `pyo3` from 0.22.6 to 0.24.1 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pyo3/pyo3/releases">pyo3's releases</a>.</em></p> <blockquote> <h2>PyO3 0.24.1</h2> <p>This release is a security fix for the <code>PyString::from_object</code> method, which passed <code>&amp;str</code> data to the Python C API without checking for a terminating nul byte. All historical PyO3 versions are affected, and we recommend you upgrade if you are using <code>PyString::from_object</code>. Thank you to <a href="https://github.com/vthib"><code>@​vthib</code></a> for the report and <a href="https://github.com/Dr-Emann"><code>@​Dr-Emann</code></a> for the fix. A RUSTSEC advisory will be published shortly.</p> <p>Aside from the security fix, this release contains a number of other non-breaking additions:</p> <ul> <li>An <code>abi3-py313</code> feature to support compiling with the Python 3.13 stable ABI.</li> <li><code>PyAnyMethods::getattr_opt</code> to get optional attributes without paying the cost of a Python exception when the attribute in question does not exist.</li> <li>Constructor for <code>PyInt::new</code>.</li> <li><code>with_critical_section2</code> for locking two objects at the same time on the free-threaded build.</li> <li>Fix for a PyO3 0.24.0 regression with <code>Option&lt;&amp;str&gt;</code> and <code>Option&lt;&amp;T&gt;</code> (where <code>T: PyClass</code>) function arguments no longer being permitted</li> </ul> <p>There are also a few other small bug fixes for edge cases, mostly related to compile errors from PyO3's macro code.</p> <p>Thank you to the following contributors for the improvements:</p> <p><a href="https://github.com/bschoenmaeckers"><code>@​bschoenmaeckers</code></a> <a href="https://github.com/davidhewitt"><code>@​davidhewitt</code></a> <a href="https://github.com/Dr-Emann"><code>@​Dr-Emann</code></a> <a href="https://github.com/emmagordon"><code>@​emmagordon</code></a> <a href="https://github.com/epontan"><code>@​epontan</code></a> <a href="https://github.com/Icxolu"><code>@​Icxolu</code></a> <a href="https://github.com/IvanIsCoding"><code>@​IvanIsCoding</code></a> <a href="https://github.com/jelmer"><code>@​jelmer</code></a> <a href="https://github.com/jonaspleyer"><code>@​jonaspleyer</code></a> <a href="https://github.com/ngoldbaum"><code>@​ngoldbaum</code></a> <a href="https://github.com/Owen-CH-Leung"><code>@​Owen-CH-Leung</code></a> <a href="https://github.com/Tpt"><code>@​Tpt</code></a> <a href="https://github.com/Trolldemorted"><code>@​Trolldemorted</code></a> <a href="https://github.com/XuehaiPan"><code>@​XuehaiPan</code></a></p> <h2>PyO3 0.24.0</h2> <p>This release is an incremental improvement of refinements and optimizations following the new APIs established in PyO3's last few releases.</p> <p>Support for <code>jiff</code> datetime conversions have been added, and also UUID conversions.</p> <p>The <code>FromPyObject</code> derive macro has gained new <code>#[pyo3(default = ...)]</code> and <code>#[pyo3(rename_all = ...)]</code> options, and the <code>IntoPyObject</code> derive macro has gained a new <code>#[pyo3(into_py_with = ...)]</code> option.</p> <p>PyO3 will now pass positional arguments to Python functions using the &quot;vectorcall&quot; protocol in many cases, which should be an optimization over the previous behaviour (of creating a Python tuple of positional arguments).</p> <p>Many methods on iterators of Python collections have been optimized.</p> <p>There are also many other incremental improvements, bug fixes and smaller features.</p> <p>Thank you to everyone who contributed code, documentation, design ideas, bug reports, and feedback. The following contributors' commits are included in this release:</p> <p><a href="https://github.com/0x676e67"><code>@​0x676e67</code></a> <a href="https://github.com/alex"><code>@​alex</code></a> <a href="https://github.com/arielb1"><code>@​arielb1</code></a> <a href="https://github.com/bschoenmaeckers"><code>@​bschoenmaeckers</code></a> <a href="https://github.com/davidhewitt"><code>@​davidhewitt</code></a></p> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/PyO3/pyo3/blob/main/CHANGELOG.md">pyo3's changelog</a>.</em></p> <blockquote> <h2>[0.24.1] - 2025-03-31</h2> <h3>Added</h3> <ul> <li>Add <code>abi3-py313</code> feature. <a href="https://redirect.github.com/PyO3/pyo3/pull/4969">#4969</a></li> <li>Add <code>PyAnyMethods::getattr_opt</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4978">#4978</a></li> <li>Add <code>PyInt::new</code> constructor for all supported number types (i32, u32, i64, u64, isize, usize). <a href="https://redirect.github.com/PyO3/pyo3/pull/4984">#4984</a></li> <li>Add <code>pyo3::sync::with_critical_section2</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4992">#4992</a></li> <li>Implement <code>PyCallArgs</code> for <code>Borrowed&lt;'_, 'py, PyTuple&gt;</code>, <code>&amp;Bound&lt;'py, PyTuple&gt;</code>, and <code>&amp;Py&lt;PyTuple&gt;</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/5013">#5013</a></li> </ul> <h3>Fixed</h3> <ul> <li>Fix <code>is_type_of</code> for native types not using same specialized check as <code>is_type_of_bound</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4981">#4981</a></li> <li>Fix <code>Probe</code> class naming issue with <code>#[pymethods]</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4988">#4988</a></li> <li>Fix compile failure with required <code>#[pyfunction]</code> arguments taking <code>Option&lt;&amp;str&gt;</code> and <code>Option&lt;&amp;T&gt;</code> (for <code>#[pyclass]</code> types). <a href="https://redirect.github.com/PyO3/pyo3/pull/5002">#5002</a></li> <li>Fix <code>PyString::from_object</code> causing of bounds reads with <code>encoding</code> and <code>errors</code> parameters which are not nul-terminated. <a href="https://redirect.github.com/PyO3/pyo3/pull/5008">#5008</a></li> <li>Fix compile error when additional options follow after <code>crate</code> for <code>#[pyfunction]</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/5015">#5015</a></li> </ul> <h2>[0.24.0] - 2025-03-09</h2> <h3>Packaging</h3> <ul> <li>Add supported CPython/PyPy versions to cargo package metadata. <a href="https://redirect.github.com/PyO3/pyo3/pull/4756">#4756</a></li> <li>Bump <code>target-lexicon</code> dependency to 0.13. <a href="https://redirect.github.com/PyO3/pyo3/pull/4822">#4822</a></li> <li>Add optional <code>jiff</code> dependency to add conversions for <code>jiff</code> datetime types. <a href="https://redirect.github.com/PyO3/pyo3/pull/4823">#4823</a></li> <li>Add optional <code>uuid</code> dependency to add conversions for <code>uuid::Uuid</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4864">#4864</a></li> <li>Bump minimum supported <code>inventory</code> version to 0.3.5. <a href="https://redirect.github.com/PyO3/pyo3/pull/4954">#4954</a></li> </ul> <h3>Added</h3> <ul> <li>Add <code>PyIterator::send</code> method to allow sending values into a python generator. <a href="https://redirect.github.com/PyO3/pyo3/pull/4746">#4746</a></li> <li>Add <code>PyCallArgs</code> trait for passing arguments into the Python calling protocol. This enabled using a faster calling convention for certain types, improving performance. <a href="https://redirect.github.com/PyO3/pyo3/pull/4768">#4768</a></li> <li>Add <code>#[pyo3(default = ...']</code> option for <code>#[derive(FromPyObject)]</code> to set a default value for extracted fields of named structs. <a href="https://redirect.github.com/PyO3/pyo3/pull/4829">#4829</a></li> <li>Add <code>#[pyo3(into_py_with = ...)]</code> option for <code>#[derive(IntoPyObject, IntoPyObjectRef)]</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4850">#4850</a></li> <li>Add FFI definitions <code>PyThreadState_GetFrame</code> and <code>PyFrame_GetBack</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4866">#4866</a></li> <li>Optimize <code>last</code> for <code>BoundListIterator</code>, <code>BoundTupleIterator</code> and <code>BorrowedTupleIterator</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4878">#4878</a></li> <li>Optimize <code>Iterator::count()</code> for <code>PyDict</code>, <code>PyList</code>, <code>PyTuple</code> &amp; <code>PySet</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4878">#4878</a></li> <li>Optimize <code>nth</code>, <code>nth_back</code>, <code>advance_by</code> and <code>advance_back_by</code> for <code>BoundTupleIterator</code> <a href="https://redirect.github.com/PyO3/pyo3/pull/4897">#4897</a></li> <li>Add support for <code>types.GenericAlias</code> as <code>pyo3::types::PyGenericAlias</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4917">#4917</a></li> <li>Add <code>MutextExt</code> trait to help avoid deadlocks with the GIL while locking a <code>std::sync::Mutex</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4934">#4934</a></li> <li>Add <code>#[pyo3(rename_all = &quot;...&quot;)]</code> option for <code>#[derive(FromPyObject)]</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4941">#4941</a></li> </ul> <h3>Changed</h3> <ul> <li>Optimize <code>nth</code>, <code>nth_back</code>, <code>advance_by</code> and <code>advance_back_by</code> for <code>BoundListIterator</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4810">#4810</a></li> <li>Use <code>DerefToPyAny</code> in blanket implementations of <code>From&lt;Py&lt;T&gt;&gt;</code> and <code>From&lt;Bound&lt;'py, T&gt;&gt;</code> for <code>PyObject</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4593">#4593</a></li> <li>Map <code>io::ErrorKind::IsADirectory</code>/<code>NotADirectory</code> to the corresponding Python exception on Rust 1.83+. <a href="https://redirect.github.com/PyO3/pyo3/pull/4747">#4747</a></li> <li><code>PyAnyMethods::call</code> and friends now require <code>PyCallArgs</code> for their positional arguments. <a href="https://redirect.github.com/PyO3/pyo3/pull/4768">#4768</a></li> <li>Expose FFI definitions for <code>PyObject_Vectorcall(Method)</code> on the stable abi on 3.12+. <a href="https://redirect.github.com/PyO3/pyo3/pull/4853">#4853</a></li> <li><code>#[pyo3(from_py_with = ...)]</code> now take a path rather than a string literal <a href="https://redirect.github.com/PyO3/pyo3/pull/4860">#4860</a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/PyO3/pyo3/commit/a213b368bd5bf859c2acb655bfed029e17c3b447"><code>a213b36</code></a> release: 0.24.1 (<a href="https://redirect.github.com/pyo3/pyo3/issues/5021">#5021</a>)</li> <li><a href="https://github.com/PyO3/pyo3/commit/d85a02d9b11f7c057e3627a0393d5d9b876dbc0a"><code>d85a02d</code></a> split <code>PyFunctionArgument</code> to specialize <code>Option</code> (<a href="https://redirect.github.com/pyo3/pyo3/issues/5002">#5002</a>)</li> <li><a href="https://github.com/PyO3/pyo3/commit/c37a50a7a33e145f6bb87f40cb89cf85f9e5fac7"><code>c37a50a</code></a> Add example of more complex exceptions (<a href="https://redirect.github.com/pyo3/pyo3/issues/5014">#5014</a>)</li> <li><a href="https://github.com/PyO3/pyo3/commit/dcacb9bbbc8c130238bd88480fc53074e445b4fc"><code>dcacb9b</code></a> Simplify PyFunctionArgument impl on &amp;Bound&lt;T&gt; (<a href="https://redirect.github.com/pyo3/pyo3/issues/5018">#5018</a>)</li> <li><a href="https://github.com/PyO3/pyo3/commit/03c31c5c7affdd8805957b5944bd8ca05d1bdec8"><code>03c31c5</code></a> fix <code>#[pyfunction]</code> option parsing (<a href="https://redirect.github.com/pyo3/pyo3/issues/5015">#5015</a>)</li> <li><a href="https://github.com/PyO3/pyo3/commit/0f49eb14b0358a8fe85c5930db84c5c404f97dd7"><code>0f49eb1</code></a> docs: Remove examples with outdated PyO3 and unmaintained projects (<a href="https://redirect.github.com/pyo3/pyo3/issues/4952">#4952</a>)</li> <li><a href="https://github.com/PyO3/pyo3/commit/1b00b0d27f1b49d4b4237bc616d99016b06c1bd8"><code>1b00b0d</code></a> implement <code>PyCallArgs</code> for borrowed types (<a href="https://redirect.github.com/pyo3/pyo3/issues/5013">#5013</a>)</li> <li><a href="https://github.com/PyO3/pyo3/commit/5caaa371dce8fe8a93c64d7a465c3c2c80ce6e2f"><code>5caaa37</code></a> fix: convert to cstrings in PyString::from_object (<a href="https://redirect.github.com/pyo3/pyo3/issues/5008">#5008</a>)</li> <li><a href="https://github.com/PyO3/pyo3/commit/4aca459fd30441fa006c3eb388c812047f5465ce"><code>4aca459</code></a> docs: guide - add link to tables and traits (<a href="https://redirect.github.com/pyo3/pyo3/issues/5001">#5001</a>)</li> <li><a href="https://github.com/PyO3/pyo3/commit/0452c0ee5299a1af42f9d966ba3d136a79edb15d"><code>0452c0e</code></a> replace quansight-labs/setup-python with actions/setup-python (<a href="https://redirect.github.com/pyo3/pyo3/issues/5007">#5007</a>)</li> <li>Additional commits viewable in <a href="https://github.com/pyo3/pyo3/compare/v0.22.6...v0.24.1">compare view</a></li> </ul> </details> <br /> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-06-10 23:01:33 -05:00
checksum = "adb6935a6f5c20170eeceb1a3835a49e12e19d792f6dd344ccc76a985ca5a6ca"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "tempfile"
version = "3.27.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
dependencies = [
"fastrand",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"getrandom 0.4.3",
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
"once_cell",
"rustix",
"windows-sys 0.61.2",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "thiserror"
version = "1.0.69"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b6aaf5339b578ea85b50e080feb250a3e8ae8cfcdff9a461c9ec2904bc923f52"
dependencies = [
"thiserror-impl 1.0.69",
]
[[package]]
name = "thiserror"
version = "2.0.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4"
dependencies = [
"thiserror-impl 2.0.18",
]
[[package]]
name = "thiserror-impl"
version = "1.0.69"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "thiserror-impl"
version = "2.0.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "thread_local"
version = "1.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f60246a4944f24f6e018aa17cdeffb7818b76356965d03b07d6a9886e8962185"
dependencies = [
"cfg-if",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "tiff"
version = "0.11.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b63feaf3343d35b6ca4d50483f94843803b0f51634937cc2ec519fc32232bc52"
dependencies = [
"fax",
"flate2",
"half",
"quick-error 2.0.1",
"weezl",
"zune-jpeg",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "tiktoken-rs"
version = "0.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fac4a168cfc1d8ed65bf17a6ee0843ad9a68f863c63c0fb2fa7eab67838782ee"
dependencies = [
"anyhow",
"base64 0.22.1",
"bstr",
"fancy-regex",
"lazy_static",
"regex",
"rustc-hash 1.1.0",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "time"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.3.51"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "85c17d80feb7334b40c484e45ed1a5273dfd8bfda537c3be2e74a06a6686f327"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
dependencies = [
"deranged",
"num-conv",
"powerfmt",
"serde_core",
"time-core",
"time-macros",
]
[[package]]
name = "time-core"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.1.9"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "9e1c906769ad99c88eaa54e728060edef082f8e358ff32030cb7c7d315e81109"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "time-macros"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.2.30"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "dcef1a61bdb119096e153208ec5cbec23944ce8bca13be5c7f60c634f7403935"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
dependencies = [
"num-conv",
"time-core",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "tinystr"
version = "0.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c8323304221c2a851516f22236c5722a72eaa19749016521d6dff0824447d96d"
dependencies = [
"displaydoc",
"zerovec",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "tinytemplate"
version = "1.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "be4d6b5f19ff7664e8c98d03e2139cb510db9b0a60b55f8e8709b689d939b6bc"
dependencies = [
"serde",
"serde_json",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "tinyvec"
version = "1.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3e61e67053d25a4e82c844e8424039d9745781b3fc4f32b8d55ed50f5f667ef3"
dependencies = [
"tinyvec_macros",
]
[[package]]
name = "tinyvec_macros"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "tokenizers"
version = "0.22.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b238e22d44a15349529690fb07bd645cf58149a1b1e44d6cb5bd1641ff1a6223"
dependencies = [
"ahash",
"aho-corasick",
"compact_str",
"dary_heap",
"derive_builder",
"esaxx-rs",
"getrandom 0.3.4",
fix(rust): audit cleanup — DiffCompressor CCR leak, CCR TOCTOU race, clippy debt, dep dedup Closes findings from the post-Phase-3g audit. Five surgical fixes plus telemetry-discoverability docs. PyO3 0.22 → 0.24 security upgrade is its own PR (issue #335). 1. DiffCompressor cache_key persistence (production bug) --------------------------------------------------------- Pre-fix: `RustDiffCompressor.compress()` minted a `cache_key`, embedded `[... hash=abc123]` in the wire marker, and returned without storing the original anywhere. Python ContentRouter then returned the compressed text with a dangling marker — every retrieval tool call from the LLM 404'd. Sibling compressors (LogCompressor, SearchCompressor) already had the right pattern: Rust mints the key, Python's `_persist_to_python_ccr` writes the original to the production `CompressionStore`. DiffCompressor was the asymmetric one. Fix: - Rust: add `DiffCompressor::compress_with_store(content, context, Option<&dyn CcrStore>)` mirroring siblings. Calls `store.put` when a key is minted; legacy `compress()` and `compress_with_stats()` delegate with `None` for parity. - Python: add `_persist_to_python_ccr` helper to `headroom/transforms/diff_compressor.py.compress()` mirroring `log_compressor.py` and `search_compressor.py`. - Pipeline `DiffOffload`: switch to `compress_with_store(Some(store))` and drop the post-hoc double-store hack that papered over this bug at the orchestrator boundary. 2. CCR store TOCTOU race in `get()` ----------------------------------- `InMemoryCcrStore::get()` checked TTL under a read lock, dropped the lock, then called `remove()`. Between drop and remove a concurrent `put()` of the same hash with fresh data could land — and our `remove` would then wipe that fresh entry. Under multi-worker proxy load this manifested as "I just stored it; why is it gone?" Fix: use `DashMap::remove_if`. Predicate runs under the shard write lock so check-and-remove is atomic. New regression test exercises a tight contention loop between writer and reader on the same key. 3. Pre-existing clippy debt in smart_crusher -------------------------------------------- - 3× `field_reassign_with_default` in `crusher.rs` test setup — switch to struct-update syntax `Config { field: x, ..Default }`. - `hash_array_for_ccr` was `#[cfg(test)]` but unused; deleted with a comment so a future test can reintroduce it as a one-liner. `cargo clippy --workspace --all-targets -- -D warnings` is now clean across the whole workspace; previous CI patches that allowed these warnings can be removed in a follow-up. 4. Tokenizers dependency dedup ------------------------------ `tokenizers 0.21` (direct dep) + `tokenizers 0.22` (transitive via fastembed) compiled twice into the binary. Bumped direct dep to `0.22` to align; API is compatible (verified by full tokenizer test suite). Saves compile time + binary bloat. 5. Telemetry-discoverability doc (no new code) ---------------------------------------------- The audit recommended a per-transform invocation counter to inform the next Python → Rust port. Discovered the infrastructure already exists at `/stats`: - `compressions_by_strategy` — invocation count per strategy - `pipeline_timing` — count + avg/max ms per transform name - `tokens_saved_by_strategy` — savings attribution Added a section to `RUST_DEV.md` showing the `curl + jq` recipes to read this data, with example output highlighting how to spot zero-invocation deferral candidates (e.g. `code_compressor`). Verification: workspace tests 734 + 14 + 5 + 4 + 6 + 5 + 2 + 2 + 3 + 4 + 2 + 2 + 1 = all green; cargo fmt clean; cargo clippy --all-targets clean; Python tests 185 pass; commitlint clean.
2026-04-30 20:54:22 -07:00
"indicatif 0.18.4",
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
"itertools 0.14.0",
"log",
"macro_rules_attribute",
"monostate",
"onig",
"paste",
"rand 0.9.4",
"rayon",
"rayon-cond",
"regex",
"regex-syntax",
"serde",
"serde_json",
"spm_precompiled",
"thiserror 2.0.18",
"unicode-normalization-alignments",
"unicode-segmentation",
"unicode_categories",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "tokio"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.52.3"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "8fc7f01b389ac15039e4dc9531aa973a135d7a4135281b12d7c1bc79fd57fffe"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"bytes",
"libc",
"mio",
"pin-project-lite",
"signal-hook-registry",
"socket2",
"tokio-macros",
"windows-sys 0.61.2",
]
[[package]]
name = "tokio-macros"
version = "2.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "385a6cb71ab9ab790c5fe8d67f1645e6c450a7ce006a33de03daa956cf70a496"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "tokio-rustls"
version = "0.26.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1729aa945f29d91ba541258c8df89027d5792d85a8841fb65e8bf0f4ede4ef61"
dependencies = [
"rustls",
"tokio",
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "tokio-stream"
version = "0.1.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32da49809aab5c3bc678af03902d4ccddea2a87d028d86392a4b1560c6906c70"
dependencies = [
"futures-core",
"pin-project-lite",
"tokio",
]
[[package]]
name = "tokio-tungstenite"
version = "0.24.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "edc5f74e248dc973e0dbb7b74c7e0d6fcc301c694ff50049504004ef4d0cdcd9"
dependencies = [
"futures-util",
"log",
"rustls",
"rustls-pki-types",
"tokio",
"tokio-rustls",
"tungstenite",
"webpki-roots 0.26.11",
]
[[package]]
name = "tokio-util"
version = "0.7.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9ae9cec805b01e8fc3fd2fe289f89149a9b66dd16786abd8b19cfa7b48cb0098"
dependencies = [
"bytes",
"futures-core",
"futures-sink",
"pin-project-lite",
"tokio",
]
fix(rust): reformat/offload pipeline + log templates + diff noise (Phase 3g rework) Replaces PR1's lossless/lossy split with ReformatTransform (pack denser, no info lost) and OffloadTransform (drop bytes, CCR-stash original via required cache_key). With CCR every transform is information-preserving end-to-end, so the lossless/lossy distinction misnamed the architecture. OffloadTransform carries a cheap, structural estimate_bloat() method scoped to its domain — generic byte-redundancy heuristics miss domain semantics. The orchestrator runs reformat phase + per-offload bloat estimation in parallel via rayon::join + par_iter, then runs offload iff bloat clears threshold OR reformat underwhelmed. Transforms shipped: REFORMATS (lossless): - JsonMinifier: serde_json round-trip whitespace stripping. - LogTemplate: Drain-inspired order-preserving template miner. Collapses consecutive runs of same-template lines into [Template Tn: ...] (Nx) + variant table. Win comes from emitting the constant-token prefix once instead of N times. Lossless: every original line reconstructible from template + variants. OFFLOADS (drop bytes, stash original via CCR): - LogOffload: wraps existing LogCompressor; bloat = repetition x uniqueness_weight + dilution x priority_dilution_weight. - DiffOffload: wraps existing DiffCompressor; bloat = context-to- change ratio. Bug-fix-on-port — persists original under the cache_key the parity-bound DiffCompressor mints (closes a leak). - DiffNoise: drops lockfile hunks (Cargo.lock, package-lock.json, yarn.lock, etc., suffix list configurable in TOML) and whitespace-only hunks. Stashes original via CCR for retrieval. Search offload exists but is not in default re-exports — modern agents (Claude Code, Codex) use scoped rg/grep, the marginal value didn't justify default registration. Reach via the explicit module path if opting in. JSON Offload is intentionally absent from this PR — already lives at SmartCrusher; Phase 3g PR3 wraps it in the OffloadTransform contract. Thresholds and weights live in config/pipeline.toml, embedded via include_str!; PipelineConfig::from_toml_str loads runtime overrides. 98 new pipeline tests; full headroom-core suite (714) and workspace tests green; cargo fmt clean. No regex, per project convention.
2026-04-30 11:02:29 -07:00
[[package]]
name = "toml"
version = "0.8.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dc1beb996b9d83529a9e75c17a1686767d148d70663143c7854d8b4a09ced362"
dependencies = [
"serde",
"serde_spanned",
"toml_datetime",
"toml_edit",
]
[[package]]
name = "toml_datetime"
version = "0.6.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "22cddaf88f4fbc13c51aebbf5f8eceb5c7c5a9da2ac40a13519eb5b0a0e8f11c"
dependencies = [
"serde",
]
[[package]]
name = "toml_edit"
version = "0.22.27"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "41fe8c660ae4257887cf66394862d21dbca4a6ddd26f04a3560410406a2f819a"
dependencies = [
"indexmap",
"serde",
"serde_spanned",
"toml_datetime",
"toml_write",
"winnow",
]
[[package]]
name = "toml_write"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5d99f8c9a7727884afe522e9bd5edbfc91a3312b36a77b5fb8926e4c31a41801"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "tower"
version = "0.5.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ebe5ef63511595f1344e2d5cfa636d973292adc0eec1f0ad45fae9f0851ab1d4"
dependencies = [
"futures-core",
"futures-util",
"pin-project-lite",
"sync_wrapper",
"tokio",
"tower-layer",
"tower-service",
"tracing",
]
[[package]]
name = "tower-http"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.6.11"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "4cfcf7e2740e6fc6d4d688b4ef00650406bb94adf4731e43c096c3a19fe40840"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"bitflags",
"bytes",
"futures-util",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
"http-body 1.0.1",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
"pin-project-lite",
"tower",
"tower-layer",
"tower-service",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"tracing",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"url",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"uuid",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
]
[[package]]
name = "tower-layer"
version = "0.3.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "121c2a6cda46980bb0fcd1647ffaf6cd3fc79a013de288782836f6df9c48780e"
[[package]]
name = "tower-service"
version = "0.3.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3"
[[package]]
name = "tracing"
version = "0.1.44"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100"
dependencies = [
"log",
"pin-project-lite",
"tracing-attributes",
"tracing-core",
]
[[package]]
name = "tracing-attributes"
version = "0.1.31"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7490cfa5ec963746568740651ac6781f701c9c5ea257c58e057f3ba8cf69e8da"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "tracing-core"
version = "0.1.36"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a"
dependencies = [
"once_cell",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"valuable",
]
fix(proxy): PR-D4 native Vertex publisher path + ADC bearer auth Adds a Rust-native Vertex AI publisher route ahead of the LiteLLM Python converter (which dropped `thinking`, `redacted_thinking`, `document`, `image`, `server_tool_use`, `mcp_tool_use` block kinds — the P4-37 / P4-38 bug). After this PR the Vertex `:rawPredict` and `:streamRawPredict` calls survive byte-equal upstream and benefit from the live-zone Anthropic dispatcher (PR-B-series) running over the body — same behaviour as `/v1/messages`. New module `crates/headroom-proxy/src/vertex/`: - `mod.rs` — single dispatch handler at the `/v1beta1/.../models/:model_action` route. Splits the trailing `:<verb>` segment with `str::rsplit_once(':')` (no regex) and flips an `attach_sse_tee` flag to dispatch to the streaming or non-streaming arm. Both verbs share one axum route shape because matchit can't distinguish two patterns that overlap on a parameter. - `envelope.rs` — `VertexEnvelope` parser. Confirms `anthropic_version` present + `model` field absent (the two fingerprints of the Vertex envelope vs `/v1/messages`). - `adc.rs` — `TokenSource` trait + `GcpAdcTokenSource` (production, `gcp_auth` 0.12) + `StaticTokenSource` (tests). Caches tokens with a 60s refresh-ahead-of-expiry window. Emits structured `event = "vertex_adc_token_refreshed"` per refresh. - `raw_predict.rs` — POST handler + shared `forward_vertex_request`. Buffers body, parses envelope, runs live-zone Anthropic compression, fetches ADC bearer, attaches `Authorization: Bearer <token>` (overwrites client-supplied Authorization header), forwards. SSE telemetry tee for the streaming verb reuses PR-C1's `AnthropicStreamState` directly (Vertex streams plain SSE, unlike Bedrock's binary EventStream). - `stream_raw_predict.rs` — module-level docs + alias to the shared dispatcher (the streaming-vs-non-streaming difference is one boolean flag inside the shared forwarder). Modifications: - `proxy.rs::build_app` — registers the single Vertex route. - `proxy.rs::AppState` — new `vertex_token_source: Arc<dyn TokenSource>` field. Production constructs `GcpAdcTokenSource` lazily (no GCP call until first `bearer()`); tests inject `StaticTokenSource` via the new `AppState::with_token_source` helper. - `config.rs` — adds `--vertex-region` / `HEADROOM_PROXY_VERTEX_REGION` (default `us-central1`, observability tag only — the upstream URL is `--upstream`) and `--vertex-adc-scope` / `HEADROOM_PROXY_VERTEX_ADC_SCOPE` (default `cloud-platform`). - `Cargo.toml` (workspace + proxy) — adds `gcp_auth = "0.12"` and `async-trait = "0.1"`. - `tests/common/mod.rs` — `start_proxy_with_state` accepts both config + state customizers; `install_static_token_source` helper for tests. `crates/headroom-proxy/tests/integration_vertex_raw_predict.rs` — all five tests pass: 1. `native_envelope_round_trip_byte_equal` — Vertex-shape body (with `anthropic_version`, no `model`) round-trips SHA-256 byte-equal upstream. 2. `adc_bearer_token_signed_correctly` — `Authorization: Bearer <static-test-token>` reaches upstream verbatim and OVERWRITES a client-supplied Authorization header. 3. `thinking_block_preserved` — request with `thinking` (incl. signature) + `redacted_thinking` (incl. opaque `data`) blocks round-trips byte-equal even with `LiveZone` compression mode enabled. This is the P4-37 / P4-38 teeth. 4. `stream_raw_predict_sse_handled` — `:streamRawPredict` proxies an Anthropic SSE response (full `message_start` → `content_block_delta` → `message_stop` sequence) back to the client without corruption; SSE content-type preserved end-to-end; bearer attached. 5. (bonus, no-silent-fallback contract) `adc_failure_returns_5xx_no_silent_forward` — when the token source returns `Err`, the proxy returns 5xx and never reaches upstream. Verifies the `event = "vertex_adc_fetch_failed"` error path. Workspace: `cargo test --workspace` green; `cargo clippy --workspace -- -D warnings` clean; `make ci-precheck` passes. - No silent fallbacks: ADC failure → structured 5xx, never an unauthenticated forward. - No hardcodes: every knob (region, ADC scope, upstream URL) is CLI-flag + env-var configurable. - No regexes: axum path parameters + `str::rsplit_once` only. - Comprehensive structured logs: `event` field on every decision point — `vertex_envelope_parsed`, `vertex_envelope_invalid`, `vertex_compression_skipped`, `vertex_compression_applied`, `vertex_adc_token_refreshed`, `vertex_adc_fetch_failed`, `vertex_streaming_pipeline_active`, `vertex_sse_stream_closed`, `vertex_forwarded`, `vertex_unknown_verb`, etc. - Performant: no body clone; ADC token cached + refreshed ahead-of-expiry, not fetched per request. - Comprehensive tests: realistic Anthropic block content (signature payload, redacted_thinking opaque blob) in `thinking_block_preserved`. The local `gcloud auth application-default print-access-token` returns no credentials, so manual validation against a real Vertex endpoint is not possible in this PR. Follow-up: the user runs `gcloud auth application-default login` once and exercises a live Vertex request — should be a no-code-change check. PR-D1 (Bedrock native) is running concurrently and will land its own envelope module at `crates/headroom-proxy/src/bedrock/envelope.rs`. The two envelope modules are intentionally siblings (not a shared trait) — the shapes differ (Bedrock has a different `anthropic_version` value, no `model` field, AWS SigV4 instead of GCP ADC), and a premature shared abstraction would obscure the provider-specific contracts. Whichever PR merges second rebases without conflict. Retires P4-38 (and the Vertex parts of P4-39); marketplace BYOC pitch (per project memory) gets one more native provider.
2026-05-03 16:22:07 -07:00
[[package]]
name = "tracing-futures"
version = "0.2.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "97d095ae15e245a057c8e8451bab9b3ee1e1f68e9ba2b4fbc18d0ac5237835f2"
dependencies = [
"pin-project",
"tracing",
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "tracing-log"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ee855f1f400bd0e5c02d150ae5de3840039a3f54b025156404e34c23c03f47c3"
dependencies = [
"log",
"once_cell",
"tracing-core",
]
[[package]]
name = "tracing-serde"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "704b1aeb7be0d0a84fc9828cae51dab5970fee5088f83d1dd7ee6f6246fc6ff1"
dependencies = [
"serde",
"tracing-core",
]
[[package]]
name = "tracing-subscriber"
version = "0.3.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cb7f578e5945fb242538965c2d0b04418d38ec25c79d160cd279bf0731c8d319"
dependencies = [
"matchers",
"nu-ansi-term",
"once_cell",
"regex-automata",
"serde",
"serde_json",
"sharded-slab",
"smallvec",
"thread_local",
"tracing",
"tracing-core",
"tracing-log",
"tracing-serde",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
]
[[package]]
name = "try-lock"
version = "0.2.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "tungstenite"
version = "0.24.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "18e5b8366ee7a95b16d32197d0b2604b43a0be89dc5fac9f8e96ccafbaedda8a"
dependencies = [
"byteorder",
"bytes",
"data-encoding",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"httparse",
"log",
"rand 0.8.6",
"rustls",
"rustls-pki-types",
"sha1",
"thiserror 1.0.69",
"utf-8",
]
[[package]]
name = "typenum"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.20.1"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "unarray"
version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "eaea85b334db583fe3274d12b4cd1880032beab409c0d774be044d4480ab9a94"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "unicode-ident"
version = "1.0.24"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "unicode-normalization-alignments"
version = "0.1.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "43f613e4fa046e69818dd287fdc4bc78175ff20331479dab6e1b0f98d57062de"
dependencies = [
"smallvec",
]
[[package]]
name = "unicode-segmentation"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.13.3"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "c6f5d3c3b1bf09027a88a6bc961fc00497d651009560b5463668dc81b0fa87a8"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "unicode-width"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254"
[[package]]
name = "unicode_categories"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "39ec24b3121d976906ece63c9daad25b85969647682eee313cb5779fdd69e14e"
chore(rust): port unidiff Tier 2 diff detector (Stage 3d PR4) Adds the second tier of the Stage-3d ContentRouter detection arch. Magika (PR3) is a probabilistic ML classifier — short, prose-prefixed, or "looks like code because the lines are code" diffs can slip past it into PlainText. PR4 catches those by running the [`unidiff`] parser as a deterministic oracle: anything that parses to ≥1 PatchedFile with ≥1 hunk is a diff. What lands: - `crates/headroom-core/src/transforms/unidiff_detector.rs`: - `is_diff(content) -> bool`: predicate. - `detect_diff(content) -> Option<ContentType>`: typed wrapper for the router (PR5) to chain after Magika. - Empty input shortcuts to false without invoking the parser. - "Found zero hunks" is treated as **not** a diff — `unidiff::PatchSet ::parse` returns Ok(()) on plain text (just finds zero files); we explicitly require non-empty patch + non-empty hunk to avoid silently routing prose through the diff compressor. - 14 unit tests: standard git diff, naked hunk without git header, multi-file, added/removed-only files, JSON/HTML/YAML/source/prose negatives, "almost looks like a diff" prose with @@/--- in passing, truncated-diff canary. Known gaps (deliberately punted to PR5+): - Combined-merge headers (`@@@ ... @@@`) — `unidiff`'s hunk regex is for plain `@@`. Rare in proxy traffic; PR5 router can fall back to the regex content_detector if needed. - Pathological CRLF-stripped inputs — `input.lines()` strips `\r` only when paired with `\n`. Acceptable. What does NOT land here (per PR scope): - No PyO3 surface — module-only. - No router rewiring — the existing regex `content_detector` still drives `ContentRouter`. PR5 chains magika → unidiff → PlainText. The `unidiff` crate brings `regex` (already in tree) and `encoding_rs` (default features) — small dep impact. `make ci-precheck` green.
2026-04-28 23:00:11 -07:00
[[package]]
name = "unidiff"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e3ae26d2e6582eb32eff85cffebf74d20b6510e8b558bbac3a23b48965cf952f"
dependencies = [
"encoding_rs",
"regex",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "unit-prefix"
version = "0.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "81e544489bf3d8ef66c953931f56617f423cd4b5494be343d9b9d3dda037b9a3"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "untrusted"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1"
feat(rust): HfTokenizer::from_pretrained — HuggingFace Hub auto-download Stage 2.1: closes the loop on the HuggingFace tokenizer story. Stage 2 shipped `HfTokenizer::from_bytes`/`from_file`, which required callers to manage their own tokenizer.json files. This adds the third constructor: let t = HfTokenizer::from_pretrained("CohereForAI/c4ai-command-r-v01")?; register_hf("command-", t); `from_pretrained` is a thin wrapper around the `hf-hub` crate's blocking `ureq` API. First call downloads `tokenizer.json` to `~/.cache/huggingface/ hub` (or `$HF_HOME` if set); subsequent calls reuse the on-disk cache. Uses the `main` revision; gated repos (Llama, Mistral) require `HF_TOKEN` in env or `~/.cache/huggingface/token`. Also adds `try_register_hf(prefix, repo)` as the obvious one-liner for proxy startup code: let _ = try_register_hf("command-", "CohereForAI/c4ai-command-r-v01"); let _ = try_register_hf("mistral-", "mistralai/Mistral-7B-v0.1"); Each call is independent — a download failure for one model (e.g. gated without a token) does not affect others. `HfTokenizerError` gains a new `Hub` variant so callers can distinguish "couldn't fetch" from "fetched but malformed" — relevant when deciding whether to retry, surface to the user, or fall back to the estimator. Why blocking, not async: `from_pretrained` is called once at startup. A sync API works from `main()`, from a `OnceLock` initializer, or from `tokio::task::spawn_blocking` if a tokio caller needs it later. The async hf-hub backend would force callers to await at startup, which doesn't fit the `register_hf` registry pattern. Why rustls, not native-tls: keeps the binary statically linkable for AWS deploys (no system OpenSSL dependency). Tests: a network-dependent integration test (`#[ignore]`d in CI; hits HF for `gpt2`, ~1.4 MB) verifies the real download + load + count path. A non-network negative test verifies that an invalid repo name surfaces as `HfTokenizerError::Hub`, not a panic. 44 unit tests + 5 proptests + 1 doctest pass; parity stays 40/40 byte-equal.
2026-04-25 15:05:09 -07:00
[[package]]
name = "ureq"
version = "2.12.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "02d1a66277ed75f640d608235660df48c8e3c19f3b4edb6a263315626cc3c01d"
dependencies = [
"base64 0.22.1",
"flate2",
"log",
"once_cell",
"rustls",
"rustls-pki-types",
"serde",
"serde_json",
"socks",
"url",
"webpki-roots 0.26.11",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "ureq"
version = "3.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dea7109cdcd5864d4eeb1b58a1648dc9bf520360d7af16ec26d0a9354bafcfc0"
dependencies = [
"base64 0.22.1",
"cookie_store",
"flate2",
"log",
"percent-encoding",
"rustls",
"rustls-pki-types",
"serde",
"serde_json",
"socks",
"ureq-proto",
"utf8-zero",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"webpki-roots 1.0.8",
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
]
[[package]]
name = "ureq-proto"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e994ba84b0bd1b1b0cf92878b7ef898a5c1760108fe7b6010327e274917a808c"
dependencies = [
"base64 0.22.1",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
"httparse",
"log",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "url"
version = "2.5.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ff67a8a4397373c3ef660812acab3268222035010ab8680ec4215f38ba3d0eed"
dependencies = [
"form_urlencoded",
"idna",
"percent-encoding",
"serde",
]
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
[[package]]
name = "urlencoding"
version = "2.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "daf8dba3b7eb870caf1ddeed7bc9d2a049f3cfdfae7cb521b087cc33ae4c49da"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "utf-8"
version = "0.7.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09cc8ee72d2a9becf2f2febe0205bbed8fc6615b7cb429ad062dc7b7ddd036a9"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "utf8-zero"
version = "0.8.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b8c0a043c9540bae7c578c88f91dda8bd82e59ae27c21baca69c8b191aaf5a6e"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "utf8_iter"
version = "1.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be"
[[package]]
name = "utf8parse"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "uuid"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.23.3"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "144d6b123cef80b301b8f72a9e2ca4370ddec21950d0a103dd22c437006d2db7"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
dependencies = [
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"getrandom 0.4.3",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"js-sys",
"wasm-bindgen",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "v_frame"
version = "0.3.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "666b7727c8875d6ab5db9533418d7c764233ac9c0cff1d469aec8fa127597be2"
dependencies = [
"aligned-vec",
"num-traits",
"wasm-bindgen",
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "valuable"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ba73ea9cf16a25df0c8caa16c51acb937d5712a8429db78a3ee29d5dcacd3a65"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "vcpkg"
version = "0.2.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "accd4ea62f7bb7a82fe23066fb0957d48ef677f6eeb8215f372f52e48bb32426"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "version_check"
version = "0.9.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
[[package]]
name = "vsimd"
version = "0.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5c3082ca00d5a5ef149bb8b555a72ae84c9c59f7250f013ac822ac2e49b19c64"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "wait-timeout"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09ac3b126d3914f9849036f826e054cbabdc8519970b8998ddaf3b5bd3c65f11"
dependencies = [
"libc",
]
[[package]]
name = "walkdir"
version = "2.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b"
dependencies = [
"same-file",
"winapi-util",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "want"
version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bfa7760aed19e106de2c7c0b581b509f2f25d3dacaf737cb82ac61bc6d760b0e"
dependencies = [
"try-lock",
]
[[package]]
name = "wasi"
version = "0.11.1+wasi-snapshot-preview1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b"
[[package]]
name = "wasip2"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.0.4+wasi-0.2.12"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "b67efb37e106e55ce722a510d6b5f9c17f083e5fc79afc2badeb12cc313d9487"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"wit-bindgen",
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
]
[[package]]
name = "wasm-bindgen"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.2.125"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "8ddb3f79143bced6de84270411622a2699cee572fc0875aeaf1e7867cf9fca1a"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"cfg-if",
"once_cell",
"rustversion",
"wasm-bindgen-macro",
"wasm-bindgen-shared",
]
[[package]]
name = "wasm-bindgen-futures"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.4.75"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "503b14d284f2c8dac03b819967e155ea753f573586193b2b2c95990cb5d69280"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"js-sys",
"wasm-bindgen",
]
[[package]]
name = "wasm-bindgen-macro"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.2.125"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "4e21a184b13fb19e157296e2c46056aec9092264fab83e4ba59e68c61b323c3d"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"quote",
"wasm-bindgen-macro-support",
]
[[package]]
name = "wasm-bindgen-macro-support"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.2.125"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "fecefd9c35bd935a20fc3fc344b5f29138961e4f47fb03297d88f2587afb5ebd"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"bumpalo",
"proc-macro2",
"quote",
"syn",
"wasm-bindgen-shared",
]
[[package]]
name = "wasm-bindgen-shared"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.2.125"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "23939e44bb9a5d7576fa2b563dc2e136628f1224e88a8deed09e04858b77871f"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"unicode-ident",
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "wasm-streams"
version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "15053d8d85c7eccdbefef60f06769760a563c7f0a9d6902a13d35c7800b0ad65"
dependencies = [
"futures-util",
"js-sys",
"wasm-bindgen",
"wasm-bindgen-futures",
"web-sys",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "web-sys"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.3.102"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "a6430a72df5eb332242960fe84b3002a241163998241eb596d4f739b9757061d"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"js-sys",
"wasm-bindgen",
]
[[package]]
name = "web-time"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb"
dependencies = [
"js-sys",
"wasm-bindgen",
]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
name = "webpki-roots"
version = "0.26.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "521bc38abb08001b01866da9f51eb7c5d647a19260e00054a8c7fd5f9e57f7a9"
dependencies = [
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"webpki-roots 1.0.8",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "webpki-roots"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.0.8"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "bf85cb06032201fa7c6f829d7db5a7e5aa45bcc0655327713065f6f0576731bf"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"rustls-pki-types",
]
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "weezl"
version = "0.1.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a28ac98ddc8b9274cb41bb4d9d4d5c425b6020c50c46f25559911905610b4a88"
feat(rust): HfTokenizer::from_pretrained — HuggingFace Hub auto-download Stage 2.1: closes the loop on the HuggingFace tokenizer story. Stage 2 shipped `HfTokenizer::from_bytes`/`from_file`, which required callers to manage their own tokenizer.json files. This adds the third constructor: let t = HfTokenizer::from_pretrained("CohereForAI/c4ai-command-r-v01")?; register_hf("command-", t); `from_pretrained` is a thin wrapper around the `hf-hub` crate's blocking `ureq` API. First call downloads `tokenizer.json` to `~/.cache/huggingface/ hub` (or `$HF_HOME` if set); subsequent calls reuse the on-disk cache. Uses the `main` revision; gated repos (Llama, Mistral) require `HF_TOKEN` in env or `~/.cache/huggingface/token`. Also adds `try_register_hf(prefix, repo)` as the obvious one-liner for proxy startup code: let _ = try_register_hf("command-", "CohereForAI/c4ai-command-r-v01"); let _ = try_register_hf("mistral-", "mistralai/Mistral-7B-v0.1"); Each call is independent — a download failure for one model (e.g. gated without a token) does not affect others. `HfTokenizerError` gains a new `Hub` variant so callers can distinguish "couldn't fetch" from "fetched but malformed" — relevant when deciding whether to retry, surface to the user, or fall back to the estimator. Why blocking, not async: `from_pretrained` is called once at startup. A sync API works from `main()`, from a `OnceLock` initializer, or from `tokio::task::spawn_blocking` if a tokio caller needs it later. The async hf-hub backend would force callers to await at startup, which doesn't fit the `register_hf` registry pattern. Why rustls, not native-tls: keeps the binary statically linkable for AWS deploys (no system OpenSSL dependency). Tests: a network-dependent integration test (`#[ignore]`d in CI; hits HF for `gpt2`, ~1.4 MB) verifies the real download + load + count path. A non-network negative test verifies that an invalid repo name surfaces as `HfTokenizerError::Hub`, not a panic. 44 unit tests + 5 proptests + 1 doctest pass; parity stays 40/40 byte-equal.
2026-04-25 15:05:09 -07:00
[[package]]
name = "winapi"
version = "0.3.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419"
dependencies = [
"winapi-i686-pc-windows-gnu",
"winapi-x86_64-pc-windows-gnu",
]
[[package]]
name = "winapi-i686-pc-windows-gnu"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6"
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "winapi-util"
version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
dependencies = [
"windows-sys 0.61.2",
]
feat(rust): HfTokenizer::from_pretrained — HuggingFace Hub auto-download Stage 2.1: closes the loop on the HuggingFace tokenizer story. Stage 2 shipped `HfTokenizer::from_bytes`/`from_file`, which required callers to manage their own tokenizer.json files. This adds the third constructor: let t = HfTokenizer::from_pretrained("CohereForAI/c4ai-command-r-v01")?; register_hf("command-", t); `from_pretrained` is a thin wrapper around the `hf-hub` crate's blocking `ureq` API. First call downloads `tokenizer.json` to `~/.cache/huggingface/ hub` (or `$HF_HOME` if set); subsequent calls reuse the on-disk cache. Uses the `main` revision; gated repos (Llama, Mistral) require `HF_TOKEN` in env or `~/.cache/huggingface/token`. Also adds `try_register_hf(prefix, repo)` as the obvious one-liner for proxy startup code: let _ = try_register_hf("command-", "CohereForAI/c4ai-command-r-v01"); let _ = try_register_hf("mistral-", "mistralai/Mistral-7B-v0.1"); Each call is independent — a download failure for one model (e.g. gated without a token) does not affect others. `HfTokenizerError` gains a new `Hub` variant so callers can distinguish "couldn't fetch" from "fetched but malformed" — relevant when deciding whether to retry, surface to the user, or fall back to the estimator. Why blocking, not async: `from_pretrained` is called once at startup. A sync API works from `main()`, from a `OnceLock` initializer, or from `tokio::task::spawn_blocking` if a tokio caller needs it later. The async hf-hub backend would force callers to await at startup, which doesn't fit the `register_hf` registry pattern. Why rustls, not native-tls: keeps the binary statically linkable for AWS deploys (no system OpenSSL dependency). Tests: a network-dependent integration test (`#[ignore]`d in CI; hits HF for `gpt2`, ~1.4 MB) verifies the real download + load + count path. A non-network negative test verifies that an invalid repo name surfaces as `HfTokenizerError::Hub`, not a panic. 44 unit tests + 5 proptests + 1 doctest pass; parity stays 40/40 byte-equal.
2026-04-25 15:05:09 -07:00
[[package]]
name = "winapi-x86_64-pc-windows-gnu"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
fix(proxy): PR-D4 native Vertex publisher path + ADC bearer auth Adds a Rust-native Vertex AI publisher route ahead of the LiteLLM Python converter (which dropped `thinking`, `redacted_thinking`, `document`, `image`, `server_tool_use`, `mcp_tool_use` block kinds — the P4-37 / P4-38 bug). After this PR the Vertex `:rawPredict` and `:streamRawPredict` calls survive byte-equal upstream and benefit from the live-zone Anthropic dispatcher (PR-B-series) running over the body — same behaviour as `/v1/messages`. New module `crates/headroom-proxy/src/vertex/`: - `mod.rs` — single dispatch handler at the `/v1beta1/.../models/:model_action` route. Splits the trailing `:<verb>` segment with `str::rsplit_once(':')` (no regex) and flips an `attach_sse_tee` flag to dispatch to the streaming or non-streaming arm. Both verbs share one axum route shape because matchit can't distinguish two patterns that overlap on a parameter. - `envelope.rs` — `VertexEnvelope` parser. Confirms `anthropic_version` present + `model` field absent (the two fingerprints of the Vertex envelope vs `/v1/messages`). - `adc.rs` — `TokenSource` trait + `GcpAdcTokenSource` (production, `gcp_auth` 0.12) + `StaticTokenSource` (tests). Caches tokens with a 60s refresh-ahead-of-expiry window. Emits structured `event = "vertex_adc_token_refreshed"` per refresh. - `raw_predict.rs` — POST handler + shared `forward_vertex_request`. Buffers body, parses envelope, runs live-zone Anthropic compression, fetches ADC bearer, attaches `Authorization: Bearer <token>` (overwrites client-supplied Authorization header), forwards. SSE telemetry tee for the streaming verb reuses PR-C1's `AnthropicStreamState` directly (Vertex streams plain SSE, unlike Bedrock's binary EventStream). - `stream_raw_predict.rs` — module-level docs + alias to the shared dispatcher (the streaming-vs-non-streaming difference is one boolean flag inside the shared forwarder). Modifications: - `proxy.rs::build_app` — registers the single Vertex route. - `proxy.rs::AppState` — new `vertex_token_source: Arc<dyn TokenSource>` field. Production constructs `GcpAdcTokenSource` lazily (no GCP call until first `bearer()`); tests inject `StaticTokenSource` via the new `AppState::with_token_source` helper. - `config.rs` — adds `--vertex-region` / `HEADROOM_PROXY_VERTEX_REGION` (default `us-central1`, observability tag only — the upstream URL is `--upstream`) and `--vertex-adc-scope` / `HEADROOM_PROXY_VERTEX_ADC_SCOPE` (default `cloud-platform`). - `Cargo.toml` (workspace + proxy) — adds `gcp_auth = "0.12"` and `async-trait = "0.1"`. - `tests/common/mod.rs` — `start_proxy_with_state` accepts both config + state customizers; `install_static_token_source` helper for tests. `crates/headroom-proxy/tests/integration_vertex_raw_predict.rs` — all five tests pass: 1. `native_envelope_round_trip_byte_equal` — Vertex-shape body (with `anthropic_version`, no `model`) round-trips SHA-256 byte-equal upstream. 2. `adc_bearer_token_signed_correctly` — `Authorization: Bearer <static-test-token>` reaches upstream verbatim and OVERWRITES a client-supplied Authorization header. 3. `thinking_block_preserved` — request with `thinking` (incl. signature) + `redacted_thinking` (incl. opaque `data`) blocks round-trips byte-equal even with `LiveZone` compression mode enabled. This is the P4-37 / P4-38 teeth. 4. `stream_raw_predict_sse_handled` — `:streamRawPredict` proxies an Anthropic SSE response (full `message_start` → `content_block_delta` → `message_stop` sequence) back to the client without corruption; SSE content-type preserved end-to-end; bearer attached. 5. (bonus, no-silent-fallback contract) `adc_failure_returns_5xx_no_silent_forward` — when the token source returns `Err`, the proxy returns 5xx and never reaches upstream. Verifies the `event = "vertex_adc_fetch_failed"` error path. Workspace: `cargo test --workspace` green; `cargo clippy --workspace -- -D warnings` clean; `make ci-precheck` passes. - No silent fallbacks: ADC failure → structured 5xx, never an unauthenticated forward. - No hardcodes: every knob (region, ADC scope, upstream URL) is CLI-flag + env-var configurable. - No regexes: axum path parameters + `str::rsplit_once` only. - Comprehensive structured logs: `event` field on every decision point — `vertex_envelope_parsed`, `vertex_envelope_invalid`, `vertex_compression_skipped`, `vertex_compression_applied`, `vertex_adc_token_refreshed`, `vertex_adc_fetch_failed`, `vertex_streaming_pipeline_active`, `vertex_sse_stream_closed`, `vertex_forwarded`, `vertex_unknown_verb`, etc. - Performant: no body clone; ADC token cached + refreshed ahead-of-expiry, not fetched per request. - Comprehensive tests: realistic Anthropic block content (signature payload, redacted_thinking opaque blob) in `thinking_block_preserved`. The local `gcloud auth application-default print-access-token` returns no credentials, so manual validation against a real Vertex endpoint is not possible in this PR. Follow-up: the user runs `gcloud auth application-default login` once and exercises a live Vertex request — should be a no-code-change check. PR-D1 (Bedrock native) is running concurrently and will land its own envelope module at `crates/headroom-proxy/src/bedrock/envelope.rs`. The two envelope modules are intentionally siblings (not a shared trait) — the shapes differ (Bedrock has a different `anthropic_version` value, no `model` field, AWS SigV4 instead of GCP ADC), and a premature shared abstraction would obscure the provider-specific contracts. Whichever PR merges second rebases without conflict. Retires P4-38 (and the Vertex parts of P4-39); marketplace BYOC pitch (per project memory) gets one more native provider.
2026-05-03 16:22:07 -07:00
[[package]]
name = "windows-core"
version = "0.62.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb"
dependencies = [
"windows-implement",
"windows-interface",
"windows-link",
"windows-result",
"windows-strings",
]
[[package]]
name = "windows-implement"
version = "0.60.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "windows-interface"
version = "0.59.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "windows-link"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
fix(proxy): PR-D4 native Vertex publisher path + ADC bearer auth Adds a Rust-native Vertex AI publisher route ahead of the LiteLLM Python converter (which dropped `thinking`, `redacted_thinking`, `document`, `image`, `server_tool_use`, `mcp_tool_use` block kinds — the P4-37 / P4-38 bug). After this PR the Vertex `:rawPredict` and `:streamRawPredict` calls survive byte-equal upstream and benefit from the live-zone Anthropic dispatcher (PR-B-series) running over the body — same behaviour as `/v1/messages`. New module `crates/headroom-proxy/src/vertex/`: - `mod.rs` — single dispatch handler at the `/v1beta1/.../models/:model_action` route. Splits the trailing `:<verb>` segment with `str::rsplit_once(':')` (no regex) and flips an `attach_sse_tee` flag to dispatch to the streaming or non-streaming arm. Both verbs share one axum route shape because matchit can't distinguish two patterns that overlap on a parameter. - `envelope.rs` — `VertexEnvelope` parser. Confirms `anthropic_version` present + `model` field absent (the two fingerprints of the Vertex envelope vs `/v1/messages`). - `adc.rs` — `TokenSource` trait + `GcpAdcTokenSource` (production, `gcp_auth` 0.12) + `StaticTokenSource` (tests). Caches tokens with a 60s refresh-ahead-of-expiry window. Emits structured `event = "vertex_adc_token_refreshed"` per refresh. - `raw_predict.rs` — POST handler + shared `forward_vertex_request`. Buffers body, parses envelope, runs live-zone Anthropic compression, fetches ADC bearer, attaches `Authorization: Bearer <token>` (overwrites client-supplied Authorization header), forwards. SSE telemetry tee for the streaming verb reuses PR-C1's `AnthropicStreamState` directly (Vertex streams plain SSE, unlike Bedrock's binary EventStream). - `stream_raw_predict.rs` — module-level docs + alias to the shared dispatcher (the streaming-vs-non-streaming difference is one boolean flag inside the shared forwarder). Modifications: - `proxy.rs::build_app` — registers the single Vertex route. - `proxy.rs::AppState` — new `vertex_token_source: Arc<dyn TokenSource>` field. Production constructs `GcpAdcTokenSource` lazily (no GCP call until first `bearer()`); tests inject `StaticTokenSource` via the new `AppState::with_token_source` helper. - `config.rs` — adds `--vertex-region` / `HEADROOM_PROXY_VERTEX_REGION` (default `us-central1`, observability tag only — the upstream URL is `--upstream`) and `--vertex-adc-scope` / `HEADROOM_PROXY_VERTEX_ADC_SCOPE` (default `cloud-platform`). - `Cargo.toml` (workspace + proxy) — adds `gcp_auth = "0.12"` and `async-trait = "0.1"`. - `tests/common/mod.rs` — `start_proxy_with_state` accepts both config + state customizers; `install_static_token_source` helper for tests. `crates/headroom-proxy/tests/integration_vertex_raw_predict.rs` — all five tests pass: 1. `native_envelope_round_trip_byte_equal` — Vertex-shape body (with `anthropic_version`, no `model`) round-trips SHA-256 byte-equal upstream. 2. `adc_bearer_token_signed_correctly` — `Authorization: Bearer <static-test-token>` reaches upstream verbatim and OVERWRITES a client-supplied Authorization header. 3. `thinking_block_preserved` — request with `thinking` (incl. signature) + `redacted_thinking` (incl. opaque `data`) blocks round-trips byte-equal even with `LiveZone` compression mode enabled. This is the P4-37 / P4-38 teeth. 4. `stream_raw_predict_sse_handled` — `:streamRawPredict` proxies an Anthropic SSE response (full `message_start` → `content_block_delta` → `message_stop` sequence) back to the client without corruption; SSE content-type preserved end-to-end; bearer attached. 5. (bonus, no-silent-fallback contract) `adc_failure_returns_5xx_no_silent_forward` — when the token source returns `Err`, the proxy returns 5xx and never reaches upstream. Verifies the `event = "vertex_adc_fetch_failed"` error path. Workspace: `cargo test --workspace` green; `cargo clippy --workspace -- -D warnings` clean; `make ci-precheck` passes. - No silent fallbacks: ADC failure → structured 5xx, never an unauthenticated forward. - No hardcodes: every knob (region, ADC scope, upstream URL) is CLI-flag + env-var configurable. - No regexes: axum path parameters + `str::rsplit_once` only. - Comprehensive structured logs: `event` field on every decision point — `vertex_envelope_parsed`, `vertex_envelope_invalid`, `vertex_compression_skipped`, `vertex_compression_applied`, `vertex_adc_token_refreshed`, `vertex_adc_fetch_failed`, `vertex_streaming_pipeline_active`, `vertex_sse_stream_closed`, `vertex_forwarded`, `vertex_unknown_verb`, etc. - Performant: no body clone; ADC token cached + refreshed ahead-of-expiry, not fetched per request. - Comprehensive tests: realistic Anthropic block content (signature payload, redacted_thinking opaque blob) in `thinking_block_preserved`. The local `gcloud auth application-default print-access-token` returns no credentials, so manual validation against a real Vertex endpoint is not possible in this PR. Follow-up: the user runs `gcloud auth application-default login` once and exercises a live Vertex request — should be a no-code-change check. PR-D1 (Bedrock native) is running concurrently and will land its own envelope module at `crates/headroom-proxy/src/bedrock/envelope.rs`. The two envelope modules are intentionally siblings (not a shared trait) — the shapes differ (Bedrock has a different `anthropic_version` value, no `model` field, AWS SigV4 instead of GCP ADC), and a premature shared abstraction would obscure the provider-specific contracts. Whichever PR merges second rebases without conflict. Retires P4-38 (and the Vertex parts of P4-39); marketplace BYOC pitch (per project memory) gets one more native provider.
2026-05-03 16:22:07 -07:00
[[package]]
name = "windows-result"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5"
dependencies = [
"windows-link",
]
[[package]]
name = "windows-strings"
version = "0.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091"
dependencies = [
"windows-link",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "windows-sys"
version = "0.52.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d"
dependencies = [
"windows-targets 0.52.6",
]
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
[[package]]
name = "windows-sys"
version = "0.59.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b"
dependencies = [
"windows-targets 0.52.6",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "windows-sys"
version = "0.60.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f2f500e4d28234f72040990ec9d39e3a6b950f9f22d3dba18416c35882612bcb"
dependencies = [
"windows-targets 0.53.5",
]
[[package]]
name = "windows-sys"
version = "0.61.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc"
dependencies = [
"windows-link",
]
[[package]]
name = "windows-targets"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973"
dependencies = [
"windows_aarch64_gnullvm 0.52.6",
"windows_aarch64_msvc 0.52.6",
"windows_i686_gnu 0.52.6",
"windows_i686_gnullvm 0.52.6",
"windows_i686_msvc 0.52.6",
"windows_x86_64_gnu 0.52.6",
"windows_x86_64_gnullvm 0.52.6",
"windows_x86_64_msvc 0.52.6",
]
[[package]]
name = "windows-targets"
version = "0.53.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4945f9f551b88e0d65f3db0bc25c33b8acea4d9e41163edf90dcd0b19f9069f3"
dependencies = [
"windows-link",
"windows_aarch64_gnullvm 0.53.1",
"windows_aarch64_msvc 0.53.1",
"windows_i686_gnu 0.53.1",
"windows_i686_gnullvm 0.53.1",
"windows_i686_msvc 0.53.1",
"windows_x86_64_gnu 0.53.1",
"windows_x86_64_gnullvm 0.53.1",
"windows_x86_64_msvc 0.53.1",
]
[[package]]
name = "windows_aarch64_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3"
[[package]]
name = "windows_aarch64_gnullvm"
version = "0.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a9d8416fa8b42f5c947f8482c43e7d89e73a173cead56d044f6a56104a6d1b53"
[[package]]
name = "windows_aarch64_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469"
[[package]]
name = "windows_aarch64_msvc"
version = "0.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b9d782e804c2f632e395708e99a94275910eb9100b2114651e04744e9b125006"
[[package]]
name = "windows_i686_gnu"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b"
[[package]]
name = "windows_i686_gnu"
version = "0.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "960e6da069d81e09becb0ca57a65220ddff016ff2d6af6a223cf372a506593a3"
[[package]]
name = "windows_i686_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66"
[[package]]
name = "windows_i686_gnullvm"
version = "0.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fa7359d10048f68ab8b09fa71c3daccfb0e9b559aed648a8f95469c27057180c"
[[package]]
name = "windows_i686_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66"
[[package]]
name = "windows_i686_msvc"
version = "0.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e7ac75179f18232fe9c285163565a57ef8d3c89254a30685b57d83a38d326c2"
[[package]]
name = "windows_x86_64_gnu"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78"
[[package]]
name = "windows_x86_64_gnu"
version = "0.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9c3842cdd74a865a8066ab39c8a7a473c0778a3f29370b5fd6b4b9aa7df4a499"
[[package]]
name = "windows_x86_64_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d"
[[package]]
name = "windows_x86_64_gnullvm"
version = "0.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0ffa179e2d07eee8ad8f57493436566c7cc30ac536a3379fdf008f47f6bb7ae1"
[[package]]
name = "windows_x86_64_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec"
[[package]]
name = "windows_x86_64_msvc"
version = "0.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650"
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
[[package]]
fix(rust): reformat/offload pipeline + log templates + diff noise (Phase 3g rework) Replaces PR1's lossless/lossy split with ReformatTransform (pack denser, no info lost) and OffloadTransform (drop bytes, CCR-stash original via required cache_key). With CCR every transform is information-preserving end-to-end, so the lossless/lossy distinction misnamed the architecture. OffloadTransform carries a cheap, structural estimate_bloat() method scoped to its domain — generic byte-redundancy heuristics miss domain semantics. The orchestrator runs reformat phase + per-offload bloat estimation in parallel via rayon::join + par_iter, then runs offload iff bloat clears threshold OR reformat underwhelmed. Transforms shipped: REFORMATS (lossless): - JsonMinifier: serde_json round-trip whitespace stripping. - LogTemplate: Drain-inspired order-preserving template miner. Collapses consecutive runs of same-template lines into [Template Tn: ...] (Nx) + variant table. Win comes from emitting the constant-token prefix once instead of N times. Lossless: every original line reconstructible from template + variants. OFFLOADS (drop bytes, stash original via CCR): - LogOffload: wraps existing LogCompressor; bloat = repetition x uniqueness_weight + dilution x priority_dilution_weight. - DiffOffload: wraps existing DiffCompressor; bloat = context-to- change ratio. Bug-fix-on-port — persists original under the cache_key the parity-bound DiffCompressor mints (closes a leak). - DiffNoise: drops lockfile hunks (Cargo.lock, package-lock.json, yarn.lock, etc., suffix list configurable in TOML) and whitespace-only hunks. Stashes original via CCR for retrieval. Search offload exists but is not in default re-exports — modern agents (Claude Code, Codex) use scoped rg/grep, the marginal value didn't justify default registration. Reach via the explicit module path if opting in. JSON Offload is intentionally absent from this PR — already lives at SmartCrusher; Phase 3g PR3 wraps it in the OffloadTransform contract. Thresholds and weights live in config/pipeline.toml, embedded via include_str!; PipelineConfig::from_toml_str loads runtime overrides. 98 new pipeline tests; full headroom-core suite (714) and workspace tests green; cargo fmt clean. No regex, per project convention.
2026-04-30 11:02:29 -07:00
name = "winnow"
version = "0.7.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "df79d97927682d2fd8adb29682d1140b343be4ac0f08fd68b7765d9c059d3945"
dependencies = [
"memchr",
]
[[package]]
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
name = "wiremock"
version = "0.6.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "08db1edfb05d9b3c1542e521aea074442088292f00b5f28e435c714a98f85031"
dependencies = [
"assert-json-diff",
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged.
2026-04-25 14:22:09 -07:00
"base64 0.22.1",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"deadpool",
"futures",
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
"http 1.4.2",
feat(rust): axum reverse proxy skeleton + http catch-all (phase-1) Builds out crates/headroom-proxy from a /healthz stub into a transparent reverse proxy: catch-all router that forwards every method/path/query to --upstream verbatim, streaming both request and response bodies through reqwest without buffering. Adds clap-based config (CLI + env), thiserror error type with sane upstream-status mapping, JSON tracing-subscriber logging, and graceful shutdown. The library surface (build_app, AppState, Config) is reused by the integration tests.
2026-04-24 15:47:07 -07:00
"http-body-util",
"hyper",
"hyper-util",
"log",
"once_cell",
"regex",
"serde",
"serde_json",
"tokio",
"url",
]
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "wit-bindgen"
version = "0.57.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e"
[[package]]
name = "writeable"
version = "0.6.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4"
fix(proxy): PR-D1 native Bedrock InvokeModel route + SigV4 Phase D PR-D1 lands the first native Rust path for AWS Bedrock, replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock non-streaming requests. Eliminates part of P4-37 and P4-39. What landed ----------- - New crates/headroom-proxy/src/bedrock/ module: - envelope.rs: parses the {"anthropic_version": "...", ...} Bedrock body shape; re-emits with anthropic_version preserved as the first key (relies on serde_json preserve_order). - sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate. Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256 is in the canonical request, hashed over the post-compression body bytes (the bytes that actually hit Bedrock). No silent fallback: signing failures return 5xx with event=bedrock_sigv4_failed. - invoke.rs: POST handler for /model/{model_id}/invoke (and /converse - same wire shape for anthropic.claude-*). Detects Anthropic vendor via literal starts_with("anthropic.") (no regex per project rule), routes Anthropic-shape bodies through the existing compress_anthropic_request live-zone dispatcher, then signs and forwards to the configured Bedrock endpoint. - Modified: - proxy.rs: routes /model/:model_id/invoke and /model/:model_id/converse when enable_bedrock_native is on (default). Adds bedrock_credentials: Option<Arc<Credentials>> to AppState. - config.rs: new flags --bedrock-region (default us-east-1, env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint (operator override for FIPS/VPC/test setups), --enable-bedrock-native (default true), --aws-profile. - main.rs: resolves AWS credentials at startup via aws_config::defaults(BehaviorVersion::latest()). Failure logs event=bedrock_credentials_unavailable at WARN; the handler refuses to forward unsigned (event=bedrock_credentials_missing). - Cargo.toml: workspace deps aws-sigv4, aws-config, aws-credential-types, aws-smithy-runtime-api. Tests ----- 8 integration tests under crates/headroom-proxy/tests/integration_bedrock_invoke.rs: 1. native_envelope_round_trip_byte_equal 2. sigv4_signed_correctly_after_compression - confirms authorization is SigV4-shape and x-amz-content-sha256 matches sha256(body received by upstream). 3. thinking_block_preserved_through_bedrock 4. redacted_thinking_preserved 5. document_block_preserved 6. tool_result_array_with_image_preserved 7. stop_sequence_null_only_when_present - pins that the proxy does NOT inject stop_sequence: null (P4-37 hardcode). 8. tool_use_input_byte_equal_preserves_key_order All eight pass. Full workspace test run is green; clippy + fmt clean. make ci-precheck (rust + python + commitlint) passes locally. Build constraints honoured -------------------------- - No silent fallbacks: missing creds / signing failures return 5xx with structured event=... log; no path ever forwards unsigned. - No hardcodes: region, endpoint, profile, enable-flag all configurable via CLI + env. - No regexes: vendor detection is str::starts_with. - Comprehensive structured logs: event=bedrock_invoke_received, bedrock_envelope_parsed, bedrock_compression_skipped, bedrock_credentials_missing, sigv4_signed, bedrock_invoke_forwarded, etc. - Performant: body buffered once, passed by &[u8] to signer (zero-copy), Bytes::clone only for ownership transfer to reqwest. Sign exactly once per request. - Elegant: 4 small focused modules mirror handlers/ + sse/. - Tests use realistic Anthropic block content (real thinking, redacted_thinking, document, base64 image fixtures). Streaming (PR-D2) and observability (PR-D3) follow.
2026-05-03 16:22:32 -07:00
[[package]]
name = "xmlparser"
version = "0.13.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "66fee0b777b0f5ac1c69bb06d361268faafa61cd4682ae064a171c16c433e9e4"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "y4m"
version = "0.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7a5a4b21e1a62b67a2970e6831bc091d7b87e119e7f9791aef9702e3bef04448"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "yoke"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.8.3"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "709fe23a0424b6a435d82152b1bd3fdfb0833487d5fa90d05d42762a9891fef5"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"stable_deref_trait",
"yoke-derive",
"zerofrom",
]
[[package]]
name = "yoke-derive"
version = "0.8.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "de844c262c8848816172cef550288e7dc6c7b7814b4ee56b3e1553f275f1858e"
dependencies = [
"proc-macro2",
"quote",
"syn",
"synstructure",
]
[[package]]
name = "zerocopy"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.8.52"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "ce1022995ff5ff5d841ad7d994facc23098cd40152f2c1d11cd607c6f530653f"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"zerocopy-derive",
]
[[package]]
name = "zerocopy-derive"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.8.52"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "1ae7f38b72ec2a254e2b87ef277cf2cd4fb97cbebf944faa6f33354da0867930"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "zerofrom"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "0.1.8"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "0ec05a11813ea801ff6d75110ad09cd0824ddba17dfe17128ea0d5f68e6c5272"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
dependencies = [
"zerofrom-derive",
]
[[package]]
name = "zerofrom-derive"
version = "0.1.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "11532158c46691caf0f2593ea8358fed6bbf68a0315e80aae9bd41fbade684a1"
dependencies = [
"proc-macro2",
"quote",
"syn",
"synstructure",
]
[[package]]
name = "zeroize"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
version = "1.9.0"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
source = "registry+https://github.com/rust-lang/crates.io-index"
feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:55:37 +02:00
checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e"
feat(rust): scaffold workspace + parity harness (phase-0) Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00
[[package]]
name = "zerotrie"
version = "0.2.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0f9152d31db0792fa83f70fb2f83148effb5c1f5b8c7686c3459e361d9bc20bf"
dependencies = [
"displaydoc",
"yoke",
"zerofrom",
]
[[package]]
name = "zerovec"
version = "0.11.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "90f911cbc359ab6af17377d242225f4d75119aec87ea711a880987b18cd7b239"
dependencies = [
"yoke",
"zerofrom",
"zerovec-derive",
]
[[package]]
name = "zerovec-derive"
version = "0.11.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "625dc425cab0dca6dc3c3319506e6593dcb08a9f387ea3b284dbd52a92c40555"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "zmij"
version = "1.0.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa"
feat(rust): real fastembed-rs EmbeddingScorer (BAAI/bge-small-en-v1.5) Replace the embedding scorer stub with a real fastembed-rs implementation. Same library + same model as the Python side will use after the next commit, giving byte-equal embeddings on identical inputs. Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX Runtime) with auto-download of the runtime binary at build time (~21s additional first-build); model weights (BAAI/bge-small-en-v1.5, ~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on first use. embedding.rs: - EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required because TextEmbedding::embed needs &mut self (single-threaded ONNX session); concurrent callers serialize on the lock, fine for the SmartCrusher hot path where inference dominates lock contention. - EmbeddingScorer::try_new() — explicit construction with HF Hub download. Returns Result; surface errors to callers. - EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your own model from fastembed's catalog. - EmbeddingScorer::default() — STUB only (model=None, is_available()=false). Mirrors Python's "sentence-transformers not installed" branch byte-for-byte. To get a real scorer, call try_new() and pass via HybridScorer::with_scorers(). Why default() is a stub: with auto-load Default, model availability would depend on whether HF Hub cache has the file — non-deterministic in tests. Explicit try_new() keeps Default cheap and predictable. cosine_similarity: - f32 vec inputs (fastembed returns Vec<Vec<f32>>). - Clamped to [0, 1] (mirrors Python _cosine_similarity — only positive similarity matters for relevance). - Defensive: zero vectors / mismatched dims → 0.0. score / score_batch: - Empty input / unavailable model → empty score with explanatory reason. - Batch encodes items + context in one model call (Python parity: amortizes model dispatch). - Inference failures degrade gracefully with empty scores rather than panicking. Tests: - 5 cosine-similarity unit tests (offline). - 3 unavailable-scorer tests (model=None path). - 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1 (semantic-match-outranks-unrelated, batch-shape, model-loads). - All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS; with it set, the gated 3 also pass. Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path remains correct (default embedding scorer reports unavailable). Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed PyPI package + record parity fixtures with real embeddings on both sides.
2026-04-26 23:22:32 -07:00
[[package]]
name = "zune-core"
version = "0.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cb8a0807f7c01457d0379ba880ba6322660448ddebc890ce29bb64da71fb40f9"
[[package]]
name = "zune-inflate"
version = "0.2.54"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "73ab332fe2f6680068f3582b16a24f90ad7096d5d39b974d1c0aff0125116f02"
dependencies = [
"simd-adler32",
]
[[package]]
name = "zune-jpeg"
version = "0.5.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "27bc9d5b815bc103f142aa054f561d9187d191692ec7c2d1e2b4737f8dbd7296"
dependencies = [
"zune-core",
]
Reference in a new issue Copy permalink