mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
1096 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
chopratejas
|
2a91cbb4b4 |
refactor: single-wheel maturin build backend (fixes #355)
Eliminates the dual-package architecture that was the root cause of #355. `pip install headroom-ai` now produces ONE wheel containing both the Python source (headroom/*.py) and the compiled Rust extension (headroom/_core.so). No more separate `headroom-core-py` package, no more chicken-and-egg with PyPI publication, no more wheelhouse / PIP_FIND_LINKS / composite-action plumbing in CI. This is the canonical pattern used by cryptography, polars, ruff, pydantic-core, and other Rust-as-core Python packages. Honors the "Rust as core engine" direction. ## What changed - pyproject.toml: `[build-system]` swapped from hatchling to maturin. `[tool.hatch.*]` deleted; `[tool.maturin]` added pointing at `crates/headroom-py/Cargo.toml` for the cdylib. `python-source = "."` picks up the root `headroom/` package directly (dashboard HTML templates and other non-Python files included automatically). - crates/headroom-py/pyproject.toml: deleted. The crate is no longer a separate published package; its Cargo.toml stays as the cdylib build target invoked via `[tool.maturin] manifest-path`. - crates/headroom-py/python/: deleted (placeholder layout for the old separate package). ## CI updates - ci.yml: `test` / `test-extras` / `test-agno` jobs simplified — Rust toolchain set up before `pip install -e .` (which now invokes maturin via build-system). Removed the "build wheel + symlink .so" dance. `build` job swapped from `python -m build` (hatch) to `maturin build` + `maturin sdist`. - release.yml: collapsed dual-package matrix into one. New `build-wheels` matrix produces cross-platform wheels for cp310/11/12/13 × {linux x86_64, linux aarch64, macos x86_64, macos aarch64}. New `collect-dist` aggregator merges artifacts. publish-pypi consumes the merged dist. - init-native-e2e.yml: dropped windows-latest from the matrix — upstream `esaxx-rs` (/MT) and `ort-sys` (/MD) link with conflicting MSVC C runtime libraries, so the Rust extension cannot build for win_amd64 today. Tracked as a follow-up; not a blocker for Linux+macOS. - headroom-e2e-setup: composite action now sets up Rust toolchain + Swatinem/rust-cache before `pip install -e .[proxy]`. - eval.yml, publish.yml, rust.yml: same pattern — rust toolchain before install. rust.yml's wheels job builds from root pyproject.toml (no more `-m crates/headroom-py/Cargo.toml`). - e2e/init/Dockerfile, e2e/wrap/Dockerfile: install rust + maturin in the build stage; copy `crates/` + workspace `Cargo.toml/lock` so the install can build the extension. Dropped `HEADROOM_REQUIRE_RUST_CORE=false` from wrap-e2e — the image now ships the full Rust core. - Dockerfile (main): simplified — no more Layer 2/3 dance with `headroom-core-py` install + symlink. Single `uv pip install` builds + installs everything. - .devcontainer/Dockerfile: rust toolchain + libssl-dev + maturin added so `uv sync` builds the extension inside the devcontainer. ## Lockfile + script - uv.lock: regenerated. No `headroom-core-py` entries remain. - scripts/build_rust_extension.sh: simplified from a symlink-into-tree workaround to a thin wrapper around `pip install -e .`. The maturin build-backend handles placement automatically. ## Local validation (all green on macOS aarch64) 1. Clean venv `pip install -e .` → `from headroom._core import …` works. 2. `maturin build --release` → 13.8 MB wheel, 336 files including `headroom/_core.cpython-311-darwin.so` (32 MB cdylib) and `headroom/dashboard/templates/dashboard.html`. 3. `pip install <wheel>` in fresh venv → import works. 4. Wheel contents verified via `unzip -l`. 5. `pytest tests/test_transforms/test_diff_compressor.py` — 29 passed. 6. `pytest tests/test_relevance.py` — 30 passed. 7. `cargo build --workspace` + `cargo test --workspace` — all green. 8. `make ci-precheck` — 176 Python tests + Rust + commitlint green. ## Migration notes Users on `pip install headroom-ai` get the Rust core automatically (linux + macos wheels). sdist installs require rust toolchain available locally — pip will build via maturin. Closes #355 Supersedes #357 (workarounds-based fix abandoned in favor of architectural fix) |
||
|
chopratejas
|
4bf559d5b5 |
fix(ci): install patchelf for maturin wheel-link repair
The Release workflow's multi-arch publish-docker job failed after
78 minutes of QEMU-emulated arm64 cargo compilation. Maturin's
wheel-link repair step needs `patchelf` to bundle external
shared libraries (libssl.so.3, libcrypto.so.3, libzstd.so.1)
into the wheel and rewrite their RPATH:
🔗 External shared libraries to be copied into the wheel:
libssl.so.3 => /usr/lib/aarch64-linux-gnu/libssl.so.3
libzstd.so.1 => /usr/lib/aarch64-linux-gnu/libzstd.so.1.5.7
libcrypto.so.3 => /usr/lib/aarch64-linux-gnu/libcrypto.so.3
💥 maturin failed
Caused by: Failed to execute 'patchelf', did you install it?
Compounding chain:
1. PR #350 added pkg-config + libssl-dev to unblock the cargo build
(openssl-sys couldn't find OpenSSL headers).
2. That made Cargo dynamically link to libssl.
3. Maturin then needs patchelf to rewrite the wheel's RPATH so the
bundled .so references resolve at runtime.
4. patchelf was never installed → fail.
Why this didn't surface in PR CI: docker-native-e2e builds only
the host platform (amd64). The Release workflow's docker-bake
builds linux/amd64 + linux/arm64 via setup-qemu-action, and the
arm64 emulation chain hits the patchelf path (different bundling
heuristic from amd64).
Follow-up that's NOT in this hotfix:
The 78-minute QEMU compile is the bigger structural issue. Switching
the Release workflow to native arm64 runners (`runs-on:
ubuntu-24.04-arm`) would cut that to ~5 min. Filing separately.
Run that failed: 25268839539
|
||
|
Tejas Chopra
|
f34e0c2e8b
|
Merge pull request #353 from chopratejas/realign-C1-rust-sse-parser
fix: C1 — byte-level SSE parser + state machines |
||
|
chopratejas
|
ddc6f6ceb0 |
fix: C1 — byte-level SSE parser + state machines
Foundation of Phase C. Delivers:
* Byte-level SSE framing (bytes::Bytes / BytesMut) with UTF-8
decoded only at \n\n event boundaries — no per-chunk decode,
no errors=ignore data loss across TCP reads.
* Three provider state machines:
- Anthropic: blocks keyed by index, all delta types
(text/thinking/input_json/citations/signature) preserved
byte-equal.
- OpenAI Chat: ToolCallState concatenation, refusal field,
include_usage final chunk handling.
- OpenAI Responses: items keyed by id (not position) for
out-of-order completion; full event coverage.
* State machine runs in parallel with byte-passthrough via a
tokio::spawn task fed by a bounded mpsc — clients see raw
bytes immediately; telemetry populates without blocking.
Retires P1-8, P1-9, P1-14, P1-15, P1-17, P4-48 in the Rust path
(Python A8 hotfix preserved as fallback until Phase H).
Per-PR-C1 plan: REALIGNMENT/05-phase-C-rust-proxy.md.
|
||
|
Tejas Chopra
|
9266ea7111
|
Merge pull request #350 from chopratejas/realign-phase-AB-cache-safety-and-live-zone
fix: Phase A+B realignment — cache safety + live-zone-only compression + production hotfixes |
||
|
chopratejas
|
6b15802c0b |
fix(ci): A0 verify must run outside /build cwd
The build-stage verify kept failing in PR #350 CI with
"ModuleNotFoundError: No module named 'headroom._core'" even after
the install order was correct. Diagnostic dump (commit
|
||
|
chopratejas
|
28a4883421 |
chore(ci): dump site-packages state before A0 build verify
Diagnostic step in the Dockerfile builder: list site-packages/headroom/ contents, run pip show -f on both headroom-core-py and headroom-ai, print sys.path and headroom.__path__ before the import-verify. Lets us see exactly what's on disk when A0's build-time verify keeps failing in PR #350 CI. Will be removed once the wheel install order issue is diagnosed. |
||
|
chopratejas
|
ce3b2f0b0b |
fix(ci): regenerate uv.lock against public PyPI (was Netflix-internal)
The validate × 3 devcontainer CI failures were NOT environmental —
they were caused by this branch.
Root cause: commit
|
||
|
chopratejas
|
f1aa12cebf |
fix(ci): reorder Dockerfile so wheel installs before headroom-ai
PR #350 CI: docker-native-e2e's wheel install succeeded but the build-stage verify (`from headroom._core import hello`) failed with `ModuleNotFoundError: No module named 'headroom._core'`. Same failure mode the customer hit in production (Finding #2) — but in CI we have the full layer trace. Root cause: the headroom-core-py wheel claims ownership of both `headroom/__init__.py` (stub from maturin's python-source layout) AND `headroom/_core.cpython-*.so`. The previous Dockerfile installed headroom-ai FIRST (which laid down the real `headroom/` tree), then the wheel SECOND with `--force-reinstall`. pip's --force-reinstall uninstalls the wheel's previously installed files before reinstalling — but the wheel's stub `__init__.py` had already overwritten headroom-ai's at first install. Net result: pip deleted `headroom/__init__.py` and `headroom/_core.so` ownership records got into a state where the .so wasn't present after the install. Fix: swap the order. Install the wheel first (lays down stub `__init__.py` + `_core.so`), then install headroom-ai (overwrites the stub with the real `__init__.py` and adds the rest of the `headroom/` tree). `_core.so` survives because headroom-ai doesn't claim ownership of it. Drop `--force-reinstall` from the wheel step since nothing is installing the wheel before it. This is the exact failure A0 was designed to catch — a deployment that ships without `_core` working. CI is now serving as a regression gate for the production install path. The remaining 3 PR check failures (validate × 3 / Dev Containers) are environmental: the runner's PyPI mirror (`pypi.netflix.net`) times out fetching `cuda-bindings==12.9.4` / `nvidia-cuda-cupti-cu12==12.8.90` / `safetensors==0.7.0`. These come from `headroom-ai[dev]` → `sentence-transformers` → `torch` → CUDA deps. Not caused by the realignment branch; the post-create script needs a `--extra dev-light` profile or the mirror needs the packages cached. Tracking separately. |
||
|
chopratejas
|
55dfc19e1d |
fix(ci): unblock A0 Docker e2e — install pkg-config + opt out wrap-e2e
Two CI failures introduced by Hotfix-A0's deployment-stage smoke test: 1. docker-native-e2e: the new maturin step in the builder stage failed with "Could not find openssl via pkg-config". The workspace transitively depends on `openssl-sys` (via reqwest's native-tls path in some dep chain). The previous Dockerfile only installed `build-essential`/`g++`/`curl`/`ca-certificates` — enough for the proxy binary build because cached target/ artefacts already had openssl-sys compiled, but the fresh maturin invocation hits a cold build and needs the dev headers. Add `pkg-config` + `libssl-dev`. 2. docker-wrap-e2e: this image is a `node:22-bookworm` base that installs headroom in editable mode for CLI-routing-only tests (aider, codex, openclaw via the wrap subcommand). It deliberately does NOT build the Rust extension. After A0, the proxy `lifespan` startup refuses to start when `headroom._core` can't import — so the wrap-e2e proxy port never opens, the harness's /health check times out, and the test fails. The wrap-e2e scope doesn't cover compression behaviour, so set `HEADROOM_REQUIRE_RUST_CORE=false` to start in degraded Python-only mode. Compression is exercised end-to-end by the smoke-test and docker-native-e2e jobs which build via the main Dockerfile. The remaining 3 PR check failures (validate * 3) were transient PyPI download failures (`nvidia-cuda-cupti-cu12==12.8.90`, `safetensors==0.7.0`) — unrelated to the realignment branch; they need a re-run, not a code change. |
||
|
chopratejas
|
cf5a71571c |
fix(security): allowlist GitGuardian-flagged test fixtures
GitGuardian flagged two strings on PR #350 as leaked secrets. Both are synthetic fixtures, NOT real credentials: 1. tests/test_cache_aligner_detector_only.py:215 — the canonical `eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxIn0.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c` JWT (header `{"alg":"HS256"}`, payload `{"sub":"1"}`) used to verify our `detect_volatile_content` recognises JWT-shaped strings. 2. tests/test_realignment_live_multi_turn.py:1091 — Anthropic-shaped tokens whose payloads literally contain "fixture" (`sk-ant-api03-payg-fixture`, `sk-ant-oat01-oauth-fixture`, `sk-ant-api03-payg-bearer-fixture`). Used to assert the auth-mode classifier routes PAYG / OAuth headers correctly. No live API call is ever made with these tokens — the test only inspects header shape. Two-layer remediation: * `.gitguardian.yaml` (new) — explicit allowlist with the literal match strings, each tagged with the file it lives in and the rationale. Anything else GG flags should be treated as a real incident; this file is the audit trail. * Inline `# ggignore` + `# noqa: S105` comments on each fixture line so a reviewer reading the test in isolation sees the intent without having to cross-reference the config. Per-feedback memory: secrets are routed via `.env`; the user's keys were never in chat or version control. These rows document the classifier-sweep false positive without weakening the detection rule. |
||
|
chopratejas
|
438620b19e |
Merge remote-tracking branch 'origin/main' into realign-phase-AB-cache-safety-and-live-zone
# Conflicts: # .claude-plugin/marketplace.json # .github/plugin/marketplace.json # plugins/headroom-agent-hooks/.claude-plugin/plugin.json # plugins/headroom-agent-hooks/.github/plugin/plugin.json |
||
|
chopratejas
|
6aacd4805a |
fix: A9 — tag protector discards wrap on placeholder loss
When a placeholder is lost during compression, restore_tags now
discards the wrap rather than appending the original tag at the
trailing edge of the output. The old "append" fallback emitted
malformed XML — an opening tag with no body and no closing tag —
on ~350 production requests over 9 days. Per the proxy log
findings, the corruption pattern was `compressed-stuff <tag>`,
which downstream models interpret as a truncated message.
Concrete changes:
* `crates/headroom-core/src/transforms/tag_protector.rs`:
- `restore_tags` no longer accumulates `tail_appends`. Lost
placeholders are silently dropped from the output bytes.
- New `restore_tags_with_request_id` entry point threads an
optional request id into the structured ERROR log so the
proxy layer can wire request context end-to-end. PyO3 binding
keeps the existing 2-arg signature (no Python caller has a
request id today).
- `tag_lost_warn` is replaced by `tag_lost_error`. Severity
moves from WARN to ERROR with structured fields
(`event=tag_protector_placeholder_lost`, `tag_preview`,
`compressed_length`, `action=discarded_wrap`, optional
`request_id`) so operators can alert on the corruption rather
than have it disappear into a WARN line.
- `parse_tag_at` gained a bounds check after consuming a
leading '/' — proptest discovered an OOB on input `</`.
- The old `restore_lost_placeholder_appended` test (which
pinned the broken behavior) is replaced with three positive
tests: wrap-discard, idempotence on full loss, and
partial-loss-keeps-present-drops-lost.
- New proptest suite enforces three invariants over arbitrary
inputs: no introduced asymmetry, idempotence on full
placeholder loss, and no orphan-byte injection.
* `headroom/transforms/tag_protector.py`: docstring updated
to document the discard-wrap semantics — the prior text
("appended on the trailing edge") is now incorrect.
* `tests/test_tag_protector_invariant.py` (new): Python-side
invariant suite that exercises the same three properties
end-to-end through the public Python API. Uses a deterministic
seeded random walk (no `hypothesis` dependency) so CI is stable
and reproducible.
* `tests/test_transforms/test_tag_protector.py`: replaces the
broken-behavior test with the new wrap-discard semantics.
Per-finding-#3: ~/Desktop/HEADROOM_PROXY_LOG_FINDINGS_2026_05_03.md.
|
||
|
chopratejas
|
00ab1ea74d |
fix: A0 — fail-loud rust core deployment smoke test
Production incident (Finding #2 of HEADROOM_PROXY_LOG_FINDINGS_2026_05_03.md): on this customer's deployment the Rust extension `headroom._core` was never installed into the runtime Docker image. Diff compression failed 54 times in a single day; "Optimization failed: ModuleNotFoundError" hit 379 times. The failure rate climbed every day and reached ~223/day on 2026-05-03 — effectively 100% of requests on the Rust path. Every Rust PR we'd merged (MessageScorer, ICM, DiffCompressor, etc.) was providing zero customer value because the module wasn't loadable at all. Root cause: the Dockerfile builder stage installed Python deps and the in-tree `headroom-ai` package but never ran `maturin build` for the `headroom-py` crate, so the runtime image shipped without `_core.so`. The Python proxy continued to start because the extension's absence is caught and routed through Python-only fallbacks that either silently no-op or raise per-request. This change makes that mode impossible by default: * `headroom.proxy.server._check_rust_core()` runs as the first step of the FastAPI lifespan. If the import fails it prints a structured diagnostic, logs `event=rust_core_missing`, and calls `sys.exit(78)` (sysexits.h `EX_CONFIG`). Process supervisors (systemd / k8s / docker) treat this as a deliberate config error and stop restart loops. * `HEADROOM_REQUIRE_RUST_CORE=false` is the explicit opt-out for Python-only `pip install -e .` developer flows; lifespan logs `event=rust_core_disabled` and continues. Any other value (including unset) keeps the fail-loud default. * `/health` now surfaces `rust_core: "loaded" | "disabled" | "missing"` (plus `rust_core_error` when non-loaded) so operators can alert on the degraded state rather than discovering it via a customer ticket. * `scripts/build_rust_extension.sh` is the single dev-time path: build → install → import-verify with the same `hello()` marker the lifespan checks. Failures are loud at every step. * `Makefile` exposes the script as `make verify-rust-core`. * `Dockerfile` now installs `rustup` + `maturin`, builds the wheel from `crates/headroom-py`, force-installs it into site-packages, and runs the same `hello()` import-verify in the build image so a broken build fails the docker-build, not the next runtime restart. Tests: * `tests/test_rust_core_smoke.py` pins all four contracts: - `_core.hello()` returns `"headroom-core"` - missing extension + default env → `SystemExit(78)` - missing extension + opt-out env → lifespan starts, `/health` returns `rust_core: "disabled"` with the underlying error - present extension + default env → `("loaded", None)` Per-finding-#2: ~/Desktop/HEADROOM_PROXY_LOG_FINDINGS_2026_05_03.md. |
||
|
chopratejas
|
dcbc921d63 |
fix: Wave 3 — multi-turn live integration tests for A+B realignment
Adds tests/test_realignment_live_multi_turn.py with 9 OPT-IN live tests
that validate the load-bearing claims of the Phase A+B megamerge against
real upstream APIs (Anthropic, OpenAI, Gemini). Each test maps to one or
more realignment PRs:
1. test_anthropic_cache_hit_across_two_turns — A2/A6/E
Identical cache_control'd system+messages on two turns must
eventually produce cache_read_input_tokens > 0. Guards the cache
hot zone invariant (I2): proxy must not mutate frozen prefix bytes.
Uses a bounded retry loop (max 4 attempts) to absorb Anthropic's
eventually-consistent prompt-cache write latency without masking
a real "proxy broke cache stability" regression.
2. test_anthropic_cache_stable_when_live_zone_compresses — B2/B3
Turn 2 mutates only the LATEST user content (8KB+ JSON tail);
cache_read on turn 2 must still be > 0 AND the proxy must emit
compression headers — proving the live-zone block dispatcher
ran on the new tail without disturbing the cached prefix.
3. test_anthropic_cache_control_passthrough_byte_faithful — A3/A4
Wraps proxy._retry_request to snapshot the upstream-bound body
and assert cache_control on system blocks survives verbatim,
and user content is not flattened from list to string form.
4. test_openai_chat_completions_multi_turn_through_proxy — A8/B
Three-turn conversation through /v1/chat/completions; each
turn returns valid content, prior assistant turns survive in
the messages list (proxy doesn't drop them).
5. test_openai_streaming_sse_chunks_arrive_in_order — A8 (SSE wire)
Streams /v1/chat/completions; asserts each event is
'data: ...\\n\\n', terminator is 'data: [DONE]\\n\\n',
reassembled content non-empty, no malformed events.
6. test_gemini_multi_turn_through_proxy — Gemini reach
Two-turn conversation through native
/v1beta/models/{model}:generateContent. Proves Gemini handler
wiring stayed intact through the megamerge.
7. test_ccr_marker_round_trip_live — B7 (CCR)
Pre-populates compression_store with a fixture entry, embeds
a CCR marker on a tool_result, verifies (a) headroom_retrieve
tool is injected into the upstream tools array (PR-B7
always-on), and (b) /v1/retrieve returns the original bytes
by hash with all rows intact. Pre-populating the Python store
(vs. driving SmartCrusher's internal Rust store) matches the
established pattern in tests/test_proxy_ccr.py and exercises
the surface served by /v1/retrieve.
8. test_memory_tail_injection_does_not_modify_system_prompt_live — B6/A2
Spins up a memory-enabled proxy with MemoryMode.AUTO_TAIL,
seeds LocalBackend, captures upstream-bound body. Asserts:
(a) system prompt byte-identical to input; (b) memory text
lands on latest user message tail; (c) earlier messages
untouched. Guards the live-zone-only injection contract.
9. test_classify_auth_mode_routes_payg_vs_oauth — Phase F-prep / B5
NOT a live API call. Sends three header shapes through the
proxy (x-api-key=..., Bearer sk-ant-oat01-..., Bearer
sk-ant-api03-...), captures dispatcher headers via a wrap on
_retry_request, and asserts the canonical auth-mode classifier
maps each correctly. Codifies the Phase F contract.
Conventions:
* file-level pytestmark = pytest.mark.live → excluded by default
via 'pytest -m "not live"'. Adds a 'live' marker registration in
pyproject.toml's [tool.pytest.ini_options].markers.
* each test skipif's on the relevant API key — no silent fallbacks,
no real-API runs against fake keys.
* uses tests/_dotenv.py helpers (load_env_overrides + autouse_apply_env)
rather than re-implementing env loading.
* model IDs and thresholds live in a top-of-file LIVE_CONFIG dict
(no hardcodes); Anthropic primary/fallback resolves at runtime per
key entitlement.
* assertions are direction-only (cache_read > 0, tokens_after <=
tokens_before) — never tied to upstream pricing/tokenizer drift.
* shared module-scoped TestClient fixture for performance; CCR and
memory tests build dedicated proxies for their config-specific paths.
Verification:
* pytest tests/test_realignment_live_multi_turn.py -v
→ 9 passed, 0 skipped, 0 failed in ~25s (with all keys set)
* pytest -m "not live" --tb=short -q
→ 4694 passed, 265 skipped, 9 deselected — same baseline as today
* make ci-precheck → green (rust + python + commitlint)
Per-realignment-plan: REALIGNMENT/04-phase-B-live-zone.md.
|
||
|
chopratejas
|
2fb905fdb0 |
fix: integrate B6+B7 — fix cross-test contamination + injector mock parity
Two follow-ups surfaced when B6 and B7 were merged onto the megamerge branch and the full suite ran: 1. tests/test_proxy_anthropic_cache_stability.py PR-B7 added `injector.scan_for_markers(optimized_messages)` to the Anthropic handler so the always-on tool-registration logic can see detected hashes for the current request. The two pre-existing `_FakeInjector` mocks (`test_ccr_system_instruction_injection_disabled_*` and `test_ccr_tool_injection_disabled_*`) didn't implement that method. Added a no-op `scan_for_markers` returning [] to both mocks — matches the real injector's contract for the not-yet-compressed request shape these tests exercise. 2. tests/test_memory_tool_mode.py::test_tool_mode_skip_emits_structured_log The B6 caplog assertion passed in isolation but failed in the full suite. Root cause: when an earlier test triggers proxy startup, `_setup_file_logging` flips `headroom.propagate=False` and attaches a RotatingFileHandler to the headroom logger. caplog captures via propagation to root, so log records stop reaching it. The conftest autouse fixture that resets `propagate=True` before every test gets shadowed by fixture-ordering edge cases. Principled fix: attach `caplog.handler` directly to `headroom.proxy.memory_handler` for the duration of the test so the capture is independent of propagation state. Restore the original level + remove the handler in `finally` to keep the test hermetic. Both B6 and B7 cherry-picks themselves are unmodified. This commit only adjusts test harness code so the pre-existing mocks/capture stay consistent with the new code paths. |
||
|
chopratejas
|
00902b8fea |
fix: B7 — CCR hardening: persistent backends + always-on tool
P2-25, P2-26: CCR (Compress-Cache-Retrieve) used an in-memory store
that fragmented across uvicorn workers and was wiped on restart, and
the `headroom_retrieve` tool was registered/unregistered per-request
based on whether the latest body happened to contain compression
markers — every flip busted the prompt cache. Both are sticky
side-channels: once a session has done CCR, the tool list bytes and
the retrieval store must stay stable. This PR fixes both.
Rust:
* Split `ccr.rs` into `ccr/` with `backends/` submodule
(`in_memory.rs`, `sqlite.rs`, `redis.rs` cfg-gated).
* `SqliteCcrStore` (production default): WAL mode, prepared upsert,
lazy TTL purge on read, persistent across worker restarts and
shareable across workers on the same host via SQLite file locking.
* `RedisCcrStore` (cfg-gated behind `feature = "redis"`): SETEX with
startup PING smoke-test, no key-prefix collision risk, no sticky
session required at the LB.
* `CcrBackendConfig::{InMemory, Sqlite, Redis}` + `from_config(...)`
factory — every init failure surfaces (no silent fallback per
`feedback_no_silent_fallbacks.md`).
* `ccr::compute_key` (BLAKE3 → first 24 hex chars) and
`ccr::marker_for("HASH") -> "<<ccr:HASH>>"` centralize the hash +
marker format; one definition for the live-zone dispatcher and the
Python regex (`headroom/ccr/tool_injection.py:211`).
* `compress_anthropic_live_zone_with_ccr` accepts
`Option<&dyn CcrStore>`. When wired, every accepted compression
puts the original bytes into the backend and appends `<<ccr:HASH>>`
to the compressed string. The token-validation gate runs on the
marker-augmented string so the `compressed_tokens >=
original_tokens` rejection stays honest.
Python:
* `SessionCcrTracker` + `apply_session_sticky_ccr_tool` mirror the
PR-A7 `SessionToolTracker` / `apply_session_sticky_memory_tools`
pattern: once a session has done CCR, every subsequent request
injects the recorded golden tool-definition bytes. Tool list bytes
are byte-stable across turns (snapshot test pins them).
* `headroom/ccr/tool_injection.py::inject_tool_definition` accepts a
new `session_has_done_ccr` kwarg per the PR-B7 spec change at line
302-328. The legacy per-request path stays intact for callers that
don't yet thread a session id (e.g. Google handler).
* Anthropic + OpenAI handlers route their CCR tool-list updates
through `apply_session_sticky_ccr_tool`, keyed off the existing
`session_tracker_store.compute_session_id(...)` plumbing.
Backend selection model: `CcrBackendConfig::Sqlite { path }` is the
production default — single host, persistent, multi-worker safe with
sticky session. `CcrBackendConfig::Redis { url }` is the multi-host
scale-out option — no stickiness needed. `InMemory` is for tests
and single-worker dev only. RUST_DEV.md "Multi-worker deployment —
CCR fragmentation" rewritten around this matrix.
Tests:
* `crates/headroom-core/tests/ccr_backends.rs` — 7 tests covering
SQLite round-trip, TTL purge, proxy-restart survival, cross-backend
byte-equal keys, `from_config` paths, and the no-redis-feature
loud-failure check (+ 2 redis tests gated behind the feature).
* `crates/headroom-core/tests/live_zone_ccr.rs` — confirms
`<<ccr:HASH>>` marker injection, store population, and
no-marker-when-no-store invariants end-to-end.
* `tests/test_ccr_tool_always_on.py` — 12 tests pinning the
always-on behaviour, session/provider isolation, LRU bound, no-
session-id fallback, and (per-acceptance-criterion) the byte-stable
tool-definition snapshot.
Per-PR-B7 plan: REALIGNMENT/04-phase-B-live-zone.md.
|
||
|
chopratejas
|
2ee05774b9 |
fix: B6 — memory injection moves to live-zone user-tail
PR-A2 locked the system prompt and routed Anthropic memory injection to the
latest non-frozen user turn. PR-B6 finishes the job: every provider handler
that auto-injects memory context now does so via the live-zone tail, and a
new MemoryMode enum makes the routing explicit and configurable.
What changed
------------
* New `MemoryMode` enum in `headroom/proxy/memory_handler.py` with two
values:
- `AUTO_TAIL` (default) — retrieval results auto-append to the latest
user message. The cache hot zone (system / instructions / frozen
prefix) is never mutated.
- `TOOL` — auto-injection is disabled entirely. The model must call
`memory_search` to retrieve. Memory is opt-in and visible.
* `MemoryConfig.mode: MemoryMode = MemoryMode.AUTO_TAIL` propagates into
`search_and_format_context`, which now short-circuits to `None` in `TOOL`
mode. This is the single chokepoint that gates every provider — Anthropic
/v1/messages, OpenAI /v1/chat/completions, OpenAI /v1/responses, and
Gemini all funnel through it, so flipping a deployment to tool mode does
not require auditing every handler.
* New `MemoryHandler._append_to_latest_user_tail(messages, context_text,
provider=..., frozen_message_count=...)` static helper provides the unified
tail-append entry point and dispatches to the existing provider-specific
helpers (`AnthropicHandlerMixin._append_context_to_latest_non_frozen_user_turn`
for Anthropic, `append_text_to_latest_user_chat_message` for OpenAI).
* Gemini handler swapped from auto-prepending memory as a system message
(the old P2-24 cache-hot-zone mutation pattern) to using
`_append_to_latest_user_tail(provider="openai")`.
* `ProxyConfig.memory_mode: Literal["auto_tail", "tool"] = "auto_tail"`
surfaces the mode for deployment configuration. Server constructs the
enum via `MemoryMode(config.memory_mode)` and raises loudly on unknown
values (no silent fallback).
* OpenAI Chat Completions, OpenAI Responses, and Anthropic handlers were
already routing to the live-zone tail via PR-A2/A3 — no code change
needed beyond inheriting the `TOOL`-mode skip from the chokepoint.
Tests
-----
* `tests/test_memory_auto_tail.py` (6 tests):
- `test_memory_appears_in_latest_user_message_tail` — Anthropic shape.
- `test_memory_appears_in_latest_user_message_tail_openai_shape` —
OpenAI string + list-content shapes.
- `test_memory_does_not_modify_system_or_tools` — system prompt and
tools list are never touched; frozen-prefix tail is a no-op.
- `test_same_query_byte_identical_across_runs` — two independent runs
with identical inputs produce byte-identical mutated message lists
(determinism gate).
- `test_default_mode_is_auto_tail` — fresh `MemoryConfig` defaults to
`AUTO_TAIL`.
- `test_unknown_provider_raises` — invalid provider strings raise
loudly per the no-silent-fallback policy.
* `tests/test_memory_tool_mode.py` (4 tests):
- `test_tool_mode_skips_auto_injection` — `search_and_format_context`
returns `None` and the backend is never queried.
- `test_tool_mode_skip_emits_structured_log` — skip emits the
`event=memory_mode_skip` log line for routing-decision auditability.
- `test_auto_tail_mode_does_query_backend` — inverse contrast pinning
down that AUTO_TAIL still works end-to-end while TOOL skips.
- `test_tool_mode_enum_value_is_stable` — string round-trip is pinned
so deployment configs do not drift on rename.
Determinism
-----------
Tests stub the backend with a fixed, ordered result set so the byte-identical
assertion isolates the tail-injection layer from upstream search non-
determinism. The vector-search layer itself (LocalBackend / HNSW) is
deterministic per-process for the same inputs but has thread-scheduling
variability across processes; per the realignment plan, request-time
determinism is guaranteed by the formatter and the tail-append helpers
(this PR's responsibility), and the backend layer's determinism stays
out-of-scope for B6.
Per-PR-B6 plan: REALIGNMENT/04-phase-B-live-zone.md.
|
||
|
chopratejas
|
6819b7e5e5 |
fix: B5 — TOIN observation-only refactor + per-tenant aggregation key
Retire the request-time hint API. PR-B5 splits TOIN into two phases:
1. Observation: TOIN keeps recording compressions/retrievals at runtime,
but `get_recommendation()` is deprecated and now returns None.
2. Publish-then-load: the new `headroom.cli.toin_publish` CLI walks the
on-disk store and emits `recommendations.toml`. The Rust proxy reads
that file once at startup via `transforms::recommendations` and
exposes `get(auth_mode, model, structure_hash) -> Option<&Rec>`.
PR-F3 will wire the loader into the live-zone dispatcher.
Per-tenant aggregation: `_patterns` is now keyed by
`(auth_mode, model_family, sig_hash)` so PAYG/OAuth/subscription tenants
no longer share buckets. Callers that don't supply auth/model land in the
`("unknown", "unknown", sig_hash)` slot. Added `_make_pattern_key` helper
+ updated tests that previously indexed by raw `structure_hash`.
AuthMode is canonical in `transforms::live_zone`; `transforms::recommendations`
re-exports it (no duplicate enum). Live-zone enum gained `Unknown`,
`as_str()`, and `Hash` derive to serve recommendations callers without a
second source of truth.
Why: per-request hint calls coupled output to mutable TOIN state, breaking
prompt-cache stability across runs (P2-27, P5-56). Pulling advice into a
startup-published TOML keeps per-request output deterministic and lets the
deploy pipeline gate publication independently of proxy uptime.
Per-PR-B5 plan: REALIGNMENT/04-phase-B-live-zone.md.
|
||
|
chopratejas
|
b3b3feff6f |
fix: B4 — token validation gate + per-content-type byte thresholds
Eliminate P3-33 / P3-34. Wraps every per-block compression in
the live-zone dispatcher with two new gates:
1. Per-content-type byte thresholds — pinned as `const` at the top
of `live_zone.rs` so the table is grep-able and reviewable in
one place. No magic numbers anywhere in the dispatch logic; a
`threshold_for(ContentType)` helper returns the value. Below
threshold → no compressor invoked, recorded as
`BlockAction::BelowByteThreshold { content_type, byte_count,
threshold_bytes }`. Thresholds:
- JSON-array tool_results: 1 KiB
- Build / log output: 512 B
- Search-result blocks: 1 KiB
- Git-diff blocks: 1 KiB
- Source code: 2 KiB (pinned for the future
Rust code-compressor port)
- Plain text: 5 KiB (pinned for Kompress wiring)
- HTML: 5 KiB (no compressor today)
2. Tokenizer-validated rejection — the byte-length proxy
(`compressed_bytes >= original_bytes`) is replaced with a
token-count check using `headroom_core::tokenizer::get_tokenizer`.
The dispatcher creates one tokenizer per request (model-aware
via the new `model: &str` parameter to
`compress_anthropic_live_zone`) and counts both the original
and compressed text. When `compressed_tokens >= original_tokens`
the candidate is rejected and the original bytes are kept.
`BlockAction::Compressed` and `BlockAction::RejectedNotSmaller`
gain `original_tokens` and `compressed_tokens` fields so the
proxy can log token-savings (the currency that actually matters
for prompt cache + provider billing) instead of bytes.
The proxy `live_zone_anthropic.rs` extracts `body["model"]` (or
falls back to `DEFAULT_MODEL = "claude-3-5-sonnet-20241022"` when
the field is missing — the chars-per-token estimator is calibrated
for the Claude family at 3.5 cpt) and threads it through. The
`Compressed` outcome now reports token counts from the manifest,
not byte counts, so the existing
`tokens_before / tokens_after` plumbing is suddenly accurate.
Tests added:
- `live_zone_thresholds.rs::below_threshold_no_compression_attempted`
— 200 B JSON array → `BelowByteThreshold` and `NoChange`.
- `live_zone_thresholds.rs::above_threshold_compression_attempted`
— 10 KB JSON array → byte-threshold gate clears and a compressor
runs (either `Compressed` or `RejectedNotSmaller`).
- `live_zone_token_validation.rs::compressed_more_tokens_falls_back`
— pathological input must not produce `Compressed` with
`compressed_tokens >= original_tokens`.
- `live_zone_token_validation.rs::compressed_fewer_tokens_accepted`
— well-formed JSON array of dicts → `Compressed` with strict
token shrinkage.
- Property test `live_zone_compression_token_count_non_increasing`
— for any well-formed body generated by `proptest`, the
dispatcher's emitted body has token-count <= input's token-count.
Pins the central PR-B4 invariant: the dispatcher never inflates
tokens.
Existing 12 unit tests in `live_zone.rs` and 6 integration tests
in `tests/live_zone_dispatch.rs` updated for the new field shape
and the `model` parameter; all pass. The diff-routing test's
fixture grew to 1.3 KiB so it clears the new GitDiff threshold
gate, exercising the dispatch path rather than short-circuiting.
Per-PR-B4 plan: REALIGNMENT/04-phase-B-live-zone.md.
|
||
|
chopratejas
|
2b55050a65 |
fix: B3 — wire type-aware compressors into live-zone dispatcher
Phase B step 3: replace PR-B2's no-op dispatcher with real per-block
compression. SmartCrusher / LogCompressor / SearchCompressor /
DiffCompressor are wired behind content-type detection. SourceCode
and PlainText remain no-op for now (Rust code-compressor port and
Kompress prose compressor land in follow-up work; they're explicit
TODOs in `dispatch_compressor`).
# What's wired
For each block in the latest user message (live zone):
| Detected type | Compressor | Strategy tag |
|---------------|------------------|--------------------|
| `JsonArray` | SmartCrusher | `smart_crusher` |
| `BuildOutput` | LogCompressor | `log_compressor` |
| `SearchResults` | SearchCompressor | `search_compressor` |
| `GitDiff` | DiffCompressor | `diff_compressor` |
| `SourceCode` | (no-op, Rust port pending) | |
| `PlainText` | (no-op, PR-B4 wires Kompress) | |
| `Html` | (no-op, no compressor) | |
Anthropic-specific block types (`tool_use`, `thinking`,
`redacted_thinking`, `compaction`) stay tagged `BlockAction::Excluded`
so they remain in the cache hot zone even when they appear in the
live-zone message.
# Cache-safety invariant — byte-range surgery
The PR replaces "deserialize → mutate → serialize" with byte-range
surgery: the dispatcher uses `serde_json::value::RawValue` borrowed
slices and pointer arithmetic to recover each block's exact byte
offset in the input buffer, then splices replacement bytes
in-place. Bytes outside any rewritten range are *literally copied*
from the input, never re-serialized.
The new integration test
`crates/headroom-core/tests/live_zone_dispatch.rs::byte_fidelity_outside_compressed_block`
pins this in CI: SHA-256 of `body[..block_start]` and
`body[block_end..]` must equal the input's, AND the block must
shrink by >2× on a 50 KB JSON-array tool_result.
# Provider scope (Phase B is Anthropic-only)
The entry point is renamed `compress_live_zone` →
`compress_anthropic_live_zone` to make scope explicit. OpenAI Chat
Completions, OpenAI Responses, and Google Gemini each need their
own dispatcher because the request shapes diverge: OpenAI puts
tool results in `role: "tool"` messages (not nested in user),
Responses uses `input` with `function_call_output` items, Gemini
uses `contents`/`parts`/`function_response`. Phase C
(`REALIGNMENT/05-phase-C-rust-proxy.md`) introduces those
dispatchers; they share `LiveZoneOutcome`, `BlockAction`,
`CompressionManifest` and the per-content-type compressor backend
from this module.
# BlockAction taxonomy (replacing PR-B2's `NoOpSkeleton`)
- `Compressed { strategy, original_bytes, compressed_bytes }` —
compressor ran and produced strictly smaller output; spliced in.
- `RejectedNotSmaller { strategy, original_bytes, compressed_bytes }`
— compressor ran but didn't shrink; original kept. PR-B4 swaps
this byte-length proxy for a tokenizer-validated count.
- `CompressorError { strategy, error }` — compressor failed loudly.
Per project memory `feedback_no_silent_fallbacks.md`, surfaced in
the manifest; proxy logs warn-level and forwards original bytes
for that block; other blocks in the same body still compress.
- `NoCompressionApplied { content_type }` — content type has no
applicable compressor (PlainText, SourceCode, Html, Image,
Unknown). Replaces PR-B2's `NoOpSkeleton` as the default.
- `Excluded { reason }` — block intentionally outside live zone
(HotZoneBlockType, BelowFrozenFloor, AboveLiveZone).
# Sequential per-block dispatch (parallelism deferred)
Per-block compression is sequential in B3. Most requests have 1-3
blocks in the latest user message; the rayon/spawn_blocking
overhead approaches the savings below ~4 blocks. PR-B4 will add
async coordination per block (since token validation needs an
async hop anyway) — that's the natural place to add parallelism
guarded by a benchmark-driven threshold.
# Observability
The proxy log line gains the new fields when bytes are rewritten:
- `decision="compressed"`, `reason="live_zone_blocks_rewritten"`
- `body_bytes_in`, `body_bytes_out`, `bytes_freed`
- `live_zone_strategies` (Vec of unique strategy tags)
- `live_zone_block_original_bytes`, `live_zone_block_compressed_bytes`
The PR-B2 `decision="no_change"` arm is preserved with
`reason="no_block_compressed"`.
# Files
- `crates/headroom-core/src/transforms/live_zone.rs` (≈1100 LOC,
+900 from B2): byte-range surgery; `dispatch_compressor` switch;
`OnceLock` singletons for SmartCrusher / Log / Search / Diff;
expanded `BlockAction` enum.
- `crates/headroom-proxy/src/compression/live_zone_anthropic.rs`:
translates `LiveZoneOutcome::Modified` → `Outcome::Compressed`
with aggregated manifest counters.
- `crates/headroom-core/tests/live_zone_dispatch.rs` (NEW):
routing tests + 50 KB byte-fidelity invariant test.
- `crates/headroom-proxy/tests/integration_compression.rs`: log
contract updated to `reason="no_block_compressed"`.
# Acceptance
- `cargo build --workspace` + `clippy` + `fmt` green.
- `cargo test --workspace --exclude headroom-py`: 881 passed.
- 6 new integration tests in `live_zone_dispatch.rs`:
json/log/diff routing, source-code no-op, unknown no-op,
byte-fidelity (50 KB → >2× reduction with byte-equal envelope).
- Existing 12 unit tests in `live_zone.rs` still pass.
Per-PR-B3 plan: REALIGNMENT/04-phase-B-live-zone.md.
|
||
|
chopratejas
|
e190544c77 |
fix: B2 — live-zone block dispatcher skeleton
Phase B step 2 of the live-zone-only realignment. Replaces PR-A1's
unconditional "passthrough" stub with a real dispatcher that
inspects the Anthropic /v1/messages body, identifies the live zone
(latest user message at index >= frozen_message_count), and routes
each block to a per-type compressor. PR-B2 wires every per-type
compressor to a no-op, so the dispatcher returns
LiveZoneOutcome::NoChange on every call — bytes-in == bytes-out.
PR-B3+ replaces the no-ops with SmartCrusher, Log, Search, Diff,
and Code compressors.
Adds:
- crates/headroom-core/src/transforms/live_zone.rs — public API:
- `compress_live_zone(body, frozen_message_count, AuthMode)`
- `LiveZoneOutcome::{NoChange, Modified}`
- `CompressionManifest` with per-block outcomes (message_index,
block_index, block_type, BlockAction).
- `BlockAction::{NoOpSkeleton, Excluded { reason }}`. The
HOT_ZONE_BLOCK_TYPES list (`tool_use`, `thinking`,
`redacted_thinking`, `compaction`) excludes blocks even when
they appear in the latest user message.
- `AuthMode::{Payg, OAuth, Subscription}` — accepted but unused
in B2; PR-F2 wires the auth-mode gate.
- 12 unit tests pin: empty messages, no messages field, invalid
JSON, latest user message selection, frozen_count respect,
hot-zone block exclusion, string-shaped content, no user msg
in live zone, AuthMode no-op, NoChange contract, manifest
counters, frozen-count clamping.
- crates/headroom-proxy/src/compression/live_zone_anthropic.rs —
new entry point. `compress_anthropic_request` parses the body,
resolves frozen_count via `resolve_frozen_count` (PR-A4 helper),
dispatches via `compress_live_zone`, and returns
`Outcome::NoCompression` on PR-B2 success / `Outcome::Passthrough
{ reason: NotJson | NoMessages | ModeOff }` on body-shape /
policy issues. Six unit tests pin: mode_off short-circuit, no
messages field, invalid JSON, valid body NoCompression,
empty body, cache_control disabled.
Modifies:
- compression/mod.rs — re-exports `compress_anthropic_request` from
`live_zone_anthropic` instead of `anthropic`. The old anthropic
module is reduced to the `resolve_frozen_count` helper only
(not deleted, because its CacheControlAutoFrozen-policy gate is
reused).
- proxy.rs — passes `state.config.cache_control_auto_frozen` into
the dispatcher. Drops the obsolete "live_zone reserved for
Phase B" warning that PR-A1 emitted on every request.
- compression/anthropic.rs — pruned to the resolve_frozen_count
helper plus its tests. The PR-A1 passthrough stub
`compress_anthropic_request` is gone (live_zone_anthropic owns
the name now).
- config.rs — `compression_mode` doc updated to reflect the wired
dispatcher (no longer "reserved for Phase B").
- tests/integration_compression.rs — `compression_decision_logged`
pins the new log contract (`decision="no_change"`,
`reason="no_op_skeleton_pr_b2"`, plus manifest fields
`frozen_message_count`, `messages_total`, `live_zone_blocks`).
Asserts the obsolete Phase A warning is NOT emitted.
- proxy.rs no longer imports CompressionMode (only used inside the
retired warning).
Benchmark cleanup (B1 leftovers that surfaced now):
- benchmarks/proxy_mode_benchmark.py + claude_session_mode_benchmark.py:
drop `intelligent_context=False` arg from ProxyConfig (the field
was retired in B1; tests/test_proxy_mode_benchmark.py and
tests/test_claude_session_mode_benchmark.py imported these
factories and started failing).
- benchmarks/bench_transforms.py: delete TestRollingWindowBenchmarks
class; rewire TestTransformPipelineBenchmarks fixture without
RollingWindow.
- benchmarks/conftest.py: drop rolling_window_config fixture.
- benchmarks/run_benchmarks.py: drop the `window` suite + table
rows referencing RollingWindow.
Cache-safety invariant:
- PR-B2 dispatcher never mutates body bytes (no-op skeleton). The
proxy forwards the original buffered bytes byte-equal. Phase A's
SHA-256 fixtures pin this.
- `passthrough_mode_live_zone_currently_passthrough_byte_equal_sha256`
retitled comment to reflect the dispatcher being live but
no-op.
Acceptance:
- cargo build --workspace + clippy + fmt: green.
- cargo test --workspace --exclude headroom-py: all green
(777 + 12 new live_zone + 6 new live_zone_anthropic tests).
- pytest: 4678 passed, 240 skipped, 0 failed.
- Anthropic decision log includes manifest fields per the
observability contract documented in
REALIGNMENT/02-architecture.md.
Per-PR-B2 plan: REALIGNMENT/04-phase-B-live-zone.md.
|
||
|
chopratejas
|
967b0db439 |
fix: B1 — retire ICM, RollingWindow, scoring, relevance + dependents
Phase B step 1 of the live-zone-only realignment. Removes ~10K LOC of
"drop messages from history" machinery that became unreachable after
PR-A1 made `/v1/messages` a passthrough on the proxy. Live-zone-only
compression (PR-B2..B7) operates on content blocks within messages;
message-list mutation no longer happens in the pipeline.
Python deletes:
- headroom/transforms/intelligent_context.py (1077 LOC)
- headroom/transforms/rolling_window.py (395 LOC)
- headroom/transforms/progressive_summarizer.py (508 LOC)
- headroom/transforms/scoring.py (459 LOC)
- headroom/transforms/tool_crusher.py (338 LOC)
- 5 corresponding tests/test_transforms/* and tests/test_proxy_intelligent_context.py
Rust deletes:
- crates/headroom-core/src/context/* (manager, config, workspace,
candidate, ccr_drop, strategy/, mod) + safety.rs replaced
- crates/headroom-core/src/scoring/* (mod, score, scorer, traits, weights)
- MessageScorerComparator from crates/headroom-parity (PR #338/#343
becomes deletable; sunk cost stays sunk)
- 13 message_scorer fixtures + record_message_scorer.py
Rust adds (move + rewrite):
- crates/headroom-core/src/transforms/safety.rs — `tool_pair_indices`
preserves the OpenAI/Anthropic tool_use ↔ tool_result pairing rule
the live-zone dispatcher (PR-B2) needs. No IcmConfig dependency.
Surface refactors:
- HeadroomConfig: drop `tool_crusher`, `rolling_window`,
`intelligent_context` fields; hoist `output_buffer_tokens` to top
level (used by client.py).
- ProxyConfig: drop `intelligent_context*` fields.
- `headroom wrap` proxy server: retire IntelligentContextManager
and RollingWindow imports + branch; pipeline is CacheAligner →
ContentRouter (smart_routing) or CacheAligner → SmartCrusher
(legacy).
- CLI: drop `--no-intelligent-context`, `--no-intelligent-scoring`,
`--no-compress-first` flags.
- LangChain memory integration: rename `_apply_rolling_window` →
`_apply_compression`, drop RollingWindowConfig dep. Threshold is
now advisory — B6 will rework the contract.
- TransformPipeline.create_pipeline now takes only cache_aligner_config.
- headroom/__init__.py + headroom/transforms/__init__.py: strip
exports of deleted symbols.
Bug fixes uncovered by full pytest sweep:
- providers/copilot/wrap.py: `environ or os.environ` collapsed
empty-dict to falsy → callers passing `environ={}` accidentally
pulled from os.environ. Use `environ if environ is not None else
os.environ`.
Test correctness fixes:
- _DummyAnthropicHandler._retry_request gains **_kwargs to match
the real handler signature post-A8.
- test_ws_http_fallback extracts JSON from `content=` (post-A3
byte-faithful) rather than the obsolete `json=` kwarg.
- test_ccr_response_handler_extra fixture joins SSE events with
`\n\n` per spec (post-A8 byte-buffer parser requirement).
- test_proxy_responses_phase_preservation: capture via direct
handler attached to the named logger, so the assertion is
order-independent (proxy `_setup_file_logging` flips
`headroom.propagate=False` once any earlier test triggers it).
- conftest.py autouse fixture resets `headroom.propagate=True`
before each test as a defensive measure for the same pollution.
- test_wrap_copilot_translated_backend_still_requires_byok:
monkeypatch.delenv every provider key so the BYOK error
actually fires.
- test_native_installers: skip when system bash < 4.3 (macOS ships 3.2).
- TestGeminiEmbedContent / TestGeminiBatchEmbedContents:
pytest.mark.skip — proxy currently has no :embedContent route;
feature gap, not regression.
Acceptance:
- cargo build --workspace + cargo clippy + cargo fmt --check: green.
- cargo test --workspace --exclude headroom-py: 777 passed.
- pytest: 4892 passed, 240 skipped, 0 failed.
- git grep returns only intentional comments referencing the deletion.
Per-PR-B1 plan: REALIGNMENT/04-phase-B-live-zone.md.
|
||
|
chopratejas
|
4073dcd231 |
fix(proxy): route Codex subscription /backend-api/* catchall to chatgpt.com
Codex CLI in subscription mode polls /backend-api/wham/usage, fetches agent identity JWKS from /backend-api/wham/agent-identities/jwks, and hits other auxiliary /backend-api/* endpoints during startup. The HTTP catchall in _select_passthrough_base_url ignored ChatGPT auth and routed all unmatched paths to api.openai.com, which 404s on every backend-api path. Codex interprets that as "session invalid" and refuses subscription auth. Add a single branch at the top of _select_passthrough_base_url: when _resolve_codex_routing_headers reports ChatGPT auth (explicit ChatGPT-Account-Id header or JWT with chatgpt_account_id claim), return https://chatgpt.com so the catchall forwards to the right host. No-op for Anthropic (x-api-key, no JWT), Gemini (x-goog-api-key, no JWT), OpenAI API key (sk- tokens fail JWT decode), and explicit-route OpenAI passthroughs (/v1/embeddings, /v1/moderations, etc. don't go through the catchall). Only behavior change is the targeted unblock for subscription Codex. |
||
|
chopratejas
|
341dcf03e9 |
fix(proxy): route Codex subscription /backend-api/* catchall to chatgpt.com
Codex CLI in subscription mode polls /backend-api/wham/usage, fetches agent identity JWKS from /backend-api/wham/agent-identities/jwks, and hits other auxiliary /backend-api/* endpoints during startup. The HTTP catchall in _select_passthrough_base_url ignored ChatGPT auth and routed all unmatched paths to api.openai.com, which 404s on every backend-api path. Codex interprets that as "session invalid" and refuses subscription auth. Add a single branch at the top of _select_passthrough_base_url: when _resolve_codex_routing_headers reports ChatGPT auth (explicit ChatGPT-Account-Id header or JWT with chatgpt_account_id claim), return https://chatgpt.com so the catchall forwards to the right host. No-op for Anthropic (x-api-key, no JWT), Gemini (x-goog-api-key, no JWT), OpenAI API key (sk- tokens fail JWT decode), and explicit-route OpenAI passthroughs (/v1/embeddings, /v1/moderations, etc. don't go through the catchall). Only behavior change is the targeted unblock for subscription Codex. |
||
|
chopratejas
|
148ded392a |
fix: A8 — SSE delta arms, UTF-8 buffer, phase preservation, request-id, 413
Eliminates the Python wire-format hotfix bugs gated on Phase A's
lockdown so the proxy is safe through Phase H's Python retirement.
Bugs retired:
- P0-7 / P4-44: Codex `phase` field is now explicitly preserved
through the Responses-API ↔ Chat-Completions round-trip; multi
text-part rebuild collapses to a single text part (no more
content doubling).
- P1-8: Bytes-level SSE event splitter
`parse_sse_events_from_byte_buffer`; emoji/CJK split across
chunks survive intact. Buffer is `bytearray`; UTF-8 decode happens
only AFTER the `\n\n` event terminator is located in bytes.
Invalid UTF-8 in a *complete* event raises (operator-visible
diagnostic, not silent corruption).
- P1-9: `_parse_sse_to_response` handles all delta types per
Anthropic guide §5.1: `thinking_delta`, `signature_delta`,
`citations_delta`. Block map keyed by `index` so out-of-order
events reconstruct correctly. `redacted_thinking.data` preserved.
- P4-47: Unknown Responses-API item types now log a structured
`unknown_responses_item_type` warning so operators see new
Codex item types in flight before they break.
- P5-57: Rust proxy captures upstream `request-id` (Anthropic) and
`x-request-id` (OpenAI); surfaced as `headroom-upstream-request-id`
on the response and as a tracing span field. Distinct from the
proxy's own `x-request-id`.
- P5-59: Body-too-large now returns 413 (was 400). Pre-checks
`Content-Length` and rejects without consuming the body when
present; chunked uploads still buffer-then-fail with 413.
Configurability (no hardcodes):
- HEADROOM_SSE_BUFFER_MAX_BYTES (default 1 MiB) — per-event cap.
- HEADROOM_PROXY_BODY_TOO_LARGE_STATUS (default 413) — operator
override for body-too-large status.
A7 follow-up: `_DummyAnthropicHandler._retry_request` accepts the
A3 byte-faithful kwargs (`original_body_bytes`, `body_mutated`,
`mutation_reasons`, `request_id`, `forwarder_name`, `path_for_log`)
so the existing 20 backpressure tests stay green against the real
handler signature.
The project-wide grep
git grep 'errors="ignore"\|errors="replace"' headroom/proxy/handlers/ headroom/ccr/
returns nothing; the single remaining lossy-decode site (response-
body diagnostics, not SSE) routes through `safe_decode_for_logging`
in `headroom/proxy/helpers.py`.
Tests:
- tests/test_sse_thinking_blocks.py (4 tests)
- tests/test_sse_utf8_split.py (3 tests)
- tests/test_proxy_responses_phase_preservation.py (4 tests)
- crates/headroom-proxy/tests/integration_request_id.rs (2 tests)
- crates/headroom-proxy/tests/integration_body_size.rs (2 tests)
|
||
|
chopratejas
|
8dcd474aca |
fix: A7 — memory tool injection session-sticky for both Anthropic and OpenAI
Closes the second half of P0-6: once memory injects memory_save / memory_search into body["tools"] for a session, every subsequent turn injects the byte-equal same definitions — even if memory is disabled mid-session. Toggling tool list mid-session busts Anthropic prefix cache per guide §6.3 #2. Adds in headroom/proxy/helpers.py: * SessionToolTracker — bounded LRU keyed by (provider, session_id) storing GOLDEN tool-definition bytes from the first injection. Tracker is provider-aware so the same session_id under Anthropic and OpenAI keeps independent state. Reentrant lock for concurrent access; LRU eviction at HEADROOM_TOOL_TRACKER_MAX_SESSIONS (default 1000). * apply_session_sticky_memory_tools — single coordination point with three paths: first-time inject (record golden bytes), sticky replay (always inject golden bytes regardless of inject_this_turn), and skip. Honors HEADROOM_TOOL_INJECTION_STICKY=disabled as a loud operator opt-in for rollback (NOT a fallback). * serialize_tool_definition_canonical — deterministic byte serialization via the same separators=(",",":")/ensure_ascii=False rules as serialize_body_canonical. * log_tool_injection_decision — structured per-decision log line; never logs the tool definition contents. Wires the helper into all four memory tool injection sites: * handlers/anthropic.py — /v1/messages * handlers/openai.py — /v1/chat/completions * handlers/openai.py — /v1/responses * handlers/openai.py — Codex WS path memory_handler.MemoryHandler gains compute_memory_tool_definitions(provider) — a pure builder that returns the tool definitions without mutating a tools list, so the proxy can route through the sticky tracker. The legacy inject_tools(...) is preserved for callers without a session_id. Tests: tests/test_memory_tool_session_sticky.py — 29 unit + integration cases covering: turn-1→turn-2 byte-equality (Anthropic + OpenAI), sticky replay after memory disabled, golden-fixture pin, LRU eviction, provider isolation under shared session_id, thread-safe concurrent access, env-var contract, disabled-mode passthrough, dedupe with client tools. Golden fixtures pin canonical bytes: * tests/fixtures/memory_tool_definitions/anthropic.json * tests/fixtures/memory_tool_definitions/openai.json No regex. No hardcodes (env-configurable: HEADROOM_TOOL_INJECTION_STICKY, HEADROOM_TOOL_TRACKER_MAX_SESSIONS). No silent fallbacks. Per-decision structured logging. Realignment build constraints satisfied. |
||
|
chopratejas
|
aec5ba3253 |
fix: A6 — anthropic-beta and openai-beta deterministic merge + session-sticky
PR-A6 of the Phase A cache-safety lockdown. Eliminates P5-50 and preps
P0-6 (memory tool injection toggling).
Two cache-killer patterns the merge + tracker defeat:
1. Mid-session mutation: when memory was enabled the proxy did an
ad-hoc concat of `context-management-2025-06-27` onto the client
value (anthropic.py:1244-1248). The order varied with the client
value, breaking byte-stable headers across turns.
2. Token drop-out across turns: clients (Claude Code, Codex CLI) MAY
drop a beta token between turn N and turn N+1 even when the proxy
mutated turn N to add it. The cache hot zone is positional, so the
next turn's prefix bytes hash differently and the prefix-cache
read misses.
Changes
-------
`headroom/proxy/helpers.py`
* `merge_anthropic_beta` / `merge_openai_beta`: pure, deterministic,
order-preserving merge. Client tokens first (in their original
order), then Headroom-required tokens (in the order passed). Dedupe
is case-insensitive but preserves the original casing of the first
occurrence. No regex.
* `SessionBetaTracker`: bounded LRU keyed by (provider, session_id),
unioning client tokens with previously-seen tokens. OrderedDict
LRU; threading.RLock for thread safety (mirrors the
CompressionCache pattern from compression_cache.py).
* `get_session_beta_tracker` / `_reset_session_beta_tracker_for_test`
process-wide singleton with test reset.
* `log_beta_header_merge`: structured log per cache-affecting merge.
* Env-var knobs (NO HARDCODES):
- HEADROOM_BETA_HEADER_STICKY=enabled|disabled (default enabled).
- HEADROOM_BETA_TRACKER_MAX_SESSIONS (default 1000).
`headroom/proxy/handlers/anthropic.py`
* After `compute_session_id` (line ~744): record client
`anthropic-beta` against the session tracker, write the sticky
value back into `headers` if changed. Order matters: sticky-merge
FIRST so memory-injection has the canonical baseline.
* Memory-injection site (line ~1244): replace the ad-hoc concat with
`merge_anthropic_beta(headers["anthropic-beta"], required_tokens)`.
`headroom/proxy/handlers/openai.py`
* Chat-completions (line ~360): record/merge `openai-beta`.
* /v1/responses HTTP (line ~1213): compute `_responses_session_id`
and record/merge `openai-beta`.
* /v1/responses WS (line ~1711): replace the ad-hoc absent-only
inject with `merge_openai_beta(sticky, ["responses_websockets=
2026-02-06"])`. Replaces any case-variants of the existing key.
Tests
-----
`tests/test_anthropic_beta_session_sticky.py` (26 tests):
* Pure helper: empty inputs, only-client, only-headroom, ordering,
dedupe casing, deterministic memory-injection order, no-double-
inject when token already present.
* Tracker: sticky-on across turns even when client drops, casing
preservation, provider namespace independence, LRU eviction at
max_sessions, env-var validation (loud failures), thread safety
under 16-thread concurrent access, blank-input rejection.
`tests/test_openai_beta_session_sticky.py` (17 tests):
* Mirror of the anthropic suite for `OpenAI-Beta`.
* Plus WS-specific coverage: sticky-then-merge of
`responses_websockets=2026-02-06` against client baseline.
`tests/test_openai_codex_routing.py`
* Add `session_tracker_store` stub to `_DummyOpenAIHandler` so the
routing tests still exercise the responses HTTP handler now that
it computes a session_id for beta-merge.
Notes
-----
Build constraints honored:
* Configurable: HEADROOM_BETA_HEADER_STICKY,
HEADROOM_BETA_TRACKER_MAX_SESSIONS.
* No regex, no hardcodes (env-var bounds), no fallbacks (disabled
mode is operator opt-in for diagnostics, loud failures on invalid
values).
* Structured tracing log via `log_beta_header_merge`.
Acceptance:
* 43 new tests pass.
* `cargo test --workspace` green (no Rust changes).
* `make ci-precheck` green.
|
||
|
chopratejas
|
2e874c5e3e |
fix: A5 — strip x-headroom-* from upstream-bound headers (P5-49)
Eliminate P5-49: every Python forwarder and the Rust transparent proxy
now drop internal `x-headroom-*` request headers (`x-headroom-bypass`,
`x-headroom-mode`, `x-headroom-user-id`, `x-headroom-stack`,
`x-headroom-base-url`) before the upstream call. Stops fingerprinting
of the proxy by subscription-revocation enforcers and prevents leakage
of internal user-id / stack / base-url internals to whichever vendor
terminates the request.
Python:
- `_strip_internal_headers(headers)` in `headroom/proxy/helpers.py`
returns a NEW dict with `x-headroom-*` keys removed (case-insensitive
prefix match, no regex). Pure function. Operator opt-in
`HEADROOM_STRIP_INTERNAL_HEADERS=disabled` keeps internal headers in
the upstream-bound dict for diagnostic shadow tracing — explicit, not
a fallback.
- Strip applied at every handler entry capture in `anthropic.py`,
`openai.py`, `batch.py`, `gemini.py` (chat completions, responses,
WebSocket handshake, Copilot passthrough, batch passthroughs, Gemini
generate / stream / countTokens / cloudcode-assist, Anthropic
passthrough + batch results). Inbound reads of x-headroom (bypass
gating, memory user-id) migrated to `request.headers.get(...)` so
they continue working off the original dict.
- `log_outbound_headers` emits `event=outbound_headers forwarder=...
stripped_count=N request_id=...` per call. Never logs header values.
Rust (crates/headroom-proxy):
- `strip_internal_headers(&mut HeaderMap)` and `is_internal_header`
helpers in `src/headers.rs`. `build_forward_request_headers` accepts
a `strip_internal: bool` so the same path serves HTTP and WebSocket.
- `Config::strip_internal_headers: StripInternalHeaders` driven by CLI
flag `--strip-internal-headers` and env var
`HEADROOM_PROXY_STRIP_INTERNAL_HEADERS` (default `enabled`).
- `proxy.rs` and `websocket.rs` call `build_forward_request_headers`
with the resolved policy; structured `tracing::info!` /
`tracing::warn!` line per request describes the strip decision.
Tests: 24 Python (`tests/test_header_isolation.py`) + 4 Rust
integration (`crates/headroom-proxy/tests/integration_headers.rs`) +
4 Rust unit tests in `headers.rs`. Covers every named header
(`bypass`, `mode`, `user-id`, `stack`, `base-url`), case-insensitive
prefix matching, legitimate-headers passthrough, the `disabled`
operator-opt-in mode, and that the inbound bypass-gating read path
is unaffected by the strip.
Acceptance: targeted `pytest -x` suite green (87 tests across
test_header_isolation, test_proxy_byte_faithful_forwarding,
test_proxy_anthropic_cache_stability, test_proxy_system_prompt_immutable,
test_proxy_openai_cache_stability, test_proxy_pipeline_lifecycle).
`cargo test -p headroom-proxy` green (23 tests across all integrations
plus 7 lib unit tests). `cargo clippy -p headroom-proxy -- -D warnings`
clean. `cargo fmt --all -- --check` clean. `cargo test --workspace`
green (~900 tests total).
Per realignment build constraints: configurable (env + CLI), no
hardcodes, no regex (pure `.lower().starts_with()` match), no silent
fallbacks (`disabled` is loud operator opt-in), structured logs
(`event=outbound_headers`).
Remaining `x-headroom-` references in `headroom/proxy/handlers/` are
inbound-read sites only: `request.headers.get("x-headroom-bypass")` /
`x-headroom-mode` for behavior gating, `request.headers.get
("x-headroom-user-id")` for memory user-id resolution, and `ws_headers
.get(...)` on the WebSocket inbound path. Response-side `X-Headroom-*`
injection (e.g. `x-headroom-tokens-saved`) is unrelated to upstream
forwarding and untouched.
|
||
|
chopratejas
|
2e9f52fda2 |
fix(providers): register claude-opus-4-7 + [1m] tier with 1M context
claude-opus-4-7[1m] (the model name Claude Code sends to select Anthropic's 1M tier) was unmapped in ANTHROPIC_CONTEXT_LIMITS and unmapped in LiteLLM's bundled JSON, so get_context_limit fell through to _PATTERN_DEFAULTS["opus"] = 200K. The under-budget made IntelligentContextManager.should_apply skip compression on typical ~50K Claude Code sessions, surfacing as "0 savings" on Opus 4.7 traffic. Register both the bare and [1m]-suffixed forms at 1M with Opus-tier pricing. |
||
|
chopratejas
|
f0dcc02775 |
fix: A3 — byte-faithful Python forwarders; serialize canonical only when mutated
Eliminates P0-2 universally. Every Python forwarder (server.py
`_retry_request`, handlers/streaming.py `_stream_response`,
handlers/openai.py `_ws_http_fallback`, handlers/batch.py `_batch_passthrough`
+ batch-create + Google batch passthrough, handlers/anthropic.py CCR
continuation + batch endpoint) now switches from `httpx ... json=body` to
`httpx ... content=raw_bytes`. The default httpx JSON encoder was
re-serializing every request with `, `/`: ` separators and `\\uXXXX` ASCII
escapes — collapsing Anthropic prompt-cache hit-rate.
Forwarder strategy:
- unmutated body → forward `await request.body()` verbatim;
- mutated body → re-serialize once via the new
`serialize_body_canonical(body) -> bytes` helper (compact separators,
`ensure_ascii=False`, dict insertion order preserved).
`HEADROOM_PROXY_PYTHON_FORWARDER_MODE` env var configures the mode:
- `byte_faithful` (default) — the new behavior;
- `legacy_json_kwarg` — explicit operator opt-in for emergency rollback.
Documented in `docs/content/docs/configuration.mdx`. NOT a fallback —
unknown values raise loudly per build constraint #4.
`BodyMutationTracker` accompanies each request through the handler so
transform sites mark the tracker (`memory_injection`,
`image_compression`, `compression_*`, `batch_compression`,
`ccr_continuation`, etc.). At forwarder dispatch we additionally compare
the final body dict against the parsed original bytes as a structural
safety net — any silent mutation we missed still triggers canonical
re-serialization.
A2 follow-up: `handlers/openai.py:534-540` (Chat Completions memory
injection) was prepending a system message; replaced with
`append_text_to_latest_user_chat_message`, the OpenAI Chat Completions
analog of `_append_context_to_latest_non_frozen_user_turn`. The cache
hot zone (system messages) is now sacrosanct on /v1/chat/completions
too. Honors `HEADROOM_MEMORY_INJECTION_MODE=disabled`.
Structured logging: every forwarder emits an `event=outbound_request`
log line with `forwarder`, `path`, `body_bytes`, `body_mutated`,
`mutation_reasons`, `source` (passthrough|canonical|legacy),
`request_id`. Never logs Authorization or full body.
`_read_request_json` factored to share `_read_request_body_bytes` with
new `read_request_json_with_bytes` so the anthropic handler can capture
both the parsed dict and the original (decompressed) bytes.
Tests:
- `tests/test_proxy_byte_faithful_forwarding.py` (28 tests):
SHA-256 byte-equality on /v1/messages and streaming, unicode
preservation, numeric precision, mutation-tracker invariants,
canonical-serializer properties, legacy-mode rollback, OpenAI
Chat memory routing.
- Existing test mocks updated to accept the new `**kwargs` on
`_retry_request` (no behavior change).
- `tests/test_proxy_handlers_batch.py` updated to read the captured
`content=` bytes (formerly `json=`).
- One A2 test corrected (`test_anthropic_tool_sort_and_context_append_helpers`)
to match the live-zone-tail semantics introduced by A2.
Constraints satisfied: configurable env var; no new regex / hardcodes;
no silent fallback (`legacy_json_kwarg` is operator opt-in);
performant (`prepare_outbound_body_bytes` is O(1) for passthrough);
elegant single-responsibility helpers; structured tracing logs.
|
||
|
chopratejas
|
704fb2f19d |
fix: A2 — system prompt immutable; memory routes to live-zone tail; cache_aligner detector-only
P0-1: Delete `_inject_system_context` from `proxy/server.py`. Memory context now routes exclusively to the first text block of the latest non-frozen user message via `_append_context_to_latest_non_frozen_user_turn` (promoted to the canonical default in handlers/anthropic.py). Mirror applied to OpenAI Responses API at handlers/openai.py: `body["instructions"]` is no longer mutated; memory context appends to the latest user item in `body["input"]`. P2-23: Replace `headroom/transforms/cache_aligner.py` with a detector-only implementation. The legacy rewrite path (~400 LOC) is removed. The volatile- content detector uses no regex — UUIDs via `uuid.UUID`, ISO 8601 via `datetime.fromisoformat`, JWT shape via base64url segment-count check, hex hashes via length + `int(token, 16)` validation. Volatile findings surface through `cache_metrics`/`warnings`/`logger.warning`; the prompt is never mutated. Configurability: new env var `HEADROOM_MEMORY_INJECTION_MODE` with values `live_zone_tail` (default) and `disabled`. No `system_prompt` value — that path is permanently retired. Structured logs: every memory injection emits `event=memory_injection` with `decision`, `bytes_injected`, `query_hash` (BLAKE2b, never raw query), `session_id`, `request_id`. Auth is never logged. Tests: - Add `tests/test_proxy_system_prompt_immutable.py` (7 tests). - Add `tests/test_cache_aligner_detector_only.py` (20 tests). - Replace `tests/test_transforms/test_cache_aligner.py` (rewrite-path tests, 58 cases) with detector-only behavior. - Update `tests/test_acceptance.py::TestDateTrap` to pin the new detector-only contract. Acceptance: - `git grep -n "_inject_system_context\|_inject_to_system_or_instructions" headroom/` returns nothing. - `git grep -n "import re\|from re import" headroom/transforms/cache_aligner.py` returns nothing. - Targeted suite (`test_proxy_system_prompt_immutable.py`, `test_cache_aligner_detector_only.py`, `test_proxy_anthropic_cache_stability.py`, `test_acceptance.py::TestDateTrap`, `test_memory*.py`, `test_cli/`) green. |
||
|
chopratejas
|
3f99128236 |
fix(rust): A4 — honor cache_control markers; serde_json arbitrary_precision + raw_value
PR-A4 of the Realignment Phase A lockdown
(REALIGNMENT/03-phase-A-lockdown.md). Eliminates P0-3 (Rust proxy
ignores customer cache_control markers) and P0-5 (numeric precision
lost via serde_json::Value round-trip) at the library level; Phase B
PR-B2 wires the helper into the live-zone block dispatcher.
Cargo.toml — add `arbitrary_precision` and `raw_value` to
`serde_json` workspace features. `arbitrary_precision` keeps `1.0`
from collapsing to `1` and preserves >2^53 integers; `raw_value`
exposes `&RawValue` so PR-B2 can forward unmodified `messages[*]`
entries as exact byte copies.
crates/headroom-core/src/cache_control.rs (new) — `compute_frozen_count`
walks `messages[i].content[*].cache_control` via serde_json
accessors only (no regex) and returns the smallest N such that
`messages[i]` is frozen for every i < N. Markers in `system` or
`tools[*]` log at debug! but never bump the floor (those fields are
unconditionally cache-hot per invariant I2). TTL ordering violations
(5m before 1h, guide §2.19) emit `tracing::warn!` but the function
computes the correct count regardless — the customer's request, not
ours to reject.
crates/headroom-core/src/lib.rs — re-export `compute_frozen_count` at
crate root so the proxy crate has a stable import path.
crates/headroom-proxy/src/compression/anthropic.rs — add
`resolve_frozen_count` thin wrapper that consults the
`cache_control_auto_frozen` config flag. When `disabled`, returns 0
regardless of body content (operator opt-out for benchmarking).
crates/headroom-proxy/src/config.rs — add `CacheControlAutoFrozen`
enum and the matching CLI flag `--cache-control-auto-frozen` /
env var `HEADROOM_PROXY_CACHE_CONTROL_AUTO_FROZEN`. Default is
`enabled`. Documented in the doc comments.
Tests
- crates/headroom-core/src/cache_control.rs (inline): 11 unit tests
covering marker detection, system/tools negative cases, ordering
state machine, defensive (missing fields, non-array messages,
non-object content blocks).
- crates/headroom-core/tests/cache_control.rs: 11 unit + 3 property
tests (monotonic non-decrease as markers are added; system/tools
markers don't change count; empty messages → 0).
- crates/headroom-proxy/tests/integration_cache_control.rs: 8 tests
exercising the proxy wrapper (configurability gate; tracing
capture for the 5m-before-1h warn path).
Acceptance gates: `cargo build --workspace`, `cargo test --workspace`
(33 new tests green), `cargo clippy --workspace -- -D warnings`,
`cargo fmt --all --check` all clean. No new `regex::` imports;
`git grep -n 'regex::' crates/{headroom-core/src/cache_control.rs,
headroom-core/tests/cache_control.rs, headroom-proxy/tests/
integration_cache_control.rs}` empty.
Honors the realignment build constraints: configurable (CLI + env),
no hardcodes (TTL strings live as const), no regex (serde_json
accessor walk), no fallbacks (one impl), structured logging
(debug!/warn! with field/index/ttl/rule context), tests
comprehensive (unit + property + integration + tracing capture).
|
||
|
chopratejas
|
a974bb153a |
fix(rust): PR-A1 — make /v1/messages compression a passthrough
Stop calling IntelligentContextManager from the Rust proxy on
/v1/messages. The proxy is now a byte-faithful passthrough on this
endpoint. Eliminates the C1+C2+C3+C4 cache-killer cluster (P0-3,
P0-4, P0-5, P1-13) by not running ICM with `frozen_message_count: 0`
hardcoded — Phase B PR-B2 brings live-zone-only compression back.
Per REALIGNMENT/03-phase-A-lockdown.md.
Changes:
- Add `--compression-mode {off,live_zone}` flag and
`HEADROOM_PROXY_COMPRESSION_MODE` env var. Default `off`. Both
modes passthrough in PR-A1; `live_zone` warns loudly because
Phase B isn't implemented yet (no silent fallback).
- Replace `compress_anthropic_request` body with a passthrough
stub that emits a structured `tracing::info!` decision log line
(request_id, path, method, compression_mode, decision,
reason="phase_a_lockdown", body_bytes) and returns
`Outcome::NoCompression`. Function signature preserved so
Phase B PR-B2 is a pure body swap.
- Delete `compression/icm.rs` (per the realignment plan: ICM
modules in headroom-core are deleted in PR-B1).
- Drop the `Arc<IntelligentContextManager>` field from `AppState`
— no longer used.
- Add request-entry `tracing::debug!` with auth_mode_placeholder
("unknown" until Phase F PR-F1 wires the auth-mode classifier).
- Add `debug_assert!` on the NoCompression branch that the
buffered bytes length is stable, locking in Phase A's
cache-safety invariant at the call site.
- Tighten existing tests from `len()` equality to SHA-256 byte
equality. Rename `compression_on_oversized_body_trims_messages`
→ `compression_on_long_body_passes_through_in_phase_a` and
flip the assertion to byte-equal.
- Add new tests: passthrough_mode_off_byte_equal_sha256,
passthrough_mode_live_zone_currently_passthrough_byte_equal_sha256,
passthrough_preserves_numeric_precision (literal-byte body so
serde_json's f64 quantization can't mask a regression),
passthrough_preserves_cache_control_markers,
passthrough_preserves_thinking_signature,
passthrough_preserves_redacted_thinking_data,
passthrough_recorded_fixture_byte_equal_sha256,
tracing_capture::compression_decision_logged.
- Add fixture
`crates/headroom-proxy/tests/fixtures/anthropic_messages_request_real.json`
with system block list + cache_control markers, tools with
nested JSON Schema, messages containing text + thinking +
signature + tool_use + tool_result + image, non-ASCII content,
large numbers. Used as the canonical SHA-256 round-trip gate.
Constraints honored: configurable (compression_mode is the only
new knob), no hardcoded thresholds, no regex usage, no silent
fallbacks (live_zone-not-implemented warns), structured tracing
on every cache-affecting decision, comprehensive tests.
Acceptance criteria from PR-A1 spec:
- `cargo build --workspace` clean
- `cargo test --workspace` green (886 tests pass)
- `cargo clippy --workspace -- -D warnings` clean
- `cargo fmt --all --check` clean
- `make ci-precheck` green
- New SHA-256 byte-equality tests pass against the recorded fixture
- `tracing::info!` decision-log line is observable
- `--compression-mode` CLI + env var work
- No regex import added
|
||
|
chopratejas
|
0ce2243dfb |
docs: add Realignment plan (40 PRs, 9 phases)
Comprehensive PR-by-PR plan to realign Headroom around live-zone-only compression with prefix-cache safety as a non-negotiable invariant. Drafted from a 10-agent deep audit against the LLM-proxy compression guide. - 14 documents under REALIGNMENT/ - 72 ranked bugs (P0 cache-killers through P6 test-infra) - 40 feature PRs + 10 test-infra PRs across 9 phases - ~25K LOC retirement (ICM + scoring + relevance + rolling-window + summarizer + tool-crusher + LiteLLM-fake-Bedrock) - Preserves TOIN, CCR, Kompress-base per user direction - Auth-mode policy gates (PAYG / OAuth / subscription) - Phase 3 cache stabilization surface (tool-sort, schema-sort, cache_control auto-place, prompt_cache_key) - Native Bedrock SigV4 + Vertex ADC handlers - Test infrastructure: SHA-256 byte-faithful gate, SSE corner cases, property tests, real-traffic shadow |
||
|
Tejas Chopra
|
51eeaf6662
|
Merge pull request #344 from chopratejas/audit-followup-c1-c2-c3
fix(proxy): cache concurrency lock, multi-worker docs, bounded compre… |
||
|
Tejas Chopra
|
4c52766860
|
Merge pull request #345 from chopratejas/rust-proxy-wire-compressors
fix(rust): wire ICM compressor into Rust proxy on /v1/messages |
||
|
chopratejas
|
456a6b33af |
fix(test): stub _run_compression_in_executor on _DummyOpenAIHandler
The bounded compression executor introduced in this PR moved every handler's compression call from `asyncio.wait_for(asyncio.to_thread(...))` to `self._run_compression_in_executor(...)`, which lives on `HeadroomProxy` (server.py) and is inherited by handler mixins at runtime. The test's `_DummyOpenAIHandler` only inherits `OpenAIHandlerMixin`, not `HeadroomProxy`, so it lacks the method. The Responses API compression path caught the AttributeError and silently fell back — which made `test_handle_openai_responses_stream_keeps_compression` fail with `apply.call_count == 0`. Add a synchronous stub that just invokes the callable; tests don't need real thread-pool semantics. |
||
|
chopratejas
|
fa5fbfabf4 |
fix(rust): wire ICM compressor into Rust proxy on /v1/messages
Adds an opt-in compression interceptor that buffers Anthropic
/v1/messages requests, runs IntelligentContextManager over the
messages array, and forwards the (possibly trimmed) body upstream.
All other paths, methods, and content-types stay on the original
streaming passthrough — so existing operators see zero change.
Behaviour gates ALL must be true to buffer + compress:
- --compression flag (or HEADROOM_PROXY_COMPRESSION=1)
- method == POST
- path == /v1/messages
- Content-Type: application/json
- ICM constructed successfully at startup
Falls through to streaming on any failure: parse, missing fields,
unknown model, body-too-large. Compression must never break a
request — that's the safety contract.
Model context windows come from a vendored LiteLLM snapshot at
crates/headroom-proxy/data/model_prices_and_context_window.json
parsed once into an OnceLock<HashMap>. Refresh via
scripts/refresh_model_limits.sh. Rationale documented inline:
hardcoded tables silently rot; LiteLLM is the canonical source
the entire LLM-tooling ecosystem relies on.
New tests:
- 16 unit tests across compression::{anthropic, icm, model_limits}
- 5 integration tests: off-passthrough, on-short-passthrough,
on-oversized-trim, on-non-json-skip, on-non-llm-path-skip
Verification:
- cargo test --workspace -> 884 passed, 0 failed
- cargo clippy --workspace -- -D warnings -> clean
- cargo fmt --check -> clean
|
||
|
Tejas Chopra
|
f726d0e280
|
Merge pull request #343 from chopratejas/rust-message-scorer-port
Rust message scorer port |
||
|
chopratejas
|
ea78cf6252 |
fix(proxy): cache concurrency lock, multi-worker docs, bounded compression executor
Three audit follow-ups from issue #327's deep-dive review. C1 — CompressionCache concurrency lock ====================================== `CompressionCache` instances are shared per `session_id` and accessed from async-dispatched threadpool workers. Pre-fix, concurrent requests for the same session raced on `_cache`, `_stable_hashes`, `_first_seen`, and `_total_tokens_saved` with no synchronization. Observable failures: * Lost-update on `_total_tokens_saved` (read-modify-write). * `RuntimeError: OrderedDict mutated during iteration` from `apply_cached` when a concurrent `store_compressed` evicts during the walk. * Lost stable-hash records — next-turn compute_frozen_count reads inconsistent state. May also explain part of SvenMeyer's `_cache: 0 entries / 1003 misses` observation: the cache was being clobbered concurrently. Added `threading.RLock` guarding all mutating methods. `RLock` (not `Lock`) so future code can call locked methods from inside another locked method without self-deadlock. Also locked `HeadroomProxy._compression_caches` dict-of-caches access via a separate `_compression_caches_lock` so two concurrent calls for the same session_id can't each create distinct CompressionCache objects (which would split the cache state between them). The `/stats` endpoint snapshots the cache list under the dict lock before iterating to avoid eviction-during-iteration. C2 — Multi-worker CCR fragmentation: documented + startup warning ================================================================= The in-memory `InMemoryCcrStore` (Rust), `_compression_caches` (Python), `session_tracker_store` (Python), and TOIN learner state are ALL per-process. Multi-worker uvicorn round-robins requests across workers, so a session whose turn-1 lands on worker A may have turn-2 land on worker B. Worker B has zero knowledge of A's CCR markers, replay cache, or prefix-cache state. Result: `Retrieve original: hash=X` markers stay in-context as opaque directives, every fresh tool_result is recompressed from scratch, and `frozen_message_count=0` causes Anthropic prefix-cache busts on every cross-worker turn. Added a "Multi-worker deployment — CCR fragmentation" section in `RUST_DEV.md` documenting the failure modes, the supported configuration (`--workers 1`), and the sticky-session workaround for horizontal scale. The proxy emits a `WARNING`-level log line on startup if `workers > 1` is detected, pointing at the doc section. C3 — Bounded compression executor with cancel-aware metrics =========================================================== `asyncio.wait_for(asyncio.to_thread(pipeline.apply), timeout=...)` cancellation does NOT propagate into the threadpool worker that's running Rust code. Once the worker has picked up the task, `concurrent.futures.Future.cancel()` returns False and the thread runs to completion. Stuck threads accumulated invisibly on asyncio's default executor, contending with unrelated `to_thread` callers (file IO, etc.). Replaced all 7 `asyncio.to_thread` call sites for `pipeline.apply()` across `proxy/handlers/anthropic.py` (3) and `proxy/handlers/openai.py` (4) with a new `HeadroomProxy._run_compression_in_executor(fn, *, timeout)` helper that: 1. Submits to a dedicated bounded `ThreadPoolExecutor` named `headroom-compress` (configurable via `ProxyConfig.compression_max_workers`; defaults to `min(32, (cpu_count or 1) * 4)`). 2. Increments `_compression_in_flight` (gauge) when work starts and decrements when work completes; tracks `_compression_in_flight_max` as a high-water mark. 3. Detects "leaked threads" by comparing wall-clock elapsed against the timeout in the worker's `finally` block. Increments `_compression_leaked_threads` when a worker finishes after its asyncio future was cancelled. Operators can see the leaked-thread rate climbing in `/stats runtime.compression_executor` BEFORE the pool fills up. Tests ===== * `TestCompressionCacheConcurrency` (3 tests) — many threads store_compressed / apply_cached / update_from_result on a single CompressionCache; assert no exceptions, no lost updates, no partial state. * `test_get_compression_cache_returns_same_instance_under_contention` — 32 concurrent `_get_compression_cache(same_id)` calls return the identical instance (would split pre-lock). * `test_proxy_compression_executor.py` (8 tests) — pool size respects config, in-flight gauge tracks running compressions, high-water mark is monotonic, timeout propagates to awaiter, leaked-thread counter increments on post-deadline completion, `/stats` surfaces all three gauges. Verification ============ * All 123 targeted regression tests pass. * `make ci-precheck` clean. * No `Co-Authored-By` trailer; conventional `fix:` prefix; no `--no-verify`. |
||
|
Tejas Chopra
|
21784f4288
|
Merge pull request #340 from chopratejas/rust-icm-core
fix(rust): IntelligentContextManager port (simplified, OSS) — PR-B |
||
|
chopratejas
|
013344f6fd |
fix(rust): IntelligentContextManager port (simplified, OSS) — PR-B
Direct port of headroom/transforms/intelligent_context.py (1077 LOC)
with a deliberately simpler architecture:
- One built-in strategy: DropByScoreStrategy. Multi-factor scoring +
safety rails + CCR-on-drop persistence. Outclasses every gateway
competitor's rolling-window behaviour.
- Minimal ContextStrategy trait so Enterprise plugs additional
strategies (compress-first, summarize, memory tiers) into the same
orchestrator without modifying it.
- 6 config fields instead of 12+. Cuts: compress_threshold,
summarize_threshold, summarization_*, memory_tiers_*, warm/cold_*.
Module layout (parallel to scoring/, signals/, transforms/):
context/
├── config.rs — IcmConfig (6 fields)
├── workspace.rs — ContextWorkspace + StrategyOutcome
├── safety.rs — system / last-N-turns / tool-pair
│ atomicity / frozen prefix
├── candidate.rs — turn / tool-unit / single candidates
├── ccr_drop.rs — serialize dropped messages, store
│ under content-hash, emit marker
├── manager.rs — cascade orchestrator
└── strategy/
├── mod.rs — ContextStrategy trait
└── drop_by_score.rs — THE OSS strategy
Quirks preserved from Python:
- Tool-call/response atomicity (OpenAI + Anthropic + Strands shapes)
- frozen_message_count protects prompt-cache prefix
- Cascade: each strategy returns tokens_freed + fully_resolved
- should_apply strict gate (no tokenization on under-budget requests)
- CCR-on-drop default ON (the OSS-defining behaviour)
44 new unit tests (50 total context tests), 811 total in headroom-core.
NO PyO3 yet (PR-C). NO Python deletion yet (PR-D).
|
||
|
Tejas Chopra
|
05f91d9adc
|
Merge pull request #338 from chopratejas/rust-message-scorer-port
fix(rust): port MessageScorer to Rust + parity harness (PR-A) |
||
|
Tejas Chopra
|
9129188d4a
|
Merge pull request #339 from chopratejas/fix-327-walker-removal
fix(proxy): remove content-keyed TTL walker that conflated content wi… |
||
|
chopratejas
|
bcef763720 |
ci: pin dtolnay/rust-toolchain@stable, set toolchain via input
The @1.95.0 git ref of dtolnay/rust-toolchain shipped action code that errors on ubuntu-latest with: failed to install component: 'clippy-preview-x86_64-unknown-linux-gnu', detected conflict: 'bin/cargo-clippy' The runner's pre-installed Rust ships cargo-clippy at $HOME/.cargo/bin, and the older action code's rustup invocation hits a path conflict when adding the clippy-preview component for 1.95.0. The @stable ref of the action has the fix; pass toolchain: 1.95.0 as input so the version stays pinned. rust-toolchain.toml continues to be the source of truth for the version (used by cargo's auto-detection); this keeps the action's install in sync. |
||
|
chopratejas
|
521fbbeabd | style: apply ruff format to test_proxy_anthropic_cache_stability lambdas | ||
|
chopratejas
|
35eaf8de7f |
fix(proxy): remove content-keyed TTL walker that conflated content with positional cache (#327)
The Anthropic token-mode handler walked past prefix_tracker.frozen_message_count
whenever an upcoming tool_result's content-hash matched comp_cache._stable_hashes
or should_defer_compression returned True. That conflated content equality with
positional cache membership.
Anthropic's prefix cache is POSITIONAL: bytes 0..K cached, anything past K is
fresh. _stable_hashes is content-keyed and grows unbounded. In long Claude Code
sessions where tool_result content rhymes across turns (repeated system prompts,
repeated file reads, repeated tool descriptions), the walker advanced
frozen_message_count to len(messages) on every turn and the pipeline produced
transforms_applied=[] on 73% of requests in user SvenMeyer's reported session
(headroom-stats-2026-05-01.json: 74 of 101 eligible requests "prefix_frozen") —
even after the prior fix in
|
||
|
chopratejas
|
c4989b1844 |
Merge remote-tracking branch 'origin/main' into rust-message-scorer-port
# Conflicts: # crates/headroom-core/src/transforms/smart_crusher/crusher.rs |
||
|
chopratejas
|
21989e3640 |
fix(rust): port MessageScorer to Rust + parity harness (PR-A)
Direct port of `headroom.transforms.scoring.MessageScorer` (459 LOC). Foundation piece for the IntelligentContext port (PR-B onward). What's wired: - Deterministic factors fully ported: recency (exp-decay), forward references (tool_call_id graph), token density (unique/total). - External-dep factors gated behind traits: `EmbeddingProvider` and `ToinProvider`. No concrete impls yet — both default to neutral values matching Python's `embedding_provider=None` / `toin=None`. PR-A1 wires fastembed; PR-A2 plugs in a PyO3 ToinProvider. - ScoringWeights + MessageScore with serde + BTreeMap-ordered breakdown for stable JSON. Parity: - 13 fixtures recorded from Python, byte-equal under the comparator. - Floats rounded to 5 decimals on both sides — absorbs f32-vs-f64 drift in the weighted sum without masking real bugs. Drive-by: re-fix three pre-existing clippy errors in smart_crusher/crusher.rs that re-emerged with new test additions (field_reassign_with_default + dead hash_array_for_ccr). |