Phase F2.1, commit 1 of 6. Pure additive change — no call site reads
this struct yet, no behavior change in main.
What lands:
- New `headroom_core::compression_policy::CompressionPolicy` struct
with `live_zone_only: bool` and `cache_aligner_enabled: bool`.
- `CompressionPolicy::for_mode(AuthMode)` returns the F2.1 per-mode
values: PAYG and OAuth aggressive (live-zone-not-only,
cache-aligner on); Subscription live-zone-only with cache aligner
disabled. The OAuth=PAYG match is intentional in F2.1 — F2.2 will
diverge once telemetry collected during F2.1's main bake shows
what OAuth users need.
- `From<crate::auth_mode::AuthMode> for crate::transforms::live_zone::AuthMode`
to bridge the two `AuthMode` enums that exist in headroom-core
(one in F1's classifier, one in the live-zone dispatcher; differ
only by the dispatcher's `Unknown` sentinel for stored
recommendation rows). Keeps the cross-module call sites clean —
`mode.into()` instead of a hand-written match every time.
Per-mode values (F2.1 only — F2.2 will tune):
| Mode | live_zone_only | cache_aligner_enabled |
|--------------|----------------|-----------------------|
| Payg | false | true |
| OAuth | false | true (= PAYG) |
| Subscription | true | false |
3 unit tests covering each variant. `oauth_matches_payg_today`
fails on purpose if F2.2 (or any well-meaning future change)
diverges OAuth from PAYG without updating the test — the divergence
must be deliberate.
Why two flags and not more in F2.1: closing #327/#388 cache-
instability complaints requires exactly these two gates. Anything
more is F2.2 tuning that benefits from real bake-time telemetry,
and a smaller F2.1 lands faster + fewer regression sites.
Why a struct instead of `match auth_mode { ... }` everywhere:
Phase E already has two PAYG-only gates (cache_control auto-
placement, prompt_cache_key injection). Adding two more without
centralisation means four duplicated match arms. The struct
collapses them and gives F2.2 one place to add fields.
Next commit (c2/6) plumbs the policy through the Rust proxy +
plugs the `auth_mode` parameter the OpenAI dispatchers are still
missing today.
Recursively sort JSON Schema object keys inside each tool's schema
so cache hits no longer depend on SDK-side serializer key-emission
order (some sort, some preserve insertion, some hash-randomize).
Wired into all three live-zone walkers, hooking the per-provider
schema location:
- Anthropic: `tool["input_schema"]`
- OpenAI Chat: `tool["function"]["parameters"]`
- OpenAI Responses: `tool["function"]["parameters"]`
Same auth-mode gate as PR-E1 (PAYG only). NO marker check — the
`cache_control` marker lives on the tool object itself, not inside
the schema, so sorting schema keys never moves the marker. PR-E2
therefore runs even on tools that PR-E1 had to skip due to a
present marker; the integration test pins this behaviour.
Array semantics preserved: `oneOf`, `anyOf`, `allOf`, `prefixItems`
and any other ordered JSON Schema array keep customer order; only
object keys move. Idempotent — sorting an already-sorted schema
yields byte-identical bytes (workspace `preserve_order` feature
pins `serde_json::Map` emission to insertion order).
Tests: unit tests for nested keys, oneOf preservation, deep
nesting, and idempotency; integration tests boot the real proxy
and assert PAYG -> sorted at every level, OAuth -> SHA-256 byte-
equal, and PAYG-with-marker -> E1 skipped but E2 still runs.
Sort `tools[]` alphabetically by name on the way out so cache hits no
longer depend on the customer-side iteration order (commonly hash-
randomized via `set()` / `dict`). Mutates request bytes only when:
1. Auth mode is PAYG (`headroom_core::auth_mode::classify`).
2. No tool already carries a `cache_control` marker (reordering
would shift cache scope and silently void customer intent).
Every gate skip emits a structured `e1_skipped` event with `reason =
auth_mode | marker_present` so dashboards can see policy adoption.
Wired into all three live-zone walkers — Anthropic `/v1/messages`,
OpenAI `/v1/chat/completions`, OpenAI `/v1/responses` — plus the
Bedrock invoke + invoke-streaming entry points. Each passes
`auth_mode` (already pre-classified by Phase F PR-F1 middleware)
into the dispatcher so the gate evaluates without re-classifying.
Sort key uses `tool["name"]` (Anthropic) or `tool["function"]["name"]`
(OpenAI). Unnamed tools (rare; malformed inputs only) fall back to
MD5 of canonical-JSON serialization for a stable in-process key —
collision odds are astronomically small and `Vec::sort_by` is stable.
Tests: unit tests for sort + marker detection + idempotency + the
permutation property; integration tests boot the real proxy in front
of a wiremock upstream and assert PAYG -> sorted, OAuth/Subscription/
marker -> byte-equal passthrough (SHA-256).
Codex treats env_key as a hard requirement: if the named env var is
absent it throws "Missing environment variable" before the session
starts. Subscription (ChatGPT Plus) users don't have OPENAI_API_KEY
set, so injecting env_key = "OPENAI_API_KEY" blocks them at startup.
With env_key absent, api_key() returns Ok(None) and Codex falls through
to the existing CodexAuth (OAuth for subscription, ApiKey for PAYG) —
both modes authenticate correctly without startup errors.
Codex subscription mode uses the built-in openai provider with
chatgpt.com/backend-api/codex as the default base URL, bypassing
OPENAI_BASE_URL and the custom model_provider setting entirely.
Setting openai_base_url in config.toml overrides that default for all
auth modes (API key and ChatGPT subscription), so both traffic paths
now flow through the Headroom proxy.
Also adds orphan-cleanup regex for openai_base_url in
_strip_codex_headroom_blocks and 5 new tests covering subscription
routing behaviour.
PR #396's first dry-run on its own changes failed three smoke matrix
entries — two fixed by PR #397's __libc_single_threaded shim (now on
main, this PR is rebased on top), one orthogonal: ubuntu:20.04 + 3.10.
Failure mode on ubuntu:20.04 + 3.10: deadsnakes PPA install path
stopped reliably provisioning python3.10-venv on focal once Ubuntu
20.04 hit End of Standard Support in May 2025. The error is from
apt-get, NOT from `import headroom._core` — the wheel never gets a
chance to load:
E: Unable to locate package python3.10-venv
Continuing to promise wheel-runtime correctness on glibc 2.31 in CI
would require either pulling ESM-tier ubuntu:focal images (paid /
auth-gated) or pinning a specific deadsnakes snapshot URL — neither
of which we want owning long-term.
Floor coverage we keep:
- manylinux_2_28_x86_64 (glibc 2.28 — the floor we promise)
- manylinux_2_28_aarch64 (glibc 2.28 — same)
- ubuntu:22.04 + 3.12 x86_64 (glibc 2.35 — issue #355's reporter env)
- ubuntu:22.04 + 3.12 aarch64 (glibc 2.35 — arm equivalent)
- macos-14 + 3.13 (Apple Silicon native)
Test pin in tests/test_release_workflows.py
(test_release_workflow_has_smoke_import_wheel_gate) only requires the
manylinux floors, ubuntu:22.04, and macos-14, so this drop doesn't
break the structural invariant.
The X1 smoke-import gate (PR #387) catches runtime symbol mismatches
on the wheel before publish, but only at release time. Recent break
patterns were upstream of that:
- #379 (docker bake `name=` regression in PR #376)
- #382 (sdist `os: ubuntu-latest` → `ubuntu-24.04` rename)
- #384 / #385 / #386 (glibc shim alias / link-order iterations)
- #387's own heredoc-indent regression that broke main on the FIRST
release run after merge — the heredoc was inside `bash -ec '...'`,
no PR-time check exercised it
X2 adds a `pull_request:` trigger to release.yml with a NARROW path
filter so the dry-run runs at PR time for changes that affect wheel
layout / release pipeline, but skips for source-only PRs to
`crates/headroom-core` / `crates/headroom-proxy`.
Path filter: release.yml, docker.yml, crates/headroom-py/**,
pyproject.toml, root Cargo.toml, Cargo.lock.
publish-pypi / publish-npm / publish-github-packages / publish-docker
/ create-release all gate on `github.event_name != 'pull_request'`,
so a PR run never publishes — the dry-run is build + collect-dist +
smoke-import only.
concurrency rules:
- PR runs: namespaced by PR number (`pr-N`), cancel-in-progress=true.
- main runs: namespaced by ref_name, cancel-in-progress=false (a
tag-push release that's mid-flight must not be cancelled).
Test pin covers all four invariants (trigger, path filter, publish
gates, concurrency split). 21 tests in test_release_workflows.py,
all green locally.
PR #396's X2 dry-run caught a wheel-import failure on the manylinux_2_28
floor matrix entry (both x86_64 and aarch64). Same class as #355:
ImportError: ... undefined symbol: __libc_single_threaded
`__libc_single_threaded` is a single-byte char added in glibc 2.32.
Newer libstdc++ (gcc 11+) reads it inside `__cxa_thread_atexit_impl`
to elide locking on the single-threaded fast path. ORT prebuilt static
archives compiled with gcc-14.2.1 against glibc-2.38+ headers bake in
the reference. Users with glibc < 2.32 hit ImportError on
`import headroom._core`.
Latent since the ORT artifact bump that started using gcc 14. X1 is
the gate that catches it at release time; X2 caught it at PR time —
exactly as designed.
Fix:
1. glibc_compat.c adds Section B: `char __libc_single_threaded = 0;`
Setting to 0 (multi-threaded) is safe; libstdc++ takes the locked
slow path. Setting to 1 would race in any multithreaded Rust wheel.
2. build.rs adds `-Wl,-u,__libc_single_threaded` so the shim's archive
members are pulled regardless of scan order.
3. audit_wheel_glibc_symbols.py POST_FLOOR_SYMBOLS adds the new
symbol — verified locally: the audit now rejects the failing
PR #396 wheel with the right message.
Companion to the MD5→SHA-256 switch in 98d458f. The hash-pinning test
asserted `hashlib.md5(content.encode()).hexdigest()[:24]`; flip it to
the new function. Also expanded the failure message so the next
person debugging this knows why this gate exists and what they need
to verify if they're tempted to change the hash function again.
CI test (3.10/3.11/3.12/3.13) failed on this single assertion after
the SHA-256 switch; with the test updated, the rest of the
compression_store regression (76 tests) stays green.
CodeQL flagged `compression_store.store()`'s default MD5 cache key as
`py/weak-sensitive-data-hashing` after the explicit_hash refactor in
PR #395 brought the line into a new diff context.
First attempt: `usedforsecurity=False` + `# lgtm[...]` comment to
silence the alert without changing the hash. Both failed — CodeQL
ignores the hashlib parameter and our LGTM marker, the alert stayed
open on the PR.
Second attempt (this commit): drop MD5 entirely. The cache key is for
deduplication / lookup, not security or integrity, so any deterministic
function works. SHA-256[:24] gives the same 96-bit collision space as
MD5[:24] (~280 trillion entries for 50% collision under birthday
bound), is FIPS-clean, and CodeQL won't flag it.
Behaviour impact: zero. The cache is in-memory (no disk persistence),
so the same content always hashes deterministically under whichever
function is in use — there is no upgrade-time mismatch to manage.
Drive-by: pre-commit's sync-plugin-versions hook bumped marketplace +
plugin manifests to 0.20.28 since v0.20.27 was tagged on main.
SmartCrusher's row-drop and opaque-blob paths emitted `<<ccr:HASH ...>>`
markers and stashed the canonical payload only in the Rust process-local
CCR store. /v1/retrieve queries the Python compression_store, so every
retrieve call for a Rust-emitted marker returned 404 even though the
data was held in the Rust store.
Changes:
- Add `explicit_hash` parameter to `CompressionStore.store()`. Required
to mirror entries keyed by hashes produced by another component
(Rust SmartCrusher emits SHA-256[:12]; the Python store's default
is MD5[:24]). Validates the hash is non-empty hex and raises
`ValueError` otherwise — no silent fallback to MD5.
- Wire a Rust → Python store bridge in `headroom/transforms/
smart_crusher.py`. After every `crush()`, `crush_array_json()`,
`compact_document_json()`, and `_smart_crush_content()` call, walk
the rendered output for `<<ccr:HASH>>` markers, fetch the canonical
via `self._rust.ccr_get(hash)`, and mirror it into the Python
compression_store under the same hash via `explicit_hash=`.
- Bridge is best-effort: failures log at debug; compression itself
is never blocked. Marker parsing is structured (JSON walk +
substring scan), no regex.
Tests:
- `tests/test_ccr_row_drop_store_bridge.py` — 9 cases covering:
(1) row-drop populates Python store keyed by marker hash;
(2) `_smart_crush_content` (the runtime path the proxy uses) also
populates the store; (3) passthrough crushes do not write;
(4) `ccr_config.enabled=False` skips both marker emission and store
write; (5) distinct payloads → distinct hashes both retrievable;
(6,7) `explicit_hash` round-trip + non-hex rejection; (8) full
/v1/compress → /v1/retrieve integration matches the issue's
reproducer; (9) unknown hashes still 404.
Addresses #389
Two telemetry env vars existed in the codebase, only one was wired to
the /v1/telemetry endpoint:
- HEADROOM_TELEMETRY (documented opt-out, used by Supabase beacon and
the Telemetry-Warning notice). Honoured: off / false / 0 / no /
disable / disabled.
- HEADROOM_TELEMETRY_DISABLED (undocumented). Was the ONLY one the
collector singleton consulted. Accepted only "1" or "true".
A user setting HEADROOM_TELEMETRY=off (the value from the docs) saw
/v1/telemetry continue to report enabled=true.
Fix: collector calls is_telemetry_enabled() (the same predicate the
beacon uses), so both env vars take effect. HEADROOM_TELEMETRY_DISABLED
remains accepted for back-compat.
Tests: parametrized regression covering all six documented OFF values
and a positive-path test for explicit ON / unset.
Drive-by: convert two pre-existing isinstance(v, (int, float)) to
isinstance(v, int | float) so the pre-commit ruff hook (which scans
the whole file) stops flagging UP038 on every commit that touches this
file.
Addresses #390 (do not auto-close — needs user confirmation in their
own environment after the next release).
The X1 smoke-import job (PR #387) embedded the smoke check as a
`<<PY ... PY` heredoc inside `bash -ec '...'`. The outer bash
single-quote preserves whitespace, so the heredoc body and the
closing `PY` retained their YAML indentation (column 14 inside
`bash -ec`). Bash never found a column-0 `PY` and read past EOF:
bash: line 71: warning: here-document at line 65 delimited by
end-of-file (wanted `PY')
IndentationError: unexpected indent
Process completed with exit code 1
Caught immediately on the post-merge release run on main,
manylinux_2_28_x86_64 / Python 3.11 (glibc 2.28-floor):
https://github.com/chopratejas/headroom/actions/runs/25361396712/job/74362427755
`python -c "<f-string>"` was the obvious next try but reintroduces
single-quote nesting (Python f-strings need quote chars; outer
`bash -ec '...'` cannot contain unescaped single quotes).
Fix: write the smoke script to `${RUNNER_TEMP}/smoke_import.py`
in a new "Stage smoke-import script" step (one heredoc at YAML
`run: |` level — uniform indent strip works fine). Linux job
mounts it via `-v ${RUNNER_TEMP}/smoke_import.py:/smoke_import.py:ro`
and runs `python /smoke_import.py`. macOS host runs the same
file directly. No quoting drift between paths.
Locally validated:
- actionlint clean
- 20/20 tests in test_release_workflows.py pass
- hand-execution of the heredoc + script roundtrip works
This is the second X1 follow-up after PR #387's shellcheck fix.
The original X1 design's gap: no PR-time release dry-run that
would have caught the heredoc on PR #387 itself. X2 (PR-time
dry-run) is the structural fix.
actionlint+shellcheck flagged the bash heredoc on the macOS host
step (release.yml:486) for SC2012 (use find instead of ls) and
SC2086 (quote variables to prevent globbing/word splitting). The
Linux container step had the same pattern but escaped detection
because actionlint can't reach into `docker run bash -ec '...'`.
Replace `ls .../headroom_ai-*-${py_tag}-${py_tag}-${arch_tag}.whl
| head -1` with `find ... -name "..." -print -quit` so the glob
expansion happens in find (handles non-alphanumeric filenames
correctly) and the variables sit inside a quoted -name argument
(no SC2086 trigger). Also replace the diagnostic `ls -la` calls
with portable find variants (-printf for GNU find on Linux,
-exec basename for BSD find on macOS).
Net behaviour identical: still picks one matching wheel (only one
exists per py_tag+arch_tag), still prints all wheels for diagnosis
on miss.
Issue #355 plus the three follow-on hotfixes (#384/#385/#386) all
share a pattern: the wheel is technically valid (clippy passes,
tests pass, auditwheel is happy, the static-symbol audit added in
#384 is happy) but FAILS at runtime on a customer's box because of
a dynamic-link symbol mismatch. None of our pre-publish gates
actually `import headroom._core` on a representative customer
environment. They only build it.
What X1 adds
------------
A `smoke-import-wheels` job that runs after `build-wheels` and
before `publish-pypi` / `publish-docker` / `create-release`.
Matrix (6 jobs in parallel, ~3 min wall-clock):
- `manylinux_2_28_x86_64` + Python 3.11 (the floor we promise)
- `ubuntu:22.04` (glibc 2.35) + Python 3.12 (issue #355's env)
- `ubuntu:20.04` (glibc 2.31) + Python 3.10 (older LTS)
- `manylinux_2_28_aarch64` + Python 3.11 (aarch64 floor)
- `ubuntu:22.04` arm64 + Python 3.12 (aarch64 customer env)
- `macos-14` host + Python 3.13 (Apple Silicon)
Each job downloads its arch's wheel artifact, installs the wheel
matching its Python version inside the container, and runs the
exact command the proxy's `_check_rust_core` runs at startup:
from headroom._core import hello as _rust_hello
If any matrix entry fails, `publish-pypi` / `publish-docker` /
`create-release` are blocked. The matrix tells us exactly which
customer environment combination breaks.
Regression test in tests/test_release_workflows.py:
`test_release_workflow_has_smoke_import_wheel_gate` pins the job's
existence, the required matrix entries, and — critically — the
gating wires (publish-pypi / publish-docker / create-release all
need-and-require-success on the smoke job). A future "this slow
CI step always passes anyway, drop it" refactor fails at PR time.
Companion tests `test_glibc_compat_shim_present_in_headroom_py`
and `test_release_workflow_audits_wheel_glibc_symbols` (added in
#384) cover the static-symbol gate; this PR is the dynamic-link
gate. Both are needed.
PR #385's shim works on x86_64 wheel build but FAILS audit on aarch64
in run 25358313722:
FAIL: headroom_ai-0.20.27-cp310-cp310-manylinux_2_28_aarch64.whl
references symbols above its glibc floor:
__isoc23_strtoll (no version tag, introduced in glibc 2.38)
Diagnosis: cargo's link order on aarch64 happens to place our shim's
static archive BEFORE the ORT prebuilt archives. When the linker
scans our archive, no UND `__isoc23_*` exists yet (ORT hasn't been
scanned), so our shim's `.o` is dropped (no symbol to satisfy). ORT
scans next, registers UND, but our archive isn't rescanned.
Result: `_core.so` still has UND `__isoc23_*` symbols and the audit
rightly rejects the wheel.
On x86_64 the order happened to be the opposite (ORT first → UND
registered → our archive scans next → satisfies → pulled in).
Order is implementation-defined and clearly arch-dependent.
Fix: emit `cargo:rustc-link-arg=-Wl,-u,<sym>` for each `__isoc23_*`
symbol in `build.rs`. `-u <sym>` (a.k.a. `--undefined`) tells the
linker to treat the symbol as undefined at the START of linking,
which forces any archive defining it to be scanned and its members
pulled in regardless of relative archive order. Shim is now
uniformly linked on both x86_64 and aarch64.
Documented inline in `build.rs`. Standard workaround for the
static-library-link-order problem when the consumer scans after
the provider.
PR #384 introduced glibc_compat.c using __attribute__((weak, alias("strtol"))) which fails to compile because GCC requires the alias TARGET to live in the same translation unit. strtol is in libc.so.6, not the .c file. Result: clippy fails on every Linux CI job for every PR + main:
glibc_compat.c: error: '__isoc23_strtol' aliased to undefined symbol 'strtol'
Two-line architectural change: (1) drop the alias attribute, give each __isoc23_* function a plain body that calls the older strtol family; (2) forward-declare the older prototypes ourselves instead of #include <stdlib.h>, otherwise GCC's __REDIRECT_NTH(strtol -> __isoc23_strtol) would silently rewrite our delegation into an infinite recursion.
Symbol-resolution semantics unchanged: on glibc 2.38+, libc's strong __isoc23_strtoll preempts ours via global-scope-first lookup; on glibc < 2.38, ours wins. Either way the symbol resolves and import succeeds.
Both traps documented inline in glibc_compat.c so a future refactor doesn't reintroduce them. PR #384's commit message overstated the validation: I tested the audit script against the broken wheel but did NOT compile the shim itself before merging. Adding the X1 smoke-import gate (separate PR) is what would have caught this.
Issue #355: published headroom_ai-*-manylinux_2_28_*.whl fails to import on Ubuntu 22.04, Debian 11/12, Conda envs with libc < 2.38: 'ImportError: undefined symbol: __isoc23_strtoll'.
Root cause: ORT prebuilt artifacts (downloaded via fastembed's ort-download-binaries-rustls-tls feature) are compiled with gcc-14.2.1 on a glibc-2.38+ host and reference __isoc23_strtoll. Our manylinux build host has glibc 2.38 so the link succeeds; end users with older glibc don't.
Two-part fix: (1) crates/headroom-py/glibc_compat.c provides weak-alias definitions for __isoc23_strtol/strtoll/strtoul/strtoull delegating to the older strtol* family, compiled by build.rs on Linux/glibc only. The dynamic linker prefers glibc's strong symbol when present (>= 2.38) and falls back to ours when not (< 2.38). (2) scripts/audit_wheel_glibc_symbols.py is a release.yml gate that runs objdump -T on every Linux wheel and rejects any UND symbol whose required glibc version exceeds the wheel's manylinux floor.
Validated: the audit correctly rejects the actually-broken v0.20.26 wheel with a precise diagnostic. The shim itself is a tiny static link with zero runtime cost.
Regression tests in tests/test_release_workflows.py pin the shim's load-bearing pieces (.c file, build.rs trigger, [build-dependencies] cc dep) and the audit invocation in release.yml. Future drift fails at PR time, not in the next release.
This is the same bug class as PR #371 (rustls-everywhere). Each instance gets fixed; the audit gate now catches the *class* — any future static linkage that introduces a post-floor symbol gets blocked before publish.
PR #376 pinned the wheel matrix's `os:` from `ubuntu-latest` to
`ubuntu-24.04` (explicit pinning, no semantic change). It silently
disabled the sdist build, whose conditional was
if: matrix.os == 'ubuntu-latest' && matrix.target == 'x86_64-unknown-linux-gnu'
The literal `'ubuntu-latest'` no longer matched, sdist never built,
`release-assets/*.tar.gz` was empty, and `create-release` failed at
gh release upload release-assets/*.tar.gz --clobber
with "no matches found". v0.20.22's release didn't get its sdist on
the GitHub Release.
Fix: key the conditional on `matrix.target` only. sdist is
platform-independent so any single matrix row is fine; using `target`
decouples the sdist build from any future host-runner rename.
Regression test in tests/test_release_workflows.py:
`test_sdist_build_conditional_keyed_on_target_not_os` pins the
target-only form. A future "let's add `os` back for clarity" refactor
will fail at PR time, not 8 minutes into a release.
This is a follow-up to PR #379 (docker bake `name=`) — same root
cause class: PR #376's matrix-shape changes silently broke a
downstream conditional whose literal value was tied to the OLD
matrix shape. Both fixes are now in place + pinned by tests.
OpenAI exposes `prompt_cache_key` to pin prefix-cache lookups to a
tenant-stable identity (preventing org-wide cache collisions). Most
clients don't set it. This PR auto-derives one from the request's
structural prefix `(model, system, tools)` and injects it on PAYG
OpenAI requests where the customer has not provided their own value.
Universal safety contract:
- Auth-mode gate: only AuthMode::Payg bodies are mutated. OAuth and
Subscription requests pass through byte-equal (preserves Phase A
passthrough invariant). Both gates emit `e4_skipped` events.
- Customer-set values win: `prompt_cache_key` already present →
skip injection. Empty strings count as absent.
- Idempotent: same `(model, system, tools)` always derives the
same key, so re-running yields identical bytes.
Key derivation: `hex(sha256(model || sha256(system) ||
sha256(tools)))[..32]` — 128 bits of collision resistance, 32 hex
chars on the wire. User/assistant message content is deliberately
excluded (they vary per turn; including would defeat caching).
Observability: every skip emits `e4_skipped` with a stable reason
(`auth_mode` / `key_present` / `not_an_object`); every successful
injection emits `e4_applied` with only the first 8 hex chars of the
key (full key is identifying material — never logged).
Hook point: `forward_http` in `crates/headroom-proxy/src/proxy.rs`,
between the live-zone dispatcher's body decision and the upstream
forward. Auth-mode is already classified at request entry.
Affects pre-existing dispatcher byte-fidelity tests (chat
completions, responses, responses streaming) — they previously
asserted byte-equality with no auth header (default PAYG). Updated
those tests to send an OAuth bearer so they keep their byte-equality
intent independent of E4. The E4 byte-mutation behaviour has its own
test matrix in `integration_e4_openai_cache_key.rs`.
Files added:
- `crates/headroom-proxy/src/cache_stabilization/openai_cache_key.rs`
- `crates/headroom-proxy/tests/integration_e4_openai_cache_key.rs`
Files modified:
- `crates/headroom-proxy/src/cache_stabilization/mod.rs` — `pub mod
openai_cache_key;` (only shared file with parallel E1/E2/E3/E6 PRs)
- `crates/headroom-proxy/src/proxy.rs` — call site + helper
- `crates/headroom-proxy/Cargo.toml` — promote sha2 to runtime dep
- 3 integration test files — auth-mode plumbing for byte-equality
invariants
Auto-place a single ephemeral cache_control marker on the last tool
definition for PAYG-classified Anthropic requests when the customer
has not placed any markers. Hand-rolled SDK callers and smaller
agents (Aider/Continue/curl) get prompt-cache hits without learning
Anthropic's marker API.
Safety contract:
1. Auth-mode (caller-side, F1 classify): PAYG only. OAuth and subscription requests pass through byte-equal — mutating their bytes risks looking like cache-evasion to upstream.
2. Customer-placement-wins: walks system (array form), messages[].content (array form), and tools[] top-level. Any pre-existing marker -> skip with reason=marker_present.
3. Idempotency: re-running on a body that already has our marker falls into gate (2).
First-ship policy: place ONE marker on the last tool. The system/message-history/4th slots are documented but require production telemetry to enable.
Bedrock invoke + invoke-streaming hard-code OAuth so AWS SigV4-signed requests never get auto-placed (Bedrock is an IAM channel, not PAYG).
Observability: tracing::info! event=e3_applied / event=e3_skipped (reason in {auth_mode, marker_present}) so dashboards can confirm the gates fire as designed.
Files added: cache_stabilization/anthropic_cache_control.rs (module + 15 unit tests); tests/integration_e3_anthropic_cache_control.rs (5 integration tests covering all three gates).
Files modified: cache_stabilization/mod.rs; compression/live_zone_anthropic.rs (new auth_mode parameter on compress_anthropic_request); proxy.rs; bedrock/invoke.rs; bedrock/invoke_streaming.rs.
Adds a Rust-native Vertex AI publisher route ahead of the LiteLLM
Python converter (which dropped `thinking`, `redacted_thinking`,
`document`, `image`, `server_tool_use`, `mcp_tool_use` block kinds —
the P4-37 / P4-38 bug). After this PR the Vertex `:rawPredict` and
`:streamRawPredict` calls survive byte-equal upstream and benefit
from the live-zone Anthropic dispatcher (PR-B-series) running over
the body — same behaviour as `/v1/messages`.
New module `crates/headroom-proxy/src/vertex/`:
- `mod.rs` — single dispatch handler at the
`/v1beta1/.../models/:model_action` route. Splits the trailing
`:<verb>` segment with `str::rsplit_once(':')` (no regex) and
flips an `attach_sse_tee` flag to dispatch to the streaming or
non-streaming arm. Both verbs share one axum route shape because
matchit can't distinguish two patterns that overlap on a
parameter.
- `envelope.rs` — `VertexEnvelope` parser. Confirms
`anthropic_version` present + `model` field absent (the two
fingerprints of the Vertex envelope vs `/v1/messages`).
- `adc.rs` — `TokenSource` trait + `GcpAdcTokenSource` (production,
`gcp_auth` 0.12) + `StaticTokenSource` (tests). Caches tokens
with a 60s refresh-ahead-of-expiry window. Emits structured
`event = "vertex_adc_token_refreshed"` per refresh.
- `raw_predict.rs` — POST handler + shared `forward_vertex_request`.
Buffers body, parses envelope, runs live-zone Anthropic
compression, fetches ADC bearer, attaches
`Authorization: Bearer <token>` (overwrites client-supplied
Authorization header), forwards. SSE telemetry tee for the
streaming verb reuses PR-C1's `AnthropicStreamState` directly
(Vertex streams plain SSE, unlike Bedrock's binary EventStream).
- `stream_raw_predict.rs` — module-level docs + alias to the
shared dispatcher (the streaming-vs-non-streaming difference is
one boolean flag inside the shared forwarder).
Modifications:
- `proxy.rs::build_app` — registers the single Vertex route.
- `proxy.rs::AppState` — new `vertex_token_source: Arc<dyn TokenSource>`
field. Production constructs `GcpAdcTokenSource` lazily (no GCP
call until first `bearer()`); tests inject `StaticTokenSource`
via the new `AppState::with_token_source` helper.
- `config.rs` — adds `--vertex-region` / `HEADROOM_PROXY_VERTEX_REGION`
(default `us-central1`, observability tag only — the upstream URL
is `--upstream`) and `--vertex-adc-scope` /
`HEADROOM_PROXY_VERTEX_ADC_SCOPE` (default `cloud-platform`).
- `Cargo.toml` (workspace + proxy) — adds `gcp_auth = "0.12"` and
`async-trait = "0.1"`.
- `tests/common/mod.rs` — `start_proxy_with_state` accepts both
config + state customizers; `install_static_token_source` helper
for tests.
`crates/headroom-proxy/tests/integration_vertex_raw_predict.rs` —
all five tests pass:
1. `native_envelope_round_trip_byte_equal` — Vertex-shape body
(with `anthropic_version`, no `model`) round-trips SHA-256
byte-equal upstream.
2. `adc_bearer_token_signed_correctly` — `Authorization: Bearer
<static-test-token>` reaches upstream verbatim and OVERWRITES a
client-supplied Authorization header.
3. `thinking_block_preserved` — request with `thinking` (incl.
signature) + `redacted_thinking` (incl. opaque `data`) blocks
round-trips byte-equal even with `LiveZone` compression mode
enabled. This is the P4-37 / P4-38 teeth.
4. `stream_raw_predict_sse_handled` — `:streamRawPredict` proxies
an Anthropic SSE response (full `message_start` →
`content_block_delta` → `message_stop` sequence) back to the
client without corruption; SSE content-type preserved end-to-end;
bearer attached.
5. (bonus, no-silent-fallback contract)
`adc_failure_returns_5xx_no_silent_forward` — when the token
source returns `Err`, the proxy returns 5xx and never reaches
upstream. Verifies the `event = "vertex_adc_fetch_failed"`
error path.
Workspace: `cargo test --workspace` green; `cargo clippy --workspace
-- -D warnings` clean; `make ci-precheck` passes.
- No silent fallbacks: ADC failure → structured 5xx, never an
unauthenticated forward.
- No hardcodes: every knob (region, ADC scope, upstream URL) is
CLI-flag + env-var configurable.
- No regexes: axum path parameters + `str::rsplit_once` only.
- Comprehensive structured logs: `event` field on every decision
point — `vertex_envelope_parsed`, `vertex_envelope_invalid`,
`vertex_compression_skipped`, `vertex_compression_applied`,
`vertex_adc_token_refreshed`, `vertex_adc_fetch_failed`,
`vertex_streaming_pipeline_active`, `vertex_sse_stream_closed`,
`vertex_forwarded`, `vertex_unknown_verb`, etc.
- Performant: no body clone; ADC token cached + refreshed
ahead-of-expiry, not fetched per request.
- Comprehensive tests: realistic Anthropic block content
(signature payload, redacted_thinking opaque blob) in
`thinking_block_preserved`.
The local `gcloud auth application-default print-access-token`
returns no credentials, so manual validation against a real Vertex
endpoint is not possible in this PR. Follow-up: the user runs
`gcloud auth application-default login` once and exercises a live
Vertex request — should be a no-code-change check.
PR-D1 (Bedrock native) is running concurrently and will land its
own envelope module at `crates/headroom-proxy/src/bedrock/envelope.rs`.
The two envelope modules are intentionally siblings (not a shared
trait) — the shapes differ (Bedrock has a different
`anthropic_version` value, no `model` field, AWS SigV4 instead of
GCP ADC), and a premature shared abstraction would obscure the
provider-specific contracts. Whichever PR merges second rebases
without conflict.
Retires P4-38 (and the Vertex parts of P4-39); marketplace BYOC
pitch (per project memory) gets one more native provider.
Per-session SHA-256 fingerprint of the cache hot zone (system / tools /
first 3 messages) with structured-log emission on drift. Detector is
read-only: never mutates request bytes, preserves the Phase A
passthrough invariant. Surfaces invisible cache busts (system prompt
edited mid-session, tools reshuffled, early message changed) without
rewriting them.
* crates/headroom-proxy/src/cache_stabilization/drift_detector.rs:
StructuralHash (system, tools, early_messages digests),
compute_structural_hash, observe_drift, derive_session_key,
DriftState (LRU bounded to 1000 sessions in production).
* Session keys derive from Authorization / x-api-key / client IP /
(IP, user-agent). Bearer tokens and API keys are SHA-256 hashed
before they ever reach the log line; the raw secret is never logged.
* Wired into forward_http after the body is buffered, before the
compression dispatcher runs. Skips paths whose wire shape is not
Anthropic / OpenAI Chat / OpenAI Responses.
* AppState gains drift_state: DriftState. Bedrock unit-test
literal-construction sites updated.
* 14 unit tests + 1 integration test covering first-request,
no-drift, per-dimension drift, multi-dim drift, LRU eviction,
non-mutation invariant, and bearer-token-never-logged.
Adds lru = "0.12" and promotes sha2 = "0.10" to a normal dependency
on headroom-proxy.
Adds an observation-only detector that scans inbound LLM request
bodies for content that busts prompt-cache hits and emits one
structured WARN log per finding. Strictly read-only — never
mutates the request, the Phase A bytes-in==bytes-out invariant
still holds.
Detection patterns (no regex, per Realignment build constraints):
- ISO-8601 timestamps (byte-position check at 4=- 7=- 10=T 13=: 16=:)
- UUID v4 (36 chars, hyphens at 8/13/18/23, version nibble 4 at pos 14,
RFC 4122 §4.4 variant nibble at pos 19)
- ID-named JSON keys (request_id / trace_id / session_id /
correlation_id) with non-empty values
Scope:
- Anthropic body shape: system, messages[].content (string or blocks),
tools[].description, tools[].input_schema (recursive).
- OpenAI body shape: messages[].content, tools[].function.description,
tools[].function.parameters (recursive).
- Other paths (Bedrock / Vertex etc.) deferred to Phase E follow-up.
Findings capped at 10 per request; sample truncated to 80 bytes
(UTF-8 boundary safe) to avoid logging bulk customer data.
Files:
- New crates/headroom-proxy/src/cache_stabilization/{mod,volatile_detector}.rs
- crates/headroom-proxy/src/lib.rs: pub mod cache_stabilization
- crates/headroom-proxy/src/proxy.rs: detector hook in the
buffered-body branch, before the compression dispatcher
- New tests/integration_volatile_detector.rs (subscriber capture
asserts the WARN line + byte-equal upstream body)
12 unit tests (timestamp, UUID, ID-field nesting, stable content,
cap, non-mutation, ApiKind shape isolation, empty-ID-value guard,
RFC 3339 space separator, non-v4 UUID rejection, UTF-8 truncate
safety, endpoint mapping) and 1 integration test (warn emitted
+ upstream body byte-equal). cargo fmt / clippy --workspace -D
warnings / workspace test all pass; full make ci-precheck green.
PR #376's per-arch fan-out correctly removed `bake-file-tags` from
the docker-build step (tags belong on the multi-arch manifest, not
on per-arch images). But that left bake without ANY reference for
the push target — no tags AND no explicit `name=` in the output
spec. Every release-time docker-build job failed with the
misleading message:
ERROR: tag is needed when pushing to registry
Buildx's actual constraint is "no tags AND no `name=` in the
output = no push target." Since push-by-digest discards tags
anyway, the fix is to specify `name=<registry>/<image>` directly
in the `*.output` spec. Tags are still applied later, only on the
multi-arch manifest by `docker-manifest`.
Regression test in tests/test_release_workflows.py:
`test_docker_per_arch_build_specifies_image_name_in_output`
pins the `name=` substring so a future "the labels block already
has the registry, surely buildx can figure it out" refactor will
fail at PR time rather than 2 minutes into release.
Phase D close. Adds the operator-facing observability surface that
PRs D1 (native invoke) and D2 (streaming EventStream) deferred, and
wires the Phase F PR-F1 auth-mode classifier into the Bedrock route
so downstream cache/compression policy gates have something to read.
Changes
-------
* New `bedrock::auth_mode_layer` middleware. Classifies every
inbound Bedrock request via F1's `classify`, coerces the result
to `AuthMode::OAuth` per the Bedrock policy matrix (SigV4 IAM is
OAuth-equivalent), and stores the resolved value in
`request.extensions()` so PR-F2/F3 can read it without
re-classifying. Mismatches are logged at WARN with
`event=bedrock_auth_mode_unexpected` — no silent coercion.
* New `observability` module with three Prometheus families:
- `bedrock_invoke_count_total{model, region, auth_mode}` (counter)
- `bedrock_invoke_latency_seconds{model, region}` (histogram)
- `bedrock_eventstream_message_count_total{model, region, event_type}`
(counter)
Registered lazily via `OnceLock` so per-request work is just
`inc_with_label_values` / `observe`. Latency observed via an
RAII `LatencyGuard` so every error path is instrumented; a
future regression that adds a new return path can't drop the
observation.
* New `GET /metrics` endpoint serves the registry in Prometheus
text format. Mounted unconditionally — no feature flag gate — so
scrape works regardless of which provider routes are mounted.
* Bedrock invoke + invoke-streaming handlers now extract
`Extension<AuthMode>`, log it in their entry breadcrumbs
(`event=bedrock_invoke_received`, `event=bedrock_invoke_streaming_received`),
and pass `model`/`region` into `translate_stream` so per-message
metrics carry the right labels.
* Operator docs at `docs/bedrock.md`: AWS credential chain,
region/endpoint config, supported model IDs (`anthropic.*`
literal-match — no regexes), compression behaviour, sample
PromQL queries, structured-log correlation, rollback path.
Tests added (6, all green)
--------------------------
Auth-mode (`integration_bedrock_authmode.rs`):
1. `bedrock_classified_as_oauth` — empty headers → OAuth in
extensions.
2. `oauth_policy_passthrough_prefer` — body byte-equal upstream;
no auto cache_control / prompt_cache_key injected.
Metrics (`integration_bedrock_metrics.rs`):
3. `metrics_increment_per_invoke` — 3 invokes → counter=3 with
correct labels.
4. `metrics_observe_latency` — 1 invoke → histogram count=1,
sum>0.
5. `eventstream_metrics_per_message_type` — 5 chunks → counter=5
with `event_type=chunk`.
6. `metrics_endpoint_serves_scrape` — `/metrics` returns 200,
`text/plain`, all three metric families' HELP/TYPE lines
present.
Each metrics test owns a unique (model, region) tuple so the
global `prometheus` registry — shared across parallel tests in
the same binary — gives each test isolated label rows. Without
isolation, parallel tests cross-contaminate counters.
Constraints honoured
--------------------
* No silent fallbacks — auth-mode coercion is logged at WARN.
* No hardcodes — region from `--bedrock-region`, model from axum
path parameter.
* No regexes — vendor prefix is literal `anthropic.`.
* Comprehensive structured logs — every metric increment paired
with `tracing::debug!` carrying the same labels for incident
correlation.
* Performant — `OnceLock`-cached descriptors, RAII guard, total
D3 overhead well under 1us per request.
* Cardinality bounded — labels driven by config + bounded enums,
never by user-controlled bytes.
Live cloud validation deferred
------------------------------
The wiremock-backed integration tests are the canonical correctness
gate for D3. A real Bedrock smoke test requires `bedrock:InvokeModel`
permissions in the developer's AWS account and is documented in
`docs/bedrock.md` — both D1 and D2 hit sandbox permission issues
trying this path; D3 follows the same convention.
Stacked on
----------
PR #364 (D1 native invoke), PR #365 (D2 streaming EventStream),
PR #366 (F1 classifier helper). Merge those first; this PR will be
rebased onto main once they land.
GitHub-hosted Linux arm64 runners (`ubuntu-24.04-arm`) went GA in Aug
2025 and are free for public repositories. Switching the aarch64
wheel + the multi-arch docker matrix off `ubuntu-latest`+QEMU onto
the native runner cuts wall-clock on both surfaces.
release.yml — build-wheels matrix
* `aarch64-unknown-linux-gnu`: `ubuntu-latest` → `ubuntu-24.04-arm`.
maturin-action still runs inside `quay.io/pypa/manylinux_2_28_aarch64`,
but the container now executes natively on an aarch64 kernel
instead of through QEMU emulation. Aarch64 wheel build drops
from ~50–60 min to ~10 min.
* `x86_64-unknown-linux-gnu`: `ubuntu-latest` → `ubuntu-24.04`
(pin the moving alias for reproducibility; no semantic change).
docker.yml — fan-out + manifest merge
* Pre-#377: one `docker-variant-tags` matrix job per variant on
`ubuntu-latest`, using bake's `platforms = [amd64, arm64]` with
QEMU for the arm64 leg. ~1h per variant, 8 variants.
* Post-#377: split into `docker-build` (variant × arch = 16
parallel jobs, each on its native runner, single-platform
push-by-digest) and `docker-manifest` (per variant, merges the
two arch digests into a multi-arch tagged manifest with
`docker buildx imagetools create`, signs the index manifest
with cosign). Wall-clock drops from ~1h per variant to ~10 min.
* `docker/setup-qemu-action` removed — there's no QEMU left.
* Per-(variant, arch) GHA cache scopes so the two arches don't
collide on cache keys.
* `promote-latest` rewired to depend on `docker-manifest`.
Behavior change: cosign now signs only the multi-arch index digest
per variant, not each per-platform image. `cosign verify <repo>:tag`
(the typical flow) is unchanged because cosign resolves the tag to
the index digest. Verifiers pinning a specific per-arch digest will
need to verify the index digest instead.
Regression tests in tests/test_release_workflows.py:
* `test_aarch64_wheel_uses_native_arm64_runner` — pins the
aarch64 row to `ubuntu-24.04-arm` (and the amd64 row to
`ubuntu-24.04`, not `-latest`), so a future "let me unify on
ubuntu-latest" refactor surfaces the QEMU regression at PR time.
* `test_docker_workflow_builds_on_native_arch_runners` — pins the
fan-out matrix's arch entries, asserts push-by-digest, asserts
`setup-qemu-action` is absent from non-comment lines, asserts
the manifest-merge job exists.
Verified:
* Both workflow files parse as valid YAML with the expected job
graph (`docker-build` → `docker-manifest` → `promote-latest`,
16 fan-out jobs, 8 manifest jobs).
* `docker buildx imagetools inspect <tag> --format '{{ json . }}'`
exposes the index digest at `.manifest.digest` (confirmed via
Docker's official reference).
* `ubuntu-24.04-arm` is the correct GitHub-hosted runner label
(GA 2025-08-07, free for public repos).
* `make ci-precheck-rust` and `make ci-precheck-python` both pass
locally; `tests/test_release_workflows.py` is 15/15 green
(13 existing + 2 new).
Root cause: `headroom-ai[all]==0.20.16` fails to install on Python 3.13
because `rapidocr-onnxruntime` 1.4.0–1.4.4 wheels declare
`requires-python: <3.13,>=3.6`. After 1.4.x the rapidocr ecosystem
split: `rapidocr-onnxruntime` (bundled-ORT, capped at <3.13) vs
`rapidocr` 3.x (engine-agnostic core, supports 3.13+, returns
RapidOCROutput dataclass instead of v1's tuple).
Fix:
1. pyproject.toml — environment-marker hybrid in [image]:
- rapidocr-onnxruntime>=1.4.0,<2; python_version<'3.13'
- rapidocr>=3.0,<4; python_version>='3.13'
- onnxruntime>=1.7,<2; python_version>='3.13'
ORT remains the engine on every Python version; bundle and speed
unchanged, just split into two packages on 3.13+.
2. headroom/image/compressor.py — runtime adapter:
_resolve_rapidocr() tries v1 first, falls back to v3 when v1 is
missing, returns (None, None) when neither installed. Cached at
module scope. Detection at runtime (not Python-version-based) so
users can install either package on any Python version.
_ocr_extract branches on resolved api_version:
- v1: (list[(box, text, score)], elapsed) tuple — unchanged
- v3: RapidOCROutput dataclass with .txts / .scores / .boxes
attrs (each may be None when nothing detected)
Defensive None-handling, length-mismatch detection, structured
log events for both branches.
Smoke test (real install verified before commit):
pip install rapidocr onnxruntime pillow
→ result type: RapidOCROutput
→ fields: txts (None when empty), scores (None when empty), boxes
Confirms the v3 None-coercion is necessary.
Tests: 11 new unit tests in tests/test_image_ocr_api_compat.py covering:
- Resolver: v1 preferred, v3 fallback, both missing
- v1 path: tuple parses, low-confidence None, empty result None
- v3 path: dataclass parses, low-confidence None, None attrs handled,
mismatched lengths logged + None
- Backend missing: returns None gracefully
All 11 pass; `make ci-precheck` PASSED.
Closes#372.
