mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
## Description The native Bedrock path (Phase D) compresses + signs Anthropic-on-Bedrock requests, but two real-world cases slipped through, and the native binary that powers it was never shipped. This PR closes those gaps as a focused set of give-backs. Aligns with the Rust migration plan (see below). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Cross-region inference-profile detection** via a new `bedrock::vendor` module (`canonical_vendor()`), following the design proposed in #953: strip a known geo prefix (`eu.`/`us.`/`apac.`/`global.`) then match the canonical vendor. Geo-prefixed Anthropic profiles (`eu.anthropic.…`) now get live-zone compression instead of being silently skipped; geo-prefixed non-Anthropic vendors stay correctly excluded. - **Converse-body compression (two parts)**: 1. `run_anthropic_compression` no longer bails to passthrough when the body lacks an InvokeModel `anthropic_version` envelope; envelope re-emit stays gated on successful parse. 2. The **live-zone dispatcher now recognizes Bedrock Converse content blocks**. Converse blocks carry no `type` discriminator (the variant is the key: `{"text": …}` vs Anthropic's `{"type":"text","text":…}`), so real Converse user-message text was still passing through uncompressed. A typeless block whose `text` is a JSON string now routes through the same surgical text path. Anthropic blocks always carry `type`, so the Anthropic path is byte-for-byte unchanged; non-text Converse blocks (`{"image":…}`, `{"toolUse":…}`) stay unrecognized and no-op. - **Correct `/converse` upstream routing**: the non-streaming handler resolved the upstream action from a hard-coded `"invoke"`, so `/converse` requests were forwarded to Bedrock's `/invoke` endpoint. It now resolves the action from the inbound path (`extract_invoke_action`), mirroring the streaming handler's `extract_streaming_action`. SigV4 signs the same URL it forwards, so the signature stays consistent. - **`aws-config` `sso` feature**: SSO profiles now resolve through the default credential chain for SigV4 — the credential chain in `docs/bedrock.md` already promised SSO; this makes the code match. - **Ship the `headroom-proxy` binary in published images** (`Dockerfile`): built in the builder stage (`--locked`, with the cargo registry cache mounted at `CARGO_HOME`) and copied into both the debian and distroless runtime images. - **Docs** (`docs/bedrock.md`): document cross-region inference profiles and a "Running the proxy" section. AWS credentials mount at `/home/nonroot/.aws` (the default nonroot image home) where the SDK looks for `~/.aws`, with a note on the root-image alternative. ## Related issues - Closes #976 — ship the `headroom-proxy` binary in published images (this PR implements the exact fix proposed there). - Addresses the **cross-region inference-profile** half of #953 via its proposed `canonical_vendor()` design. Non-Anthropic vendor compression parity (Nova/GLM/MiniMax/ Kimi) is the natural follow-up — `bedrock::vendor` is the shared resolver it can build on. - Extends the native Bedrock InvokeModel compression requested in #734 (the Bedrock slice of #510) to cross-region profiles and Converse bodies. - Partially enables #181 (native, Python-free packaging): the native binary now ships in the images, though full Python-free distribution remains out of scope. ## Alignment with the Rust migration plan Per `docs/spec/022-rust-migration.md`, the migration is **proxy-first**: `headroom-proxy` is the deployable Rust artifact, native routes replace Python passthroughs one at a time (Stage 4 = provider expansion, Bedrock included), and the binary is meant to be "built, tested, and **released together with the Python package**." Two ways this PR advances that: - The binary-in-images change makes the codebase do what the spec already states (ship the artifact) — closing the gap that forced downstreams to build from source. - Hardening the native Bedrock route (cross-region, Converse routing + body compression) is exactly the Stage-4 provider-expansion work, keeping the native path at parity with real traffic so it can be the default rather than a passthrough. ## Testing - [x] Unit tests pass (`cargo test -p headroom-core -p headroom-proxy` — full suites, 0 failures) - [x] Linting passes (`cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings`) - [x] Formatting passes (`cargo fmt -- --check`) - [x] New tests added — `bedrock::vendor` (foundation + inference-profile matching), `extract_invoke_action` + converse upstream URL, and live-zone Converse text-block routing (`block_has_string_text_field`, converse-vs-anthropic dispatch equivalence). - [x] Manual testing performed ### Test Output ```text $ cargo test -p headroom-core -p headroom-proxy # all suites: ok, 0 failed $ cargo clippy -p headroom-core -p headroom-proxy --all-targets -- -D warnings # Finished, no warnings $ cargo fmt -- --check # clean # image validation (local, proxy/code extras): $ docker build --target runtime ... # debian: /usr/local/bin/headroom-proxy, --help OK $ docker build --target runtime-slim ... # distroless: binary links + --help OK ``` ## Real Behavior Proof - Environment: native Bedrock proxy against `bedrock-runtime.eu-west-2`, SSO profile, model `eu.anthropic.claude-haiku-4-5-20251001-v1:0`. - Exact command / steps: POST a large multi-turn Converse body to `/model/eu.anthropic.claude-haiku-4-5-20251001-v1:0/converse`; separately build the `runtime` + `runtime-slim` targets and run `/usr/local/bin/headroom-proxy --help`. - Observed result: before — `bedrock_compression_skipped` (geo-prefixed id not recognized), forwarded uncompressed to the wrong `/invoke` upstream; after — geo-prefixed id recognized, `/converse` forwarded to the `/converse` upstream, live-zone dispatcher compresses the Converse user-message text, measurable token savings. Images contain a runnable `headroom-proxy` in both variants. - Not tested: non-Anthropic vendor compression parity (#953 follow-up); Converse `toolResult` nested-text compression (follow-up — only top-level Converse text blocks compress today). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - An earlier revision flipped the EventStream `Accept` default (`*/*`/absent → passthrough); **dropped** — `*/*` is what most clients (incl. reqwest and the proxy's own metrics tests) send while expecting SSE, so forcing passthrough breaks the standard SSE path. - The binary build adds the native-proxy compile to the image build; happy to gate it behind a build arg if maintainers prefer it opt-in. - Addressed a Copilot review round: corrected the `/converse` upstream routing, the stale `run_anthropic_compression` comment, the Dockerfile cargo cache mount + `--locked`, and the nonroot AWS-credentials docs example.
140 lines
5.3 KiB
Docker
140 lines
5.3 KiB
Docker
ARG PYTHON_VERSION=3.13
|
|
ARG UV_VERSION=0.11.18
|
|
ARG DISTROLESS_IMAGE=gcr.io/distroless/python3-debian13
|
|
ARG PYTHON_SITE_PACKAGES=/usr/local/lib/python${PYTHON_VERSION}/site-packages
|
|
|
|
# ---- Build stage: compile native extensions, build wheel ----
|
|
FROM python:${PYTHON_VERSION}-slim AS builder
|
|
|
|
ARG UV_VERSION
|
|
|
|
# build-essential / g++ for any C extension wheels uv may need to build
|
|
# from source. curl + ca-certificates are required by the rustup
|
|
# bootstrap below. patchelf for maturin's wheel-link repair on linux.
|
|
# No OpenSSL system deps required: the rustls-everywhere refactor
|
|
# eliminated `openssl-sys` from our build tree by switching fastembed
|
|
# to `hf-hub-rustls-tls` + `ort-download-binaries-rustls-tls`.
|
|
RUN apt-get update && \
|
|
apt-get install -y --no-install-recommends \
|
|
build-essential \
|
|
g++ \
|
|
curl \
|
|
ca-certificates \
|
|
patchelf \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
RUN python -m pip install --no-cache-dir uv==${UV_VERSION}
|
|
|
|
# Rust toolchain for the headroom._core extension. With single-wheel
|
|
# architecture (post-#355), `pip install -e .` invokes maturin via
|
|
# pyproject.toml's [build-system], which calls cargo. No more separate
|
|
# headroom-core-py package.
|
|
ENV CARGO_HOME=/usr/local/cargo \
|
|
RUSTUP_HOME=/usr/local/rustup \
|
|
PATH=/usr/local/cargo/bin:${PATH}
|
|
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs \
|
|
| sh -s -- -y --no-modify-path --profile minimal -c rustfmt -c clippy --default-toolchain 1.95.0
|
|
|
|
WORKDIR /build
|
|
|
|
# Copy the full set of files maturin needs to build the wheel: the root
|
|
# pyproject.toml + Cargo workspace + Rust crates + Python source. The
|
|
# uv install builds + installs the wheel in one shot.
|
|
COPY pyproject.toml uv.lock README.md ./
|
|
COPY Cargo.toml Cargo.lock rust-toolchain.toml ./
|
|
COPY crates/ crates/
|
|
COPY headroom/ headroom/
|
|
|
|
ARG HEADROOM_EXTRAS=proxy,code
|
|
RUN --mount=type=cache,target=/root/.cache/uv \
|
|
--mount=type=cache,target=/root/.cargo/registry \
|
|
--mount=type=cache,target=/build/target \
|
|
uv pip install --system ".[${HEADROOM_EXTRAS}]"
|
|
|
|
# Build-stage smoke check: verify the extension loads end-to-end inside
|
|
# the build image before we copy site-packages into the runtime image.
|
|
# If this fails, the runtime image would fail Phase A0's fail-loud
|
|
# startup check on every restart. Run from /tmp so cwd doesn't shadow
|
|
# site-packages with /build/headroom/ (which has no _core.so since
|
|
# maturin installed the .so into site-packages).
|
|
RUN cd /tmp && python -c "from headroom._core import DiffCompressor, SmartCrusher; \
|
|
print(f'build-stage rust core verify OK: {DiffCompressor.__name__}, {SmartCrusher.__name__}')"
|
|
|
|
# Build the native Rust reverse proxy binary and stage it for the runtime
|
|
# images (issue #976). These images already run "the proxy"; bundling the
|
|
# native `headroom-proxy` binary lets operators front the Python proxy with
|
|
# the Rust SigV4 / live-zone compression path from the same image. The
|
|
# binary is copied out of the cache-mounted target dir into a persistent
|
|
# path so the COPY in the runtime stages can pick it up.
|
|
RUN --mount=type=cache,target=/usr/local/cargo/registry \
|
|
--mount=type=cache,target=/build/target \
|
|
cargo build --release --locked --bin headroom-proxy && \
|
|
cp target/release/headroom-proxy /usr/local/bin/headroom-proxy
|
|
|
|
# ---- Runtime stage (python-slim): supports root/nonroot via build arg ----
|
|
FROM python:${PYTHON_VERSION}-slim AS runtime-slim-base
|
|
|
|
ARG RUNTIME_USER=nonroot
|
|
ARG PYTHON_SITE_PACKAGES
|
|
|
|
RUN apt-get update && \
|
|
apt-get install -y --no-install-recommends curl && \
|
|
rm -rf /var/lib/apt/lists/*
|
|
|
|
COPY --from=builder ${PYTHON_SITE_PACKAGES} ${PYTHON_SITE_PACKAGES}
|
|
COPY --from=builder /usr/local/bin/headroom /usr/local/bin/headroom
|
|
# Native Rust reverse proxy binary (issue #976).
|
|
COPY --from=builder /usr/local/bin/headroom-proxy /usr/local/bin/headroom-proxy
|
|
|
|
RUN mkdir -p /home/nonroot /data && \
|
|
if [ "$RUNTIME_USER" = "nonroot" ]; then \
|
|
groupadd --gid 1000 nonroot && \
|
|
useradd --uid 1000 --gid nonroot --create-home nonroot && \
|
|
mkdir -p /home/nonroot/.headroom && \
|
|
chown -R nonroot:nonroot /data /home/nonroot; \
|
|
else \
|
|
mkdir -p /root/.headroom; \
|
|
fi
|
|
|
|
USER ${RUNTIME_USER}
|
|
WORKDIR /home/nonroot
|
|
|
|
ENV HEADROOM_HOST=0.0.0.0 \
|
|
PYTHONUNBUFFERED=1 \
|
|
PYTHONDONTWRITEBYTECODE=1
|
|
|
|
EXPOSE 8787
|
|
|
|
HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
|
|
CMD ["curl", "--fail", "--silent", "http://127.0.0.1:8787/readyz"]
|
|
|
|
ENTRYPOINT ["headroom", "proxy"]
|
|
CMD ["--host", "0.0.0.0", "--port", "8787"]
|
|
|
|
FROM ${DISTROLESS_IMAGE} AS runtime-slim
|
|
|
|
ARG RUNTIME_USER=nonroot
|
|
ARG PYTHON_SITE_PACKAGES
|
|
|
|
COPY --from=builder ${PYTHON_SITE_PACKAGES} ${PYTHON_SITE_PACKAGES}
|
|
# Native Rust reverse proxy binary (issue #976).
|
|
COPY --from=builder /usr/local/bin/headroom-proxy /usr/local/bin/headroom-proxy
|
|
|
|
USER ${RUNTIME_USER}
|
|
WORKDIR /app
|
|
|
|
ENV HEADROOM_HOST=0.0.0.0 \
|
|
PYTHONUNBUFFERED=1 \
|
|
PYTHONDONTWRITEBYTECODE=1 \
|
|
PYTHONPATH=${PYTHON_SITE_PACKAGES}
|
|
|
|
EXPOSE 8787
|
|
|
|
HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
|
|
CMD ["python3", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8787/readyz', timeout=5)"]
|
|
|
|
ENTRYPOINT ["python3", "-m", "headroom.cli", "proxy"]
|
|
CMD ["--host", "0.0.0.0", "--port", "8787"]
|
|
|
|
# Default published image remains python-slim runtime
|
|
FROM runtime-slim-base AS runtime
|