mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
Adds a Rust-native Vertex AI publisher route ahead of the LiteLLM
Python converter (which dropped `thinking`, `redacted_thinking`,
`document`, `image`, `server_tool_use`, `mcp_tool_use` block kinds —
the P4-37 / P4-38 bug). After this PR the Vertex `:rawPredict` and
`:streamRawPredict` calls survive byte-equal upstream and benefit
from the live-zone Anthropic dispatcher (PR-B-series) running over
the body — same behaviour as `/v1/messages`.
New module `crates/headroom-proxy/src/vertex/`:
- `mod.rs` — single dispatch handler at the
`/v1beta1/.../models/:model_action` route. Splits the trailing
`:<verb>` segment with `str::rsplit_once(':')` (no regex) and
flips an `attach_sse_tee` flag to dispatch to the streaming or
non-streaming arm. Both verbs share one axum route shape because
matchit can't distinguish two patterns that overlap on a
parameter.
- `envelope.rs` — `VertexEnvelope` parser. Confirms
`anthropic_version` present + `model` field absent (the two
fingerprints of the Vertex envelope vs `/v1/messages`).
- `adc.rs` — `TokenSource` trait + `GcpAdcTokenSource` (production,
`gcp_auth` 0.12) + `StaticTokenSource` (tests). Caches tokens
with a 60s refresh-ahead-of-expiry window. Emits structured
`event = "vertex_adc_token_refreshed"` per refresh.
- `raw_predict.rs` — POST handler + shared `forward_vertex_request`.
Buffers body, parses envelope, runs live-zone Anthropic
compression, fetches ADC bearer, attaches
`Authorization: Bearer <token>` (overwrites client-supplied
Authorization header), forwards. SSE telemetry tee for the
streaming verb reuses PR-C1's `AnthropicStreamState` directly
(Vertex streams plain SSE, unlike Bedrock's binary EventStream).
- `stream_raw_predict.rs` — module-level docs + alias to the
shared dispatcher (the streaming-vs-non-streaming difference is
one boolean flag inside the shared forwarder).
Modifications:
- `proxy.rs::build_app` — registers the single Vertex route.
- `proxy.rs::AppState` — new `vertex_token_source: Arc<dyn TokenSource>`
field. Production constructs `GcpAdcTokenSource` lazily (no GCP
call until first `bearer()`); tests inject `StaticTokenSource`
via the new `AppState::with_token_source` helper.
- `config.rs` — adds `--vertex-region` / `HEADROOM_PROXY_VERTEX_REGION`
(default `us-central1`, observability tag only — the upstream URL
is `--upstream`) and `--vertex-adc-scope` /
`HEADROOM_PROXY_VERTEX_ADC_SCOPE` (default `cloud-platform`).
- `Cargo.toml` (workspace + proxy) — adds `gcp_auth = "0.12"` and
`async-trait = "0.1"`.
- `tests/common/mod.rs` — `start_proxy_with_state` accepts both
config + state customizers; `install_static_token_source` helper
for tests.
`crates/headroom-proxy/tests/integration_vertex_raw_predict.rs` —
all five tests pass:
1. `native_envelope_round_trip_byte_equal` — Vertex-shape body
(with `anthropic_version`, no `model`) round-trips SHA-256
byte-equal upstream.
2. `adc_bearer_token_signed_correctly` — `Authorization: Bearer
<static-test-token>` reaches upstream verbatim and OVERWRITES a
client-supplied Authorization header.
3. `thinking_block_preserved` — request with `thinking` (incl.
signature) + `redacted_thinking` (incl. opaque `data`) blocks
round-trips byte-equal even with `LiveZone` compression mode
enabled. This is the P4-37 / P4-38 teeth.
4. `stream_raw_predict_sse_handled` — `:streamRawPredict` proxies
an Anthropic SSE response (full `message_start` →
`content_block_delta` → `message_stop` sequence) back to the
client without corruption; SSE content-type preserved end-to-end;
bearer attached.
5. (bonus, no-silent-fallback contract)
`adc_failure_returns_5xx_no_silent_forward` — when the token
source returns `Err`, the proxy returns 5xx and never reaches
upstream. Verifies the `event = "vertex_adc_fetch_failed"`
error path.
Workspace: `cargo test --workspace` green; `cargo clippy --workspace
-- -D warnings` clean; `make ci-precheck` passes.
- No silent fallbacks: ADC failure → structured 5xx, never an
unauthenticated forward.
- No hardcodes: every knob (region, ADC scope, upstream URL) is
CLI-flag + env-var configurable.
- No regexes: axum path parameters + `str::rsplit_once` only.
- Comprehensive structured logs: `event` field on every decision
point — `vertex_envelope_parsed`, `vertex_envelope_invalid`,
`vertex_compression_skipped`, `vertex_compression_applied`,
`vertex_adc_token_refreshed`, `vertex_adc_fetch_failed`,
`vertex_streaming_pipeline_active`, `vertex_sse_stream_closed`,
`vertex_forwarded`, `vertex_unknown_verb`, etc.
- Performant: no body clone; ADC token cached + refreshed
ahead-of-expiry, not fetched per request.
- Comprehensive tests: realistic Anthropic block content
(signature payload, redacted_thinking opaque blob) in
`thinking_block_preserved`.
The local `gcloud auth application-default print-access-token`
returns no credentials, so manual validation against a real Vertex
endpoint is not possible in this PR. Follow-up: the user runs
`gcloud auth application-default login` once and exercises a live
Vertex request — should be a no-code-change check.
PR-D1 (Bedrock native) is running concurrently and will land its
own envelope module at `crates/headroom-proxy/src/bedrock/envelope.rs`.
The two envelope modules are intentionally siblings (not a shared
trait) — the shapes differ (Bedrock has a different
`anthropic_version` value, no `model` field, AWS SigV4 instead of
GCP ADC), and a premature shared abstraction would obscure the
provider-specific contracts. Whichever PR merges second rebases
without conflict.
Retires P4-38 (and the Vertex parts of P4-39); marketplace BYOC
pitch (per project memory) gets one more native provider.
113 lines
3.7 KiB
Rust
113 lines
3.7 KiB
Rust
//! Shared test harness: spin up a Rust proxy bound to an ephemeral port
|
|
//! pointed at an arbitrary upstream URL.
|
|
|
|
use std::net::SocketAddr;
|
|
use std::sync::Arc;
|
|
use std::time::Duration;
|
|
|
|
use headroom_proxy::vertex::TokenSource;
|
|
use headroom_proxy::{build_app, AppState, Config};
|
|
use tokio::sync::oneshot;
|
|
use url::Url;
|
|
|
|
#[allow(dead_code)]
|
|
pub struct ProxyHandle {
|
|
pub addr: SocketAddr,
|
|
pub shutdown: Option<oneshot::Sender<()>>,
|
|
pub task: tokio::task::JoinHandle<()>,
|
|
}
|
|
|
|
#[allow(dead_code)]
|
|
impl ProxyHandle {
|
|
pub fn url(&self) -> String {
|
|
format!("http://{}", self.addr)
|
|
}
|
|
pub fn ws_url(&self) -> String {
|
|
format!("ws://{}", self.addr)
|
|
}
|
|
pub async fn shutdown(mut self) {
|
|
if let Some(tx) = self.shutdown.take() {
|
|
let _ = tx.send(());
|
|
}
|
|
let _ = self.task.await;
|
|
}
|
|
}
|
|
|
|
#[allow(dead_code)]
|
|
pub async fn start_proxy(upstream: &str) -> ProxyHandle {
|
|
start_proxy_with(upstream, |_| {}).await
|
|
}
|
|
|
|
/// Start a proxy with a customized `Config`. The closure receives a
|
|
/// mutable reference to the default `Config::for_test` and may toggle
|
|
/// flags like `compression` before the proxy is built.
|
|
#[allow(dead_code)]
|
|
pub async fn start_proxy_with<F>(upstream: &str, customize: F) -> ProxyHandle
|
|
where
|
|
F: FnOnce(&mut Config),
|
|
{
|
|
start_proxy_with_state(upstream, customize, |s| s).await
|
|
}
|
|
|
|
/// Start a proxy with both a Config customizer and an AppState
|
|
/// post-processor. PR-D1: tests that exercise the Bedrock route
|
|
/// inject credentials via `with_bedrock_credentials` here.
|
|
/// PR-D4: Vertex tests inject a `StaticTokenSource` via
|
|
/// `install_static_token_source` here (chain-style) so they never
|
|
/// hit real GCP.
|
|
#[allow(dead_code)]
|
|
pub async fn start_proxy_with_state<F, G>(
|
|
upstream: &str,
|
|
customize: F,
|
|
customize_state: G,
|
|
) -> ProxyHandle
|
|
where
|
|
F: FnOnce(&mut Config),
|
|
G: FnOnce(AppState) -> AppState,
|
|
{
|
|
let upstream_url: Url = upstream.parse().expect("valid upstream url");
|
|
let mut config = Config::for_test(upstream_url);
|
|
customize(&mut config);
|
|
let state = AppState::new(config.clone()).expect("app state");
|
|
let state = customize_state(state);
|
|
let app = build_app(state).into_make_service_with_connect_info::<SocketAddr>();
|
|
let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
|
|
.await
|
|
.expect("bind ephemeral");
|
|
let addr = listener.local_addr().expect("local addr");
|
|
let (tx, rx) = oneshot::channel::<()>();
|
|
let task = tokio::spawn(async move {
|
|
let _ = axum::serve(listener, app)
|
|
.with_graceful_shutdown(async move {
|
|
let _ = rx.await;
|
|
})
|
|
.await;
|
|
});
|
|
// Tiny delay to let the listener start accepting on slow CI.
|
|
tokio::time::sleep(Duration::from_millis(20)).await;
|
|
ProxyHandle {
|
|
addr,
|
|
shutdown: Some(tx),
|
|
task,
|
|
}
|
|
}
|
|
|
|
/// Convenience: replace the default `vertex_token_source` with a
|
|
/// `StaticTokenSource` returning the supplied bearer string. Used by
|
|
/// the PR-D4 Vertex integration tests so they never hit real GCP.
|
|
#[allow(dead_code)]
|
|
/// PR-D4: chain-style helper to install a `StaticTokenSource` on an
|
|
/// `AppState`. Returns the modified state so it composes with
|
|
/// `start_proxy_with_state`'s `FnOnce(AppState) -> AppState`.
|
|
pub fn install_static_token_source(mut state: AppState, bearer: &str) -> AppState {
|
|
state.vertex_token_source = Arc::new(headroom_proxy::vertex::StaticTokenSource::new(
|
|
bearer.to_string(),
|
|
)) as Arc<dyn TokenSource>;
|
|
state
|
|
}
|
|
|
|
/// Hold a reference to the config so dead_code doesn't strip its use.
|
|
#[allow(dead_code)]
|
|
pub fn _config_ref() -> Arc<Config> {
|
|
Arc::new(Config::for_test(Url::parse("http://127.0.0.1:1").unwrap()))
|
|
}
|