mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
fix: Wave 3 — multi-turn live integration tests for A+B realignment
Adds tests/test_realignment_live_multi_turn.py with 9 OPT-IN live tests
that validate the load-bearing claims of the Phase A+B megamerge against
real upstream APIs (Anthropic, OpenAI, Gemini). Each test maps to one or
more realignment PRs:
1. test_anthropic_cache_hit_across_two_turns — A2/A6/E
Identical cache_control'd system+messages on two turns must
eventually produce cache_read_input_tokens > 0. Guards the cache
hot zone invariant (I2): proxy must not mutate frozen prefix bytes.
Uses a bounded retry loop (max 4 attempts) to absorb Anthropic's
eventually-consistent prompt-cache write latency without masking
a real "proxy broke cache stability" regression.
2. test_anthropic_cache_stable_when_live_zone_compresses — B2/B3
Turn 2 mutates only the LATEST user content (8KB+ JSON tail);
cache_read on turn 2 must still be > 0 AND the proxy must emit
compression headers — proving the live-zone block dispatcher
ran on the new tail without disturbing the cached prefix.
3. test_anthropic_cache_control_passthrough_byte_faithful — A3/A4
Wraps proxy._retry_request to snapshot the upstream-bound body
and assert cache_control on system blocks survives verbatim,
and user content is not flattened from list to string form.
4. test_openai_chat_completions_multi_turn_through_proxy — A8/B
Three-turn conversation through /v1/chat/completions; each
turn returns valid content, prior assistant turns survive in
the messages list (proxy doesn't drop them).
5. test_openai_streaming_sse_chunks_arrive_in_order — A8 (SSE wire)
Streams /v1/chat/completions; asserts each event is
'data: ...\\n\\n', terminator is 'data: [DONE]\\n\\n',
reassembled content non-empty, no malformed events.
6. test_gemini_multi_turn_through_proxy — Gemini reach
Two-turn conversation through native
/v1beta/models/{model}:generateContent. Proves Gemini handler
wiring stayed intact through the megamerge.
7. test_ccr_marker_round_trip_live — B7 (CCR)
Pre-populates compression_store with a fixture entry, embeds
a CCR marker on a tool_result, verifies (a) headroom_retrieve
tool is injected into the upstream tools array (PR-B7
always-on), and (b) /v1/retrieve returns the original bytes
by hash with all rows intact. Pre-populating the Python store
(vs. driving SmartCrusher's internal Rust store) matches the
established pattern in tests/test_proxy_ccr.py and exercises
the surface served by /v1/retrieve.
8. test_memory_tail_injection_does_not_modify_system_prompt_live — B6/A2
Spins up a memory-enabled proxy with MemoryMode.AUTO_TAIL,
seeds LocalBackend, captures upstream-bound body. Asserts:
(a) system prompt byte-identical to input; (b) memory text
lands on latest user message tail; (c) earlier messages
untouched. Guards the live-zone-only injection contract.
9. test_classify_auth_mode_routes_payg_vs_oauth — Phase F-prep / B5
NOT a live API call. Sends three header shapes through the
proxy (x-api-key=..., Bearer sk-ant-oat01-..., Bearer
sk-ant-api03-...), captures dispatcher headers via a wrap on
_retry_request, and asserts the canonical auth-mode classifier
maps each correctly. Codifies the Phase F contract.
Conventions:
* file-level pytestmark = pytest.mark.live → excluded by default
via 'pytest -m "not live"'. Adds a 'live' marker registration in
pyproject.toml's [tool.pytest.ini_options].markers.
* each test skipif's on the relevant API key — no silent fallbacks,
no real-API runs against fake keys.
* uses tests/_dotenv.py helpers (load_env_overrides + autouse_apply_env)
rather than re-implementing env loading.
* model IDs and thresholds live in a top-of-file LIVE_CONFIG dict
(no hardcodes); Anthropic primary/fallback resolves at runtime per
key entitlement.
* assertions are direction-only (cache_read > 0, tokens_after <=
tokens_before) — never tied to upstream pricing/tokenizer drift.
* shared module-scoped TestClient fixture for performance; CCR and
memory tests build dedicated proxies for their config-specific paths.
Verification:
* pytest tests/test_realignment_live_multi_turn.py -v
→ 9 passed, 0 skipped, 0 failed in ~25s (with all keys set)
* pytest -m "not live" --tb=short -q
→ 4694 passed, 265 skipped, 9 deselected — same baseline as today
* make ci-precheck → green (rust + python + commitlint)
Per-realignment-plan: REALIGNMENT/04-phase-B-live-zone.md.
This commit is contained in:
parent
2fb905fdb0
commit
dcbc921d63
2 changed files with 1141 additions and 0 deletions
|
|
@ -318,6 +318,7 @@ asyncio_mode = "auto"
|
|||
markers = [
|
||||
"slow: slow tests (model loads, large fixtures)",
|
||||
"real_llm: tests that hit real LLM APIs; skipped unless explicitly enabled",
|
||||
"live: opt-in multi-turn tests that hit real upstream APIs; require provider keys",
|
||||
]
|
||||
|
||||
[tool.coverage.run]
|
||||
|
|
|
|||
1140
tests/test_realignment_live_multi_turn.py
Normal file
1140
tests/test_realignment_live_multi_turn.py
Normal file
File diff suppressed because it is too large
Load diff
Loading…
Add table
Add a link
Reference in a new issue