mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-10 14:27:00 -04:00
fix(ccr): skip Anthropic marker emission when tool injection is deferred (#1273)
## Description Anthropic request-side CCR can still compress a turn into retrieval markers after the frozen-prefix cache guard suppresses `headroom_retrieve` registration. That leaves the model with marker-only context it cannot redeem, so the proxy silently drops recoverable data on exactly the turns where cache preservation deferred tool injection. This change couples the Anthropic request-side CCR path to tool availability so a turn never emits retrieve-only markers without the retrieval tool, even when token mode or cache-mode prefix replay could otherwise reuse already-compressed marker text. Closes #1006 After a collaborator merged current `main` into this branch, CI also picked up unrelated offline-memory failures from the merged base. Those follow-up changes are test-only: they keep the offline Hugging Face cache lanes skipping cleanly instead of failing in memory tests that are outside the CCR runtime path. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Couple Anthropic request-side CCR compression to the same frozen-prefix guard that already defers `headroom_retrieve` registration. - Keep the existing cache-preservation behavior: frozen-prefix turns stop emitting CCR retrieval markers instead of forcing tool injection into the cached prefix. - Make the skip decision use the effective frozen prefix after token-mode reclamping, so turns that genuinely reclamp to zero still keep normal reversible CCR behavior. - Bypass cached marker reuse in both token mode and cache-mode prefix replay when tool injection is deferred. - Add focused regressions for the Anthropic request-path seams under this bug: - frozen-prefix turns do not emit marker-only payloads - unfrozen turns still keep normal reversible CCR behavior - token-mode reclamp back to zero still compresses normally - existing `headroom_retrieve` tools keep reversible CCR on frozen turns - cache-mode delta reuse and exact-prefix replay both forward original content when retrieval is unavailable - Add a `CHANGELOG.md` entry because the proxy's user-visible Anthropic CCR behavior changes. - Add a shared test skip helper for offline Hugging Face cache misses and apply it to the merged `main` memory tests that were failing only in the offline CI shards after the branch picked up current `main`. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_proxy/test_anthropic_ccr_deferred_injection.py`) - [x] Linting passes (`uv run ruff check . && uv run ruff format . --check`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text uv run pytest tests/test_proxy/test_anthropic_ccr_deferred_injection.py 14 passed, 1 warning in 34.19s uv run pytest tests/test_memory/test_skip_helpers.py tests/test_memory/test_embedder_mps_serialization.py::test_cpu_uses_shared_executor tests/test_memory/test_hierarchical.py::TestLocalEmbedder::test_embed_single tests/test_memory_bridge.py::TestMemoryBridgeImport::test_import_claude_code_memory tests/test_memory_handler_concurrent_init.py::test_real_localbackend_initializes_via_public_entrypoint tests/test_memory_system.py::TestLocalBackend::test_save_memory_basic -q 4 passed, 5 skipped, 11 warnings in 7.65s uv run ruff check . All checks passed! uv run ruff format . --check 968 files already formatted ``` ## Real Behavior Proof - Environment: local FastAPI `TestClient` for the Anthropic request path, plus Windows Python 3.12 offline-memory repros with `TRANSFORMERS_OFFLINE=1` - Exact command / steps: Run the focused CCR regression command and the offline-memory repro subset below on the merged branch state. - `uv run pytest tests/test_proxy/test_anthropic_ccr_deferred_injection.py` - `uv run pytest tests/test_memory/test_skip_helpers.py tests/test_memory/test_embedder_mps_serialization.py::test_cpu_uses_shared_executor tests/test_memory/test_hierarchical.py::TestLocalEmbedder::test_embed_single tests/test_memory_bridge.py::TestMemoryBridgeImport::test_import_claude_code_memory tests/test_memory_handler_concurrent_init.py::test_real_localbackend_initializes_via_public_entrypoint tests/test_memory_system.py::TestLocalBackend::test_save_memory_basic -q` - Scenario coverage from the CCR pytest command: - frozen prefix with deferred tool injection and cached marker text available in token mode - unfrozen turn with normal CCR marker emission - token mode where the tracked frozen prefix reclamps back to zero - frozen prefix where the client already supplied `headroom_retrieve` - cache-mode append-only delta reuse with a previously forwarded compressed prefix - cache-mode exact-prefix replay where the previous forwarded prefix already contained a marker - Scenario coverage from the offline-memory repro command: - direct local embedder startup on CPU with no cached HF model - hierarchical memory embedder startup with offline model cache missing - bridge import through `LocalBackend` - `MemoryHandler` public init warmup path - `LocalBackend` save path under the offline lane - Observed result: The CCR regression keeps marker-free forwarding on frozen turns without tool availability, and the merged-`main` offline-memory lanes now skip cleanly instead of failing unrelated CI shards. - frozen-prefix Anthropic turns without tool availability forward the original long transcript across both token-mode and cache-mode reuse paths, while unfrozen turns, reclamped token-mode turns, and frozen turns that already advertise `headroom_retrieve` keep the reversible CCR marker path - the merged-`main` offline-memory regressions now skip cleanly when the Hugging Face cache is unavailable instead of failing unrelated CI shards - Not tested: live Zed session cache-hit behavior, provider latency under real Anthropic upstreams, and online Hugging Face download lanes ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - Runtime scope is still intentionally narrow to Anthropic request-side CCR. OpenAI, Gemini, streaming, and response-side CCR behavior are unchanged. - The only non-CCR diff is the test-only offline-memory follow-up required after current `main` was merged into the branch. - The focused proxy pytest run still emits the Windows-local `StarletteDeprecationWarning` from `fastapi.testclient`'s `httpx` bridge. The offline-memory repro command also emits existing datetime deprecation warnings and a pytest teardown warning around skipped offline lanes; none of those warnings were introduced by the CCR runtime change.
This commit is contained in:
Rod Boev
GitHub
parent
b0146c4ccd
commit
2cae13dd79
9 changed files with 1566 additions and 143 deletions
|
|
@ -32,8 +32,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
|
||||
* **proxy:** force Responses API `store=true` when Headroom injects memory tools so `previous_response_id` continuations work after memory tool calls from clients that requested `store=false` ([#1103](https://github.com/chopratejas/headroom/pull/1103)).
|
||||
* **proxy:** build SSL contexts for custom CA bundles so enterprise/private PKI roots work with Python/OpenSSL strict verification.
|
||||
* **tokenizers:** bound token-counting of oversized tool-content blobs instead of running `count_text` over the whole serialized string. `count_messages` runs on the proxy request path; serializing is cheap, but `count_text` over a multi-megabyte `tool_result` / `tool_use` string took seconds and could freeze `/health` and in-flight requests. For payloads over ~50KB serialized, `count_text` now runs on an even-spread sample of the string and scales by length — model-accurate (tracks the active tokenizer), bounded for any blob shape, and biased to under-count (the safe direction). Smaller payloads stay exact.
|
||||
* **tokenizers:** bound token-counting of oversized tool-content blobs instead of running `count_text` over the whole serialized string. `count_messages` runs on the proxy request path; serializing is cheap, but `count_text` over a multi-megabyte `tool_result` / `tool_use` string took seconds and could freeze `/health` and in-flight requests. For payloads over ~50KB serialized, `count_text` now runs on an even-spread sample of the string and scales by length; it stays model-accurate, bounded for any blob shape, and biased to under-count. Smaller payloads stay exact.
|
||||
* **codex:** stop persisting a project-specific `--db` path in the global `headroom_memory` MCP config, so `headroom wrap codex --memory` falls back to the active cwd's `.headroom/memory.db` at runtime while keeping the current project's local bootstrap work scoped correctly ([#1147](https://github.com/chopratejas/headroom/issues/1147)).
|
||||
* **ccr:** stop emitting Anthropic request-side retrieval markers on frozen-prefix turns when `headroom_retrieve` injection is deferred, so cache-preserving requests forward original content instead of irrecoverable marker-only payloads ([#1006](https://github.com/chopratejas/headroom/issues/1006)).
|
||||
* **proxy:** route Codex OAuth image generation and edit requests through the ChatGPT Codex image backend, while preserving OpenAI API-key image passthrough ([#1215](https://github.com/chopratejas/headroom/pull/1215)).
|
||||
* **wrap (codex):** keep RTK guidance in the global Codex `AGENTS.md` instead of modifying the shared project `AGENTS.md` ([#1235](https://github.com/chopratejas/headroom/issues/1235)).
|
||||
* **proxy:** enable SSO credential resolution in the native Bedrock route via the `aws-config` `sso` feature flag, making the credential chain match what `docs/bedrock.md` already documented ([#999](https://github.com/chopratejas/headroom/pull/999)).
|
||||
|
|
|
|||
|
|
@ -1029,13 +1029,27 @@ class AnthropicHandlerMixin:
|
|||
from headroom.transforms.compression_policy import resolve_policy
|
||||
|
||||
compression_policy = resolve_policy(getattr(request.state, "auth_mode", None))
|
||||
from headroom.ccr.tool_injection import CCR_TOOL_NAME
|
||||
|
||||
existing_tool_names = {
|
||||
tool.get("name") or tool.get("function", {}).get("name")
|
||||
for tool in (body.get("tools") or [])
|
||||
if isinstance(tool, dict)
|
||||
}
|
||||
|
||||
def should_skip_ccr_request_compression(
|
||||
current_frozen_message_count: int,
|
||||
) -> bool:
|
||||
# If the tool is already present, CCR stays reversible even on frozen turns.
|
||||
return (
|
||||
self.config.ccr_inject_tool
|
||||
and current_frozen_message_count > 0
|
||||
and CCR_TOOL_NAME not in existing_tool_names
|
||||
)
|
||||
|
||||
if is_token_mode(self.config.mode):
|
||||
comp_cache = self._get_compression_cache(session_id)
|
||||
|
||||
# Zone 1: Swap cached compressed versions into working copy
|
||||
working_messages = comp_cache.apply_cached(messages)
|
||||
|
||||
# Re-freeze boundary: consecutive stable messages from start.
|
||||
# Safety: never freeze beyond provider-confirmed cached prefix.
|
||||
# `prefix_tracker.frozen_message_count` (set above) is the
|
||||
|
|
@ -1064,64 +1078,27 @@ class AnthropicHandlerMixin:
|
|||
# Record all tool_results in the verified frozen prefix as stable
|
||||
comp_cache.mark_stable_from_messages(messages, frozen_message_count)
|
||||
|
||||
# Phase 3 (#1171): off-path deferral gate. On a cold-
|
||||
# start-large request (frozen=0 + large live zone) the
|
||||
# synchronous kompress run would blow the 30s budget and
|
||||
# leak a non-preemptible worker. Forward the cache-
|
||||
# swapped messages uncompressed NOW and compress off the
|
||||
# request path; the result lands in the SAME
|
||||
# CompressionCache and apply_cached swaps it in (live
|
||||
# zone) on a later turn. Byte-identity holds — see the
|
||||
# frozen/live invariant; one-time upstream cache miss
|
||||
# when the compressed form first lands, then stable.
|
||||
if (
|
||||
getattr(self, "_background_compression_enabled", False)
|
||||
and frozen_message_count == 0
|
||||
and original_tokens >= self._background_compression_min_tokens
|
||||
):
|
||||
# Snapshot refs for the async job. The handler must
|
||||
# NOT mutate these lists/dicts in-place after this
|
||||
# point -- the background job reads them on a later
|
||||
# turn. It doesn't today; keep it that way.
|
||||
_bg_messages = messages
|
||||
_bg_working = working_messages
|
||||
_bg_frozen = frozen_message_count
|
||||
# Dedup key: the gate only fires at frozen==0 (the
|
||||
# first in-flight deferral of a session episode), so
|
||||
# session_id alone is the right granularity -- one
|
||||
# background job per session in flight -- and avoids
|
||||
# JSON-serializing the large message list for a key.
|
||||
accepted = self._background_compressor.enqueue(
|
||||
session_id,
|
||||
lambda: self.anthropic_pipeline.apply(
|
||||
messages=_bg_working,
|
||||
model=model,
|
||||
model_limit=context_limit,
|
||||
context=extract_user_query(_bg_working),
|
||||
frozen_message_count=_bg_frozen,
|
||||
biases=biases,
|
||||
request_id=request_id,
|
||||
compression_policy=compression_policy,
|
||||
**proxy_pipeline_kwargs(self.config),
|
||||
),
|
||||
lambda result: comp_cache.update_from_result(
|
||||
_bg_messages, result.messages
|
||||
),
|
||||
skip_ccr_request_compression = should_skip_ccr_request_compression(
|
||||
frozen_message_count
|
||||
)
|
||||
if skip_ccr_request_compression:
|
||||
logger.info(
|
||||
f"[{request_id}] CCR: skipping request-side compression "
|
||||
f"(frozen prefix={frozen_message_count}) because tool injection is deferred"
|
||||
)
|
||||
# Forward uncompressed either way (the request can't
|
||||
# wait); only CLAIM deferral when the job was actually
|
||||
# queued. A full-queue drop is visible in telemetry as
|
||||
# "deferred:dropped" and self-heals on a later turn.
|
||||
optimized_messages = working_messages
|
||||
transforms_applied = [
|
||||
"deferred:background_compression"
|
||||
if accepted
|
||||
else "deferred:dropped"
|
||||
]
|
||||
pipeline_timing = {}
|
||||
if skip_ccr_request_compression:
|
||||
optimized_messages = messages
|
||||
optimized_tokens = tokenizer.count_messages(optimized_messages)
|
||||
else:
|
||||
async with stage_timer.measure("compression_first_stage"):
|
||||
result = await self._run_compression_in_executor(
|
||||
# Zone 1: Swap cached compressed versions into working copy
|
||||
working_messages = comp_cache.apply_cached(messages)
|
||||
if (
|
||||
getattr(self, "_background_compression_enabled", False)
|
||||
and frozen_message_count == 0
|
||||
and original_tokens >= self._background_compression_min_tokens
|
||||
):
|
||||
accepted = self._background_compressor.enqueue(
|
||||
session_id,
|
||||
lambda: self.anthropic_pipeline.apply(
|
||||
messages=working_messages,
|
||||
model=model,
|
||||
|
|
@ -1133,57 +1110,103 @@ class AnthropicHandlerMixin:
|
|||
compression_policy=compression_policy,
|
||||
**proxy_pipeline_kwargs(self.config),
|
||||
),
|
||||
timeout=COMPRESSION_TIMEOUT_SECONDS,
|
||||
lambda bg_result: comp_cache.update_from_result(
|
||||
messages, bg_result.messages
|
||||
),
|
||||
)
|
||||
|
||||
class _DeferredCompressionResult:
|
||||
messages = working_messages
|
||||
transforms_applied = [
|
||||
"deferred:background_compression"
|
||||
if accepted
|
||||
else "deferred:dropped"
|
||||
]
|
||||
timing = {}
|
||||
|
||||
result = _DeferredCompressionResult()
|
||||
else:
|
||||
async with stage_timer.measure("compression_first_stage"):
|
||||
result = await self._run_compression_in_executor(
|
||||
lambda: self.anthropic_pipeline.apply(
|
||||
messages=working_messages,
|
||||
model=model,
|
||||
model_limit=context_limit,
|
||||
context=extract_user_query(working_messages),
|
||||
frozen_message_count=frozen_message_count,
|
||||
biases=biases,
|
||||
request_id=request_id,
|
||||
compression_policy=compression_policy,
|
||||
**proxy_pipeline_kwargs(self.config),
|
||||
),
|
||||
timeout=COMPRESSION_TIMEOUT_SECONDS,
|
||||
)
|
||||
|
||||
# Cache newly compressed messages (index-aligned diff)
|
||||
if result.messages != working_messages:
|
||||
comp_cache.update_from_result(messages, result.messages)
|
||||
|
||||
# Always use pipeline result — Zone 1 swaps applied
|
||||
# Always use pipeline result — Zone 1 swaps are already applied
|
||||
optimized_messages = result.messages
|
||||
transforms_applied = result.transforms_applied
|
||||
pipeline_timing = result.timing
|
||||
# Issue #327 / Bug 3: pipeline.apply uses the provider-
|
||||
# side tokenizer (AnthropicProvider tiktoken estimator),
|
||||
# which counts ~25% higher than the proxy-side
|
||||
# EstimatingTokenCounter used to set `original_tokens`
|
||||
# at line 634. Reusing `result.tokens_after` here
|
||||
# produced an apples-vs-oranges comparison against
|
||||
# `original_tokens` in the inflation guard below
|
||||
# (line ~901): even after a real 12% compression the
|
||||
# provider-tokenizer figure was higher than the proxy-
|
||||
# tokenizer baseline, triggering a spurious revert.
|
||||
# Recount optimized_messages with the proxy tokenizer
|
||||
# so original_tokens vs optimized_tokens is self-
|
||||
# consistent. The recount cost (~ms on a 50K-token
|
||||
# request) is paid once per request and is dwarfed by
|
||||
# the upstream call latency.
|
||||
optimized_tokens = tokenizer.count_messages(optimized_messages)
|
||||
# Issue #327 / Bug 3: pipeline.apply uses the provider-
|
||||
# side tokenizer (AnthropicProvider tiktoken estimator),
|
||||
# which counts ~25% higher than the proxy-side
|
||||
# EstimatingTokenCounter used to set `original_tokens`
|
||||
# at line 634. Reusing `result.tokens_after` here
|
||||
# produced an apples-vs-oranges comparison against
|
||||
# `original_tokens` in the inflation guard below
|
||||
# (line ~901): even after a real 12% compression the
|
||||
# provider-tokenizer figure was higher than the proxy-
|
||||
# tokenizer baseline, triggering a spurious revert.
|
||||
# Recount optimized_messages with the proxy tokenizer
|
||||
# so original_tokens vs optimized_tokens is self-
|
||||
# consistent. The recount cost (~ms on a 50K-token
|
||||
# request) is paid once per request and is dwarfed by
|
||||
# the upstream call latency.
|
||||
optimized_tokens = tokenizer.count_messages(optimized_messages)
|
||||
elif not is_cache_mode(self.config.mode):
|
||||
async with stage_timer.measure("compression_first_stage"):
|
||||
result = await self._run_compression_in_executor(
|
||||
lambda: self.anthropic_pipeline.apply(
|
||||
messages=messages,
|
||||
model=model,
|
||||
model_limit=context_limit,
|
||||
context=extract_user_query(messages),
|
||||
frozen_message_count=frozen_message_count,
|
||||
biases=biases,
|
||||
request_id=request_id,
|
||||
compression_policy=compression_policy,
|
||||
**proxy_pipeline_kwargs(self.config),
|
||||
),
|
||||
timeout=COMPRESSION_TIMEOUT_SECONDS,
|
||||
skip_ccr_request_compression = should_skip_ccr_request_compression(
|
||||
frozen_message_count
|
||||
)
|
||||
if skip_ccr_request_compression:
|
||||
logger.info(
|
||||
f"[{request_id}] CCR: skipping request-side compression "
|
||||
f"(frozen prefix={frozen_message_count}) because tool injection is deferred"
|
||||
)
|
||||
if not skip_ccr_request_compression:
|
||||
async with stage_timer.measure("compression_first_stage"):
|
||||
result = await self._run_compression_in_executor(
|
||||
lambda: self.anthropic_pipeline.apply(
|
||||
messages=messages,
|
||||
model=model,
|
||||
model_limit=context_limit,
|
||||
context=extract_user_query(messages),
|
||||
frozen_message_count=frozen_message_count,
|
||||
biases=biases,
|
||||
request_id=request_id,
|
||||
compression_policy=compression_policy,
|
||||
**proxy_pipeline_kwargs(self.config),
|
||||
),
|
||||
timeout=COMPRESSION_TIMEOUT_SECONDS,
|
||||
)
|
||||
|
||||
if result.messages != messages:
|
||||
optimized_messages = result.messages
|
||||
transforms_applied = result.transforms_applied
|
||||
pipeline_timing = result.timing
|
||||
original_tokens = result.tokens_before
|
||||
optimized_tokens = result.tokens_after
|
||||
if result.messages != messages:
|
||||
optimized_messages = result.messages
|
||||
transforms_applied = result.transforms_applied
|
||||
pipeline_timing = result.timing
|
||||
original_tokens = result.tokens_before
|
||||
optimized_tokens = result.tokens_after
|
||||
else:
|
||||
skip_ccr_request_compression = should_skip_ccr_request_compression(
|
||||
frozen_message_count
|
||||
)
|
||||
if skip_ccr_request_compression:
|
||||
logger.info(
|
||||
f"[{request_id}] CCR: skipping request-side compression "
|
||||
f"(frozen prefix={frozen_message_count}) because tool injection is deferred"
|
||||
)
|
||||
previous_original_messages = prefix_tracker.get_last_original_messages()
|
||||
previous_forwarded_messages = prefix_tracker.get_last_forwarded_messages()
|
||||
delta = self._extract_cache_stable_delta(
|
||||
|
|
@ -1194,27 +1217,35 @@ class AnthropicHandlerMixin:
|
|||
if delta is not None:
|
||||
stable_forwarded_prefix, delta_messages = delta
|
||||
if delta_messages:
|
||||
result = await self._run_compression_in_executor(
|
||||
lambda: self.anthropic_pipeline.apply(
|
||||
messages=delta_messages,
|
||||
model=model,
|
||||
model_limit=context_limit,
|
||||
context=extract_user_query(delta_messages),
|
||||
frozen_message_count=0,
|
||||
biases=biases,
|
||||
request_id=request_id,
|
||||
compression_policy=compression_policy,
|
||||
**proxy_pipeline_kwargs(self.config),
|
||||
),
|
||||
timeout=COMPRESSION_TIMEOUT_SECONDS,
|
||||
)
|
||||
optimized_messages = stable_forwarded_prefix + result.messages
|
||||
transforms_applied = result.transforms_applied
|
||||
pipeline_timing = result.timing
|
||||
optimized_tokens = tokenizer.count_messages(optimized_messages)
|
||||
if skip_ccr_request_compression:
|
||||
optimized_messages = messages
|
||||
optimized_tokens = tokenizer.count_messages(optimized_messages)
|
||||
else:
|
||||
result = await self._run_compression_in_executor(
|
||||
lambda: self.anthropic_pipeline.apply(
|
||||
messages=delta_messages,
|
||||
model=model,
|
||||
model_limit=context_limit,
|
||||
context=extract_user_query(delta_messages),
|
||||
frozen_message_count=0,
|
||||
biases=biases,
|
||||
request_id=request_id,
|
||||
compression_policy=compression_policy,
|
||||
**proxy_pipeline_kwargs(self.config),
|
||||
),
|
||||
timeout=COMPRESSION_TIMEOUT_SECONDS,
|
||||
)
|
||||
optimized_messages = stable_forwarded_prefix + result.messages
|
||||
transforms_applied = result.transforms_applied
|
||||
pipeline_timing = result.timing
|
||||
optimized_tokens = tokenizer.count_messages(optimized_messages)
|
||||
else:
|
||||
optimized_messages = stable_forwarded_prefix
|
||||
optimized_tokens = tokenizer.count_messages(optimized_messages)
|
||||
if skip_ccr_request_compression:
|
||||
optimized_messages = messages
|
||||
optimized_tokens = tokenizer.count_messages(optimized_messages)
|
||||
else:
|
||||
optimized_messages = stable_forwarded_prefix
|
||||
optimized_tokens = tokenizer.count_messages(optimized_messages)
|
||||
else:
|
||||
# Conservative rule for cache mode:
|
||||
# only replay exact stable message-prefix extensions.
|
||||
|
|
|
|||
45
tests/_skip_helpers.py
Normal file
45
tests/_skip_helpers.py
Normal file
|
|
@ -0,0 +1,45 @@
|
|||
"""Helpers for skipping tests when external model dependencies are unavailable."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterator
|
||||
|
||||
|
||||
def _iter_exception_chain(exc: BaseException) -> Iterator[BaseException]:
|
||||
"""Yield an exception and its direct cause/context chain."""
|
||||
seen: set[int] = set()
|
||||
current: BaseException | None = exc
|
||||
while current is not None and id(current) not in seen:
|
||||
yield current
|
||||
seen.add(id(current))
|
||||
current = current.__cause__ or current.__context__
|
||||
|
||||
|
||||
def external_model_skip_reason(exc: BaseException) -> str | None:
|
||||
"""Return a pytest skip reason for transient or offline model dependency errors."""
|
||||
try:
|
||||
import httpx
|
||||
except ImportError: # pragma: no cover
|
||||
httpx = None
|
||||
|
||||
try:
|
||||
from huggingface_hub.errors import LocalEntryNotFoundError
|
||||
except ImportError: # pragma: no cover
|
||||
LocalEntryNotFoundError = None
|
||||
|
||||
for candidate in _iter_exception_chain(exc):
|
||||
if httpx is not None and isinstance(candidate, httpx.ReadTimeout):
|
||||
return "Skipped due to network timeout (flaky CI)"
|
||||
|
||||
if LocalEntryNotFoundError is not None and isinstance(candidate, LocalEntryNotFoundError):
|
||||
return "Skipped because required Hugging Face model files are unavailable offline"
|
||||
|
||||
if isinstance(candidate, OSError):
|
||||
message = str(candidate)
|
||||
if (
|
||||
"couldn't connect to 'https://huggingface.co'" in message
|
||||
and "couldn't find them in the cached files" in message
|
||||
):
|
||||
return "Skipped because required Hugging Face model files are unavailable offline"
|
||||
|
||||
return None
|
||||
|
|
@ -14,14 +14,7 @@ from unittest.mock import Mock
|
|||
|
||||
import pytest
|
||||
|
||||
# Import httpx for timeout handling (will be available since it's a dependency)
|
||||
try:
|
||||
import httpx
|
||||
|
||||
HTTPX_AVAILABLE = True
|
||||
except ImportError:
|
||||
HTTPX_AVAILABLE = False
|
||||
|
||||
from tests._skip_helpers import external_model_skip_reason
|
||||
|
||||
# =============================================================================
|
||||
# Global test hooks
|
||||
|
|
@ -30,19 +23,21 @@ except ImportError:
|
|||
|
||||
@pytest.hookimpl(hookwrapper=True)
|
||||
def pytest_runtest_call(item):
|
||||
"""Wrap test execution to catch httpx.ReadTimeout and skip instead of fail.
|
||||
"""Wrap test execution to skip transient or offline external model failures.
|
||||
|
||||
This handles flaky network timeouts that occur when:
|
||||
This handles model-loading failures that occur when:
|
||||
- HuggingFace Hub is slow during model downloads (sentence-transformers)
|
||||
- Required HuggingFace model files were not restored into the offline CI cache
|
||||
- External embedding APIs timeout
|
||||
- Network connectivity issues in CI
|
||||
"""
|
||||
outcome = yield
|
||||
|
||||
if HTTPX_AVAILABLE and outcome.excinfo is not None:
|
||||
if outcome.excinfo is not None:
|
||||
exc_type, exc_value, exc_tb = outcome.excinfo
|
||||
if isinstance(exc_value, httpx.ReadTimeout):
|
||||
pytest.skip("Skipped due to network timeout (flaky CI)")
|
||||
reason = external_model_skip_reason(exc_value)
|
||||
if reason is not None:
|
||||
pytest.skip(reason)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
|
|
|
|||
|
|
@ -9,27 +9,23 @@ os.environ["TOKENIZERS_PARALLELISM"] = "false"
|
|||
|
||||
import pytest
|
||||
|
||||
# Import httpx for type checking (will be available since it's a dependency)
|
||||
try:
|
||||
import httpx
|
||||
|
||||
HTTPX_AVAILABLE = True
|
||||
except ImportError:
|
||||
HTTPX_AVAILABLE = False
|
||||
from tests._skip_helpers import external_model_skip_reason
|
||||
|
||||
|
||||
@pytest.hookimpl(hookwrapper=True)
|
||||
def pytest_runtest_call(item):
|
||||
"""Wrap test execution to catch httpx.ReadTimeout and skip instead of fail.
|
||||
"""Wrap test execution to skip transient or offline external model failures.
|
||||
|
||||
This handles flaky network timeouts that occur when:
|
||||
This handles model-loading failures that occur when:
|
||||
- HuggingFace Hub is slow during model downloads (sentence-transformers)
|
||||
- Required HuggingFace model files were not restored into the offline CI cache
|
||||
- External embedding APIs timeout
|
||||
- Network connectivity issues in CI
|
||||
"""
|
||||
outcome = yield
|
||||
|
||||
if HTTPX_AVAILABLE and outcome.excinfo is not None:
|
||||
if outcome.excinfo is not None:
|
||||
exc_type, exc_value, exc_tb = outcome.excinfo
|
||||
if isinstance(exc_value, httpx.ReadTimeout):
|
||||
pytest.skip("Skipped due to network timeout (flaky CI)")
|
||||
reason = external_model_skip_reason(exc_value)
|
||||
if reason is not None:
|
||||
pytest.skip(reason)
|
||||
|
|
|
|||
29
tests/test_memory/test_skip_helpers.py
Normal file
29
tests/test_memory/test_skip_helpers.py
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import httpx
|
||||
from huggingface_hub.errors import LocalEntryNotFoundError
|
||||
|
||||
from tests._skip_helpers import external_model_skip_reason
|
||||
|
||||
|
||||
def test_external_model_skip_reason_handles_httpx_timeout() -> None:
|
||||
reason = external_model_skip_reason(httpx.ReadTimeout("slow network"))
|
||||
assert reason == "Skipped due to network timeout (flaky CI)"
|
||||
|
||||
|
||||
def test_external_model_skip_reason_handles_hf_local_cache_miss() -> None:
|
||||
reason = external_model_skip_reason(LocalEntryNotFoundError("not cached"))
|
||||
assert reason == "Skipped because required Hugging Face model files are unavailable offline"
|
||||
|
||||
|
||||
def test_external_model_skip_reason_handles_transformers_offline_oserror() -> None:
|
||||
exc = OSError(
|
||||
"We couldn't connect to 'https://huggingface.co' to load the files, "
|
||||
"and couldn't find them in the cached files."
|
||||
)
|
||||
reason = external_model_skip_reason(exc)
|
||||
assert reason == "Skipped because required Hugging Face model files are unavailable offline"
|
||||
|
||||
|
||||
def test_external_model_skip_reason_ignores_unrelated_errors() -> None:
|
||||
assert external_model_skip_reason(RuntimeError("boom")) is None
|
||||
|
|
@ -8,7 +8,9 @@ Run with: pytest tests/test_memory_bridge.py -v
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
import functools
|
||||
import json
|
||||
import os
|
||||
import uuid
|
||||
|
||||
import pytest
|
||||
|
|
@ -24,6 +26,7 @@ from headroom.memory.bridge_parsers import (
|
|||
parse_generic_markdown,
|
||||
parse_markdown,
|
||||
)
|
||||
from tests._skip_helpers import external_model_skip_reason
|
||||
|
||||
# Sample content for testing
|
||||
CLAUDE_CODE_MEMORY = """\
|
||||
|
|
@ -63,6 +66,40 @@ The system uses FastAPI for the proxy layer.
|
|||
"""
|
||||
|
||||
|
||||
def skip_offline_model_failures(func):
|
||||
"""Skip bridge integration tests when the local embedder cannot start offline."""
|
||||
|
||||
@functools.wraps(func)
|
||||
async def wrapper(*args, **kwargs):
|
||||
try:
|
||||
return await func(*args, **kwargs)
|
||||
except Exception as exc:
|
||||
reason = external_model_skip_reason(exc)
|
||||
if reason is not None:
|
||||
pytest.skip(reason)
|
||||
raise
|
||||
|
||||
return wrapper
|
||||
|
||||
|
||||
def decorate_async_test_methods(cls):
|
||||
"""Wrap every async test method on a class with the offline-model skip helper."""
|
||||
for name, value in vars(cls).items():
|
||||
if name.startswith("test_"):
|
||||
setattr(cls, name, skip_offline_model_failures(value))
|
||||
return cls
|
||||
|
||||
|
||||
def skip_if_offline_bridge_import_failed(stats) -> None:
|
||||
"""Skip bridge assertions when every section failed only because the model cache is offline."""
|
||||
if (
|
||||
os.environ.get("TRANSFORMERS_OFFLINE") == "1"
|
||||
and stats.sections_imported == 0
|
||||
and stats.sections_failed > 0
|
||||
):
|
||||
pytest.skip("Skipped because required Hugging Face model files are unavailable offline")
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Parser Tests (pure functions, no backend)
|
||||
# =============================================================================
|
||||
|
|
@ -273,6 +310,7 @@ def bridge(bridge_config, backend):
|
|||
return MemoryBridge(bridge_config, backend)
|
||||
|
||||
|
||||
@decorate_async_test_methods
|
||||
class TestMemoryBridgeImport:
|
||||
@pytest.mark.asyncio
|
||||
async def test_import_claude_code_memory(self, bridge, tmp_dir, backend):
|
||||
|
|
@ -281,6 +319,7 @@ class TestMemoryBridgeImport:
|
|||
md_path.write_text(CLAUDE_CODE_MEMORY, encoding="utf-8")
|
||||
|
||||
stats = await bridge.import_from_markdown(paths=[md_path], user_id="test_user")
|
||||
skip_if_offline_bridge_import_failed(stats)
|
||||
|
||||
assert stats.files_processed == 1
|
||||
assert stats.sections_imported > 0
|
||||
|
|
@ -297,6 +336,7 @@ class TestMemoryBridgeImport:
|
|||
md_path.write_text(CLAUDE_CODE_MEMORY, encoding="utf-8")
|
||||
|
||||
stats1 = await bridge.import_from_markdown(paths=[md_path], user_id="test_user")
|
||||
skip_if_offline_bridge_import_failed(stats1)
|
||||
assert stats1.sections_imported > 0
|
||||
|
||||
stats2 = await bridge.import_from_markdown(paths=[md_path], user_id="test_user")
|
||||
|
|
@ -316,6 +356,7 @@ class TestMemoryBridgeImport:
|
|||
md_path.write_text(modified, encoding="utf-8")
|
||||
|
||||
stats = await bridge.import_from_markdown(paths=[md_path], user_id="test_user")
|
||||
skip_if_offline_bridge_import_failed(stats)
|
||||
assert stats.files_processed == 1
|
||||
assert stats.sections_imported >= 1 # At least the new section
|
||||
|
||||
|
|
@ -339,6 +380,7 @@ class TestMemoryBridgeImport:
|
|||
|
||||
bridge._config.md_format = MarkdownFormat.CHATGPT
|
||||
stats = await bridge.import_from_markdown(paths=[md_path], user_id="test_user")
|
||||
skip_if_offline_bridge_import_failed(stats)
|
||||
assert stats.sections_imported > 0
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
|
@ -366,6 +408,7 @@ class TestMemoryBridgeImport:
|
|||
assert "source_file" in metadata
|
||||
|
||||
|
||||
@decorate_async_test_methods
|
||||
class TestMemoryBridgeExport:
|
||||
@pytest.mark.asyncio
|
||||
async def test_export_claude_code_style(self, bridge, tmp_dir, backend):
|
||||
|
|
@ -422,6 +465,7 @@ class TestMemoryBridgeExport:
|
|||
assert "No memories" in markdown
|
||||
|
||||
|
||||
@decorate_async_test_methods
|
||||
class TestMemoryBridgeSync:
|
||||
@pytest.mark.asyncio
|
||||
async def test_sync_imports_and_exports(self, bridge, tmp_dir, backend):
|
||||
|
|
@ -432,6 +476,7 @@ class TestMemoryBridgeSync:
|
|||
|
||||
# First sync: imports from file
|
||||
stats = await bridge.sync(user_id="test_user")
|
||||
skip_if_offline_bridge_import_failed(stats.import_stats)
|
||||
assert stats.import_stats.sections_imported > 0
|
||||
|
||||
# Add an organic memory (not from bridge)
|
||||
|
|
@ -465,6 +510,7 @@ class TestMemoryBridgeSync:
|
|||
assert stats.memories_exported == 0
|
||||
|
||||
|
||||
@decorate_async_test_methods
|
||||
class TestSyncStatePersistence:
|
||||
@pytest.mark.asyncio
|
||||
async def test_state_saved_and_loaded(self, tmp_dir, backend):
|
||||
|
|
@ -496,6 +542,7 @@ class TestSyncStatePersistence:
|
|||
assert stats.files_skipped_unchanged == 1
|
||||
|
||||
|
||||
@decorate_async_test_methods
|
||||
class TestRoundTrip:
|
||||
@pytest.mark.asyncio
|
||||
async def test_import_export_preserves_facts(self, bridge, tmp_dir, backend):
|
||||
|
|
|
|||
|
|
@ -12,6 +12,8 @@ Covers:
|
|||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import functools
|
||||
import os
|
||||
from typing import Any
|
||||
from unittest.mock import patch
|
||||
|
||||
|
|
@ -22,6 +24,24 @@ from headroom.proxy.memory_handler import (
|
|||
MemoryConfig,
|
||||
MemoryHandler,
|
||||
)
|
||||
from tests._skip_helpers import external_model_skip_reason
|
||||
|
||||
|
||||
def skip_offline_model_failures(func):
|
||||
"""Skip real-backend smoke tests when the local embedder cannot start offline."""
|
||||
|
||||
@functools.wraps(func)
|
||||
async def wrapper(*args, **kwargs):
|
||||
try:
|
||||
return await func(*args, **kwargs)
|
||||
except Exception as exc:
|
||||
reason = external_model_skip_reason(exc)
|
||||
if reason is not None:
|
||||
pytest.skip(reason)
|
||||
raise
|
||||
|
||||
return wrapper
|
||||
|
||||
|
||||
# -------------------------------------------------------------------
|
||||
# Singleflight under concurrent callers
|
||||
|
|
@ -224,6 +244,7 @@ async def test_ensure_initialized_cancellation_propagates_and_resets_state(tmp_p
|
|||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@skip_offline_model_failures
|
||||
async def test_real_localbackend_initializes_via_public_entrypoint(tmp_path):
|
||||
"""End-to-end sanity check: the public ``ensure_initialized`` path works
|
||||
against a real LocalBackend. This catches regressions where the new
|
||||
|
|
@ -253,6 +274,8 @@ async def test_real_localbackend_initializes_via_public_entrypoint(tmp_path):
|
|||
|
||||
# warmup_embedder is best-effort; on a real backend it should succeed.
|
||||
warmed = await handler.warmup_embedder()
|
||||
if not warmed and os.environ.get("TRANSFORMERS_OFFLINE") == "1":
|
||||
pytest.skip("Skipped because required Hugging Face model files are unavailable offline")
|
||||
assert warmed is True
|
||||
await handler.close()
|
||||
|
||||
|
|
|
|||
1256
tests/test_proxy/test_anthropic_ccr_deferred_injection.py
Normal file
1256
tests/test_proxy/test_anthropic_ccr_deferred_injection.py
Normal file
File diff suppressed because it is too large
Load diff
Loading…
Add table
Add a link
Reference in a new issue