akadmin/headroom
1
0
Fork 0
mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
headroom/tests
History
Adryan Eka Vandra a31d81a426
feat(proxy): track Codex WS sessions and cancel relay tasks deterministically 
Unit 3 of the Codex proxy resilience plan. Eliminates the "aged process
has leaked relay tasks" hypothesis by making every WS session explicitly
tracked and both relay tasks deterministically cancelled when either
exits.

- New headroom/proxy/ws_session_registry.py: dict-backed
  WebSocketSessionRegistry + WSSessionHandle with register /
  deregister / attach_tasks / snapshot. Deregister is idempotent and
  clears relay-task references so coroutine frames are not retained
  past session end.
- HeadroomProxy exposes proxy.ws_sessions so /debug/ws-sessions
  (Unit 5) can read the live snapshot.
- handle_openai_responses_ws now registers on websocket.accept()
  success and deregisters in the outermost finally so no leak can
  survive handshake-phase, mid-stream, or upstream-error paths. The
  session_id / termination_cause is threaded through both relay
  halves and both sides raise asyncio.CancelledError cleanly.
- Replaced asyncio.gather(_client_to_upstream(), _upstream_to_client(),
  return_exceptions=True) with explicit asyncio.create_task(...)
  (named codex-ws-c2u-<sid> / codex-ws-u2c-<sid>) +
  asyncio.wait(FIRST_COMPLETED) + cancel-and-await on the survivor.
  Termination cause is classified as client_disconnect /
  client_error / upstream_disconnect / upstream_error /
  response_completed from which task completed first plus inline
  error captures from the halves.
- Prometheus metrics: new active_ws_sessions and active_relay_tasks
  gauges plus ws_session_duration_ms_{sum,count,max} histogram
  bucketed by termination cause. Mirrors the Unit 2 stage_timing_*
  shape.

Preserved: upstream WS retry loop, WS→HTTP fallback, memory-context
timeout, compression pipeline, Unit 2 stage timings. Memory-tool
execution inside _upstream_to_client still runs when the client task
exits first; however, if the client disconnects *before* the upstream
emits response.completed, pending memory writes in `pending_fcs` are
dropped (unchanged from prior behavior — a crashing upstream has the
same effect). Note: handle_openai_responses (HTTP, line ~800) is a
single-shot HTTP request; lifecycle tracking isn't added there
(scope boundary).

Tests:
- tests/test_ws_session_registry.py: 8 registry unit tests
  (register/deregister idempotency, snapshot shape, attach merging,
  reference release).
- tests/test_openai_codex_ws_lifecycle.py: 6 integration tests
  using real relay tasks (only upstream WS endpoint mocked):
  happy-path, failing-test-first "client disconnect cancels upstream
  relay within 100 ms", upstream-closes-first, upstream-error mid-
  stream, handshake-failure deregister, 50 concurrent sessions.
- Regression: test_openai_codex_ws_timings, test_openai_codex_routing,
  test_proxy_codex_route_aliases, test_ws_memory_relay all pass.
- Tests pass under python -W error::RuntimeWarning (no "coroutine
  was never awaited").
2026-04-20 22:01:41 +07:00
..
cli Add Click-based CLI with memory management commands 2026-01-29 21:30:21 -08:00
integrations/test_strands feat: Add AWS Strands Agents SDK integration 2026-01-31 00:31:37 -08:00
test_cache fix: eliminate prefix cache busts from frozen count underestimation 2026-04-07 17:10:20 -07:00
test_cli Fix CI lint errors and test failures 2026-04-14 18:23:06 -07:00
test_compression Remove LLMLingua: Kompress is the sole text compressor 2026-03-26 11:11:00 -07:00
test_evals Remove LLMLingua: Kompress is the sole text compressor 2026-03-26 11:11:00 -07:00
test_install feat(docker): forward HEADROOM_WORKSPACE_DIR and HEADROOM_CONFIG_DIR into containers 2026-04-16 19:19:25 -05:00
test_integrations fix: remove unused imports in test_langgraph.py (ruff F401) 2026-03-30 09:27:49 -07:00
test_learn Fix headroom learn failing on project paths with underscores (#159) 2026-04-14 15:59:49 -07:00
test_memory Fix ruff lint errors in test files 2026-03-24 15:54:12 -07:00
test_providers Harden cache validation reporting and TTL analysis 2026-04-06 20:11:21 -05:00
test_storage Fix all ruff lint and format errors for CI 2026-01-10 15:33:44 -08:00
test_transforms Update number array tests for schema-preserving compression 2026-04-15 19:00:43 -07:00
__init__.py Initial commit: Headroom SDK - LLM context optimization toolkit 2026-01-06 23:16:58 -08:00
conftest.py Add global httpx.ReadTimeout handler for all tests 2026-01-30 16:37:34 -08:00
test_acceptance.py Fix all ruff lint and format errors for CI 2026-01-10 15:33:44 -08:00
test_adapter_hooks.py Add pluggable adapter hooks for CCR, Storage, and TOIN backends 2026-02-15 17:47:30 -08:00
test_adaptive_sizer.py Diversity-aware SmartCrusher: keep unique items, compress text within 2026-03-25 23:57:24 -07:00
test_anthropic_stage_timings.py feat(proxy): add per-stage timings for Codex WS and Anthropic HTTP paths 2026-04-20 22:01:41 +07:00
test_backend_bugs.py Fix Bedrock/Vertex auth regression: don't forward api_key for env-auth providers (#105) 2026-04-07 14:24:19 -07:00
test_bedrock_region.py Fix ruff lint errors in test files 2026-03-24 15:54:12 -07:00
test_ccr.py Fix all ruff lint and format errors for CI 2026-01-10 15:33:44 -08:00
test_ccr_batch_processor.py Harden Anthropic prefix cache stability across proxy and batch paths 2026-04-04 13:45:37 -05:00
test_ccr_context_tracker.py v0.2.2: Add CCR Response Handler, Context Tracker, and restructure docs 2026-01-14 13:03:41 -08:00
test_ccr_feedback.py Fix all ruff lint and format errors for CI 2026-01-10 15:33:44 -08:00
test_ccr_mcp_server.py fix: restore Windows mypy compatibility 2026-04-11 18:40:53 -05:00
test_ccr_response_handler.py Fix security vulnerabilities in memory and CCR systems 2026-02-04 12:05:50 -08:00
test_ccr_tool_injection.py Remove LLMLingua: Kompress is the sole text compressor 2026-03-26 11:11:00 -07:00
test_claude_session_branch_compare.py Harden anthropic cache-mode replay stability 2026-04-05 16:11:39 -05:00
test_claude_session_mode_benchmark.py Harden cache validation reporting and TTL analysis 2026-04-06 20:11:21 -05:00
test_cli_proxy_env.py feat(cli): add Docker-native install flow and parity docs 2026-04-10 23:27:24 -05:00
test_codex_rate_limits.py feat: add OpenAI Codex rate-limit window tracking 2026-04-11 23:34:19 -05:00
test_compress_api.py Fix CI: guard starlette imports, asyncio.run(), deprecate datetime.utcnow() 2026-02-19 11:03:24 -08:00
test_compression_cache.py fix(ci): harden wrap e2e validation 2026-04-07 23:27:31 -05:00
test_compression_store.py chore: add nosec B324 annotations to non-cryptographic MD5 usages and update temporary database path to use system temp directory 2026-04-07 13:07:26 +06:00
test_compression_summary.py Add compression summaries, multi-provider headers, Dockerfile fix 2026-02-18 16:54:20 -08:00
test_compression_summary_eval.py Add compression summaries, multi-provider headers, Dockerfile fix 2026-02-18 16:54:20 -08:00
test_compression_summary_hard_eval.py Add compression summaries, multi-provider headers, Dockerfile fix 2026-02-18 16:54:20 -08:00
test_compression_summary_integration.py Add Query Echo: re-inject user question after compressed tool outputs 2026-02-18 23:39:26 -08:00
test_compression_summary_tool_eval.py Add compression summaries, multi-provider headers, Dockerfile fix 2026-02-18 16:54:20 -08:00
test_config.py fix: dashboard metrics, TTFB tracking, eager LLMLingua loading, and multi-provider consistency 2026-03-07 23:33:45 -08:00
test_copilot_quota.py style: apply ruff format to copilot_quota and test_copilot_quota 2026-04-12 00:08:56 -05:00
test_cost_tracker_counterfactual.py Fix dashboard cost savings: use model list price, not moving average 2026-03-31 08:50:25 -07:00
test_critical_fixes.py Add Query Echo: re-inject user question after compressed tool outputs 2026-02-18 23:39:26 -08:00
test_critical_gaps.py Update test for lowered TOIN confidence threshold default 2026-01-21 21:40:11 -08:00
test_crushability.py Fix all ruff lint and format errors for CI 2026-01-10 15:33:44 -08:00
test_dashboard_cache_ttl_playwright.py Add observed cache TTL metrics and dashboard coverage 2026-04-06 21:13:30 -05:00
test_google_multimodal.py Fix Google Gemini conversion to preserve non-text content (Gap #4) 2026-01-24 12:20:13 -08:00
test_google_multimodal_e2e.py Add E2E tests for Google multimodal content preservation 2026-01-24 21:01:25 -08:00
test_hnsw_only.py Add bounded memory support to HNSWVectorIndex with LRU eviction 2026-02-01 21:12:06 -08:00
test_hooks.py Add Compression Hooks — extension points for SaaS and advanced customization 2026-02-19 08:17:30 -08:00
test_image_compression.py Skip image tests when Pillow not installed (CI fix) 2026-04-09 16:24:15 -07:00
test_image_compressor.py Slim core dependencies: 2.5GB → 195MB install size 2026-03-13 15:35:49 -07:00
test_log_compressor.py fix: improve error handling and add comprehensive test coverage 2026-01-27 16:08:36 -08:00
test_memory_bridge.py Add Memory Bridge: bidirectional sync between markdown files and Headroom memory 2026-02-23 22:45:38 -08:00
test_memory_eval.py Add hierarchical memory system with graph + vector storage 2026-01-26 21:58:47 -08:00
test_memory_handler_concurrent_init.py feat(proxy): share cold-start warmup across pipelines + lock memory init 2026-04-20 22:01:41 +07:00
test_memory_integration.py Add centralized ML model configuration 2026-02-01 23:47:42 -08:00
test_memory_sync.py Fix CI lint errors and test failures 2026-04-14 18:23:06 -07:00
test_memory_system.py Fix CI: guard starlette imports, asyncio.run(), deprecate datetime.utcnow() 2026-02-19 11:03:24 -08:00
test_memory_tracker.py Add memory observability system (Phase 1) 2026-02-01 19:49:53 -08:00
test_memory_tracker_integration.py Add wrap commands for Codex/Cursor/Aider with rtk instructions, fix savings metrics 2026-03-13 22:02:47 -07:00
test_memory_usage_integration.py Add SQLite vector backend as default and fix HNSW test skipping 2026-02-01 22:12:45 -08:00
test_models.py Fix pricing lookup for retired Claude models 2026-03-12 21:11:18 -07:00
test_oauth_bearer_routing.py fix: support OAuth Bearer token routing for proxy endpoints 2026-04-19 12:50:09 +07:00
test_observability_metrics.py feat: add OTEL observability core 2026-04-09 21:20:34 -05:00
test_observability_tracing.py feat: add OTEL observability core 2026-04-09 21:20:34 -05:00
test_openai_codex_routing.py fix(proxy): retry responses transport without bypassing compression 2026-04-20 22:01:39 +07:00
test_openai_codex_ws_lifecycle.py feat(proxy): track Codex WS sessions and cancel relay tasks deterministically 2026-04-20 22:01:41 +07:00
test_openai_codex_ws_timings.py feat(proxy): add per-stage timings for Codex WS and Anthropic HTTP paths 2026-04-20 22:01:41 +07:00
test_openai_streaming_backend.py Commit message: 2026-03-13 16:49:18 -07:00
test_package_init_lazy.py Fix CI lint errors and test failures 2026-04-14 18:23:06 -07:00
test_parser.py Add multi-provider memory system with auto-detection 2026-02-01 14:42:50 -08:00
test_paths.py test(paths): close coverage gaps for derived-only helpers 2026-04-16 19:59:30 -05:00
test_paths_backward_compat.py test(paths): add adversarial backward-compat stress suite 2026-04-16 19:59:47 -05:00
test_provider_model_fallback.py Fix CI regressions for cache benchmark work 2026-04-04 22:33:44 -05:00
test_proxy_anthropic_cache_stability.py Fix 19 test failures: missing security attr, nosec inside f-strings, stale test mock 2026-04-07 18:30:29 -07:00
test_proxy_batch_integration.py Add multi-provider batch API support with CCR post-processing 2026-01-24 11:41:18 -08:00
test_proxy_cache_ttl_metrics.py feat: expose proxy OTEL metrics and Langfuse status 2026-04-09 21:20:46 -05:00
test_proxy_ccr.py Fix ruff formatting 2026-01-17 17:14:12 -08:00
test_proxy_codex_route_aliases.py test(proxy): cover codex responses subpath aliases 2026-04-08 21:22:05 -05:00
test_proxy_compress_endpoint.py Add TypeScript SDK (headroom-ai npm package) 2026-03-26 15:41:56 -07:00
test_proxy_compression_headers.py fix: strip accept-encoding from forwarded proxy headers 2026-04-15 11:40:11 -07:00
test_proxy_count_tokens_integration.py Add multi-provider batch API support with CCR post-processing 2026-01-24 11:41:18 -08:00
test_proxy_gemini_integration.py Add multi-provider batch API support with CCR post-processing 2026-01-24 11:41:18 -08:00
test_proxy_gemini_native_integration.py Add multi-provider batch API support with CCR post-processing 2026-01-24 11:41:18 -08:00
test_proxy_healthchecks.py Fix --memory flag silently dropped when proxy already running 2026-04-13 14:36:30 -07:00
test_proxy_hooks_regression.py fix: restore anthropic compression when hooks enabled 2026-03-30 11:25:15 +02:00
test_proxy_intelligent_context.py feat(toin): add TOIN + CCR integration to IntelligentContextManager 2026-01-27 16:08:36 -08:00
test_proxy_memory_integration.py fix: correct cost calculation for cache tokens and simplify memory DevEx 2026-01-26 22:49:31 -08:00
test_proxy_mode_benchmark.py Rebrand proxy modes to token/cache and harden cache-mode stability 2026-04-04 14:32:07 -05:00
test_proxy_modes.py Harden cache-mode immutability for OpenAI and fix stats mode reporting 2026-04-04 14:36:29 -05:00
test_proxy_openai_cache_stability.py Fix CI regressions for cache benchmark work 2026-04-04 22:33:44 -05:00
test_proxy_openai_responses_integration.py Add compression for OpenAI Responses API (/v1/responses) 2026-03-30 20:13:00 -07:00
test_proxy_passthrough_integration.py Add multi-provider batch API support with CCR post-processing 2026-01-24 11:41:18 -08:00
test_proxy_savings_history.py Apply Ruff formatting to rebased stack 2026-04-02 13:40:57 +02:00
test_proxy_scalability.py Fix Codex WebSocket HTTP 500: ChatGPT auth routing, correct beta header, HTTP fallback (#71) 2026-04-06 12:48:46 -07:00
test_proxy_streaming_ratelimit_headers.py fix(proxy): retry responses transport without bypassing compression 2026-04-20 22:01:39 +07:00
test_proxy_streaming_resilience.py Refactor: extract 7 modules from server.py (Steps 2-4) 2026-04-03 16:18:08 -07:00
test_proxy_telemetry_env.py feat: allow overriding proxy telemetry sdk via HEADROOM_SDK 2026-03-27 14:23:32 +01:00
test_proxy_warmup.py feat(proxy): share cold-start warmup across pipelines + lock memory init 2026-04-20 22:01:41 +07:00
test_quality_retention.py Fix all ruff lint and format errors for CI 2026-01-10 15:33:44 -08:00
test_query_echo.py Add Query Echo: re-inject user question after compressed tool outputs 2026-02-18 23:39:26 -08:00
test_quota_registry.py feat: pluggable QuotaTracker ABC + QuotaTrackerRegistry 2026-04-12 00:58:01 -05:00
test_release_version.py fix: preserve empty-body commits in release bump parsing 2026-04-16 23:46:06 -05:00
test_relevance.py Fix all mypy type errors and flaky embedding test 2026-01-10 18:27:33 -08:00
test_responses_converter.py Add compression for OpenAI Responses API (/v1/responses) 2026-03-30 20:13:00 -07:00
test_rtk_installer.py feat(cli): add Docker-native install flow and parity docs 2026-04-10 23:27:24 -05:00
test_search_compressor.py fix: improve error handling and add comprehensive test coverage 2026-01-27 16:08:36 -08:00
test_security_validations.py Fix security vulnerabilities in memory and CCR systems 2026-02-04 12:05:50 -08:00
test_shared_context.py Add SharedContext for multi-agent, rewrite README, fix proxy cleanup 2026-03-17 16:31:04 -07:00
test_sqlite_graph_store.py Add SQLiteGraphStore for bounded, persistent graph storage 2026-02-01 20:48:57 -08:00
test_sqlite_vector_index.py Add SQLiteVectorIndex using sqlite-vec for bounded vector search 2026-02-01 21:23:09 -08:00
test_stage_timer.py feat(proxy): add per-stage timings for Codex WS and Anthropic HTTP paths 2026-04-20 22:01:41 +07:00
test_strands_tokenizer.py Count Strands reasoningContent, image, document, video tokens (#111 follow-up) 2026-04-08 17:31:00 -07:00
test_subscription_tracker.py fix: ruff lint in test_subscription_tracker (unused imports, ordering) 2026-04-11 22:06:20 -05:00
test_tag_protection_integration.py Fix API integration test: use tool output not user message for tags 2026-03-26 11:46:55 -07:00
test_telemetry.py Fix all ruff lint and format errors for CI 2026-01-10 15:33:44 -08:00
test_telemetry_warning.py fix: format telemetry warning tests for CI 2026-04-09 23:29:56 -05:00
test_text_compressors.py Enhanced MCP server with compress/stats tools, fix proxy torch crash 2026-03-16 22:56:30 -07:00
test_toin.py Add comprehensive evaluation framework for compression accuracy 2026-01-22 09:31:54 -08:00
test_toin_feedback.py fix: close TOIN feedback loop — retrievals now flow back for learning 2026-03-22 14:18:30 -07:00
test_toin_field_learning.py Fix ruff formatting 2026-01-17 17:14:12 -08:00
test_toin_fixes.py Fix all ruff lint and format errors for CI 2026-01-10 15:33:44 -08:00
test_toin_full_integration.py Update test for lowered TOIN confidence threshold default 2026-01-21 21:40:11 -08:00
test_toin_integration.py Add pluggable storage backend abstraction for CompressionStore 2026-01-20 23:25:28 -08:00
test_token_headroom_mode.py Rebrand proxy modes to token/cache and harden cache-mode stability 2026-04-04 14:32:07 -05:00
test_tokenizers.py Fix all ruff lint and format errors for CI 2026-01-10 15:33:44 -08:00
test_ws_http_fallback.py fix(proxy): retry responses transport without bypassing compression 2026-04-20 22:01:39 +07:00
test_ws_memory_relay.py Fix CI lint errors and test failures 2026-04-14 18:23:06 -07:00
test_ws_session_registry.py feat(proxy): track Codex WS sessions and cancel relay tasks deterministically 2026-04-20 22:01:41 +07:00