1 commit

Author	SHA1	Message	Date
sfc-gh-nashukla	f00ace6da5	fix(cortex-code): migrate to current Cortex REST API endpoints + add e2e benchmarks (#1474) ... ## Description Follow-up to #1190 (Cortex Code provider). Three issues found during post-merge testing, plus full MCP and Proxy+MCP validation added. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [x] Documentation update ## Changes Made - `docs/cortex-code.md`: corrected legacy endpoint references (`inference:complete` → `/v1/chat/completions`), fixed incorrect claim that `role:"tool"` is unsupported (works on Chat Completions, not Messages path), updated proxy mode instructions - `tests/e2e_cortex_savings.py`: migrated from deprecated `inference:complete` to `/api/v2/cortex/v1/chat/completions` + `max_completion_tokens` - `tests/e2e_cortex_latency.py`: new — TTFT + E2E latency benchmark, streaming API, N-run median - `tests/e2e_cortex_quality.py`: new — answer accuracy benchmark; 0 quality regressions at 44–68% compression - `tests/e2e_cortex_proxy.py`: new — proxy-in-the-loop multi-turn test via FastAPI proxy - `tests/e2e_cortex_mcp.py`: new — **MCP mode** test using official MCP Python SDK (stdio transport, same protocol as Cortex Code); verifies `headroom_compress`, `headroom_retrieve`, `headroom_stats` - `tests/e2e_cortex_proxy_mcp.py`: new — **Proxy + MCP** test; starts FastAPI proxy and MCP server simultaneously, exercises both paths in same session ## Testing - [x] Linting passes (`ruff check .`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # MCP mode (e2e_cortex_mcp.py) [1/6] Connecting to headroom MCP server ... OK [2/6] Listing MCP tools ... found: ['headroom_compress', 'headroom_retrieve', 'headroom_stats'] [3/6] Test 1 - dbt run results (40 models) Direct Cortex call ... prompt=2,112 tokens MCP headroom_compress ... saved 0 tokens hash=825cf6f2... Cortex call (MCP-compressed) ... prompt=2,112 saved 0 (0.0%) [4/6] Test 2 - INFORMATION_SCHEMA tables (59 rows) Direct Cortex call ... prompt=3,203 tokens MCP headroom_compress ... saved 1,280 tokens (37.2%) Cortex call (MCP-compressed) ... prompt=1,163 saved 2,040 (63.7%) [5/6] headroom_retrieve CCR round-trip ... original content retrieved [6/6] headroom_stats ... compressions: 2, total_tokens_saved: 1280 MCP TEST PASSED - 38.4% avg token reduction via MCP tools # Proxy + MCP mode (e2e_cortex_proxy_mcp.py) [1/7] Starting headroom proxy ... OK [2/7] Connecting to headroom MCP server ... OK MCP tools: ['headroom_compress', 'headroom_retrieve', 'headroom_stats'] [3/7] Baseline: dbt=2,107 tables=3,203 [4/7] Proxy-only: dbt=2,107 (0.0%) tables=3,203 (0.0%) [5/7] MCP+Proxy: dbt=2,107 (0.0%) tables=1,163 (63.7% saved) [6/7] CCR round-trip: original content retrieved Components verified: Proxy starts (FastAPI + uvicorn) and routes to Cortex MCP server connects (MCP Python SDK client) headroom_compress works via MCP headroom_retrieve (CCR) works via MCP Proxy + MCP run simultaneously in same session ``` ## Real Behavior Proof - Environment: macOS, Python 3.11, Snowflake account SFSENORTHAMERICA-NAVNIT_AWS_CAPSTONE - Exact command / steps: `pip install mcp "starlette>=0.37.2,<0.41.0"` then `SF_CONN=<conn> python3 tests/e2e_cortex_savings.py`, `SF_CONN=<conn> python3 tests/e2e_cortex_quality.py`, `SF_CONN=<conn> python3 tests/e2e_cortex_latency.py`, `SF_CONN=<conn> python3 tests/e2e_cortex_proxy.py`, `SF_CONN=<conn> python3 tests/e2e_cortex_mcp.py`, `PROXY_PORT=8798 SF_CONN=<conn> python3 tests/e2e_cortex_proxy_mcp.py` - Observed result: MCP server connects via stdio, tools verified, 63.7% token reduction on table payloads, CCR retrieval works, proxy and MCP run simultaneously without conflict - Not tested: Windows; Cortex Code with live agentic tool calls (simulated via MCP SDK client) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works ## Additional Notes - `role:"tool"` correction: Chat Completions endpoint supports it; Messages endpoint does not (use `user` message with `tool_result` block instead) - MCP tests require `pip install mcp` - Starlette compatibility: `mcp` may install starlette 1.3.1 which conflicts with headroom proxy; fix with `pip install "starlette>=0.37.2,<0.41.0"` --------- Co-authored-by: Cortex Code <noreply@snowflake.com>	2026-06-30 14:14:36 -05:00