2 commits

Author	SHA1	Message	Date
sfc-gh-nashukla	f00ace6da5	fix(cortex-code): migrate to current Cortex REST API endpoints + add e2e benchmarks (#1474) ... ## Description Follow-up to #1190 (Cortex Code provider). Three issues found during post-merge testing, plus full MCP and Proxy+MCP validation added. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [x] Documentation update ## Changes Made - `docs/cortex-code.md`: corrected legacy endpoint references (`inference:complete` → `/v1/chat/completions`), fixed incorrect claim that `role:"tool"` is unsupported (works on Chat Completions, not Messages path), updated proxy mode instructions - `tests/e2e_cortex_savings.py`: migrated from deprecated `inference:complete` to `/api/v2/cortex/v1/chat/completions` + `max_completion_tokens` - `tests/e2e_cortex_latency.py`: new — TTFT + E2E latency benchmark, streaming API, N-run median - `tests/e2e_cortex_quality.py`: new — answer accuracy benchmark; 0 quality regressions at 44–68% compression - `tests/e2e_cortex_proxy.py`: new — proxy-in-the-loop multi-turn test via FastAPI proxy - `tests/e2e_cortex_mcp.py`: new — **MCP mode** test using official MCP Python SDK (stdio transport, same protocol as Cortex Code); verifies `headroom_compress`, `headroom_retrieve`, `headroom_stats` - `tests/e2e_cortex_proxy_mcp.py`: new — **Proxy + MCP** test; starts FastAPI proxy and MCP server simultaneously, exercises both paths in same session ## Testing - [x] Linting passes (`ruff check .`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # MCP mode (e2e_cortex_mcp.py) [1/6] Connecting to headroom MCP server ... OK [2/6] Listing MCP tools ... found: ['headroom_compress', 'headroom_retrieve', 'headroom_stats'] [3/6] Test 1 - dbt run results (40 models) Direct Cortex call ... prompt=2,112 tokens MCP headroom_compress ... saved 0 tokens hash=825cf6f2... Cortex call (MCP-compressed) ... prompt=2,112 saved 0 (0.0%) [4/6] Test 2 - INFORMATION_SCHEMA tables (59 rows) Direct Cortex call ... prompt=3,203 tokens MCP headroom_compress ... saved 1,280 tokens (37.2%) Cortex call (MCP-compressed) ... prompt=1,163 saved 2,040 (63.7%) [5/6] headroom_retrieve CCR round-trip ... original content retrieved [6/6] headroom_stats ... compressions: 2, total_tokens_saved: 1280 MCP TEST PASSED - 38.4% avg token reduction via MCP tools # Proxy + MCP mode (e2e_cortex_proxy_mcp.py) [1/7] Starting headroom proxy ... OK [2/7] Connecting to headroom MCP server ... OK MCP tools: ['headroom_compress', 'headroom_retrieve', 'headroom_stats'] [3/7] Baseline: dbt=2,107 tables=3,203 [4/7] Proxy-only: dbt=2,107 (0.0%) tables=3,203 (0.0%) [5/7] MCP+Proxy: dbt=2,107 (0.0%) tables=1,163 (63.7% saved) [6/7] CCR round-trip: original content retrieved Components verified: Proxy starts (FastAPI + uvicorn) and routes to Cortex MCP server connects (MCP Python SDK client) headroom_compress works via MCP headroom_retrieve (CCR) works via MCP Proxy + MCP run simultaneously in same session ``` ## Real Behavior Proof - Environment: macOS, Python 3.11, Snowflake account SFSENORTHAMERICA-NAVNIT_AWS_CAPSTONE - Exact command / steps: `pip install mcp "starlette>=0.37.2,<0.41.0"` then `SF_CONN=<conn> python3 tests/e2e_cortex_savings.py`, `SF_CONN=<conn> python3 tests/e2e_cortex_quality.py`, `SF_CONN=<conn> python3 tests/e2e_cortex_latency.py`, `SF_CONN=<conn> python3 tests/e2e_cortex_proxy.py`, `SF_CONN=<conn> python3 tests/e2e_cortex_mcp.py`, `PROXY_PORT=8798 SF_CONN=<conn> python3 tests/e2e_cortex_proxy_mcp.py` - Observed result: MCP server connects via stdio, tools verified, 63.7% token reduction on table payloads, CCR retrieval works, proxy and MCP run simultaneously without conflict - Not tested: Windows; Cortex Code with live agentic tool calls (simulated via MCP SDK client) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works ## Additional Notes - `role:"tool"` correction: Chat Completions endpoint supports it; Messages endpoint does not (use `user` message with `tool_result` block instead) - MCP tests require `pip install mcp` - Starlette compatibility: `mcp` may install starlette 1.3.1 which conflicts with headroom proxy; fix with `pip install "starlette>=0.37.2,<0.41.0"` --------- Co-authored-by: Cortex Code <noreply@snowflake.com>	2026-06-30 14:14:36 -05:00
sfc-gh-nashukla	d9d0bf4b79	feat(providers): add Cortex Code (Snowflake CoCo) as a supported agent (#1190) ... ## Description Adds **Cortex Code (CoCo)** — Snowflake's AI coding CLI — as a first-class headroom provider alongside Claude Code, Codex, and Cursor. Cortex Code routes requests to Snowflake's Cortex inference endpoint via the OpenAI-compatible pipeline. This PR adds the provider slice, registers it under `"cortex-code"`, and ships tests that measure real token savings against `claude-sonnet-4-6`. Closes # ## Type of Change - [x] New feature (non-breaking change that adds functionality) - [x] Documentation update ## Changes Made - `headroom/providers/cortex_code/__init__.py` — new provider package - `headroom/providers/cortex_code/runtime.py` — `proxy_base_url()`, `build_launch_env()`, `default_api_url()` (reads `SNOWFLAKE_HOST` / `SNOWFLAKE_ACCOUNT`) - `headroom/providers/cortex_code/install.py` — `build_install_env()` sets `OPENAI_BASE_URL`; `render_setup_lines()` - `headroom/providers/install_registry.py` — registers `"cortex-code"` in `_ENV_BUILDERS` - `tests/test_provider_cortex_code.py` — 15 unit tests - `tests/test_cortex_code_compression.py` — 5 compression benchmark tests (no API key needed) - `tests/e2e_cortex_savings.py` — real REST API benchmark; reads `SF_CONN`/`SF_HOST` from env, no hardcoded identifiers - `docs/cortex-code.md` — integration guide (quick start, library mode, auth, limitations) - `README.md` — Cortex Code row added to agent compatibility matrix ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --with pytest pytest tests/test_provider_cortex_code.py tests/test_cortex_code_compression.py -v tests/test_provider_cortex_code.py::test_cortex_code_proxy_base_url_is_openai_compatible PASSED tests/test_provider_cortex_code.py::test_cortex_code_proxy_base_url_uses_given_port PASSED tests/test_provider_cortex_code.py::test_cortex_code_build_install_env_sets_openai_base_url PASSED tests/test_provider_cortex_code.py::test_cortex_code_build_launch_env_does_not_mutate_input PASSED tests/test_provider_cortex_code.py::test_cortex_code_build_launch_env_applies_project_prefix PASSED tests/test_provider_cortex_code.py::test_cortex_code_build_launch_env_ignores_blank_project PASSED tests/test_provider_cortex_code.py::test_cortex_code_render_setup_lines_contains_proxy_url PASSED tests/test_provider_cortex_code.py::test_cortex_code_render_setup_lines_project_attribution PASSED tests/test_provider_cortex_code.py::test_cortex_code_default_api_url_reads_snowflake_host_env PASSED tests/test_provider_cortex_code.py::test_cortex_code_default_api_url_constructs_url_from_account_name PASSED tests/test_provider_cortex_code.py::test_cortex_code_default_api_url_host_takes_priority_over_account PASSED tests/test_provider_cortex_code.py::test_cortex_code_default_api_url_falls_back_when_no_env PASSED tests/test_provider_cortex_code.py::test_cortex_code_default_api_url_preserves_https_prefix PASSED tests/test_provider_cortex_code.py::test_cortex_code_install_registry_includes_cortex_code PASSED tests/test_provider_cortex_code.py::test_cortex_code_install_registry_unknown_target_skipped PASSED tests/test_cortex_code_compression.py::test_cortex_code_headroom_compression_saves_tokens PASSED tests/test_cortex_code_compression.py::test_cortex_code_tool_results_are_compressed_not_user_turns PASSED tests/test_cortex_code_compression.py::test_cortex_code_tables_json_compresses PASSED tests/test_cortex_code_compression.py::test_cortex_code_rag_search_json_compresses PASSED tests/test_cortex_code_compression.py::test_cortex_code_compression_is_lossless_on_key_content PASSED 20 passed, 1 warning in 1.91s ``` ## Real Behavior Proof - Environment: macOS, Python 3.11, headroom 0.27.0, Snowflake Cortex (claude-sonnet-4-6) - Exact command / steps: `SF_CONN=<connection-name> python3 tests/e2e_cortex_savings.py` - Observed result: 62% average token reduction across 4 payload types; usage.prompt_tokens confirmed in live API responses (full output in Test Output above) - Not tested: headroom wrap cortex-code proxy mode — Cortex REST API path /api/v2/cortex/inference:complete differs from /v1/chat/completions; library mode is the supported path (documented in docs/cortex-code.md Limitations) ```text Tokens saved : 22,077 prompt tokens (4 calls) Avg per call : 5,519 tokens / $0.01656 At 1k/day : $16.56/day | $6,044/year ``` ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes Pre-commit hooks skipped locally due to a GPG signing / ruff-format stash conflict in the dev environment. `ruff check` passes clean on all new files. --------- Co-authored-by: Cortex Code <noreply@snowflake.com>	2026-06-21 22:18:47 -07:00