mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
2493 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Radhakrishnan Pachyappan
|
fd4628d821
|
fix(memory): sync FTS5 and vector indexes on CLI delete/edit/prune/purge
## Problem `headroom memory delete`, `prune`, `edit`, and `purge` all operate on the bare `SQLiteMemoryStore` — they update the primary `memories` table but never touch the FTS5 full-text index (`memory_fts` in `memory.db`) or the vector index (`vec_metadata` / `vec_embeddings` in `memory_vectors.db`). The index maintenance path lives in `HierarchicalMemory.delete()` / `.update()`, which the CLI never instantiates. **Symptoms (from #2856):** ```sql -- After deleting 16 of 46 memories via CLI: SELECT COUNT(*) FROM memories; -- 30 SELECT COUNT(*) FROM memory_fts; -- 46 ← orphans -- memory_vectors.db SELECT COUNT(*) FROM vec_metadata; -- 46 ← orphans ``` Deleted memories keep surfacing in `memory_search` results even after a full server restart, because server startup only re-embeds memories whose `embedding IS NULL` — it never removes orphaned index entries. Fixes #2856. ## Solution Add two best-effort helpers to `headroom/cli/memory.py` that use **direct SQLite** (no `sqlite-vec` extension, no embedder): - **`_remove_from_search_indexes(db_path, memory_ids)`**: removes specific IDs from `memory_fts` and from `vec_metadata` / `vec_embeddings`. Skips silently if an index doesn't exist. - **`_clear_all_search_indexes(db_path)`**: truncates both indexes completely (for purge). Wire these up in four commands: | Command | Change | |---|---| | `delete` | `_remove_from_search_indexes` after `store.delete_batch()` | | `prune` | `_remove_from_search_indexes` after `store.delete_batch()` | | `purge` | `_clear_all_search_indexes` after `store.clear_all()` | | `edit` | If content changed: remove stale entries, clear `embedding` (server re-embeds on next startup), re-add FTS5 entry with new content immediately | The edit path re-adds the FTS5 entry right away so keyword search reflects the new content without requiring a server restart. Vector search is deferred to the next startup re-embed cycle (same as what the server already does for missing embeddings). ## Changes - `headroom/cli/memory.py` — two new helpers; four command call sites - `tests/test_cli_memory_index_sync.py` (new) — 9 unit tests covering both helpers with FTS5 and a stub vector DB. No `sqlite-vec` or embedder required; tests run locally. ## Testing ``` $ python -m pytest tests/test_cli_memory_index_sync.py -v ... 9 passed in 2.38s ``` --------- Signed-off-by: Radhakrishnan Pachyappan <radhakrishnan.p@op.tech> Signed-off-by: Radhakrishnan Pachyappan <gingeekrishna@gmail.com> Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> |
||
|
Abhinav Kumar Singh
|
65961827cf
|
fix(memory): close DirectMem0 resources
## Description `DirectMem0Adapter.close()` now deterministically drains or cancels background writes and releases every initialized client/driver. Fixes #2897 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Initialize the OpenAI client field to `None` so cleanup is safe before or after initialization. - Drain background tasks within a configurable 60-second default, cancel tasks that exceed the timeout, await cancellation, and retain completed/cancelled task status. - Close Mem0, OpenAI, Qdrant, Neo4j, embedder, and graph resources independently, including async close methods, while continuing cleanup if one resource fails. - Clear task and client references and keep `close()` idempotent. - Add regression tests for task draining, timeout cancellation, all resource cleanup, and repeated close calls. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text python -m pytest -q tests/test_memory/test_direct_mem0.py tests/test_memory/test_qdrant_env.py 52 passed ruff check . All checks passed! ruff format --check . 1383 files already formatted python -m mypy headroom Success: no issues found in 515 source files python -m pytest -q Collection blocked in this Windows environment by 174 errors, primarily missing compiled headroom._core; 18 tests skipped. ``` ## Real Behavior Proof - Environment: Windows, Python 3.12, local DirectMem0Adapter instance using real `httpx.Client` resources. - Exact command / steps: Assigned real `httpx.Client()` instances to the adapter's OpenAI and Qdrant resource slots, registered an asynchronous background task, awaited `adapter.close(timeout=1.0)`, then checked both clients' `is_closed` state and the task status. - Observed result: `real httpx clients closed and background task drained`; both clients reported closed, no pending task IDs remained, and the task status was `completed`. - Who maintains it: Headroom Labs maintains this active upstream repository and memory backend. - Install surface: No dependencies or install behavior changed. The fix uses the standard-library asyncio/inspect modules and existing resource close methods; no native code or runtime network access is introduced. - Not tested: The complete test suite could not run past collection because this Windows environment lacks the compiled `headroom._core` extension. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes (full suite blocked by missing native extension; targeted tests pass) - [x] I did not edit `CHANGELOG.md` - it is generated by release-please from my Conventional Commit PR title. ## Screenshots (if applicable) Not applicable. ## Additional Notes The default close timeout is 60 seconds and can be overridden by callers that need a shorter shutdown budget. --------- Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> |
||
|
Suliman Abdulrazzaq
|
e044139001
|
fix(install): trust Docker bridge for dashboard metadata
## Summary Closes #2909. The `persistent-docker` installer now discovers Docker's default bridge gateway and passes the exact `/32` gateway CIDR to the proxy's dashboard metadata allowlist when no explicit `HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS` value is configured. This keeps the existing metadata gate intact while allowing the first-party loopback-published container to see its own Recent Requests and Per-Project Savings data. Explicit user configuration continues to take precedence. Both native wrappers (POSIX and PowerShell) use the same behavior, and installer integration coverage verifies the generated Docker command. ## Validation - `python -m pytest tests/test_install/test_native_installers.py -q -k bash` (1 skipped on Windows because Bash is unavailable) - PowerShell wrapper smoke test with the repository fake Docker shim: verified `docker network inspect bridge` is called and `HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS=172.17.0.1/32` is passed to `docker run` - Explicit allowlist smoke test: verified an existing `HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS` is preserved without adding a discovered default - `git diff --check` ## Real behavior proof Setup tested: Windows 11 host, PowerShell wrapper, repository fake Docker shim (Docker CLI is not installed in this environment). Exact command: `headroom.ps1 install apply --profile smoke --port 18999 --image fake/headroom:test`. Observed result: the generated Docker invocation included `docker network inspect bridge --format ...` and `--env HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS=172.17.0.1/32`, and the installer completed successfully. Not tested: a live Docker daemon/dashboard request on this host. --------- Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> |
||
|
Abhinav Kumar Singh
|
c85abf7a87
|
fix(oauth2): make repository lint checks pass
## Description Fixes #2895 The repository-wide Ruff command failed on the bundled OAuth2 plugin. This change sorts the public export list, narrows the optional LiteLLM setup exception handling to expected failures, and replaces the silent HTTP error-body drain with explicit handling and debug logging. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Sorted headroom_oauth2.__all__ according to Ruff RUF022. - Replaced the blind install-time Exception catch with explicit ImportError, AttributeError, OSError, TypeError, and ValueError handling. - Replaced the silent HTTPError body-drain pass with explicit HTTPException, OSError, and ValueError handling plus debug logging. - Added regression coverage for body-drain failures and invalid LiteLLM header state. ## Testing - [x] Unit tests pass - [x] Linting passes (ruff check .) - [x] Type checking passes (mypy headroom) - [x] New tests added - [x] Manual testing performed ### Test Output ruff 0.15.17 ruff check . All checks passed! ruff format --check . 1382 files already formatted python -m mypy headroom Success: no issues found in 515 source files PYTHONPATH=plugins/headroom-oauth2/src python -m pytest -q plugins/headroom-oauth2/tests 39 passed in 11.12s Full Python pytest was attempted: 8,878 tests were collected, but collection stopped with 174 environment errors because the required compiled headroom._core extension is unavailable in this Windows checkout. 18 tests were skipped. ## Real Behavior Proof - Environment: Windows PowerShell, Python 3.12, Ruff 0.15.17. - Exact command / steps: Ran the OAuth2 test suite with PYTHONPATH pointing to plugins/headroom-oauth2/src. Its local HTTPServer fixture exercised real urllib token minting, cached refresh, HTTP error handling, and middleware injection. - Observed result: 39 tests passed, including real loopback token minting and the new failure-path tests; repository-wide Ruff completed with no diagnostics. - Not tested: External identity-provider traffic and the full Python suite after native extension build, because the local Windows toolchain cannot build headroom._core. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of the code - [x] I have commented my code where needed - [ ] I have made corresponding changes to the documentation (not needed; behavior and lint handling are covered by existing comments/tests) - [x] My changes generate no new warnings - [x] I have added tests that prove the fix is effective - [ ] New and existing full-repository unit tests pass locally (blocked by missing native headroom._core) - [x] I did not edit CHANGELOG.md ## Additional Notes No dependencies or public API behavior changed. Expected environment and transport failures remain handled; unexpected programmer errors now propagate instead of being silently swallowed. The OAuth2 plugin remains standard-library-only. --------- Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> |
||
|
Abhinav Kumar Singh
|
07d89a751d
|
fix(litellm): close shared cloud client
## Description Adds an explicit, idempotent async cleanup lifecycle for the LiteLLM callback's shared cloud HTTP client. Fixes #2894 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `HeadroomCallback.aclose()` to close the lazily-created `httpx.AsyncClient` and clear its reference. - Made cleanup safe when cloud mode was never used and when shutdown cleanup is invoked more than once. - Added regression coverage for initialized-client cleanup, reference clearing, and repeated/no-op cleanup. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text python -m pytest -q tests/test_integrations/test_litellm_callback.py 5 passed ruff check . All checks passed! ruff format --check . 1382 files already formatted python -m mypy headroom Success: no issues found in 515 source files python -m pytest -q Collection blocked in this Windows environment by 174 errors, primarily missing compiled headroom._core; one unrelated test also lacks respx. ``` ## Real Behavior Proof - Environment: Windows, Python 3.12, loopback HTTP server, real `httpx.AsyncClient`. - Exact command / steps: Started a local HTTP server, configured `HeadroomCallback(api_key="hdr_test", api_url="http://127.0.0.1:<port>")`, ran `_cloud_compress()` against it, saved the created client, awaited `callback.aclose()`, then awaited `callback.aclose()` again. - Observed result: The real cloud request succeeded; the client was open during the request, reported closed after `aclose()`, the callback reference became `None`, and repeated cleanup was harmless. - Who maintains it: Headroom Labs maintains this active upstream repository and its LiteLLM integration. - Install surface: No dependencies or install behavior changed. Cloud mode continues to use the existing optional `httpx` dependency; no native code or runtime network access is introduced by this fix. - Not tested: The complete test suite could not run past collection because the local Windows environment lacks the compiled `headroom._core` extension. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] Documentation changes are not required; `aclose()` is documented in its public docstring and the host owns shutdown sequencing - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes (full suite blocked by missing native extension; targeted tests pass) - [x] I did not edit `CHANGELOG.md` - it is generated by release-please from my Conventional Commit PR title. ## Screenshots (if applicable) Not applicable. ## Additional Notes The callback exposes `aclose()` for the host application's async shutdown lifecycle, matching the existing ASGI integration pattern. |
||
|
Sudhindra Desai
|
99f07e7bbd
|
fix(proxy): cache litellm model resolution to stop repeated Provider List spam
## Description The proxy repeatedly prints LiteLLM's `Provider List: https://docs.litellm.ai/docs/providers` banner during normal operation, with no explanation or way to suppress it (#2851). Root cause: `_resolve_litellm_model()` in `headroom/proxy/savings_tracker.py` runs on every savings-tracking update (i.e. every request). For any model LiteLLM can't price (a custom/local/gateway model name — e.g. the reporter's local oMLX setup), the uncached fallback path calls `litellm.cost_per_token(...)` purely to probe resolvability. When that probe fails, LiteLLM prints the banner as an internal side effect before raising, and since the probe was never cached, it re-fires on every single request for the same unresolvable model. **Update:** review flagged that the first version of this fix cached into a plain, unbounded `dict` keyed by the (client-controlled) model name — a memory-retention path on a request-facing proxy, since a caller can grow it without limit by sending a new model string on every request. Replaced with a bounded `functools.lru_cache`; see Changes Made below. Closes #2851 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/savings_tracker.py`: `_resolve_litellm_model()` is now decorated with `@lru_cache(maxsize=256)` instead of backing onto a hand-rolled unbounded `dict`. An evicted model name simply re-probes LiteLLM on next use — never a correctness issue, only whether the noisy failure banner reruns for that specific name. - `tests/conftest.py`: added a global `autouse` fixture, `_reset_litellm_model_resolution_cache`, that clears the cache before and after every test. It's process-lifetime and module-global, and several existing tests monkeypatch `savings_tracker.litellm` with different behavior per test while reusing common model names like `"gpt-4o"` — without a reset, whichever test resolves a name first silently wins that cache slot for the rest of the run and later tests stop exercising their own fake. - `tests/test_savings_tracker_litellm_resolution_cache.py` (new): regression tests for the three properties that actually matter — repeated resolution of one unknown model only probes LiteLLM once, resolving far more distinct names than the bound never grows the cache past it, and an evicted name is transparently re-probed rather than reusing a slot it no longer owns. - No behavior change for models LiteLLM can already price (fast path via `model_cost` lookup) — only the noisy uncached probe path is memoized, same as before. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) — not run; `mypy` isn't installed in this environment - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python3 -m pytest tests/test_proxy_savings_history.py tests/test_savings_tracker_zero_price.py \ tests/test_savings_tracker_litellm_resolution_cache.py -q tests/test_proxy_savings_history.py .................................... [ 73%] ... [ 79%] tests/test_savings_tracker_zero_price.py ....... [ 93%] tests/test_savings_tracker_litellm_resolution_cache.py ... [100%] 49 passed, 1 warning in 1.26s # Re-run in reversed file order to check for the exact order-dependence the # review flagged — same 49 passed, no failures either direction: $ python3 -m pytest tests/test_savings_tracker_litellm_resolution_cache.py \ tests/test_savings_tracker_zero_price.py tests/test_proxy_savings_history.py -q 49 passed, 1 warning in 1.11s $ python3 -m ruff check headroom/proxy/savings_tracker.py tests/conftest.py \ tests/test_savings_tracker_litellm_resolution_cache.py All checks passed! ``` ## Real Behavior Proof - Environment: macOS, Python 3.12.3, this repo checked out locally. - What changed since the last review pass: I got the compiled `headroom._core` Rust extension in hand (by installing the published `headroom-ai[all]` wheel into a separate venv and copying its `_core.abi3.so` next to this local source tree — same Python ABI, pure-Python edits in `savings_tracker.py` don't touch the compiled boundary). That unblocked the full test files this fix touches, including `tests/test_proxy_savings_history.py`, which was previously reported as untestable here. - Exact command / steps: three properties asserted directly against the real (now-bounded) cache in `tests/test_savings_tracker_litellm_resolution_cache.py`: 1. Resolve the same unresolvable model 5 times → assert the underlying `litellm.cost_per_token` probe fired exactly once. 2. Resolve `_MODEL_RESOLUTION_CACHE_MAXSIZE + 50` distinct model names → assert `_resolve_litellm_model.cache_info().currsize` stays at exactly `_MODEL_RESOLUTION_CACHE_MAXSIZE` (256), never higher — this is the actual memory-retention fix the review asked for. 3. Resolve one model, push exactly `maxsize` other distinct names through to evict it via LRU, then resolve it again → assert it re-probed (call count went 1 → 2), proving eviction is real and not just an untested cache_info number. - Observed result: all three pass; full affected-file suite (49 tests) passes in both forward and reversed run order, confirming the new `conftest.py` fixture actually fixes the cross-test leakage risk (verified by literally reordering the files, not just by inspection). - Not tested: a live HTTP request against a running `headroom proxy` process specifically re-exercising this bounded-cache commit — the earlier "20 simulated requests" proof against the previous (unbounded-dict) version of this fix was via a standalone script, not a real server; I have not repeated that specific live-server pass against this commit. The unit-level proof above exercises the exact same function (`_resolve_litellm_model`) the real proxy calls per-request from `headroom/proxy/server.py`, so I'm confident it generalizes, but flagging the gap rather than implying I re-ran it live. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas — the bound/eviction rationale is commented above `_resolve_litellm_model`, and the cross-test leakage rationale is commented above the new `conftest.py` fixture - [ ] I have made corresponding changes to the documentation — N/A, internal implementation detail with no user-facing API/doc surface - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes - `mypy` still hasn't been run — not installed in this sandbox, and I didn't want to widen the PR further by installing/configuring it just for this. Flagging rather than silently skipping. - The earlier "Additional Notes" gap about `test_proxy_savings_history.py` being untestable in this environment is resolved (see Real Behavior Proof) — it now runs and passes, including the pre-existing `test_litellm_resolution_and_savings_estimation_fallbacks` test that exercises `_resolve_litellm_model` with a mutated `model_cost` dict across several assertions in one test. - Deliberately did not also bound `headroom/pricing/litellm_pricing.py`'s sibling `_resolved_model_cache` — same shape of cache, arguably the same exposure — since it's outside this PR's diff and touching it wasn't asked for. Flagging in case a maintainer wants it as a fast follow-up rather than silently leaving it unmentioned. --------- Co-authored-by: connectsudhindra-gif <connectsudhindra-gif@users.noreply.github.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Abhinav Kumar Singh
|
4bd8ecd1e3
|
fix(memory): close MCP backend on shutdown
## Description Closes the initialized LocalBackend and cancels in-flight initialization whenever the memory MCP stdio transport exits. Fixes #2898 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added an explicit server cleanup callback that cancels and awaits pending backend initialization. - Closes an initialized backend exactly once and clears the backend/task references. - Runs cleanup in `_run()` through a `finally` block after the stdio transport exits, including transport errors. - Added regression coverage for initialized cleanup, pending initialization cancellation, idempotence, and `_run()` shutdown behavior. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text python -m pytest -q tests/test_memory/test_mcp_server.py 15 passed, 20 warnings ruff check . All checks passed! ruff format --check . 1382 files already formatted python -m mypy headroom Success: no issues found in 515 source files python -m pytest -q Collected 8881 items / 174 errors / 18 skipped. Interrupted during collection because this Windows environment lacks the compiled headroom._core extension. ``` ## Real Behavior Proof - Environment: Windows, Python 3.12, async MCP server lifecycle test with the real `create_memory_server()` closure and an embedded server transport stub. - Exact command / steps: Ran `python -m pytest -q tests/test_memory/test_mcp_server.py`; the regression tests initialized a backend through the server's registered tool lifecycle, returned the stdio transport, and invoked the cleanup callback from `_run()`'s `finally` path. - Observed result: 15 tests passed. Initialized backends were closed once, pending initialization was cancelled and awaited, and transport exit invoked cleanup even when the server run returned. - Who maintains it: Headroom Labs maintains this active upstream repository and memory MCP server. - Install surface: No dependencies or install behavior changed. The fix uses existing asyncio lifecycle handling and `LocalBackend.close()`; no native code or runtime network access is introduced. - Not tested: The complete repository suite could not run past collection because this Windows environment lacks the compiled `headroom._core` extension. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes (full suite blocked by missing native extension; targeted tests pass) - [x] I did not edit `CHANGELOG.md` - it is generated by release-please from my Conventional Commit PR title. ## Screenshots (if applicable) Not applicable. ## Additional Notes Cleanup is attached to each created memory MCP server and is idempotent, so embedded callers can invoke the same lifecycle callback safely if needed. |
||
|
Suliman Abdulrazzaq
|
620028fa18
|
fix(proxy): emit request log timestamps in UTC
## Description `RequestLog.timestamp` was serialized with `datetime.now().isoformat()`, which omits timezone information. Browsers then interpret the value as local time, so requests from a UTC container can display negative ages in non-UTC dashboards. Closes #2910 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Emit request-log timestamps from `datetime.now(timezone.utc)` so the ISO-8601 value includes `+00:00`. - Add a regression test that parses the emitted timestamp and requires a UTC offset. ## Testing - [x] New tests added for the regression - [x] `python -m compileall -q headroom/proxy/outcome.py tests/test_request_outcome.py` - [x] `git diff --check` - [ ] Unit tests pass (`pytest`) — the repository's Rust extension cannot build in this Windows environment because `link.exe` (MSVC) is unavailable; the focused test is included for CI. ### Test Output ```text python -m compileall -q headroom/proxy/outcome.py tests/test_request_outcome.py (pass) git diff --check (pass) uv run pytest tests/test_request_outcome.py -q blocked while building headroom-py: linker `link.exe` not found ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.11; proxy timestamps are generated in `headroom/proxy/outcome.py`. - Exact command / steps: traced the Recent Requests write path and added a timestamp assertion in `tests/test_request_outcome.py` (CI will run with the project's Rust toolchain). - Observed result: the production call now emits an ISO-8601 timestamp with `+00:00`; the regression assertion requires an offset-aware UTC value, preventing browser timezone skew. - Not tested: full pytest suite locally because the MSVC linker is unavailable. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review - [x] I have added tests that prove my fix is effective - [x] I did not edit `CHANGELOG.md` Signed-off-by: Suliman Abdulrazzaq <suliman9000a@gmail.com> |
||
|
Joseph Benno
|
0ae948c151
|
fix(cache): bound compression cache bookkeeping
## Description `CompressionCache.max_entries` bounded the main compression cache, but not `_stable_hashes` or `_first_seen`. A long-lived session could therefore retain every unique tool-result hash even while `_cache` stayed empty. This change applies the same bounded retention to both side tables. It also cleans up expired first-seen entries and resets the timing window when compression occurs near the TTL boundary. Fixes #2874 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Store stable hashes and first-seen timestamps in ordered mappings. - Evict oldest entries when either side table exceeds `max_entries`. - Keep all bookkeeping under the existing reentrant lock. - Reset first-seen timing after compression near the TTL boundary. - Add tests covering size limits, TTL behavior, frozen-prefix safety, and concurrency. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run ruff format --check . Passed uv run ruff check . All checks passed! uv run mypy headroom Success: no issues found in 515 source files uv run pytest Passed ``` Focused cache tests on macOS 26.5.2 arm64 with Python 3.11.14: ```text uv run pytest tests/test_compression_cache.py::TestCompressionCacheRetention -v 5 passed in 0.30s uv run pytest tests/test_compression_cache.py -q 38 passed in 5.76s ``` After the final formatting-only commit, the cache test file was also run on Linux with Python 3.12.13: ```text 37 passed, 1 skipped in 32.70s ``` ## Real Behavior Proof - Environment: Linux 6.18 x86_64, Python 3.12.13, `CompressionCache(max_entries=100)`. - Exact command / steps: Created a `CompressionCache(max_entries=100)`, generated 20,000 unique content hashes, and passed each hash through `mark_stable()` and `should_defer_compression()`. Store sizes were sampled after 100, 1,000, 5,000, and 20,000 results. - Observed result: `_cache=0`, `_stable_hashes=100`, and `_first_seen=100` at every sample after reaching the configured limit. At 20,000 results, traced memory was approximately 0.03 MB current and 0.04 MB peak. Before the fix, the same workload retained all 20,000 hashes and timestamps. - Not tested: A live multi-hour proxy/provider session. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented the code where retention behavior is not obvious - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove the fix is effective - [x] New and existing unit tests pass locally - [x] I did **not** edit `CHANGELOG.md` ## Screenshots N/A — internal cache bookkeeping change. ## Additional Notes No changes to dependencies, public APIs, or configuration. No user-facing behavior changes. |
||
|
Abhinav Kumar Singh
|
739fdef423
|
fix(proxy): cancel periodic TOIN task on shutdown
## Description Retains the periodic TOIN statistics task on application state and reaps it during proxy lifespan shutdown. Fixes #2896 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Store the periodic TOIN task as `app.state.periodic_toin_stats_task` when enabled. - Cancel and await the task with the existing bounded shutdown helper before stopping proxy resources. - Clear the application state reference after shutdown. - Add regression coverage proving the task is canceled and reaped when the FastAPI lifespan exits. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text python -m pytest -q tests/test_proxy_telemetry_env.py 0 items / 1 error ModuleNotFoundError: No module named 'headroom._core' Temporary in-process native-core stub + real FastAPI TestClient: python -m pytest -q tests/test_proxy_telemetry_env.py 8 passed ruff check . All checks passed! ruff format --check . 1382 files already formatted python -m mypy headroom Success: no issues found in 515 source files python -m pytest -q Collected 8878 items / 174 errors / 18 skipped. Interrupted during collection because this Windows environment lacks the compiled headroom._core extension. ``` ## Real Behavior Proof - Environment: Windows, Python 3.12, real FastAPI `TestClient` lifespan; only the unavailable native `headroom._core` import was replaced with an in-process test stub. - Exact command / steps: Ran the telemetry test module with the temporary core stub. The new test enabled periodic TOIN stats, held the real lifespan open, observed the stored task, exited the `TestClient` context, and checked that the task was canceled and the state reference cleared. - Observed result: 8 telemetry tests passed, including the new shutdown regression test; the periodic task reported canceled after lifespan exit and no task reference remained on application state. - Who maintains it: Headroom Labs maintains this active upstream repository and proxy lifecycle. - Install surface: No dependencies or install behavior changed. The fix uses existing asyncio and FastAPI lifecycle APIs; no native code or runtime network access is introduced. - Not tested: The complete suite and the unmodified proxy test command cannot run in this Windows environment without the compiled `headroom._core` extension. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes (full suite blocked by missing native extension; stubbed focused tests pass) - [x] I did not edit `CHANGELOG.md` - it is generated by release-please from my Conventional Commit PR title. ## Screenshots (if applicable) Not applicable. ## Additional Notes The shutdown uses the existing three-second `_timed()` bound and handles the disabled configuration without creating a task. |
||
|
Suliman Abdulrazzaq
|
5e53b8aa0a
|
fix(opencode): keep Claude models off OpenAI provider
## Description The injected `headroom` OpenCode provider uses `@ai-sdk/openai-compatible` and the proxy's `/v1/chat/completions` route. It currently advertises Claude model IDs in that provider, so OpenCode sends Claude requests to the OpenAI upstream and receives `invalid_api_key` errors. Keep Claude on OpenCode's native `anthropic` provider, which Headroom already redirects to the proxy. Closes #2911 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (bug fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Remove Claude IDs from the injected OpenAI-compatible provider model map. - Keep GPT models available through the `headroom/<id>` namespace. - Add regression assertions that generated config never advertises Claude models on this endpoint. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text python -m pytest tests/test_providers_opencode_config.py -q -k "not build_launch_env_with_project" 40 passed, 1 deselected python -m ruff check headroom/providers/opencode/config.py tests/test_providers_opencode_config.py All checks passed! python -m compileall -q headroom/providers/opencode/config.py tests/test_providers_opencode_config.py (pass) ``` The full config test module also exposes an unrelated pre-existing Windows path assertion failure in `test_build_launch_env_with_project`; the failure is caused by comparing a native `Path` string with JSON-escaped backslashes and is outside this change. ## Real Behavior Proof - Environment: Windows 11, Python 3.11; no external API credentials used. - Exact command / steps: `python -c "from headroom.providers.opencode.config import headroom_provider_entry; print(sorted(headroom_provider_entry(8787)['models']))"` - Observed result: `['gpt-4.1', 'gpt-4o']`; the generated OpenAI-compatible provider no longer advertises any `claude-*` IDs. - Not tested: live OpenCode request routing or a vendor API call, because they require external credentials. The regression suite verifies the generated configuration consumed by OpenCode. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas (not needed; the provider routing rationale is documented inline) - [ ] I have made corresponding changes to the documentation (the generated provider behavior is documented in code; existing docs describe the separate npm provider) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing relevant unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes The native `anthropic` and `openai` provider entries both continue to point at the Headroom proxy, so this change only removes an invalid duplicate Claude route and does not affect native Claude traffic. |
||
|
Abhay Singh
|
702dbc5902
|
fix(opencode): ship the transport hook-shim so wheel installs route Node child traffic
## Description The OpenCode transport plugin injects `NODE_OPTIONS=--import=<...>/hook-shim/handler.js` into every spawned Node child so its `fetch`/`http` traffic routes through the proxy (`transport.ts` wraps those globals only in the plugin's own process; a spawned `npx` MCP server or `tokensave serve` is a fresh process). That shim was never shipped in the wheel: - Only `headroom/providers/opencode/_dist/entry.opencode.js` is committed and packaged. - The shim source at `plugins/opencode/hook-shim/handler.js` imports the non-bundled `../dist/index.js`, which a pip install (no `node_modules`) cannot resolve. Before #2806, the missing file crashed every Node MCP under `headroom wrap opencode` with `ERR_MODULE_NOT_FOUND` at the ESM loader, before the stdio handshake. #2806 added an `existsSync` guard so the loader is not injected when the shim is absent, which stopped the crash but left child-process routing silently disabled for all wheel installs (#2850). This ships the shim. It builds a self-contained variant in the standalone tsup config (`src/hook-shim.ts`, with the transport bundled inline like the entry, since site-packages has no `node_modules`), and commits it to `headroom/providers/opencode/hook-shim/handler.js` -- the sibling of `_dist/` that `transport.ts`'s `shimImportSpecifier()` resolves via `../hook-shim/handler.js`. maturin packages every file under `headroom/`, so the wheel now carries it, and `existsSync` finds it, so the loader routes spawned Node children again. Fixes #2850 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `plugins/opencode/src/hook-shim.ts` (new): self-contained Node `--import` loader that installs the transport from the inlined `./transport.js`. - `plugins/opencode/tsup.standalone.config.ts`: add `hook-shim/handler` as a second standalone entry. - `headroom/providers/opencode/hook-shim/handler.js` (new): the committed self-contained shim (output of `npm run build:standalone`), shipped by maturin. - `.github/workflows/opencode-plugin.yml`: byte-compare the committed shim against a fresh build (mirrors the existing `entry.opencode.js` guard), and add the shim path to the workflow triggers. - `tests/test_providers_opencode_plugin_path.py`: added `test_hook_shim_is_committed_next_to_the_entry_bundle` asserting the shim ships as a sibling of `_dist/` and is the self-contained build. ## Testing - [x] Unit tests pass (`pytest` + `vitest`) - [x] Type checking passes (`tsc --noEmit`) - [x] New tests added for new functionality - [x] Committed shim rebuilt and byte-matches the standalone build - [ ] Manual testing performed ### Test Output ```text # Fail-before (shim removed from the package): tests/test_providers_opencode_plugin_path.py::test_hook_shim_is_committed_next_to_the_entry_bundle FAILED # Pass-after: tests/test_providers_opencode_plugin_path.py tests/test_providers_opencode_install.py tests/test_providers_opencode_config.py 49 passed, 1 pre-existing failure # the 1 failure (test_build_launch_env_with_project) fails identically on pristine main: # a Windows path-escaping quirk in OPENCODE_CONFIG_CONTENT, unrelated to this diff. # TypeScript: npm run typecheck (clean), npm test -> 14 passed # Standalone build: entry.opencode.js byte-unchanged vs the committed blob; # dist-standalone/hook-shim/handler.js cmp-matches the committed shim. # Shim runtime sanity (node): # with HEADROOM_OPENCODE_TRANSPORT_PROXY_URL set -> loads, exit 0, wraps globalThis.fetch # without it -> throws "loaded without HEADROOM_OPENCODE_TRANSPORT_PROXY_URL", exit 1 ``` ## Real Behavior Proof - Environment: Windows 11, Node v24.11.0, npm 11.5.2, tsup 8.5.1 / esbuild 0.28.1 (pinned via `npm ci`), Python 3.12.11, pytest 9.1.1, ruff 0.15.17. - Exact command / steps: confirmed `transport.ts` resolves `../hook-shim/handler.js` next to the loaded entry (so the wheel needs it at `providers/opencode/hook-shim/handler.js`), that the current wheel ships only `_dist/entry.opencode.js`, and that maturin packages every file under `headroom/`. Added the standalone shim entry, ran `npm run typecheck` and `npm test` (clean), `npm run build:standalone`, verified `entry.opencode.js` is byte-identical to the committed git blob (the standalone build is reproducible; my working copy was only autocrlf-inflated), copied the built shim to the wheel path, and exercised it in Node: it installs the transport (wraps `fetch`) with the proxy env set and throws without it. Fail-before by removing the shim (the new Python test fails); pass-after restored. - Observed result: `headroom/providers/opencode/hook-shim/handler.js` now ships in the package as a self-contained module, so a pip-installed `headroom wrap opencode` routes spawned Node children (npx MCPs, `tokensave serve`) through the proxy instead of leaving them unrouted, and never crashes them. - Not tested: a full pip-install-and-spawn on Linux with a live OpenCode session (no OpenCode client here). The shim is verified to load and wrap `fetch` under Node, the bundle is reproducible and byte-checked by CI, and the packaging path is maturin's standard file inclusion under `headroom/`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md`: it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes The checkout keeps using `plugins/opencode/hook-shim/handler.js` (which imports `../dist/index.js` from the regular build), so dev behavior is unchanged; only the wheel gains the self-contained sibling. `entry.opencode.js` is byte-unchanged, so its existing CI guard still passes. The committed shim is stored with LF endings so the Linux CI byte-compare matches. |
||
|
Abhay Singh
|
d7b25ae3bb
|
fix(wrap/serena): install Serena from the serena-agent PyPI wheel, not the git source
## Description `headroom/mcp_registry/install.py` (`build_serena_spec`) and the wrap-time Serena pre-index in `headroom/cli/wrap.py` both ran: ``` uvx --from git+https://github.com/oraios/serena serena ... ``` The git source forces a from-source build. On proot-based filesystems (Termux + proot-distro on Android, some restricted Linux) `uv` cannot hardlink build dependencies into a fresh build venv, so the build fails immediately and Serena's MCP server fails to start on every `headroom wrap codex` launch: ``` × Failed to download and build `serena-agent @ git+https://github.com/oraios/serena@<commit>` ╰─▶ failed to hardlink file ... Operation not permitted (os error 1) ``` Setting `UV_LINK_MODE=copy` fixes it in an interactive shell, but Codex strips most env vars from the MCP subprocesses it spawns, so that workaround does not reliably reach Serena's launch. Serena publishes the official `serena-agent` package to PyPI with prebuilt wheels, and it exposes the same `serena` console script (`serena = "serena.cli:top_level"` in the project's `pyproject.toml`), so `uvx --from serena-agent serena ...` runs the identical command without a build step. On platforms where the git build already worked there is no functional difference. Fixes #2871 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/mcp_registry/install.py` (`build_serena_spec`): `--from git+https://github.com/oraios/serena` -> `--from serena-agent`. - `headroom/cli/wrap.py` (Serena `project index` pre-warm): same swap. - `tests/test_mcp_registry/test_install.py`: updated the spec assertion and added `test_build_serena_spec_uses_pypi_not_git_source` (asserts `serena-agent` is used and no `git+` source remains). - `tests/test_cli/test_wrap_serena_boost.py`: the pre-index test now asserts `serena-agent` is in the command and the git source is not. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # Fail-before (source swap stashed, updated tests kept): tests/test_mcp_registry/test_install.py::test_build_serena_spec_uses_agent_context FAILED tests/test_mcp_registry/test_install.py::test_build_serena_spec_uses_pypi_not_git_source FAILED tests/test_cli/test_wrap_serena_boost.py::test_preindex_runs_serena_in_cwd FAILED # Pass-after: tests/test_mcp_registry/ tests/test_cli/test_wrap_serena_boost.py tests/test_cli/test_serena_migrate.py tests/test_cli/test_serena_disable.py 135 passed # uvx ruff@0.15.17 check -> All checks passed! # uvx mypy@1.20.2 headroom/mcp_registry/install.py -> Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1, ruff 0.15.17 and mypy 1.20.2 via uvx. - Exact command / steps: confirmed `serena-agent` exists on PyPI (v1.6.1, homepage github.com/oraios/serena) and that its `pyproject.toml` declares `[project.scripts] serena = "serena.cli:top_level"`, so the `serena start-mcp-server ...` invocation is unchanged. Swapped both `--from` sources, then fail-before with `git stash push headroom/mcp_registry/install.py headroom/cli/wrap.py` (the two production-asserting tests fail on the old git source) and pass-after with `git stash pop` (135 serena-suite tests pass). Verified no `git+https://github.com/oraios/serena` references remain in `headroom/`. - Observed result: `build_serena_spec` and the pre-index command now install Serena from the `serena-agent` PyPI wheel, so a proot environment gets the prebuilt wheel instead of a from-source build that cannot hardlink. The migration/ledger tests, which use the old git spec as a deliberately-stale fixture, are unaffected. - Not tested: a live `headroom wrap codex` on a real proot/Termux device (not available here). The change is a package-source swap verified against Serena's own published package metadata and the existing spec/command tests. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md`: it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes The git source was unpinned (tracked the repo default branch), so switching to `serena-agent` from PyPI does not lose a version pin; if anything it is more reproducible. The issue reporter also noted that `headroom wrap codex` force-rewrites the Serena block in `~/.codex/config.toml` from this template on every launch, which is why the fix has to live in the package source rather than a user config edit -- this PR puts it there. |
||
|
gglucass
|
c6f99482e1
|
fix(proxy/anthropic): run tool-search history repair after turn hooks
## Description
`strip_unsupported_tool_search_blocks` (#2807) validates every replayed
`tool_reference` in the transcript against the request's `tools` array.
It ran *before* the turn-hooks block in `handlers/anthropic.py`, and a
registered turn hook may rewrite that array — the hook surface is
documented as "a registered hook may inspect or rewrite the outbound
tools/messages before we send upstream".
So a hook that drops a tool named by a replayed reference leaves the
repair having validated against a stale view, and upstream rejects the
request:
```text
400 Tool reference 'X' not found in available tools
```
The repair's correctness argument is that it validates against exactly
the `tools` array upstream will see. That was true at the old call site
and stopped being true one block later.
### Fix
Move the repair to after the turn-hooks block, so it is the last stage
that can invalidate a reference:
- It still runs **after** the deferral injection, so the tool just
injected counts as present — the main loop strips nothing and the frozen
prefix stays byte-identical.
- Nothing past the new call site mutates `body["tools"]` on the outbound
path. (The two later `continuation_body["tools"]` assignments build a
*derived* body from the already-repaired `body`, so they inherit the
repair.)
- It still runs before the consistency token re-count, so `tok_after`
continues to reflect the repaired messages.
- It remains unconditional (not gated on `HEADROOM_TOOL_SEARCH`, not
gated on `_bypass`), so transcripts poisoned before the flag was turned
off still recover.
`strip_unsupported_tool_search_blocks` is copy-on-write and returns the
original `messages` object by identity when nothing is removed, so
relocating the call does not change the no-op path.
### Severity
Latent. No turn hook ships in-tree, so this cannot fire on a default
install — it is reachable only through a third-party registered hook
that shrinks the tools array. Filing the fix now so the ordering
constraint is enforced by a test rather than rediscovered.
Closes #2888
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/anthropic.py`: the tool-search history repair
block moves from just after the deferral injection to just after the
turn-hooks block. The comment now states the ordering constraint in both
directions (after injection, after hooks) so the next person to add a
stage knows where the boundary is. No logic change.
- `tests/test_proxy/test_tool_search_repair_after_turn_hooks.py` (new):
two handler-level regressions. Ordering is the whole property under
test, so a unit test of the helper cannot see it — these drive the real
handler through `TestClient` and assert on the forwarded body.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed — no in-tree turn hook exists to exercise
this against a live API key; the handler-level test below is the
substitute, see Not tested.
### Test Output
```text
$ uv run --extra dev pytest tests/test_proxy/ -q
======================= 241 passed, 1 warning in 35.82s ========================
$ uvx ruff check headroom tests
All checks passed!
$ uv run --extra dev mypy headroom
Success: no issues found in 515 source files
```
## Real Behavior Proof
- Environment: macOS 15 (darwin 24.6.0), Python 3.10.18, pytest 9.0.3,
in a worktree off `upstream/main` at
|
||
|
Tejas Chopra
|
1a04c957f5
|
fix(cache): stabilize Anthropic block-growing lineages (#2917)
## Description Fixes the remaining Anthropic prompt-cache failure in #2671 and the newly reported parallel-tool-profile variant. The production failure has three connected parts: 1. `SessionTrackerStore.resolve_tracker` only recognized whole-message prefixes. A caller that grows or regenerates blocks inside one message therefore received a fresh tracker every turn, so previous forwarded state was always empty and breakpoint relocation could never run. 2. `normalize_message_cache_control` always moved the message breakpoint to the newest block. That is correct for a pure block append, but a message that rewrites its tail can never match the prior newest-block write and repeatedly rewrites the full message prefix. 3. Parallel Anthropic sub-calls can carry identical messages but different tools. Because tools precede messages in the provider cache key, sharing one frozen-prefix tracker across those calls cross-contaminates cache state even when message lineage is identical. This PR deliberately combines the valid parts of #2699 and #2702, fixes the discriminator between their two shapes, and adds cache-key affinity for the second pattern reported on #2671. In particular, a pure append is identified by `stable_prefix_blocks == previous_block_count`; rewritten-tail relocation is only possible when `stable_prefix_blocks < previous_block_count`. This prevents a pure append from being pinned to an old boundary. Closes #2671. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added one canonical history classifier with distinct exact, whole-message append, pure block append, rewritten-tail, and diverged outcomes. - Kept pure block appends on newest-block breakpoint placement so each request reads the old prefix and writes only its appended blocks. - Added block-level replay of the prior forwarded bytes for pure appends; the whole-message delta path explicitly refuses this shape so it cannot silently discard appended blocks. - Kept a rewritten-tail request on its existing tracker and anchored its breakpoint to the end of the byte-stable leading run. - Made rewritten-tail matching conservative: one changed message, unchanged message count, no shrink, at least 8 stable leading blocks covering at least half of old and new content, and a fixed suffix of at least 2 blocks. - Required a unique best rewritten-tail lineage match. Ambiguity creates a fresh lineage instead of making sibling sub-calls ping-pong one tracker. - Added a stable affinity fingerprint over model, deterministically forwarded tools, tool choice, thinking, and output configuration. Different provider cache-key profiles cannot share frozen-prefix state. - Snapshotted previous original/forwarded messages once in the Anthropic handler and reused that exact state for delta extraction, replay, and breakpoint placement. - Added `HEADROOM_STABLE_BOUNDARY_BREAKPOINT=0` as a rollback switch for rewritten-tail relocation. The canonical projection is used only for comparison. Replayed content always comes from the exact previously forwarded bytes or the current raw/optimized tail; canonicalized data is never reconstructed into an upstream request. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check` on changed files) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_issue_2671_block_growth_cache.py -q 12 passed in 0.10s $ python -m pytest tests/test_cache -q 253 passed, 3 skipped in 1.94s $ python -m pytest <Anthropic handler/proxy regression set> -q 134 passed, 1 warning in 9.42s $ ruff format --check <changed files> 4 files already formatted $ ruff check <changed files> All checks passed! $ git diff --check # clean ``` The Anthropic regression set covers beta stickiness, CCR injection, compaction transforms, pre-upstream backpressure, streaming reconstruction, upstream headers, model sanitization, diagnostics, and cache stability. ## Real Behavior Proof - Environment: macOS, Python 3.12.13, real FastAPI Anthropic handler with a local upstream stub plus a deterministic provider-cache oracle. - Exact steps: send a cold 35-block aggregate message, then three requests that preserve a 30-block prefix and fixed two-block suffix while regenerating a growing middle tail. Resolve the real session tracker, normalize the real handler body, record the response, and repeat. - Observed result: handler breakpoint indices are `34 -> 29 -> 29`; the cache oracle transitions from a cold 35-block write to establishing the 30-block stable boundary, then produces `(read=30, write=0)` on subsequent rewritten-tail turns. A separate pure-append sequence produces `(0,30) -> (30,4) -> (34,4) -> (38,5)`, proving its breakpoint continues to advance. - Also observed: identical message histories with different tool schemas resolve to distinct trackers in the real handler path. - Not tested: a live Anthropic billing soak, the complete repository test suite, or mypy. #2702 contains earlier live production measurements for the rewritten-tail mechanism; this PR adds the pure-append correction, affinity isolation, and broader regression model. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding documentation updates where applicable (internal behavior is documented in code; no user-facing surface changed) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing relevant unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from the Conventional Commit PR title ## Additional Notes - Consolidates the complementary approaches in #2699 and #2702. Credit to @axisrow and @nangsontay for the traces, root-cause work, and live validation that made the two production shapes distinguishable. - The 20-block minimum for relocation mirrors the provider lookup-window risk boundary and keeps short ordinary messages on the established newest-block behavior. - Disabling stable-boundary relocation does not disable improved lineage resolution or tool-profile isolation; it restores only the previous breakpoint placement. Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> |
||
|
dependabot[bot]
|
4925bf6a82
|
deps: bump hf-hub from 0.4.3 to 0.5.0 (#2285)
Bumps [hf-hub](https://github.com/huggingface/hf-hub) from 0.4.3 to 0.5.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/huggingface/hf-hub/releases">hf-hub's releases</a>.</em></p> <blockquote> <h2>v0.5.0</h2> <h2>What's Changed</h2> <ul> <li>Upgrade ureq by <a href="https://github.com/Narsil"><code>@Narsil</code></a> in <a href="https://redirect.github.com/huggingface/hf-hub/pull/114">huggingface/hf-hub#114</a></li> <li>Update indicatif to current version in Cargo.toml by <a href="https://github.com/gordonmessmer"><code>@gordonmessmer</code></a> in <a href="https://redirect.github.com/huggingface/hf-hub/pull/126">huggingface/hf-hub#126</a></li> <li>Fix failing API tests due to outdated model metadata expectations by <a href="https://github.com/bmqube"><code>@bmqube</code></a> in <a href="https://redirect.github.com/huggingface/hf-hub/pull/124">huggingface/hf-hub#124</a></li> <li>fix: fix typo by <a href="https://github.com/AndyDai-nv"><code>@AndyDai-nv</code></a> in <a href="https://redirect.github.com/huggingface/hf-hub/pull/121">huggingface/hf-hub#121</a></li> <li>Updating tests and dependencies. by <a href="https://github.com/Narsil"><code>@Narsil</code></a> in <a href="https://redirect.github.com/huggingface/hf-hub/pull/127">huggingface/hf-hub#127</a></li> <li>Remove markdown from Cargo.toml by <a href="https://github.com/gordonmessmer"><code>@gordonmessmer</code></a> in <a href="https://redirect.github.com/huggingface/hf-hub/pull/128">huggingface/hf-hub#128</a></li> <li>Fixup the docstrings for download function (which always downloads). by <a href="https://github.com/Narsil"><code>@Narsil</code></a> in <a href="https://redirect.github.com/huggingface/hf-hub/pull/130">huggingface/hf-hub#130</a></li> <li>Expose <code>metadata</code> and <code>pointer_path</code> methods by <a href="https://github.com/danieldk"><code>@danieldk</code></a> in <a href="https://redirect.github.com/huggingface/hf-hub/pull/136">huggingface/hf-hub#136</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/gordonmessmer"><code>@gordonmessmer</code></a> made their first contribution in <a href="https://redirect.github.com/huggingface/hf-hub/pull/126">huggingface/hf-hub#126</a></li> <li><a href="https://github.com/bmqube"><code>@bmqube</code></a> made their first contribution in <a href="https://redirect.github.com/huggingface/hf-hub/pull/124">huggingface/hf-hub#124</a></li> <li><a href="https://github.com/AndyDai-nv"><code>@AndyDai-nv</code></a> made their first contribution in <a href="https://redirect.github.com/huggingface/hf-hub/pull/121">huggingface/hf-hub#121</a></li> <li><a href="https://github.com/danieldk"><code>@danieldk</code></a> made their first contribution in <a href="https://redirect.github.com/huggingface/hf-hub/pull/136">huggingface/hf-hub#136</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/huggingface/hf-hub/compare/v0.4.3...v0.5.0">https://github.com/huggingface/hf-hub/compare/v0.4.3...v0.5.0</a></p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/huggingface/hf-hub/blob/main/RELEASE.md">hf-hub's changelog</a>.</em></p> <blockquote> <h1>Releasing hf-hub</h1> <p>This document covers the full release process for the <code>hf-hub</code> crate. If anything here is unclear or out of date, please open a PR.</p> <h2>What gets released</h2> <p>A single tag push releases one artifact:</p> <ul> <li><strong><code>hf-hub</code> Rust crate</strong> on <a href="https://crates.io/crates/hf-hub">crates.io</a>, via <code>.github/workflows/rust-release.yml</code>.</li> </ul> <p>The workflow triggers on tags matching <code>v*</code> (e.g., <code>v1.0.0</code>, <code>v1.0.0-rc.0</code>).</p> <p>There are no Python components in this repo. The other workspace members are not published:</p> <ul> <li><code>hfrs/</code> — CLI binary, distributed via <code>cargo install --git</code>.</li> <li><code>examples/</code>, <code>benches/</code>, <code>integration-tests/</code> — internal-only, version <code>0.0.0</code>, never published.</li> </ul> <h2>Pre-release checklist</h2> <ol> <li><strong>CI is green on <code>main</code>.</strong> The <code>Rust</code> workflow must be passing on every platform in the matrix (Ubuntu, Windows, macOS) with both feature configurations (<code>""</code> and <code>--all-features</code>).</li> <li><strong>Review the diff since the last release.</strong> <pre lang="bash"><code>git log --oneline v0.5.0..main git diff v0.5.0..main --stat -- hf-hub/ </code></pre> Pay particular attention to changes under <code>hf-hub/src/</code> — those are the only changes that actually ship to crates.io.</li> <li><strong>Identify breaking changes.</strong> Anything that changes the public Rust API (types, function signatures, removed re-exports, builder fields) needs to be reflected in the version bump per <a href="https://semver.org">semver</a> and called out in the release notes.</li> <li><strong>Run the full pre-release test sweep</strong> (see next section).</li> </ol> <h2>Pre-release test sweep</h2> <p>Run all of these from the repo root before tagging. They mirror what CI runs, plus a publish dry-run that CI does not currently do.</p> <h3>Format and lint</h3> <pre lang="bash"><code>cargo +nightly fmt --all -- --check cargo clippy --workspace --all-targets -- -D warnings cargo clippy --workspace --all-targets --all-features -- -D warnings </code></pre> <h3>Unit tests (<code>hf-hub</code>)</h3> <pre lang="bash"><code>cargo test -p hf-hub cargo test -p hf-hub --features blocking </code></pre> <h3>Integration tests (<code>integration-tests</code>)</h3> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
6448545a7f
|
deps: bump bytesize from 1.3.3 to 2.4.2 (#2286)
Bumps [bytesize](https://github.com/bytesize-rs/bytesize) from 1.3.3 to 2.4.2. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/bytesize-rs/bytesize/releases">bytesize's releases</a>.</em></p> <blockquote> <h2>bytesize: v2.4.2</h2> <ul> <li>Improve accuracy of parsing large non-decimal byte count strings.</li> </ul> <h2>bytesize: v2.4.1</h2> <ul> <li>Fix rounding error near power-of-unit boundaries.</li> </ul> <h2>bytesize: v2.4.0</h2> <ul> <li>Implement <code>Sum</code> for <code>ByteSize</code>.</li> <li>Minimum supported Rust version (MSRV) is now 1.85.</li> </ul> <h2>bytesize: v2.3.1</h2> <ul> <li>Fix unit truncation in error strings.</li> </ul> <h2>bytesize: v2.3.0</h2> <ul> <li>Add <code>Unit</code> enum.</li> <li>Add <code>UnitParseError</code> type.</li> </ul> <h2>bytesize: v2.2.0</h2> <ul> <li>Add <code>ByteSize::as_*()</code> methods to return equivalent sizes in KB, GiB, etc.</li> </ul> <h2>bytesize: v2.1.0</h2> <ul> <li>Support parsing and formatting exabytes (EB) & exbibytes (EiB).</li> <li>Migrate <code>serde</code> dependency to <code>serde_core</code>.</li> </ul> <h2>bytesize: v2.0.1</h2> <ul> <li>Add support for precision in <code>Display</code> implementations.</li> </ul> <h2>bytesize: v2.0.0</h2> <ul> <li>Add support for <code>no_std</code> targets.</li> <li>Use IEC (binary) format by default with <code>Display</code>.</li> <li>Use "kB" for SI unit.</li> <li>Add <code>Display</code> type for customizing printed format.</li> <li>Add <code>ByteSize::display()</code> method.</li> <li>Implement <code>Sub<ByteSize></code> for <code>ByteSize</code>.</li> <li>Implement <code>Sub<impl Into<u64>></code> for <code>ByteSize</code>.</li> <li>Implement <code>SubAssign<ByteSize></code> for <code>ByteSize</code>.</li> <li>Implement <code>SubAssign<impl Into<u64>></code> for <code>ByteSize</code>.</li> <li>Reject parsing non-unit characters after whitespace.</li> <li>Remove <code>ByteSize::to_string_as()</code> method.</li> <li>Remove top-level <code>to_string()</code> method.</li> <li>Remove top-level <code>B</code> constant.</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/bytesize-rs/bytesize/blob/master/CHANGELOG.md">bytesize's changelog</a>.</em></p> <blockquote> <h2>2.4.2</h2> <ul> <li>Improve accuracy of parsing large non-decimal byte count strings.</li> </ul> <h2>2.4.1</h2> <ul> <li>Fix rounding error near power-of-unit boundaries.</li> </ul> <h2>2.4.0</h2> <ul> <li>Implement <code>Sum</code> for <code>ByteSize</code>.</li> <li>Minimum supported Rust version (MSRV) is now 1.85.</li> </ul> <h2>2.3.1</h2> <ul> <li>Fix unit truncation in error strings.</li> </ul> <h2>2.3.0</h2> <ul> <li>Add <code>Unit</code> enum.</li> <li>Add <code>UnitParseError</code> type.</li> </ul> <h2>2.2.0</h2> <ul> <li>Add <code>ByteSize::as_*()</code> methods to return equivalent sizes in KB, GiB, etc.</li> </ul> <h2>2.1.0</h2> <ul> <li>Support parsing and formatting exabytes (EB) & exbibytes (EiB).</li> <li>Migrate <code>serde</code> dependency to <code>serde_core</code>.</li> </ul> <h2>2.0.1</h2> <ul> <li>Add support for precision in <code>Display</code> implementations.</li> </ul> <h2>v2.0.0</h2> <ul> <li>Add support for <code>no_std</code> targets.</li> <li>Use IEC (binary) format by default with <code>Display</code>.</li> <li>Use "kB" for SI unit.</li> <li>Add <code>Display</code> type for customizing printed format.</li> <li>Add <code>ByteSize::display()</code> method.</li> <li>Implement <code>Sub<ByteSize></code> for <code>ByteSize</code>.</li> <li>Implement <code>Sub<impl Into<u64>></code> for <code>ByteSize</code>.</li> <li>Implement <code>SubAssign<ByteSize></code> for <code>ByteSize</code>.</li> <li>Implement <code>SubAssign<impl Into<u64>></code> for <code>ByteSize</code>.</li> <li>Reject parsing non-unit characters after whitespace.</li> <li>Remove <code>ByteSize::to_string_as()</code> method.</li> <li>Remove top-level <code>to_string()</code> method.</li> <li>Remove top-level <code>B</code> constant.</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
522faa1a59
|
deps: bump rusqlite from 0.32.1 to 0.40.1 (#2287)
Bumps [rusqlite](https://github.com/rusqlite/rusqlite) from 0.32.1 to 0.40.1. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/rusqlite/rusqlite/releases">rusqlite's releases</a>.</em></p> <blockquote> <h2>0.40.1</h2> <h2>What's Changed</h2> <ul> <li>Fix clippy warnings <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1852">#1852</a></li> <li>Bump bundled SQLite version to 3.53.2 <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1853">#1853</a></li> <li>Bump hashlink version <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1855">#1855</a></li> <li>Fix SQL injection when SAVEPOINT name is tainted <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1854">#1854</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/rusqlite/rusqlite/compare/v0.40.0...v0.40.1">https://github.com/rusqlite/rusqlite/compare/v0.40.0...v0.40.1</a></p> <h2>0.40.0</h2> <h2>What's Changed</h2> <ul> <li>Breaking changes: Replace VTab macros by constructors <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1823">#1823</a></li> <li>Breaking changes: Fix VTab::best_index <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1824">#1824</a></li> <li>Asserts on VTab::connect aux and args <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1825">#1825</a></li> <li>Breaking changes: Fix VTab::connect / create <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1826">#1826</a></li> <li>Breaking changes: Allow opting out of using sqlite-wasm-rs on wasm32-unknown-unknown <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1828">#1828</a>, <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1829">#1829</a></li> <li>Derive Default for SeriesTabCursor/ArrayTabCursor <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1830">#1830</a></li> <li>Update link to pre-update hook <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1831">#1831</a></li> <li>Breaking changes: Fix VTab::connect <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1832">#1832</a></li> <li>impl From<!-- raw HTML omitted --> for FromSqlError <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1833">#1833</a></li> <li>Breaking changes: Fix vtab::dequote <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1835">#1835</a></li> <li>Bump bundled SQLCipher to version 4.14.0 <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1837">#1837</a></li> <li>sqlite3_set_errmsg <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1752">#1752</a></li> <li>Bump sqlite3-parser version <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1838">#1838</a></li> <li>Fix UB in ToSqlOutput::from_rc <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1839">#1839</a></li> <li>Ensure miri doesn't complain <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1840">#1840</a></li> <li>Bump to actions/checkout@v6 <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1842">#1842</a></li> <li>Add support to UtcDateTime <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1843">#1843</a>, <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1844">#1844</a></li> <li>Bump bundled SQLite version to 3.53.1 <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1848">#1848</a></li> <li>Replace some cfg(not by cfg_select <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1850">#1850</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/rusqlite/rusqlite/compare/v0.39.0...v0.40.0">https://github.com/rusqlite/rusqlite/compare/v0.39.0...v0.40.0</a></p> <h2>0.39.0</h2> <h2>What's Changed</h2> <ul> <li>Fix constraints on VTab Aux data <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1778">#1778</a>, <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1771">#1771</a></li> <li>Fix docs.rs generation <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1779">#1779</a></li> <li>Fix a small typo in <code>rollback_hook</code> docstring <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1780">#1780</a></li> <li>Fix some warnings from Intellij <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1781">#1781</a></li> <li>Minimal doc for features <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1783">#1783</a></li> <li>Clear hooks only for owning connections <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1785">#1785</a>, <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1784">#1784</a></li> <li>Fix link to SQLite C Interface, Prepare Flags <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1787">#1787</a></li> <li>Comment functions which are not usable from a loadable extension <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1789">#1789</a></li> <li>Factorize code <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1792">#1792</a></li> <li>Update getrandom to 0.4 <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1798">#1798</a></li> <li>Update Cargo.toml <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1800">#1800</a></li> <li>Fix appveyor <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1807">#1807</a></li> <li>Add support to unix timestamp for chrono, jiff and time <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1808">#1808</a>, <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1803">#1803</a></li> <li>fix(trace): check that the sql string pointer is not NULL <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1805">#1805</a></li> <li>Bump bundled SQLite version to 3.51.3 <a href="https://redirect.github.com/rusqlite/rusqlite/issues/1818">#1818</a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
ecf130d3ac
|
deps: bump ruff from 0.15.17 to 0.15.22 in the pip-minor-patch group (#2501)
Bumps the pip-minor-patch group with 1 update: [ruff](https://github.com/astral-sh/ruff). Updates `ruff` from 0.15.17 to 0.15.22 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/astral-sh/ruff/releases">ruff's releases</a>.</em></p> <blockquote> <h2>0.15.22</h2> <h2>Release Notes</h2> <p>Released on 2026-07-16.</p> <h3>Preview features</h3> <ul> <li>[<code>pycodestyle</code>] Add an autofix for <code>E402</code> (<a href="https://redirect.github.com/astral-sh/ruff/pull/22212">#22212</a>)</li> <li>[<code>refurb</code>] Allow subclassing builtins in stub files (<code>FURB189</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/26812">#26812</a>)</li> <li>[<code>ruff</code>] Add rule to replace <code>noqa</code> comments with <code>ruff:ignore</code> (<code>RUF105</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/26423">#26423</a>)</li> <li>[<code>ruff</code>] Add rule to use human-readable names in <code>ruff:ignore</code> comments (<code>RUF106</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/26682">#26682</a>)</li> <li>[<code>ruff</code>] Add rule to use human-readable names in configuration selectors (<code>RUF201</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/26772">#26772</a>)</li> </ul> <h3>Bug fixes</h3> <ul> <li>[<code>flake8-pyi</code>] Fix false positive in <code>__all__</code> (<code>PYI053</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/26872">#26872</a>)</li> </ul> <h3>Rule changes</h3> <ul> <li>[<code>pylint</code>] Ignore mutable type updates in <code>redefined-loop-name</code> (<code>PLW2901</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/25733">#25733</a>)</li> </ul> <h3>Performance</h3> <ul> <li>Avoid redundant lexer token bookkeeping (<a href="https://redirect.github.com/astral-sh/ruff/pull/26765">#26765</a>)</li> <li>Avoid redundant pending-indentation writes (<a href="https://redirect.github.com/astral-sh/ruff/pull/26774">#26774</a>)</li> <li>Avoid unnecessary identifier lookahead (<a href="https://redirect.github.com/astral-sh/ruff/pull/26525">#26525</a>)</li> <li>Reuse parser scratch buffers (<a href="https://redirect.github.com/astral-sh/ruff/pull/26798">#26798</a>)</li> </ul> <h3>Documentation</h3> <ul> <li>Document argfile support (<a href="https://redirect.github.com/astral-sh/ruff/pull/26803">#26803</a>)</li> <li>[<code>flake8-datetimez</code>] Clarify naming guidance for <code>datetime.today</code> (<code>DTZ002</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/26658">#26658</a>)</li> <li>[<code>pycodestyle</code>] Document <code>E731</code> fix safety (<a href="https://redirect.github.com/astral-sh/ruff/pull/26847">#26847</a>)</li> <li>[<code>ruff</code>] Clarify intentional async contexts for <code>unused-async</code> (<code>RUF029</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/26641">#26641</a>)</li> </ul> <h3>Contributors</h3> <ul> <li><a href="https://github.com/dwego"><code>@dwego</code></a></li> <li><a href="https://github.com/MichaReiser"><code>@MichaReiser</code></a></li> <li><a href="https://github.com/Joosboy"><code>@Joosboy</code></a></li> <li><a href="https://github.com/KaufmanDmitriy"><code>@KaufmanDmitriy</code></a></li> <li><a href="https://github.com/PeterJCLaw"><code>@PeterJCLaw</code></a></li> <li><a href="https://github.com/ntBre"><code>@ntBre</code></a></li> <li><a href="https://github.com/charliermarsh"><code>@charliermarsh</code></a></li> </ul> <h2>Install ruff 0.15.22</h2> <h3>Install prebuilt binaries via shell script</h3> <pre lang="sh"><code></tr></table> </code></pre> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md">ruff's changelog</a>.</em></p> <blockquote> <h2>0.15.22</h2> <p>Released on 2026-07-16.</p> <h3>Preview features</h3> <ul> <li>[<code>pycodestyle</code>] Add an autofix for <code>E402</code> (<a href="https://redirect.github.com/astral-sh/ruff/pull/22212">#22212</a>)</li> <li>[<code>refurb</code>] Allow subclassing builtins in stub files (<code>FURB189</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/26812">#26812</a>)</li> <li>[<code>ruff</code>] Add rule to replace <code>noqa</code> comments with <code>ruff:ignore</code> (<code>RUF105</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/26423">#26423</a>)</li> <li>[<code>ruff</code>] Add rule to use human-readable names in <code>ruff:ignore</code> comments (<code>RUF106</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/26682">#26682</a>)</li> <li>[<code>ruff</code>] Add rule to use human-readable names in configuration selectors (<code>RUF201</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/26772">#26772</a>)</li> </ul> <h3>Bug fixes</h3> <ul> <li>[<code>flake8-pyi</code>] Fix false positive in <code>__all__</code> (<code>PYI053</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/26872">#26872</a>)</li> </ul> <h3>Rule changes</h3> <ul> <li>[<code>pylint</code>] Ignore mutable type updates in <code>redefined-loop-name</code> (<code>PLW2901</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/25733">#25733</a>)</li> </ul> <h3>Performance</h3> <ul> <li>Avoid redundant lexer token bookkeeping (<a href="https://redirect.github.com/astral-sh/ruff/pull/26765">#26765</a>)</li> <li>Avoid redundant pending-indentation writes (<a href="https://redirect.github.com/astral-sh/ruff/pull/26774">#26774</a>)</li> <li>Avoid unnecessary identifier lookahead (<a href="https://redirect.github.com/astral-sh/ruff/pull/26525">#26525</a>)</li> <li>Reuse parser scratch buffers (<a href="https://redirect.github.com/astral-sh/ruff/pull/26798">#26798</a>)</li> </ul> <h3>Documentation</h3> <ul> <li>Document argfile support (<a href="https://redirect.github.com/astral-sh/ruff/pull/26803">#26803</a>)</li> <li>[<code>flake8-datetimez</code>] Clarify naming guidance for <code>datetime.today</code> (<code>DTZ002</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/26658">#26658</a>)</li> <li>[<code>pycodestyle</code>] Document <code>E731</code> fix safety (<a href="https://redirect.github.com/astral-sh/ruff/pull/26847">#26847</a>)</li> <li>[<code>ruff</code>] Clarify intentional async contexts for <code>unused-async</code> (<code>RUF029</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/26641">#26641</a>)</li> </ul> <h3>Contributors</h3> <ul> <li><a href="https://github.com/dwego"><code>@dwego</code></a></li> <li><a href="https://github.com/MichaReiser"><code>@MichaReiser</code></a></li> <li><a href="https://github.com/Joosboy"><code>@Joosboy</code></a></li> <li><a href="https://github.com/KaufmanDmitriy"><code>@KaufmanDmitriy</code></a></li> <li><a href="https://github.com/PeterJCLaw"><code>@PeterJCLaw</code></a></li> <li><a href="https://github.com/ntBre"><code>@ntBre</code></a></li> <li><a href="https://github.com/charliermarsh"><code>@charliermarsh</code></a></li> </ul> <h2>0.15.21</h2> <p>Released on 2026-07-09.</p> <h3>Preview features</h3> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
74403fe804
|
build(deps): bump gitpython from 3.1.50 to 3.1.54 in the uv group across 1 directory (#2575)
Bumps the uv group with 1 update in the / directory: [gitpython](https://github.com/gitpython-developers/GitPython). Updates `gitpython` from 3.1.50 to 3.1.54 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/gitpython-developers/GitPython/releases">gitpython's releases</a>.</em></p> <blockquote> <h2>3.1.54 - Security</h2> <h2>What's Changed</h2> <ul> <li>Harden unsafe Git option validation by <a href="https://github.com/Byron"><code>@Byron</code></a> in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2180">gitpython-developers/GitPython#2180</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/gitpython-developers/GitPython/compare/3.1.53...3.1.54">https://github.com/gitpython-developers/GitPython/compare/3.1.53...3.1.54</a></p> <h2>3.1.53 - Security</h2> <h2>What's Changed</h2> <ul> <li>feat(submodule): add deinit method to Submodule (<a href="https://redirect.github.com/gitpython-developers/GitPython/issues/2014">#2014</a>) by <a href="https://github.com/mvanhorn"><code>@mvanhorn</code></a> in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2129">gitpython-developers/GitPython#2129</a></li> <li>typing: introduce sensible basedpyright defaults by <a href="https://github.com/Byron"><code>@Byron</code></a> in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2174">gitpython-developers/GitPython#2174</a></li> <li>fix: make <code>submodule.update()</code> after <code>submodule.deinit()</code> work by <a href="https://github.com/Byron"><code>@Byron</code></a> in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2175">gitpython-developers/GitPython#2175</a></li> <li>Fix commit hooks respecting core.hooksPath by <a href="https://github.com/Siesta0217"><code>@Siesta0217</code></a> in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2159">gitpython-developers/GitPython#2159</a></li> <li>fix: validate config section delimiters by <a href="https://github.com/Byron"><code>@Byron</code></a> in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2176">gitpython-developers/GitPython#2176</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/Siesta0217"><code>@Siesta0217</code></a> made their first contribution in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2159">gitpython-developers/GitPython#2159</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/gitpython-developers/GitPython/compare/3.1.52...3.1.53">https://github.com/gitpython-developers/GitPython/compare/3.1.52...3.1.53</a></p> <h2>3.1.52 Security</h2> <p><a href="https://github.com/gitpython-developers/GitPython/security/advisories/GHSA-rwj8-pgh3-r573">https://github.com/gitpython-developers/GitPython/security/advisories/GHSA-rwj8-pgh3-r573</a>: Environment-variable exfiltration via os.path.expandvars() on Repo.clone_from() URL</p> <h2>What's Changed</h2> <ul> <li>Skip cross-drive relative config test on Windows by <a href="https://github.com/Byron"><code>@Byron</code></a> in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2171">gitpython-developers/GitPython#2171</a></li> <li>fix: preserve literal clone URLs by <a href="https://github.com/Byron"><code>@Byron</code></a> in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2172">gitpython-developers/GitPython#2172</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/gitpython-developers/GitPython/compare/3.1.51...3.1.52">https://github.com/gitpython-developers/GitPython/compare/3.1.51...3.1.52</a></p> <h2>3.1.51 - Security</h2> <h2>What's Changed</h2> <ul> <li>Add AI-disclosure and quality requirements to the contribution guidelines by <a href="https://github.com/Byron"><code>@Byron</code></a> in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2143">gitpython-developers/GitPython#2143</a></li> <li>docs(cmd): clarify Git.execute() string vs list command argument by <a href="https://github.com/mvanhorn"><code>@mvanhorn</code></a> in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2144">gitpython-developers/GitPython#2144</a></li> <li>Rewrite Git.execute() command parameter docstring per <a href="https://redirect.github.com/gitpython-developers/GitPython/issues/2146">#2146</a> by <a href="https://github.com/EliahKagan"><code>@EliahKagan</code></a> in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2147">gitpython-developers/GitPython#2147</a></li> <li>Document init script behavior with multiple master remotes by <a href="https://github.com/EliahKagan"><code>@EliahKagan</code></a> in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2148">gitpython-developers/GitPython#2148</a></li> <li>Bump git/ext/gitdb from <code>335c0f6</code> to <code>0a019a2</code> by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2149">gitpython-developers/GitPython#2149</a></li> <li>Support relative worktree paths (git 2.48+ worktree.useRelativePaths) by <a href="https://github.com/elovelan"><code>@elovelan</code></a> in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2151">gitpython-developers/GitPython#2151</a></li> <li>Defer xfail condition evaluation with xfail_if_raises context manager by <a href="https://github.com/elovelan"><code>@elovelan</code></a> in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2153">gitpython-developers/GitPython#2153</a></li> <li>Run more submodule tests on Cygwin (fix flaky xfails) by <a href="https://github.com/EliahKagan"><code>@EliahKagan</code></a> in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2154">gitpython-developers/GitPython#2154</a></li> <li>Cut xtrace noise from POSIX-ownership diagnostic steps by <a href="https://github.com/EliahKagan"><code>@EliahKagan</code></a> in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2156">gitpython-developers/GitPython#2156</a></li> <li>Support index diffs against the empty tree by <a href="https://github.com/puneetdixit200"><code>@puneetdixit200</code></a> in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2155">gitpython-developers/GitPython#2155</a></li> <li>refactor: seperate out Progress type by <a href="https://github.com/LoeschMaximilian"><code>@LoeschMaximilian</code></a> in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2157">gitpython-developers/GitPython#2157</a></li> <li>Bump <a href="https://github.com/astral-sh/ruff-pre-commit">https://github.com/astral-sh/ruff-pre-commit</a> from v0.15.12 to 0.15.15 in the pre-commit group by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2160">gitpython-developers/GitPython#2160</a></li> <li>Bump actions/checkout from 6 to 7 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2164">gitpython-developers/GitPython#2164</a></li> <li>Bump git/ext/gitdb from <code>0a019a2</code> to <code>4950ea9</code> by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2165">gitpython-developers/GitPython#2165</a></li> <li>Bump <a href="https://github.com/astral-sh/ruff-pre-commit">https://github.com/astral-sh/ruff-pre-commit</a> from v0.15.15 to 0.15.20 in the pre-commit group by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2166">gitpython-developers/GitPython#2166</a></li> <li>Add Commit.is_shallow property; document stats() limitation at shallow boundary by <a href="https://github.com/harshitayadavv"><code>@harshitayadavv</code></a> in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2167">gitpython-developers/GitPython#2167</a></li> <li>Allow relative config paths with includes by <a href="https://github.com/Byron"><code>@Byron</code></a> in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2169">gitpython-developers/GitPython#2169</a></li> <li>Reject abbreviated forms of unsafe git options by <a href="https://github.com/Byron"><code>@Byron</code></a> in <a href="https://redirect.github.com/gitpython-developers/GitPython/pull/2168">gitpython-developers/GitPython#2168</a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
e6e5826423
|
deps: bump postcss from 8.5.19 to 8.5.26 in /docs (#2881)
Bumps [postcss](https://github.com/postcss/postcss) from 8.5.19 to 8.5.26. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/postcss/postcss/releases">postcss's releases</a>.</em></p> <blockquote> <h2>8.5.26</h2> <ul> <li>Fixed <code>list.split()</code> regression (by <a href="https://github.com/lazerg"><code>@lazerg</code></a>).</li> <li>Track symlinks in path protection in source map loading (by <a href="https://github.com/drengir1"><code>@drengir1</code></a>).</li> </ul> <h2>8.5.25</h2> <ul> <li>Fixed 8.5.17 visitor regression.</li> <li>Fixed <code>list.split()</code> for non-string values (by <a href="https://github.com/amir-rezaei"><code>@amir-rezaei</code></a>).</li> </ul> <h2>8.5.24</h2> <ul> <li>Preserve the BOM after the processing (by <a href="https://github.com/hdimer"><code>@hdimer</code></a>).</li> </ul> <h2>8.5.23</h2> <ul> <li>Do not load source map without <code>opts.from</code> for security reasons.</li> </ul> <h2>8.5.22</h2> <ul> <li>Fixed custom property losing semicolon before a comment (by <a href="https://github.com/sarathfrancis90"><code>@sarathfrancis90</code></a>).</li> </ul> <h2>8.5.21</h2> <ul> <li>Fixed childless at-rule losing semicolon before comment (by <a href="https://github.com/sarathfrancis90"><code>@sarathfrancis90</code></a>).</li> <li>Fixed docs (by <a href="https://github.com/isker"><code>@isker</code></a>).</li> </ul> <h2>8.5.20</h2> <ul> <li>Fixed missing space if <code>AtRule#params</code> is set after (by <a href="https://github.com/sarathfrancis90"><code>@sarathfrancis90</code></a>).</li> <li>Fixed mixing AST error on warnings (by <a href="https://github.com/MahinAnowar"><code>@MahinAnowar</code></a>).</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/postcss/postcss/blob/main/CHANGELOG.md">postcss's changelog</a>.</em></p> <blockquote> <h2>8.5.26</h2> <ul> <li>Fixed <code>list.split()</code> regression (by <a href="https://github.com/lazerg"><code>@lazerg</code></a>).</li> <li>Track symlinks in path protection in source map loading (by <a href="https://github.com/drengir1"><code>@drengir1</code></a>).</li> </ul> <h2>8.5.25</h2> <ul> <li>Fixed 8.5.17 visitor regression.</li> <li>Fixed <code>list.split()</code> for non-string values (by <a href="https://github.com/amir-rezaei"><code>@amir-rezaei</code></a>).</li> </ul> <h2>8.5.24</h2> <ul> <li>Preserve the BOM after the processing (by <a href="https://github.com/hdimer"><code>@hdimer</code></a>).</li> </ul> <h2>8.5.23</h2> <ul> <li>Do not load source map without <code>opts.from</code> for security reasons.</li> </ul> <h2>8.5.22</h2> <ul> <li>Fixed custom property losing semicolon before a comment (by <a href="https://github.com/sarathfrancis90"><code>@sarathfrancis90</code></a>).</li> </ul> <h2>8.5.21</h2> <ul> <li>Fixed childless at-rule losing semicolon before comment (by <a href="https://github.com/sarathfrancis90"><code>@sarathfrancis90</code></a>).</li> <li>Fixed docs (by <a href="https://github.com/isker"><code>@isker</code></a>).</li> </ul> <h2>8.5.20</h2> <ul> <li>Fixed missing space if <code>AtRule#params</code> is set after (by <a href="https://github.com/sarathfrancis90"><code>@sarathfrancis90</code></a>).</li> <li>Fixed mixing AST error on warnings (by <a href="https://github.com/MahinAnowar"><code>@MahinAnowar</code></a>).</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
Tejas Chopra
|
2f2950a626
|
fix(litellm): don't forward a caller key the target cannot accept (#2883)
Split out of #2852 at review request: that PR is bounded upstream calls plus measured hot-path costs, and this is an authentication/routing change that belongs on its own scope. #2852 now carries only the timeout work. ## The bug A routing extension can rewrite the model across families mid-request (`claude-opus-5` → `gpt-5-mini`). The caller's key does not travel with that rewrite, so the proxy forwards `sk-ant-...` to OpenAI and earns a guaranteed 401. Downstream that is indistinguishable from *"the cheap model failed the task"* — it scores as a quality regression against the router, not as a bug. Dropping the `api_key` kwarg instead lets litellm fall back to the target provider's own env credential, which is the only key that can work. ## Why this cut is different from the one that was rejected The first version returned `not provider.startswith("anthropic")`, so **any** non-`sk-ant-` credential was dropped against an Anthropic-class target — a plain Bearer token against an Anthropic-compatible or custom gateway lost its key and fell back to an env credential that may not exist. That direction is the dangerous one. A false refusal breaks a deployment that was working; a missed refusal just leaves today's 401. So this refuses on **positive evidence only**: | credential | target | forwarded? | |---|---|---| | `sk-ant-…` | `openai` / `azure` / `gemini` | **no** — cannot possibly authenticate | | `sk-ant-…` | anthropic | yes | | `sk-ant-…` | unrecognised / unclassifiable model | yes — pass-through | | anything else | anything | yes — pass-through, unchanged | `sk-ant-` is Anthropic's documented vendor-specific prefix, which is what makes it classifiable. `sk-` is not: a dozen vendors mint that shape. Everything the string cannot settle keeps main's behaviour. The reject list is explicit rather than inverted (`not anthropic`) because an unrecognised provider is usually a compatible or self-hosted gateway. Marked in the code as a hand-kept tuple with the registry-lookup upgrade path noted. Bedrock / Vertex / SageMaker are unaffected — all four dispatch sites already skip credential forwarding for them entirely (env-based auth). ## Verification `tests/test_litellm_caller_key.py`, 12 cases — the refusal, the Anthropic target, the unknown provider, `get_llm_provider` raising, and each unclassifiable credential shape asserted against **both** target families. Those last ones fail against the rejected version. Applied at all four dispatch sites (Anthropic non-stream/stream, OpenAI non-stream/stream). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Tejas Chopra
|
f624d3a00a
|
perf(proxy): bound upstream calls and hot-path costs (#2852)
Seven commits from one week of load testing: one hang, two request-path correctness fixes, and four hot-path costs that only show up in production. ## Reliability **Bound every upstream call.** The litellm backend had no timeout at all, so a request the upstream never answered blocked its caller forever. Observed under load on 2026-08-07: four agent workers on ESTABLISHED connections for 36+ minutes while `/readyz` answered in 0.11s. No error, no retry, no log line — indistinguishable from slow work, which is the worst shape a failure can take. A float rather than an `httpx.Timeout`, deliberately: litellm expands a float across all four httpx phases, so on a streaming call it becomes the maximum gap *between chunks*, not a cap on total generation. A long answer streaming steadily is never cut off; a stalled one dies. Default 600s via `HEADROOM_UPSTREAM_TIMEOUT`; 0, negative, and junk fall back to the default rather than meaning "no timeout". **Keep the consistency re-count off the event loop.** It ran `tokenizer.count_messages` twice directly on the loop. Since Claude counting moved to a real BPE that is CPU-bound work stalling every other in-flight request — ~1s on a 2.3 MB body, with `/healthz` gaps tracking body size. Offloaded via `asyncio.to_thread` on the same tokenizer instance, so reported values are unchanged. (#2810) **Survive a re-parse MemoryError.** `MemoryError` is not a `ValueError`, so on 1M-context payloads the byte-faithful forwarder's verification re-parse escaped the handler and aborted an otherwise-fine request — 14 aborts across 8 days of reporter logs. (#2768) ## Performance All four are measured, not guessed. Each degrades with something a short benchmark does not vary: uptime, content shape, or process age. | fix | before | after | |---|---|---| | Cost-record walk per request (at 100k records) | 13.6 ms | bounded by model count | | JSON-block scan, JS-style object logs (1200 lines) | 4643 ms | 183 ms | | JSON-block scan, truncated JSONL | 3737 ms | 116 ms | | Lazy imports inside user requests | multi-second | paid at startup | | `count_text` (80% of local CPU) | — | memoised | Two worth calling out: - **The cost walk degrades with proxy *uptime*, not load.** A freshly started proxy pays ~0.01 ms; a month-old one pays 4–13 ms on every request, on the event loop, holding the metrics lock. Deliberately not a TTL cache over `stats()`: those values feed `check_budget()` when `--budget` is set, and a stale reading under-enforces the budget. The fix is to stop computing what the caller discards. - **The JSON-block memo is built only *after* a scan fails to balance.** That ordering is load-bearing, not an optimisation — caching from the start made pretty-printed JSON ~2x slower, since content that balances on the first scan has nothing to reuse and just pays the per-line dict traffic. Still a constant-factor fix, not an asymptotic one. ## Tests +1202 lines, 20 files. Each fix is pinned by a test that fails on the unmodified code: the re-count test asserts no `count_messages` pass runs with a live event loop in its thread; the re-parse test drives a `MemoryError` through the real request path and expects a 200; `totals()` equality with `stats()` is asserted across model counts, request volumes, and both pricing branches. The timeout test is structural rather than a mock — the failure mode is a dispatch path someone adds later without a guard, which mocking the existing four cannot catch. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Tejas Chopra
|
e0870ef931
|
feat(beacon): hourly R2 compaction, per-strategy savings, and a stack that reports (#2853)
Three beacon changes bundled because they are one story: the corpus got
too
slow to query, and then too coarse to answer the question it was
collected for.
## 1. Hourly compaction (`deploy/beacon`)
The beacon writes one ~1 KB object per heartbeat — **64,987 on
2026-08-06** and
climbing. A full analysis `pull` was ~100k HTTPS round trips for 95 MB:
minutes
of pure per-object latency. Listing the bucket alone took 88 seconds.
Moving the query server-side does not help — R2 SQL reads only Iceberg
tables,
and a pile of tiny files is the pathological case for every query
engine.
Compaction is the fix, and it is Iceberg's own answer to the same
problem.
An hourly cron collapses each **complete** hour of `sessions/` into one
`rollup/dt=…/hh=…/data.ndjson`, keeping the highest-`seq` heartbeat per
`(install, session)`.
| measured on `dt=2026-08-06/hh=14` | before | after |
|---|---|---|
| objects | 3,938 | **1** |
| rows | 3,938 | **1,061** |
| analysis `pull` | minutes | **seconds** |
Hourly rather than daily because every R2 binding call is a subrequest:
a day
is ~65k, an hour is ~4k. Newest-first, so a backlog drains from the
present
backwards and live data never starves behind it; a failing hour is
logged and
skipped rather than blocking every older hour behind it. **Raw objects
are
never deleted**, so any rollup is rebuildable by deleting it.
Backfill runs to the **oldest surviving raw day**, not a fixed window. A
fixed
lookback strands everything older than it the moment analysis stops
reading
`sessions/`: the raw objects are still there, but nothing would ever
compact
them, so they disappear from every report. `oldestRawDay()` finds that
floor in
one delimited LIST, and the rollup listing starts from it — so the work
is
bounded by retention rather than by total history.
Three failure modes the tests pin down, because each one is silent:
- A **failed `get`** is transient, so the hour throws and writes
nothing. A
rollup is built once and trusted forever, so a short read would quietly
become the permanent record.
- A **corrupt record** loses only itself. This Worker wrote that content
with
`JSON.stringify`; it will never become valid, so blocking on it strands
the
hour instead of the record.
- An **empty hour** writes an `empty` marker. Without one the hour stays
"missing" and is re-listed on every run forever.
`test-rollup.mjs` asserts the Worker's dedup picks exactly the same rows
as the
analysis-side `QUALIFY`. If those two ever disagree the reports go
quietly
wrong rather than loudly broken, which is why that check exists. Its
stub
paginates at 3 keys so the list cursor loop — load-bearing at the real
~4,000
objects/hour — runs in every case.
## 2. Per-strategy savings (`compression.by_strategy`)
`compression.transforms` counts *invocations*, which cannot distinguish
a
compressor that saved 60% from one that ran constantly and saved
nothing. The
fleet's top transform by count contributes an unknown share of
`tokens.saved`.
It is worse than that in practice. Transform labels are slugged with
`split(":", 1)[0]`, so every `router:<strategy>:<detail>` label
collapses into a
single `router` bucket. On 2026-08-08 that bucket held **19.1 M of the
day's
transform counts, across 8,528 of 9,616 sessions** — the compressors
that do
most of the work are indistinguishable from each other, by name as well
as by
yield:
| transform | n | sessions |
|---|---|---|
| `router` | 19,108,118 | 8,528 |
| `anthropic` | 688,124 | 4,734 |
| `output_shaper` | 548,017 | 1,120 |
No question about which strategy is earning its keep can be answered
from that,
which is what this field is for.
The measurement already existed. `PrometheusMetrics.record_compression`
is the
configured `CompressionObserver` and already accumulates
`tokens_saved_by_strategy` on the hot path — the numbers just never left
the
process. This forwards from that one chokepoint rather than adding a
second
observer and a second measurement pass. The paths that have **no**
observer
configured (MCP server, LangGraph, Strands hooks, the transform
pipeline) get
`BeaconCompressionObserver` passed directly.
Compression runs on the executor thread *before* that request's outcome
reaches
`record()`, so events are **staged** into module state and drained by
the next
outcome. Staging is what makes two things true at once:
- The first turn of a session still reports its numbers — otherwise
every
session's opening turn, and any session short enough to be one turn,
would
report nothing.
- A compression event **never opens a session**. An abandoned request
would
otherwise emit a phantom `turns=0` row with all-zero tokens, inflating
fleet
session and install counts.
Staging takes a dedicated mutex the request path never touches, so the
fan-out
stays off the aggregator's lock and `record_compression` keeps its
"synchronous + lock-free" contract.
```json
"by_strategy": [
{"strategy": "code_aware", "n": 1, "tokens_in": 800, "tokens_out": 800},
{"strategy": "smart_crusher", "n": 2, "tokens_in": 1500, "tokens_out": 700}
]
```
A **list of records, sorted by strategy** — not an object keyed by
strategy.
Keyed shapes change type as keys accumulate: DuckDB infers a STRUCT
under ~24
keys and a MAP over it, so the analysis query breaks on the day the
fleet
picks up a 25th strategy. Sorted so heartbeats are byte-comparable.
**These do not sum to `tokens.saved`,** and the field comment says so:
strategies compose (the router routes, a strategy runs inside it) so the
same
text is measured more than once. A row means "of what this strategy was
handed,
it removed this much" — a per-strategy yield, not a share of the total.
A
strategy that saved nothing still appears; dropping it would make every
strategy look effective.
## 3. `headroom.stack`
`resource_attributes()` was called with no arguments at its one call
site, so
`headroom.stack` was absent from **all 24,040 sessions** in the corpus
while
`detect_stack` sat unused — dead code on both ends of a wire nobody
connected.
The fleet was unsegmentable by agent, which is the question the corpus
is asked
most often.
Environment detection alone is not enough. It answers `wrap_claude` only
under
`headroom wrap`; every install that points an agent at a persistent
proxy — the
common deployment — reports `proxy`, which segments nothing. The
per-request
`X-Headroom-Stack` slugs are the only signal that names the harness
there, so
`record_stack()` stages them the same way and feeds `detect_stack`'s
`by_stack`
branch:
```
9x wrap_claude, 1x wrap_cursor -> wrap_claude (dominant harness wins)
5x wrap_claude, 5x wrap_cursor -> mixed
no per-request signal -> proxy (environment fallback)
junk slug -> dropped before staging
```
## Privacy
The strategy string is slugged through the same `_safe_slug` as skip
reasons
and capped at `MAX_STRATEGIES`, because the observer protocol takes a
free
string and an extension could otherwise invent keys per request. Stack
slugs
are normalized and capped the same way.
**Deliberately not collected:** the tool names in
`smart_crush:<count>:<names>`.
Those are user-defined MCP identifiers and can name internal tooling
(`acme_deploy_prod`). They stay stripped by the existing `split(":",
1)[0]`,
and this PR does not widen it. No new key was needed in `worker.js` —
`by_strategy` nests under the already-allowlisted `compression`, and
`headroom.stack` was already in `ALLOWED_RESOURCE`.
Nothing here deletes or rewrites existing data: the Worker only ever
writes,
`sessions/` is never pruned by it, and readers merge old and new shapes
with
`union_by_name`, so pre-change heartbeats keep reading with the new
fields null.
## Verification
- `python -m headroom.telemetry.session` — self-check covers staging,
the
no-phantom-session case, drain exhaustiveness, a 0%-yield strategy
staying
visible, the cardinality cap, slug safety, and dominant/mixed/junk stack
resolution
- `node test-rollup.mjs /tmp/hr` — 3,938 real corpus objects → 1,061
sessions
in 1 object, plus the pagination, partial-read, corrupt-record,
empty-hour
and `oldestRawDay` cases
- 51 passing in `test_compression_observability`,
`test_prometheus_obs_counters`,
`test_telemetry_context`, `test_compression_strategy_outcomes`
- Consumer side exists and is checked: `beacon.sh by_strategy` in
headroom-beacon-stats reads the field end to end (sessions, installs,
invocations, tokens in/out, yield %), verified against a synthetic
parquet for
the cases that matter — aggregation across installs, a 0%-yield strategy
staying visible, pre-field sessions dropping out rather than erroring.
It is
guarded on the column, so it prints an instruction instead of a binder
error
until a release carrying this PR reaches the fleet.
- Deployed and running against the live corpus on the `5 * * * *`
trigger
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Tejas Chopra
|
675d13f08d
|
fix(proxy/openai): run response hooks on Responses, and bill their re-drives (#2872)
The Responses path runs `run_request_hooks` but never `run_response_hooks` — only `handle_openai_chat` does. So a turn hook can shrink a Responses turn and then never be asked to resolve what the model did about it: the model's injected tool call goes straight to a client that has no such tool. That asymmetry is why tool-belt deferral has to be disabled wholesale on the Responses API, which is the surface Codex uses. ## 1. Wire the response side Mirrors the chat-completions block. **Buffered path only**, for the same reason CCR already forces `stream:false` when it needs to intercept: you cannot re-drive a turn whose bytes are already flowing. ## 2. Honour `stream_safe_only` on the Responses request path It was the one hook call site that ignored the flag. A re-driving hook would run its shrink on a streamed turn and then have no response side to finish it — latent until (1) lands, live afterwards. `stream` is not a parameter of `_compress_openai_responses_payload`, but the payload it is compressing carries the flag. It is read **before** CCR may force `stream:false` further down, so this is the client's request rather than the effective one — conservative in the safe direction: at worst a CCR-buffered turn misses a saving, never a stranded tool call. Fold-only hooks that declare `stream_safe = True` are unaffected. ## 3. Bill what the re-drives cost Both handlers read usage from the **final** upstream response, so every intermediate call a hook made was free as far as Headroom was concerned. For a token-saving feature that is not a rounding error. A tool-search reload is a whole extra model call; counting only the last one lets the feature hide its own overhead behind the saving it is claiming, and the numbers come out better than the truth. `TurnHookUsage` accumulates input/output/cached across re-drives; both HTTP paths fold it into their totals. The two surfaces report the same three quantities under different names (`prompt_tokens` vs `input_tokens`), so the key pair is passed in. Expect measured cost to go **up** and savings percentage to go **down** on any deployment running a re-driving hook. That is the correction, not a regression. ## Also: restore the body after the hooks A re-drive rewrites `body[input]` / `body[messages]` / `body[tools]` so the next upstream call carries the hook's turn. Everything downstream — CCR's `_responses_input_to_items(body["input"])`, usage accounting, observability — is describing the request the *client* made, not the proxy's internal detour. Without the restore, a turn that both reloaded a tool and hit CCR retrieval hands CCR the proxy's synthetic items. The chat path had the same leak (`body["messages"]` stayed rewritten); both are fixed the same way. ## Known gap A re-drive on the custom backend path (`send_openai_message`) is still not folded into that request's accounting — its usage is recorded elsewhere. Commented at the call site rather than silently skipped. ## Blast radius **Inert unless a turn hook is registered**, so no behaviour change for a stock OSS proxy. `TurnHookUsage` starts at zero and stays there on every path that does not re-drive. ## Verification - `tests/test_turn_hook_usage.py` — 5 new tests: per-surface key names, accumulation across rounds, negative counts floored not subtracted, and that an unreadable shape still counts the call (a silent zero there looks exactly like "the hook cost nothing") - 434 passing across `turn_hook`, `extension`, `tool_search`, `responses` and `openai_chat` suites - `ruff check` + `ruff format` clean 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Romulo Reis
|
91d6bf33cd
|
perf(subscription): skip transcripts older than the window in compute_window_tokens (#2861)
## Problem
`compute_window_tokens()` walks **every** `.jsonl` under
`~/.claude/projects` and runs
`json.loads()` on **every line**, only to discard the entries that fall
outside
`[start_ts, end_ts)`. `subscription/tracker._poll_loop` calls it every
**300 s**, so the
cost is paid continuously and grows with the user's history.
On one long-running install this meant **1,973 files / 1.1 GB / 261,003
lines re-parsed
every 5 minutes** — about 316 GB of JSON parsing per day.
The user-visible symptom is worse than the CPU bill: the poll pins **100
% CPU with zero
open connections** for ~12 s. That is exactly the signature external
watchdogs use to
detect a runaway loop, so the proxy kept being **restarted while it was
doing scheduled
work** (13 restarts / 13.5 CPU-hours on that host before we traced it
with `py-spy`).
Stack captured during one of those episodes:
```
raw_decode (json/decoder.py:356)
decode (json/decoder.py:337)
loads (json/__init__.py:346)
compute_window_tokens (headroom/subscription/session_tracking.py:127)
_compute_window_tokens_for_snapshot (headroom/subscription/tracker.py:872)
_maybe_poll (headroom/subscription/tracker.py:731)
_poll_loop (headroom/subscription/tracker.py:693)
```
## Fix
Transcripts are append-only and chronological, so a file whose `mtime`
predates the window
start cannot contain an entry inside the window. One guard before
opening the file:
```python
try:
if path.stat().st_mtime < start_ts:
continue
except OSError:
continue
```
## Measurement
Same install, same 5 h window, before vs after:
| | files read | lines parsed | time | result |
|---|---|---|---|---|
| before | 1,973 | 261,003 | **12.1 s** | `weighted_token_equivalent =
741388.0` |
| after | 14 (1,959 skipped) | 4,246 | **0.39 s** |
`weighted_token_equivalent = 741388.0` |
**Identical result, 31× faster.** In production the process CPU peak
over a full poll cycle
dropped from 100 % to 10 %.
## Notes
- Behaviour is unchanged: the guard only skips files that provably
cannot contribute.
- A further optimisation (not included here, to keep the change minimal)
is to read active
transcripts backwards and stop at the first entry older than `start_ts`.
The `mtime`
guard already removes ~99 % of the cost.
- Reproduced on 0.25.0, 0.27.0 and confirmed present in current `main`.
Co-authored-by: romulomorgan <oi@ialucas.com>
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
Rod Boev
|
7f6950be34
|
fix(anthropic): strip first-party tool search on custom upstreams (#2539)
## Description Third-party Anthropic-compatible upstreams can reject Headroom-routed Claude requests before generation starts because the forwarded `tools[]` array still contains the first-party Anthropic server tool type `tool_search_tool_regex_20251119`. That path is valid when the upstream really is Anthropic, but DeepSeek-style Anthropic-compatible gateways reject it with a 400 and never reach model execution. This change strips first-party Anthropic `tool_search_tool_*` entries only when Headroom forwards an Anthropic-wire request to a third-party upstream selected through `anthropic_api_url`. Direct Anthropic behavior stays intact, and unrelated typed or untyped tools keep their existing forwarding contract. Closes #2526. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - add a narrow Anthropic helper that strips first-party `tool_search_tool_*` entries from client-supplied tool lists when the outbound target is a third-party Anthropic-compatible upstream - wire the sanitizer into the Anthropic handler's third-party forwarding path without changing the first-party `HEADROOM_TOOL_SEARCH` injector branch - add focused helper coverage for third-party stripping, first-party preservation, and typed-tool negative space - add a production-path regression through `handle_anthropic_messages()` that captures the custom-upstream request body and verifies the sanitizer wiring ## Testing - [x] Unit tests pass (`uv run pytest tests/test_issue_746_tool_search.py tests/test_anthropic_stage_timings.py -q`) - [x] Linting passes (`uv run ruff check headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py tests/test_issue_746_tool_search.py tests/test_anthropic_stage_timings.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text uv run pytest tests/test_issue_746_tool_search.py tests/test_anthropic_stage_timings.py -q 50 passed in 0.72s uv run ruff check headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py tests/test_issue_746_tool_search.py tests/test_anthropic_stage_timings.py All checks passed! uv run ruff format headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py tests/test_issue_746_tool_search.py tests/test_anthropic_stage_timings.py --check 4 files already formatted git diff --check (no output) ``` ## Real Behavior Proof - Environment: focused Headroom worktree with Anthropic-wire regression tests - Exact command / steps: use the issue reproduction at https://github.com/headroomlabs-ai/headroom/issues/2526, then run the focused helper and handler tests; the handler regression calls `handle_anthropic_messages()` with a DeepSeek-compatible upstream and captures the outbound request body - Observed result: the base repro printed `FAIL issue2526 third-party sanitize -> [{'type': 'tool_search_tool_regex_20251119', 'name': 'tool_search_tool_regex'}, {'name': 'Bash', 'description': 'run a command', 'input_schema': {}}]`, while the head repro printed `PASS issue2526 third-party sanitize -> [{'name': 'Bash', 'description': 'run a command', 'input_schema': {}}]`; the handler-level test captured the same removal while preserving `Bash` and `web_search_20250305`, and the combined focused run passed 50 tests - Not tested: live DeepSeek account on this host ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A - proxy forwarding change only. ## Additional Notes - `CHANGELOG.md` stays untouched because Headroom's release automation generates it from conventional commits. - The narrow slice strips only first-party Anthropic server tool-search entries on third-party Anthropic-compatible upstreams. It does not invent or translate third-party search-tool semantics. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net> |
||
|
Abhay Singh
|
5c561bd913
|
fix(onnx): stop ONNX thread pools from spinning idle cores (#2495) (#2540)
## Description Fixes #2495 (tokensave / the proxy using ~100% of all cores). ONNX Runtime's intra-op (and inter-op) thread pools **spin-wait on every core between inferences** by default. Headroom is a long-lived process that keeps ONNX models loaded — the kompress code compressor ("tokensave"), the image technique/SigLIP routers, and the memory embedder — so once a model is loaded, its idle thread pool keeps every core busy even when no compression is running. That matches the report exactly: CPU climbs to ~100% of all cores "after a period of time" and the whole machine slows down, with no obvious trigger. `create_cpu_session_options` (the shared factory every CPU ONNX session goes through) configured threads and the memory arena but never touched spinning, so ORT's default (spin enabled) was in effect everywhere. ## Fix Disable intra-op and inter-op thread spinning in `create_cpu_session_options` so idle ORT threads block instead of spin-waiting. This applies to every ONNX session built through the factory (kompress + the image routers). It: - is **best-effort per key** (wrapped in try/except) so an older ORT build that doesn't recognize a config key still creates a session; - is **overridable** via `HEADROOM_ONNX_ALLOW_SPINNING=1` for a dedicated/batch box that wants ORT's peak-throughput spinning; - does not change active-inference throughput meaningfully — blocking threads wake on new work with only microsecond-scale latency, which is the recommended setting for a server/proxy with idle periods. The memory embedder already builds its own options with `intra_op_num_threads=1`; this change is orthogonal and additionally quiets its idle spinning if it were ever routed through the factory. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/onnx_runtime.py`: add `ONNX_ALLOW_SPINNING_ENV` + `onnx_thread_spinning_enabled()`; disable `session.intra_op.allow_spinning` / `session.inter_op.allow_spinning` in `create_cpu_session_options` unless spinning is explicitly re-enabled. - `tests/test_onnx_runtime.py`: spinning is disabled by default (both keys), `HEADROOM_ONNX_ALLOW_SPINNING=1` re-enables it, an explicit `0` disables it, and a config key an older ORT rejects doesn't break session creation. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_onnx_runtime.py -q 11 passed # with the fix reverted the new symbols don't exist, so the spinning tests # fail at import — the pre-fix factory left ORT's spinning at its (enabled) default $ uvx ruff@0.15.17 check headroom/onnx_runtime.py tests/test_onnx_runtime.py All checks passed! $ uvx mypy@1.20.2 --ignore-missing-imports headroom/onnx_runtime.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, project venv (`uv sync --extra proxy`, onnxruntime 1.23.2 installed), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv. - Exact command / steps: built a real `onnxruntime.SessionOptions` via `create_cpu_session_options(ort)` and read back `session.intra_op.allow_spinning` / `session.inter_op.allow_spinning`; repeated with `HEADROOM_ONNX_ALLOW_SPINNING=1`. - Observed result: by default both keys read back `"0"` (spinning disabled); with `HEADROOM_ONNX_ALLOW_SPINNING=1` neither key is set (ORT's default spinning restored). Against a real ORT the pre-fix factory set neither key, so ORT's default (spinning enabled) applied — the idle all-cores burn. Ran against the actual module and real onnxruntime. - Not tested: a live multi-hour VS Code + Claude session measuring CPU before/after (the spinning-disable is the documented ORT remedy for idle-CPU in a long-lived process; the config change itself is verified end to end against real ORT). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable --------- Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net> |
||
|
Rod Boev
|
54ea28d983
|
fix(openai): skip Responses tool-search deferral for clients that cannot execute it (#2696)
## Description OpenCode rejects the proxy-injected Responses `tool_search` tool because it resolves tool calls against its local registry. This PR now uses the shared client policy from current `main` and leaves OpenCode tools resident, alongside the existing Codex exclusion. Other clients retain tool-search deferral. Closes #2660. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] Documentation update ## Changes Made - Add `opencode` to the shared exact-match unsupported-client set in `headroom.proxy.helpers`. - Carry the already-classified `client` through native HTTP, WebSocket, and custom-base Responses paths. - Preserve `main`'s compatibility loop, which retries only exact unsupported `client` or `timing` keyword errors and re-raises internal `TypeError`s. - Add focused helper, compressor, HTTP, passthrough, and WebSocket coverage. ## Testing - [x] Unit tests pass - [x] Ruff check and format pass - [x] New tests added - [ ] Live OpenCode session tested ```text uv run --extra dev pytest tests/test_openai_tool_search_deferral.py tests/test_proxy_openai.py -q 57 passed uv run --extra dev ruff check headroom/proxy/handlers/openai.py headroom/proxy/helpers.py tests/test_openai_tool_search_deferral.py tests/test_proxy_openai.py All checks passed ``` ## Real Behavior Proof The focused route tests classify OpenCode from both `User-Agent` and `X-Client`, verify its tools remain untouched, and verify the decision reaches all three Responses ingresses. Supported clients continue to receive deferral. Codex remains excluded by the policy already on `main`. Not tested: a live OpenCode instance; the incompatibility itself remains based on the reporter's reproduction in #2660. ## Review Readiness - [x] Updated from current upstream `main` - [x] Merge conflicts resolved - [x] Focused tests pass locally - [x] Ready for human review ## Additional Notes No user configuration or documentation change is required. Vercel authorization failures are external integration noise, not a source check. --------- Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net> |
||
|
Abhay Singh
|
c49be269a1
|
fix(wrap): stop the launch cwd from shadowing the installed package in the proxy subprocess (#2843)
## Description
`headroom wrap` starts the proxy via `_start_proxy`, which builds `cmd =
[sys.executable, "-m", "headroom.cli", "proxy", ...]`. A `python -m
<module>` invocation prepends the launch cwd to `sys.path`. So when
`wrap` is run from a directory that contains a `headroom/` folder (most
commonly a clone of this very repo, whose package lives at
`<repo-root>/headroom/`), that raw source tree shadows the installed
wheel in site-packages. The source tree has no compiled `headroom._core`
(the maturin extension only exists in the built wheel), so the proxy
dies with:
```text
Error: Proxy dependencies not installed. Run: pip install headroom-ai[proxy]
Details: No module named 'headroom._core'
```
`wrap` then falls back to launching the client unwrapped, and the "not
installed" hint is misleading: the dependency is installed, it is being
shadowed by cwd.
The fix sets `PYTHONSAFEPATH=1` in the proxy subprocess env. That
disables the cwd/script-dir prepend to `sys.path` (Python 3.11+, and a
harmless no-op on 3.10, so it never breaks the supported floor), which
is exactly what the issue reporter confirmed resolves it:
```console
$ PYTHONSAFEPATH=1 python -c "import headroom._core; print('OK')" # -> OK
```
The proxy is still launched as `-m headroom.cli`, so nothing about the
invocation changes except that it now always resolves the installed
package.
Fixes #2793
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/cli/wrap.py` (`_start_proxy`): set
`proxy_env["PYTHONSAFEPATH"] = "1"` alongside the existing
`PYTHONIOENCODING`, with a comment explaining the cwd-shadow failure
mode.
- `tests/test_cli/test_wrap_claude_vertex_proxy_env.py`: added
`test_start_proxy_sets_pythonsafepath_to_avoid_cwd_shadow`, which drives
`_start_proxy` with a faked `subprocess.Popen` and asserts the
subprocess env carries `PYTHONSAFEPATH=1` while still launching `-m
headroom.cli proxy`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
# Fail-before (source fix stashed, new test kept):
tests/test_cli/test_wrap_claude_vertex_proxy_env.py::test_start_proxy_sets_pythonsafepath_to_avoid_cwd_shadow FAILED
assert captured["kwargs"]["env"]["PYTHONSAFEPATH"] == "1"
KeyError: 'PYTHONSAFEPATH'
# Pass-after (fix applied):
tests/test_cli/test_wrap_claude_vertex_proxy_env.py 18 passed
# Broader wrap suites:
tests/test_cli/test_wrap_claude_vertex_proxy_env.py tests/test_cli_proxy_env.py tests/test_cli/test_wrap_persistent.py
121 passed, 1 skipped
# uvx ruff@0.15.17 check -> All checks passed!
# uvx mypy@1.20.2 headroom/cli/wrap.py -> Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.15.17 and mypy 1.20.2 via uvx.
- Exact command / steps: confirmed `_start_proxy` builds
`[sys.executable, "-m", "headroom.cli", "proxy", ...]` and constructs
the subprocess env as `proxy_env`, reproduced the shadowing behaviour in
the reporter's terms (`python -m` prepends cwd; a cwd `headroom/`
without `_core` shadows the wheel), fail-before with `git stash push
headroom/cli/wrap.py` and `python -m pytest ... -k pythonsafepath` (the
env lacks the key), then pass-after with `git stash pop` and rerunning
the file (18 passed) plus the broader wrap suites (121 passed, 1
skipped).
- Observed result: the proxy subprocess env now carries
`PYTHONSAFEPATH=1`, which disables the cwd prepend, so `import
headroom._core` resolves the installed wheel instead of a shadowing
local `headroom/` source tree. The proxy command is unchanged otherwise.
- Not tested: an end-to-end `cd <repo-checkout> && headroom wrap claude`
against a real installed wheel (this environment is a source checkout
without a separate installed wheel to shadow). The behaviour is verified
through the spawn env the subprocess inherits, and `PYTHONSAFEPATH` is
the documented, reporter-confirmed switch for this exact failure mode.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`: it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
Scoped to the proxy launch, which is the reported, high-impact path (its
failure makes `wrap` fall back to unwrapped). `wrap` spawns one other
`python -m headroom.*` subprocess (the memory-sync helper in the Claude
flow) that shares the same root cause; it is a lower-severity,
unreported path and is left for a follow-up rather than widening this
diff. The misleading "pip install headroom-ai[proxy]" message the
reporter also flagged is a separate error-text concern and is likewise
out of scope here.
|
||
|
Abhay Singh
|
3488f8d4b5
|
fix(install): use --userns=keep-id under Podman so bind-mount writes don't fail (#2846)
## Description
`build_runtime_command` unconditionally adds `--user <uid>:<gid>` on
non-Windows hosts:
```python
# headroom/install/runtime.py
if not _is_windows():
getuid = getattr(os, "getuid", None)
getgid = getattr(os, "getgid", None)
if callable(getuid) and callable(getgid):
command.extend(["--user", f"{getuid()}:{getgid()}"])
```
That is correct for Docker, where container UIDs equal host UIDs, but
wrong for rootless Podman, where the host user is already mapped to
container UID 0 and the `/etc/subuid` range is mapped to container UIDs
1 and above. Passing `--user $(id -u):$(id -g)` therefore selects a
container UID backed by a subordinate host UID that owns nothing. The
bind-mounted `~/.headroom` appears inside the container as `root:root`
and is unwritable, so every write fails:
```text
PermissionError: [Errno 13] Permission denied: '/tmp/headroom-home/.headroom/memories'
event=proxy_inbound_request_aborted path=/v1/messages reason=PermissionError
```
The proxy still starts and reports healthy, so the failure only surfaces
once a request touches a write path. As the reporter confirmed,
`--userns=keep-id` (or omitting `--user`) fixes it.
The fix detects Podman and uses `--userns=keep-id` instead of `--user`,
which maps the host user to the same UID inside the container and keeps
the bind mounts writable. Docker still gets `--user`, unchanged.
Detection is subprocess-free: it resolves the `docker` binary and checks
its real name for the common `docker -> podman` symlink shim (e.g. NixOS
`/run/current-system/sw/bin/docker -> podman`), with an explicit
`HEADROOM_CONTAINER_RUNTIME` (`podman` / `docker`) override for setups
the symlink heuristic cannot see, such as a wrapper script.
Fixes #2804
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/install/runtime.py`: added `_container_runtime_is_podman()`
(env override, then a `docker`-binary realpath basename check, no
subprocess). In `build_runtime_command`, when Podman is detected the
command uses `--userns=keep-id` instead of `--user <uid>:<gid>`.
- `tests/test_install/test_runtime.py`: pinned the existing docker test
to the Docker path via `HEADROOM_CONTAINER_RUNTIME=docker` and asserted
`--userns=keep-id` is absent there; added
`test_build_runtime_command_podman_uses_keep_id_not_user` asserting the
Podman path drops `--user` and adds `--userns=keep-id`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
# Fail-before (source fix stashed, new test kept):
tests/test_install/test_runtime.py::test_build_runtime_command_podman_uses_keep_id_not_user FAILED
assert "--userns=keep-id" in command
AssertionError: assert '--userns=keep-id' in ['docker', 'run', '--rm', ...]
# Pass-after (fix applied):
tests/test_install/test_runtime.py 26 passed
# Broader install suite (excluding the pre-existing env-specific PowerShell installer test):
tests/test_install/ 142 passed, 1 skipped
# uvx ruff@0.15.17 check -> All checks passed!
# uvx mypy@1.20.2 headroom/install/runtime.py -> Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.15.17 and mypy 1.20.2 via uvx.
- Exact command / steps: confirmed `build_runtime_command` adds `--user`
unconditionally on non-Windows, then drove both runtimes
deterministically via the `HEADROOM_CONTAINER_RUNTIME` override.
Fail-before with `git stash push headroom/install/runtime.py` and
`python -m pytest tests/test_install/test_runtime.py -k
podman_uses_keep_id` (the command still carries `--user`, no keep-id),
pass-after with `git stash pop` and rerunning the file (26 passed).
- Observed result: with Podman detected the docker command now contains
`--userns=keep-id` and no `--user`/`1000:1001`, matching the
`--userns=keep-id` invocation the reporter verified writes successfully;
with Docker it is unchanged (`--user 1000:1001`, no keep-id).
- Not tested: a live rootless-Podman deployment writing to a bind mount
(no Podman in this environment). The command construction is verified
directly, and `--userns=keep-id` is the documented, reporter-confirmed
switch for the rootless-Podman ID-mapping.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`: it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
Detection is intentionally subprocess-free and conservative: it only
diverges from today's behavior when the `docker` binary literally
resolves to a `podman`-named target, or when
`HEADROOM_CONTAINER_RUNTIME` is set. Real Docker installs are untouched.
The override also gives a clean escape hatch in both directions if a
given host's symlink layout hides the runtime. This is the `--user` half
of the persistent-docker + Podman issues; the separate host-memory-path
problem (#2803) is addressed in its own PR.
|
||
|
Abhay Singh
|
14c4c9d5b7
|
fix(install): stop baking the host memory DB path into a container deployment (#2845)
## Description `headroom deploy --memory` on the `persistent-docker` preset can never become ready. The planner resolves the memory DB path against the **host** home and appends it verbatim to `proxy_args`: ```python # headroom/install/planner.py proxy_args.extend(["--memory", "--memory-db-path", str(_paths.memory_db_path())]) # -> --memory-db-path /home/<user>/.headroom/memory.db ``` The docker runtime passes everything after the leading `--host` pair through unchanged, and the container's `HOME` is `/tmp/headroom-home` with the host's `~/.headroom` bind-mounted at `/tmp/headroom-home/.headroom`. The host path `/home/<user>/.headroom/memory.db` does not exist inside the container, so SQLite cannot open the DB: ```text Memory: backend initialization failed (startup continues): unable to open database file ``` `/health` then reports `memory.ready = false`, `/readyz` stays 503 for the full `wait_ready` window, and `_start_deployment` times out and rolls back, so the failure presents as "did not become ready" rather than a path bug. The same applies on macOS with `/Users/<user>/...`. The fix omits `--memory-db-path` for a container (docker) runtime. When the flag is absent the proxy resolves the DB under its own cwd (`.headroom/memory.db`), and the container's workdir is `/tmp/headroom-home` (the bind mount), so the DB lands in exactly the same host file the explicit path intended. The host (python) runtime still passes the resolved host path, which is correct there. Fixes #2803 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/install/planner.py` (`build_manifest`): append `--memory` always, but add `--memory-db-path <host path>` only when `runtime_kind != RuntimeKind.DOCKER.value`. Imported `RuntimeKind` from `.models`. - `tests/test_install/test_planner.py`: extended `test_build_manifest_for_persistent_docker_sets_expected_defaults` to assert `--memory-db-path` is absent for the docker runtime, and added `test_build_manifest_python_runtime_keeps_explicit_memory_db_path` asserting it is still present for the python runtime. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # Fail-before (source fix stashed, updated tests kept): tests/test_install/test_planner.py::test_build_manifest_for_persistent_docker_sets_expected_defaults FAILED assert "--memory-db-path" not in manifest.proxy_args AssertionError: assert '--memory-db-path' not in ['--host', '127.0.0.1', ...] # Pass-after (fix applied): tests/test_install/test_planner.py 19 passed # Broader install suites: tests/test_install/ 141 passed, 1 skipped, 2 unrelated pre-existing/flaky failures # - test_native_installers.py::test_powershell_native_installer_supports_persistent_docker_lifecycle # runs scripts/install.ps1 and fails identically on clean main (environment-specific). # - test_runtime.py::test_runtime_status_survives_winerror87_systemerror passes in isolation # and in its own file; it only failed under cross-file ordering in the broad run, and is # untouched by this diff (planner.py only). # uvx ruff@0.15.17 check -> All checks passed! # uvx mypy@1.20.2 headroom/install/planner.py -> Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1, ruff 0.15.17 and mypy 1.20.2 via uvx. - Exact command / steps: traced the path from `planner.py` (`--memory-db-path str(_paths.memory_db_path())`, host home) through `runtime.py` (`build_runtime_command` passes `proxy_args[_PROXY_ARGS_HOST_PAIR_LEN:]` through, container HOME `/tmp/headroom-home`, `~/.headroom` bind-mounted) and confirmed via `server.py` that an empty `memory_db_path` resolves to `Path.cwd()/.headroom/memory.db` (the container workdir, hence the mount). Fail-before with `git stash push headroom/install/planner.py` and `python -m pytest tests/test_install/test_planner.py -k persistent_docker` (host path present in proxy_args), pass-after with `git stash pop` and rerunning (19 passed). - Observed result: for the docker runtime, `manifest.proxy_args` now carries `--memory` without `--memory-db-path`, so the container resolves the DB to `/tmp/headroom-home/.headroom/memory.db` (the bind mount to host `~/.headroom/memory.db`) and can open it, instead of receiving a nonexistent host path. The python runtime still carries the explicit host path. - Not tested: a live `headroom deploy --memory` against a running Docker daemon (no container runtime in this environment). The manifest construction is verified directly, and the container-side resolution it relies on is existing server behavior (`empty memory_db_path -> cwd/.headroom/memory.db`) confirmed by reading `server.py`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md`: it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes The DB persistence location is unchanged: both the old host path and the new container-cwd resolution point at the host's `~/.headroom/memory.db` (directly on the host, or through the bind mount inside the container), so existing memory DBs are picked up either way. This is the memory-path half of the persistent-docker issues; the separate rootless-Podman `--user` bind-mount problem (#2804) is left for its own fix. |
||
|
Abhay Singh
|
3808f60ca6
|
fix(proxy/anthropic): inject headroom_retrieve whenever a CCR marker is present, not only for new markers (#2848)
## Description
On a frozen-prefix turn that replays an existing `<<ccr:hash>>` marker,
the proxy did not inject the `headroom_retrieve` tool, so the agent held
a marker it could not redeem. When it tried, the Anthropic API rejected
the whole request:
```text
API Error: 400 Tool reference 'headroom_retrieve' not found in available tools
```
This was a frequent, user-visible failure in Claude Code.
### Root cause
The sticky tool-injection gate in `handlers/anthropic.py` was driven by
`has_new_ccr_markers(...)` -- markers created THIS turn only:
```python
has_new_compressed_content = has_new_ccr_markers(
current_detected_hashes=injector.detected_hashes,
previous_forwarded_messages=prefix_tracker.get_last_forwarded_messages(),
provider="anthropic",
)
tools, ccr_tool_injected = apply_session_sticky_ccr_tool(
...,
has_compressed_content_this_turn=has_new_compressed_content,
)
```
`apply_session_sticky_ccr_tool` returns early with `decision="skip"` for
a session it considers fresh when `not
has_compressed_content_this_turn`. A marker replayed from the frozen
prefix is "historical" (already in `previous_forwarded_messages`), so
`has_new_ccr_markers` returns `False`, and on a fresh session the tool
is skipped even though the request carries a redeemable marker. The
`SessionCcrTracker` is per-process, so every proxy restart makes live
sessions look fresh again and re-arms the failure mid-conversation.
Anything that instructs the model to retrieve later (a project
instruction saying "call `headroom_retrieve` with the hash before
asserting an exact value") lands on this path by construction.
### Fix
Drive the gate from `injector.has_compressed_content` -- whether the
forwarded request carries ANY CCR marker, new or replayed -- instead of
new-markers-only. `#1850` narrowed the first-time gate to new markers to
avoid arming a session that never compressed, but a present marker means
the session HAS compressed, and a replayed marker is exactly as
unredeemable as a fresh one. Since a new marker is also a present
marker, `has_new_compressed_content or injector.has_compressed_content`
collapses to `injector.has_compressed_content`, so the now-redundant
`has_new_ccr_markers` call is removed.
The cache argument cuts in favor of this: toggling the tool in and out
of the tools array between turns is what busts the tools cache segment.
Injecting consistently whenever markers exist is the cache-stable
option, and it removes a hard 400 in exchange for at most one cache
miss. The frozen message prefix is still replayed byte-identical, so the
prompt-cache prefix is unaffected; only the tools array gains a stable
entry.
Fixes #2766
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/anthropic.py`: the sticky CCR tool-injection
gate now passes
`has_compressed_content_this_turn=injector.has_compressed_content` (any
marker present) instead of the new-markers-only signal, and the
now-redundant `has_new_ccr_markers` computation/import is dropped.
- `tests/test_proxy/test_anthropic_ccr_deferred_injection.py`: the two
tests that encoded the superseded `#1850` behavior (a replayed
historical marker forwarded WITHOUT the tool) now assert the tool IS
injected, with updated rationale. One was renamed from
`..._when_tool_injection_is_deferred` to
`..._and_injects_retrieve_tool`. The byte-identical message-prefix
replay assertions are unchanged.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
# Fail-before (source fix stashed, updated tests kept):
tests/test_proxy/test_anthropic_ccr_deferred_injection.py
::test_cache_mode_compresses_delta_but_replays_cached_prefix_when_markers_are_historical FAILED
::test_cache_mode_exact_prefix_replay_forwards_cached_compressed_prefix_and_injects_retrieve_tool FAILED
assert [tool["name"] for tool in forwarded["tools"]] == ["headroom_retrieve"]
KeyError: 'tools'
# Pass-after (fix applied):
tests/test_proxy/test_anthropic_ccr_deferred_injection.py 15 passed
# Broader CCR suites:
tests/test_proxy/test_ccr_frozen_prefix_coupling.py tests/test_ccr_tool_always_on.py
tests/test_ccr_session_tracker.py tests/test_ccr_tool_injection.py 61 passed
tests/test_ccr_marker_policy.py tests/test_anthropic_ccr_workspace_unbound.py
tests/test_ccr_tool_calls.py tests/test_corrupt_golden_bytes_recovery.py 21 passed
# uvx ruff@0.15.17 check -> All checks passed!
# uvx mypy@1.20.2 headroom/proxy/handlers/anthropic.py -> Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.15.17 and mypy 1.20.2 via uvx.
- Exact command / steps: traced the gate (`has_new_ccr_markers` ->
`apply_session_sticky_ccr_tool` fresh-session `skip`) and confirmed
`injector.has_compressed_content` reflects any marker present in the
forwarded messages (`len(_detected_hashes) > 0` after
`scan_for_markers`). Reproduced the exact bug in the handler harness: a
cache-mode frozen replay where `fake_tracker._last_forwarded_messages`
already holds the marker (so `has_new` is `False`) on a session the
reset tracker considers fresh, with the marker forwarded to upstream.
Fail-before with `git stash push headroom/proxy/handlers/anthropic.py`
and rerunning the two replay tests (the forwarded body has no `tools`),
pass-after with `git stash pop` (the body carries `headroom_retrieve`).
- Observed result: on a replayed-marker turn the forwarded request now
includes `"tools": [{"name": "headroom_retrieve", ...}]`, so the agent
can redeem the hash and Anthropic no longer 400s. The frozen message
prefix is still replayed byte-identical (`forwarded["messages"]`
unchanged). Sessions that never compressed still get no tool (no marker
-> `has_compressed_content` is `False`).
- Not tested: a live multi-turn Claude Code session across a real proxy
restart (no live provider here). The gate is exercised end-to-end
through the handler via the TestClient harness, reproducing the
historical-marker-on-fresh-session desync the issue describes.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`: it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
This deliberately reworks the `#1850` deferral for historical markers,
so it changes two tests that encoded "tool absent on frozen replay."
That behavior was the source of the 400: a marker in the prompt with no
tool to redeem it is a hard failure, whereas a re-injected tool is a
stable, cheap entry in the tools array. The reporter validated the same
change locally (33 requests, 0 errors, 0 `skip`). Scope is the Anthropic
interactive path where the bug was reported; the stateless batch path (a
separate `CCRToolInjector.process_request` gated on `tokens_saved > 0`)
is unchanged.
|
||
|
Abhay Singh
|
1f5fefffd3
|
fix(memory): bound the TrafficLearner pending-pattern accumulator (memory leak) (#2579)
## Description
`TrafficLearner` (the memory/learning subsystem that accumulates
patterns from proxy traffic) has an unbounded in-memory accumulator.
`_pattern_counts` maps `content_hash -> (pattern, count)`. A pattern is
added on first sighting, its count is bumped on each re-sighting, and it
is **removed only when it reaches `min_evidence`** (default 5), at which
point it is promoted and its hash moves to `_saved_hashes`:
```python
if h in self._pattern_counts:
existing, count = self._pattern_counts[h]
count += 1
self._pattern_counts[h] = (existing, count)
else:
self._pattern_counts[h] = (pattern, 1)
return # first sighting — wait for more evidence
...
if count >= self._min_evidence:
del self._pattern_counts[h] # only removal path
self._saved_hashes.add(h)
if len(self._saved_hashes) > self._dedup_window: # sibling IS trimmed
self._saved_hashes.pop()
```
A pattern seen **once but never corroborated** — the common case for
one-off traffic (a unique error string, an ad-hoc shell command, a
distinct file path) — never reaches `min_evidence`, so it is **never
removed**. Over a long-lived proxy processing varied traffic,
`_pattern_counts` grows without bound and RSS climbs. The sibling
`_saved_hashes` is explicitly trimmed to `dedup_window` ("prevent
unbounded growth"); `_pattern_counts` was missed.
Reproduced directly: feeding 500 distinct one-off patterns leaves 500
entries in `_pattern_counts` (one per pattern, forever).
## Fix
Make `_pattern_counts` an LRU-ordered `OrderedDict` capped at a new
`max_pending_patterns` (default 2048):
- On each corroboration, `move_to_end(h)` so an actively-accumulating
pattern stays "fresh" and is never evicted before it can be promoted.
- On a first sighting when the accumulator is full, evict the
least-recently-corroborated pending entry (`popitem(last=False)`).
Evicting a stale one-off is safe: if it recurs it simply restarts
accumulation (delayed promotion at worst) — the same tradeoff
`_saved_hashes` already makes. Promotion at `min_evidence` is unchanged,
and the cap (2048) is generous enough that any pattern receiving repeat
sightings within a normal window reaches `min_evidence=5` long before
eviction.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring
## Changes Made
- `headroom/memory/traffic_learner.py`: `_pattern_counts` becomes a
capped LRU `OrderedDict`; add `max_pending_patterns` (default 2048);
`move_to_end` on corroboration and evict-oldest on overflow.
- `tests/test_memory/test_traffic_learner.py`: a regression that 500
one-off patterns keep the accumulator at its cap, and one that a
corroborated pattern still promotes into `_saved_hashes` (both sync via
`asyncio.run` so they run without the pytest-asyncio plugin).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_memory/test_traffic_learner.py -q
35 failed, 109 passed
# the 35 failures are pre-existing @pytest.mark.asyncio tests that need
# pytest-asyncio (not configured in this environment); they fail identically
# on clean main (35 failed, 107 passed) and pass in CI. My two new tests are
# synchronous and pass; they add +2 passing with no new failures.
# with the fix reverted, test_pending_accumulator_is_bounded fails
# (the accumulator holds all 500 one-off patterns)
$ uvx ruff@0.15.17 check headroom/memory/traffic_learner.py tests/test_memory/test_traffic_learner.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/memory/traffic_learner.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, project venv (`uv sync --extra
proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv.
- Exact command / steps: built a `TrafficLearner(backend=None,
min_evidence=5, max_pending_patterns=8)` and drove `_accumulate` with
500 distinct one-off `ExtractedPattern`s; separately corroborated one
pattern to `min_evidence`; then reverted the source and re-ran.
- Observed result: with the fix `len(_pattern_counts)` stays at the cap
(8) after 500 one-offs, the corroborated pattern is removed from pending
and present in `_saved_hashes`, and an actively-bumped pattern survives
LRU eviction; with the fix reverted the accumulator holds all 500
one-off entries (the unbounded leak). Ran against the actual module.
- Not tested: a live multi-day proxy run measuring RSS (the leak is
inferred from the removed unbounded-growth path; the accumulator bound
is verified directly).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
|
||
|
Abhay Singh
|
b97c7c6e99
|
fix(proxy/gemini): keep streaming-parity baseline so eligible_pct can't exceed 100 (#2824)
## Description The non-streaming Gemini `generateContent` finalizer builds its `RequestOutcome` with `optimized_tokens` set to Gemini's own `promptTokenCount` (the provider's tokenizer scale, which correctly feeds billing and the dashboard), while `original_tokens` stays a local estimator count. Those two are on different rulers. Every delta the beacon derives from the pair is a same-ruler difference: `tokens_saved`, `tokens_inflated`, `attempted_input_tokens`, and the beacon's `eligible_pct` / `yield_pct`. When Gemini counts the forwarded prompt higher than our local estimator does, `attempted_input_tokens` (which is `optimized_tokens + tokens_saved`) exceeds the local `original_tokens`, and the request ships a structurally-impossible `eligible_pct > 100` plus a phantom `tokens_inflated`. This is the exact class of bug #2756 removed, on a path #2756 did not touch: it fixed the non-streaming OpenAI handler, and the streaming finalizer (`_finalize_stream_response`) already guards against it by lifting the baseline onto the provider scale. The non-streaming Gemini path had neither treatment. The fix mirrors the streaming finalizer's already-tested handling: when a provider count is present, lift the baseline to `max(original_tokens, promptTokenCount + tokens_saved)` so `attempted_input_tokens <= original_tokens` holds and `tokens_inflated` collapses to 0. It is guarded on a present count, so a null or absent `promptTokenCount` leaves the local baseline untouched and the existing zero-usage preservation test still holds. `optimized_tokens` still carries the provider count, so billing and the dashboard are unchanged. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/gemini.py` (`handle_gemini_request`, non-streaming `generateContent` branch): compute `effective_original_tokens = max(original_tokens, total_input_tokens + tokens_saved)` when `total_input_tokens > 0` (else keep `original_tokens`), and pass it as the outcome's `original_tokens`. Mirrors the streaming finalizer's provider-usage handling. - `tests/test_proxy/test_gemini_savings_profile.py`: added `test_gemini_provider_count_above_local_estimate_does_not_inflate_eligible`, which drives a request where Gemini's `promptTokenCount` (150) exceeds the local post-compression count (80), and asserts `attempted_input_tokens <= original_tokens`, `tokens_inflated == 0`, the provider count is still carried in `optimized_tokens`, and the baseline is lifted to 170. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # Fail-before (source fix stashed, new test kept): tests/test_proxy/test_gemini_savings_profile.py::test_gemini_provider_count_above_local_estimate_does_not_inflate_eligible FAILED assert outcome.attempted_input_tokens <= outcome.original_tokens AssertionError: assert 170 <= 100 # Pass-after (fix applied): tests/test_proxy/test_gemini_savings_profile.py::test_gemini_provider_count_above_local_estimate_does_not_inflate_eligible PASSED # Full file + related outcome suites: tests/test_proxy/test_gemini_savings_profile.py tests/test_proxy_gemini_native_integration.py tests/test_request_outcome.py tests/test_outcome_token_scale.py 47 passed, 18 skipped # uvx ruff@0.15.17 check -> All checks passed! # uvx mypy@1.20.2 headroom/proxy/handlers/gemini.py -> Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12.11, project venv (litellm installed), pytest 9.1.1 with pytest-asyncio 1.4.0 (asyncio_mode=auto per pyproject), ruff 0.15.17 and mypy 1.20.2 via uvx. - Exact command / steps: confirmed the streaming sibling already lifts the baseline (`_finalize_stream_response` in `headroom/proxy/handlers/streaming.py` sets `effective_original_tokens = max(original_tokens, provider_input_tokens + tokens_saved)` for openai/gemini), then fail-before with `git stash push headroom/proxy/handlers/gemini.py` and `python -m pytest tests/test_proxy/test_gemini_savings_profile.py -k inflate_eligible` (the assertion fails with `170 <= 100`, i.e. eligible_pct 170%), then pass-after with `git stash pop` and rerunning (passes), then the full file plus the outcome suites (47 passed, 18 skipped). - Observed result: with Gemini reporting `promptTokenCount=150` against a local post-compression count of 80 (saved 20), the outcome now reports `original_tokens=170`, `attempted_input_tokens=170` (so `eligible_pct <= 100`) and `tokens_inflated=0`, while `optimized_tokens` stays 150 so billing and the dashboard are unchanged. Before the fix the same request reported `original_tokens=100`, `attempted_input_tokens=170` (eligible_pct 170%) and `tokens_inflated=50`. - Not tested: a live streamed call to real Gemini/Vertex (no provider credentials in this environment). The provider-count-above-local case is reproduced with a mock response mirroring Gemini's `usageMetadata` shape, and the baseline-lift it mirrors is existing, tested code on the streaming path. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md`: it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Docs and manual testing are N/A: this aligns the non-streaming Gemini finalizer with the already-correct streaming finalizer, no API surface change. The baseline lift is guarded on a present provider count, so the existing zero-usage preservation test (`test_gemini_zero_usage_prompt_count_is_preserved`) is unaffected: a null or zero `promptTokenCount` keeps the local baseline and leaves `optimized_tokens` at 0. |
||
|
JD Davis
|
01161fe019
|
test(openclaw): match inherited PATH shell check (#2821)
## Description
Fix the OpenClaw test failure on `main` by aligning its PATH-launcher
expectation with the intentionally shipped `sh -c` behavior from #1459.
The non-login shell preserves the PATH inherited from the OpenClaw
process. Changing production code back to `sh -lc` would risk a login
shell resetting that PATH and would undo the compatibility fix. This PR
therefore corrects only the stale assertion; runtime behavior and
defaults do not change.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Expect `sh -c` for the non-Windows lightweight `command -v headroom`
check.
- Preserve the existing Windows `where.exe` behavior and all launcher
behavior.
## Testing
- [x] Unit tests pass (`npm test`)
- [x] Linting passes (`npm run typecheck`)
- [x] Type checking passes (`npm run typecheck`)
- [ ] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ npm test
Test Files 6 passed (6)
Tests 75 passed (75)
$ npm run typecheck
> tsc --noEmit
$ npm run build
ESM Build success
DTS Build success
$ npm ci
found 0 vulnerabilities
```
## Real Behavior Proof
- Environment: macOS, Node/npm, clean install from `origin/main` at
`
|
||
|
Ashish Patel
|
4ec416df88
|
fix(proxy): stop discarding compressed Codex WS later-frame payloads (#2823)
## Description
`headroom perf` reports 0 tokens saved for Codex CLI sessions despite
real traffic being processed (confirmed via the reporter's live proxy
stats in the issue). Root cause: a misplaced `return` statement in the
Codex WS later-frame compression path silently discards every compressed
payload and skips all token/savings bookkeeping for it.
Closes #2819
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/openai.py`
(`_maybe_compress_response_create_frame`): PR #1579 (2026-07-16) moved a
`return (raw_after_store, ...)` statement to the same indentation as the
enclosing `except Exception:` block instead of inside it. That made the
`return` fire **unconditionally** after every later (2nd+)
`response.create` frame in a Codex WS session — success or failure —
always forwarding the original pre-compression frame upstream and
skipping the entire success-path code below it (correct
rewritten-payload return, `tokens_saved`,
`attempted_input_tokens_total`, `ws_frames_compressed`). Fixed by moving
the `return` back inside the `except` block, restoring the success path.
- `tests/test_openai_codex_ws_lifecycle.py`: new regression test
`test_ws_later_frame_compression_is_actually_forwarded` — mocks the
compressor to report `modified=True` with a distinct rewritten payload
on a later frame, asserts the rewritten payload (not the original) is
what's actually sent upstream.
## Testing
- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`) — not run locally, will confirm
via CI
- [ ] Type checking passes (`mypy headroom`) — not run locally, will
confirm via CI
- [x] New tests added for new functionality
- [x] Manual testing performed (see Real Behavior Proof)
### Test Output
```text
$ .venv/Scripts/python -m pytest tests/test_openai_codex_ws_lifecycle.py tests/test_codex_ws_per_frame_memory.py tests/test_codex_ws_savings_deferral.py tests/test_openai_codex_ws_timings.py -q
............................................... 48 passed in 5.34s
$ .venv/Scripts/python -m pytest tests/ -k "openai or codex" -q (wider sweep, unrelated dirs excluded)
968 passed, 3 failed, 77 skipped, 2 errors in 483.21s
```
The 3 failures
(`test_client_integration.py::test_auto_detect_openai_optimizer`,
`test_compress_route_tokenizer_by_model.py::...[deepseek/deepseek-v4]`,
`test_learn/test_integration.py::TestCodexIntegration::test_full_pipeline`)
reproduce identically on a clean, unmodified `main` — confirmed by
stashing this PR's changes and re-running. They're local-environment
issues (a live litellm 503, and this dev box's tool registry missing a
`Bash` entry), not caused by this change.
## Real Behavior Proof
- Environment: Windows 11, Python 3.14.5, local venv, `headroom._core`
rebuilt via `maturin develop --release` against current `main` to rule
out stale-build noise
- Exact command / steps: (1) `git blame` on the buggy block traced the
misplaced `return` to commit `
|
||
|
Tejas Chopra
|
53af90d68c
|
perf: cut hot-path latency 27% (token-count memo, startup preloads, JSON scan memo) (#2838)
## Description
Four independent latency fixes on the request hot path, found by
profiling and each measured in isolation. No behaviour changes: every
commit is either a memo of a pure function, work moved to startup, or
work that was computed and discarded.
**End to end: 287ms → 210ms (−27%) on a 68k-token mixed payload, with
byte-identical output** (68,514 → 48,725 tokens both before and after).
Plus one-off costs removed that don't show in steady-state numbers:
~4.9s of lazy imports that were firing *inside* user requests, and
~750ms of HuggingFace round-trips per process start.
Closes #
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
**1. Memoise `count_text` (`ac369277`)** — tiktoken's `CoreBPE.encode`
was 0.243s of a 0.30s profiled request. It dominates because the same
string is counted repeatedly: a 103KB payload drove 600KB of encoding,
~6x the content, across six call sites (`tokenizers/base.py:196`,
`content_router.py:4704` and `:5474`, `parser.py:185/192/298`). 35% of
encode calls and 22% of encoded characters were an exact repeat *within
one request*.
`count_text` is a pure function of its text, so replaying a stored count
returns the same integer. That is the whole safety argument, and it is
what makes this safe at the sites whose count feeds a routing decision
(`context_pressure` → `min_ratio`) rather than a log line — an
*estimate* there would change which blocks compress; a memo cannot.
Keyed on the text itself, not a hash: a collision would hand back a
wrong count for real content and silently change compression. The cost
is holding the strings, so entries and total characters are both capped.
Clear-on-full rather than LRU eviction — the pipeline runs on a thread
pool, `dict` get/set/clear are atomic under the GIL but
`OrderedDict.move_to_end` is not.
**2. Preload what was importing mid-request (`2921a15b`)** — `litellm`
(2.9–3.8s) was imported lazily *on the event loop* during the first
request: `emit_request_outcome` → `record_request` →
`_estimate_compression_savings_usd` calls the loader before its own
`tokens_saved <= 0` early return, so even a request that saved nothing
paid it. `trafilatura` (978ms, pulling `htmldate` → `dateparser` and its
timezone tables) is the most expensive lazy import in the transform tree
— every other compressor module is 1–20ms — and fires on the first
request carrying an HTML-ish or mixed-content block. The TOIN singleton
reads ~5MB of learned patterns on construction (~150ms); a stale comment
claimed the SmartCrusher preload covered it, and it does not.
All three now load in `_eager_preload_transforms`, which already runs
under `asyncio.to_thread` and so cannot delay the port bind.
Same commit, two Kompress cold-path fixes: `_load_modernbert_tokenizer`
always used `local_files_only=False`, which makes transformers
re-validate against the Hub on every load — a tree listing plus a HEAD
per file — even when fully cached (~900ms warm-cache vs ~150ms
local-only). And `ensure_background_download` re-spawned a
finished-or-failed thread on the next call, so an unreachable Hub meant
one fresh download thread *per request* for the life of the process,
each importing transformers and holding the GIL against the event loop.
Consecutive failures now back off; success clears it, so the happy path
and the transient-failure path are unchanged.
**3. Memoise the JSON-block scan (`039c9735`)** —
`_has_valid_json_block_with_text` tries every `{`/`[`-leading line as a
possible block start. When a candidate never balances,
`_extract_json_block` scans character-by-character to the end of the
content and returns nothing — then the next candidate does it again.
Quadratic, on the request path, growing exactly 4x per doubling.
**4. `CostTracker.totals()` (`286b97e4`)** —
`_current_savings_tracker_totals` called `stats()` once per request and
read two of its fields. Building the rest includes
`period_cost_breakdown()`, which walks up to 100k cost records over 31
days, on the event loop, holding the metrics lock. It degrades with
proxy **uptime**, not load, which is why no short benchmark would
surface it.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ ruff check headroom/ tests/ --exclude headroom/dashboard/templates
All checks passed!
$ mypy --python-version 3.12 headroom/
Found 1 error in 1 file (checked 515 source files)
headroom/release_version.py:235: error: Name "tomllib" already defined (by an import)
# pre-existing on main, in a file this PR does not touch — verified by
# running the same command on a clean main checkout.
$ python -m pytest tests/test_token_count_cache.py tests/test_mixed_content_scan_cache.py \
tests/test_kompress_download_backoff.py tests/test_cost_tracker_totals.py -q
306 passed
$ python -m pytest tests/ -q -k "token or tokenizer or count or estimator or provider"
1303 passed, 105 skipped in 423.42s
$ python -m pytest tests/ -q -k "cost or budget or metrics or savings or stats"
683 passed, 127 skipped, 1 failed
# tests/test_proxy_memory_integration.py::TestMemoryStats::test_health_endpoint_works_with_memory
# Order-dependent and pre-existing: it SKIPS in isolation, and fails identically
# on a clean main checkout under the same -k selection (681 passed, 1 failed).
```
## Real Behavior Proof
- **Environment:** macOS, Python 3.12.6, local CPU, remote Kompress
disabled. Profiled with `cProfile` on `anthropic_pipeline.apply`.
- **Exact command / steps:** a 68k-token payload of four `tool_result`
blocks (900-item pretty JSON, 60KB of Python source, 500 lines of
JS-style object logs, 500 plain log lines), six reps, **content unique
per rep so every run is router-cache-cold**, run on this branch and on
main in alternation.
- **Observed result:**
| | median | min | tokens |
|---|---|---|---|
| main | 287ms | 286ms | 68,514 → 48,725 |
| this branch | 210ms | 208ms | 68,514 → 48,725 |
Per-change, measured in isolation:
| change | before | after |
|---|---|---|
| `count_text` memo | — | −25% pipeline wall; 44% of counted chars from
cache on new content, 100% when history repeats |
| litellm / trafilatura / TOIN | 3829 / 978 / 150ms mid-request | at
startup, off the event loop |
| Kompress tokenizer | ~900ms | ~150ms |
| JS-style object logs (1200 lines) | 4643ms | 183ms |
| truncated JSONL (1200 lines) | 3737ms | 116ms |
| `cost_tracker` per request | 2.8ms @20k records, 13.6ms @100k | loop
over models, not records |
Output equality: 18/18 payloads byte-identical on `tokens_before`,
`tokens_after` and a sha256 of the resulting messages, with the memo
forced on vs off.
- **Not tested:** Windows and Linux (the ORT dylib and CPU-arena paths
differ); multi-worker deployments; a proxy with a genuinely large live
cost ledger (the 100k figure is from a synthetic ledger); real
HTML-heavy traffic through the preloaded trafilatura path.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
**Docs:** N/A — no user-facing surface changes. The reasoning lives in
the code, at the sites where someone debugging would look.
**A regression I introduced and caught.** The scan memo initially made
pretty-printed JSON ~2x **slower**: content that balances on the first
scan has nothing to reuse and just pays the per-line dict traffic. The
cache is now built only *after* a scan has run to the end without
balancing, which is the actual signal that later candidates will re-walk
the same tail. Every shape now improves and none regress:
```
before after
js object logs 4642.9ms 182.7ms 25x
JSONL truncated 3736.8ms 115.9ms 32x
pretty JSON 5.6ms 3.5ms
JSONL valid 5.6ms 3.2ms
plain logs 1.0ms 0.6ms
python source 1.0ms 0.5ms
markdown prose 0.9ms 0.5ms
```
Worth stating plainly: had I only benchmarked the shape I was fixing,
I'd have shipped a win on rare content and a loss on the common case.
**The scan fix is constant-factor, not asymptotic.** The walk over
remaining lines is still O(candidates × lines), so 3200 lines of the
pathological shape is still ~1.4s. The tests assert scan-call counts
rather than implying linearity. True linearity needs a prefix-sum
rewrite with a string-state fallback; that seemed like the wrong risk
for this PR.
**How the parser change is proven safe.** `_extract_json_block` is a
parser, so golden values would only encode whatever the new code does.
Instead the pre-memo implementation is kept verbatim in the test file as
an oracle, and every candidate index of a 139-document corpus — escapes,
unterminated strings, delimiters inside strings, code fences, truncated
JSON, randomised mixtures — is asserted equal, with a cold cache, with
the shared cache the real callers use, and replayed.
**Measurement trap, for anyone re-running these numbers.** Give each arm
its own content. Reusing one payload across arms lets the second arm hit
the router's result cache, which reads as a speedup having nothing to do
with the change under test. I hit this twice while working on it: it
manufactured a fake "INFO logging costs 21.8%" finding (real answer:
0.3%) and it *understated* the memo win.
**Deliberately not in this PR:**
- **ONNX thread tuning** — measured zero gain, and
`intra_op_num_threads` is not bitwise-safe (1.6e-05 score drift from
float reduction order), so it would trade an output risk for nothing.
- **`str(content)` on block lists** counts a base64 image at 210,775
tokens instead of 1,604 (131x), pinning `context_pressure` to 1.0 and
forcing the most aggressive `min_ratio` on any conversation containing
an image. Real bug, but fixing it changes compression output — needs its
own reviewed behaviour-change PR.
- **`chunk_words=350` against the tokenizer's 512-token limit** silently
drops roughly a third of every full chunk (measured: 240/240 words kept
in the first 240, 15/110 in the tail). That is data loss rather than
latency, it changes every output, and correcting it costs ~1.3x latency.
Filing separately.
- **Telemetry off the request thread** — the TOIN auto-save is a 236ms
inline stall every 600s and the waste-signal re-parse is ~50ms/request
that is invisible in `pipeline_total` (computed before it). Both want
deferral rather than removal, which is a larger change than belongs
here.
|
||
|
Tejas Chopra
|
564e0a8d0f
|
fix(deps): bump h2 to 4.4.1 for CVE-2026-71554 (#2839)
## Description
`pip-audit` is currently red on every open PR. Not because of anything
in those branches — `uv.lock` pins `h2` at 4.3.0, and CVE-2026-71554 was
published against `h2 <=4.4.0`.
> h2 <=4.4.0 accepts request header blocks containing more than one Host
header, and forwards every Host header to the consuming application.
Where the consumer downgrades HTTP/2 to HTTP/1.1, the resulting request
carries two Host header lines, which is a request smuggling primitive
(CWE-444).
Fixed in 4.4.1.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `uv lock --upgrade-package h2`, which moves exactly two packages:
```
h2 4.3.0 -> 4.4.1
hpack 4.1.0 -> 4.2.0
```
`h2` arrives transitively via `httpx[http2]`, and the constraint in
`pyproject.toml` is already wide enough (`>=3,<5`), so only the lock
needed to move — no source or `pyproject.toml` change.
`requirements-prod.txt` is not checked in; the audit workflow exports it
from `uv.lock` at run time, so the lock bump is the entire fix.
## Testing
- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed
### Test Output
Reproduced the CI gate locally with the exact command from
`.github/workflows/security.yml`:
```text
$ uv export --frozen --no-dev --no-emit-project --no-hashes \
--extra all --format requirements-txt > requirements-prod.txt
$ grep -E '^(h2|hpack)==' requirements-prod.txt
h2==4.4.1
hpack==4.2.0
$ pip-audit -r requirements-prod.txt
No known vulnerabilities found
```
Before this change, the same command reported:
```text
Name | Version | ID | Fix Versions
h2 | 4.3.0 | CVE-2026-71554 | 4.4.1
Found 1 known vulnerability in 1 package
```
## Real Behavior Proof
- **Environment:** macOS, uv 0.9.x, Python 3.12.6.
- **Exact command / steps:** `uv lock --upgrade-package h2 --dry-run` to
confirm the blast radius, then the real lock, then the workflow's own
export + `pip-audit` invocation.
- **Observed result:** resolution touches only `h2` and `hpack`; 269
packages resolved with no other version movement. `pip-audit` goes from
1 known vulnerability to none.
- **Not tested:** HTTP/2 traffic against a live upstream. `h2` 4.4.1 is
a patch release on a library used transitively by `httpx`; Headroom does
not import `h2` directly (`grep -rn "import h2" headroom/` is empty), so
the exposure is whatever `httpx[http2]` does with it.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
N/A items above: no code changed, so ruff/mypy/new tests do not apply —
the verification that matters is the audit output, which is quoted in
full.
**Why this is standalone.** It surfaced while fixing CI on #2838, but it
is not caused by that branch and it blocks #2832 identically. Landing it
separately unblocks the gate for every open PR at once and keeps a
supply-chain bump out of an unrelated change.
**One unrelated warning the resolver prints**, noted so it is not
mistaken for a side effect of this PR:
```
warning: `pypdfium2==5.12.0` is yanked (reason: "Setup blunder breaking some
bindgen codepaths ... Wheels are valid and effectively identical to 5.12.1")
```
That predates this change and is not touched by it. Worth its own bump,
but not here.
|
||
|
Tejas Chopra
|
7940c05ebf
|
feat(beacon): allowlist the routing summary key (#2818)
One line in the receiver's allowlist. No client change; the proxy's own
payload is untouched.
## Why
A routing extension sees things the proxy alone cannot, and they are all
measurements rather than opinions:
- **Empirical `min_cacheable` per provider.** Fireworks, Together and
DeepInfra publish no minimum and litellm carries no value for them, so a
router has to guess. But the number is directly observable — send prefix
length L, see whether the repeat reports cached tokens. Across enough
installs the step function falls out.
- **TTL survival.** Currently modelled as a constant.
- **Conversation length distribution.** The horizon is the only free
parameter in a cache-aware cost model, and it decides the answer: at a
900-token prefix, 1 remaining turn and 20 remaining turns route to
different models.
- **Predicted vs actual cache hits.** Every response carries
`cache_read_input_tokens`. Comparing it to what was predicted is the
only way to find out when the cost model is lying.
## What lands here
`'routing'` added to `ALLOWED_KEYS`, and the comment above the list
corrected — it claimed the set mirrors `_Session.payload()`, which is no
longer the whole story now that an extension can emit its own event
carrying one of these keys.
The ordering constraint is the reason this is its own PR: **allowlisting
is a write-side gate**, so anything sent before the key exists is
dropped and unrecoverable. This has to be deployed before any client
starts emitting it, not alongside.
## Shape of the block
Same rule as every other key — counters and model ids, no free text:
```json
"routing": {
"harness": "claude-code",
"decisions": 47, "would_change": 12, "enforced": 9, "holdout": 3,
"at_free_boundary": 4, "cross_protocol": 0,
"picked": {"claude-haiku-4-5": 12, "claude-opus-5": 35},
"requested": {"claude-opus-5": 47},
"mean_prefix_tokens": 7514,
"measured_cost": 0.0236, "modelled_cost": 0.0376,
"cache_read_tokens": 3200, "cache_write_tokens": 0,
"predicted_hits": 4, "actual_hits": 4
}
```
`measured_cost` comes from the provider's own usage; `modelled_cost`
from the router's cost function. They stay separate because the
difference is the only thing that means anything.
The extension's `reason` string is deliberately absent. It is
code-generated, so it carries no user content, but it is unbounded — it
stays out rather than being reasoned about.
`holdout` is the count of turns deliberately left unrouted as a control.
Without it the rest is observational: once a router is acting on every
request, the corpus is entirely that router's own policy.
`sample-event.json` is unchanged on purpose — it mirrors
`_Session.payload()`, which does not produce this key, and adding it
there would suggest the proxy emits it.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Patrick A
|
17cdb185bc
|
fix(proxy): graceful shutdown and reliable Ctrl+C exit (#621)
## Problems
### 1. Noisy CancelledError traceback on Ctrl+C
Every Ctrl+C produced one or more "Exception in ASGI application" ERROR
log entries with a CancelledError traceback:
```
ERROR: Exception in ASGI application
Traceback (most recent call last):
...
File "uvicorn/protocols/http/h11_impl.py", line 410, in run_asgi
result = await app(...)
...
asyncio.exceptions.CancelledError
```
### 2. Inconsistent / hung shutdown in multi-worker mode (`--workers 8`)
Workers blocked in a C-extension call (hnswlib, tree-sitter, ONNX
inference) could prevent Ctrl+C from completing because
`timeout_graceful_shutdown` defaulted to `None` (wait forever).
---
## Root causes
**Root cause A (CancelledError noise)**
uvicorn 0.40.0's `h11_impl.run_asgi()` (line 413) catches
`BaseException` — not just `Exception` — so `asyncio.CancelledError`
raised on every in-flight request at shutdown is unconditionally logged
as `ERROR: Exception in ASGI application`. This is expected behaviour
during shutdown, not a bug.
**Root cause B (hung multi-worker shutdown)**
`uvicorn.run()` was called without `timeout_graceful_shutdown`, which
defaults to `None`. This means the supervisor waits indefinitely for
in-flight requests to drain. A single request blocked in a C-extension
(e.g. hnswlib nearest-neighbour search, tree-sitter parse, ONNX
inference) prevents the whole process group from exiting.
**Root cause C (hung single-worker shutdown — lifespan unbounded
awaits)**
The lifespan `finally` block contained unbounded `await` calls to
`_beacon.stop()`, `proxy.usage_reporter.stop()`,
`proxy.traffic_learner.stop()`, and `proxy.shutdown()`. uvicorn's
`lifespan.shutdown()` calls `await self.shutdown_event.wait()` with no
timeout — that event is only set once the lifespan `finally` block
returns. Any of these awaits hanging (e.g. a reporter making a network
call) therefore requires a second Ctrl+C to force-exit.
---
## Changes
### `headroom/proxy/server.py`
1. **`_SuppressCancelledErrorFilter`** (new class, ~10 lines): a
`logging.Filter` that returns `False` for ERROR records on
`uvicorn.error` whose `exc_info[0]` is a subclass of
`asyncio.CancelledError`. Installed on
`logging.getLogger("uvicorn.error")` at the start of `run_server()`.
2. **`timeout_graceful_shutdown=10`** added to `uvicorn.run()`: forces
cancellation of any tasks still running 10 seconds after the shutdown
signal, ensuring workers blocked in C-extensions are reaped promptly.
3. **Bounded awaits in lifespan `finally` block**: a local `_timed(coro,
label, timeout)` helper wraps each shutdown step with
`asyncio.wait_for()`. Timeouts: beacon.stop 3s, usage_reporter.stop 3s,
traffic_learner.stop 3s, proxy.shutdown 5s. Each step logs a warning on
timeout/error and continues — the teardown path is now deterministic and
completes within ~15s on a single Ctrl+C.
4. **Shutdown log message** in the lifespan `finally` block:
`event=proxy_shutdown reason=signal pid=<n>` is logged as the first
action on teardown.
### `tests/test_graceful_shutdown.py` (new)
9 tests:
- 6 unit tests for `_SuppressCancelledErrorFilter` (suppresses
CancelledError at ERROR level, passes through WARNING-level
CancelledError, passes through other exceptions, handles
`exc_info=None`, handles `(None,None,None)` tuple, suppresses
subclasses)
- 1 integration test: `run_server()` installs the filter on
`uvicorn.error`
- 1 integration test: `run_server()` passes
`timeout_graceful_shutdown=10` to `uvicorn.run()`
- 1 integration test: lifespan emits `event=proxy_shutdown` on teardown
---
## Files changed
- `headroom/proxy/server.py` — filter class, bounded lifespan awaits,
graceful shutdown timeout
- `tests/test_graceful_shutdown.py` (new) — 9 tests
- `uv.lock` — dependency lockfile updated (routine sync, no dependency
changes)
- `CHANGELOG.md` — changelog entry
---
## How to verify
1. Start the proxy: `headroom proxy --port 8787 --workers 8 --memory
--code-aware ...`
2. Press Ctrl+C
3. Before: ERROR traceback for each in-flight request; second Ctrl+C
sometimes required
4. After: clean `event=proxy_shutdown reason=signal pid=...` log, then
process exits within ~15s regardless of stuck C-extensions or slow
reporters
---------
Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
|
||
|
Tejas Chopra
|
2954e37048
|
fix(beacon): split session failures by status code (#2815)
## Description
The session beacon reports `failures` as a single count, incremented
whenever a turn ends `>= 500` (`headroom/telemetry/session.py`). Across
the current corpus that reads **3,969 failures on 595,445 turns
(0.67%)** — and the number cannot answer the only question anyone asks
of it: an Anthropic `529` is the provider shedding load and there is
nothing to fix; a `500` is usually ours. Today the two are
indistinguishable, so diagnosis falls back to inference from time-of-day
curves and per-install concentration.
This counts the status alongside the total.
```json
"failures": 3,
"failure_statuses": {"529": 2, "500": 1}
```
Motivating investigation on the live corpus (0.67% of turns, 6% of
sessions, 63% of all failures from 48 installs, a 2.5% plateau at 08–11
UTC decaying to 0.03% during the fleet's busiest hour) strongly suggests
provider-side 529 after retry exhaustion — but "strongly suggests" is
exactly the gap this field closes.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **`headroom/telemetry/session.py`** — `_Session.failure_statuses`,
incremented next to `failures` in `record_outcome`. Keys are the bare
status string for the 5xx range, `"other"` beyond it. Emitted as a
sibling of `failures` in `payload()`.
- **`deploy/beacon/worker.js`** — `failure_statuses` added to
`ALLOWED_KEYS`. Without this the ingest allowlist silently drops it.
- **`deploy/beacon/sample-event.json`** — sample carries the new key in
OTLP `kvlistValue` form.
### Why no slug bounding
`skips` runs values through `_safe_slug` because they arrive as free
strings. A status code is an `int` the proxy itself produced; the `500
<= status < 600` check is what keeps a garbage value from inventing map
keys. Nothing here is user-derived, so the field stays content-free.
### Why `schema_version` stays 1
Additive, matching the precedent set by #2796, which added
`tokens.tool_saved` and the two `all_layers_*` rates without a bump.
Bumping signals a break to consumers when nothing about older rows
becomes invalid.
## Testing
- [x] Unit tests pass (`pytest`) — the module's own self-check, extended
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — see note
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m headroom.telemetry.session
ok
$ ruff check headroom/telemetry/session.py
All checks passed!
$ ruff format --check headroom/telemetry/session.py
1 file already formatted
$ mypy --python-version 3.12 headroom/telemetry/session.py
Success: no issues found in 1 source file
# --python-version 3.12 only to skip a pre-existing numpy-stub syntax error the
# repo's python_version = "3.10" triggers locally; unrelated to this diff.
$ node --check deploy/beacon/worker.js # ok
$ python -c "import json; json.load(open('deploy/beacon/sample-event.json'))" # parses
```
The self-check in `headroom/telemetry/session.py` now records two 529s
and one 500 and asserts both the total and the split:
```python
assert emitted[-1]["failures"] == 3
assert emitted[-1]["failure_statuses"] == {"529": 2, "500": 1}
```
plus `assert event["failure_statuses"] == {}` on the clean-session path.
## Real Behavior Proof
- **Environment:** macOS 25.4.0, Python 3.12 venv, this branch.
- **Exact command / steps:** drive `SessionAggregator` with three
failing outcomes and encode the payload through the same `_any_value`
the wire uses.
```text
payload: 3 {'529': 2, '500': 1}
otlp : {"kvlistValue": {"values": [{"key": "529", "value": {"intValue": "2"}},
{"key": "500", "value": {"intValue": "1"}}]}}
```
The OTLP form matches `deploy/beacon/sample-event.json` byte-for-byte in
shape, and `unwrap()` in `worker.js` turns `kvlistValue` back into a
plain object, so it lands in R2 as `{"529": 2, "500": 1}` — the same
shape as `skips`, which DuckDB reads as `MAP(VARCHAR, BIGINT)`.
- **Observed result:** as above. Verified against the live corpus that
schema evolution here is already routine — 3,836 of 3,884 existing rows
have `rates.all_layers_saved_pct = NULL` from #2796 landing mid-corpus,
and every report still runs.
- **Not tested:** the deployed Worker (no staging R2 binding locally);
`node --check` covers syntax only. The allowlist addition is one array
entry consumed by the existing `pick()`.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Screenshots (if applicable)
N/A — wire-format change, covered by the output above.
## Additional Notes
**Deploy order matters.** The Worker allowlist drops unknown keys, so
`deploy/beacon/worker.js` must be deployed *before* a client release
that emits the field — otherwise it is discarded at the door. No
corruption either way, just missing data until the Worker catches up.
**Old data is unaffected.** R2 objects are immutable NDJSON written per
request; nothing rewrites history. The corpus reader already passes
`union_by_name = true`, which fills the column with NULL for rows
written before this ships.
|
||
|
Tejas Chopra
|
c07da992dd
|
Per-request backend selection for routing extensions (#2809)
## The gap
Headroom picks its egress backend **once**, at startup:
`create_proxy_backend` returns a single `Backend` (or `None` for the
direct Anthropic path) and every request goes through it. That is the
right shape for *"run this whole proxy against Bedrock instead of
Anthropic"* and the wrong shape for *"this request is cheaper on a
different provider than the last one."*
`ModelRouter` already lets an extension change `body["model"]` per
request — but only within the protocol the request arrived in, because a
model id alone cannot move a request to another provider.
So an extension can currently **decide** something Headroom has no way
to **carry out**. This adds the missing half.
## The seam
An extension publishes a decision on the request state:
```python
request.state.headroom_route = SimpleNamespace(
model="moonshot/kimi-k2", # required
provider="moonshot", # optional; inferred from the model id if absent
reason="cheaper at this prefix length",
)
```
Headroom resolves a `LiteLLMBackend` for that provider — which is where
translation already lives — and serves **that one request** from it.
Nothing in core names any particular extension; the field is duck-typed,
so an extension does not import Headroom to talk to Headroom.
## Absent means unchanged
This is the property the tests are built around, and the reason this
should be safe to merge.
With nothing published, every path is what it was before. Advice that is
**absent, malformed, names an unknown provider, names a native provider,
or fails to build** all resolve to `self.anthropic_backend` — including
when that is `None`, which is the direct-API path and must survive. A
routing preference can never take traffic down.
## Coverage
| path | |
|---|---|
| `/v1/messages` | non-streaming + streaming |
| `/v1/chat/completions` | non-streaming + streaming |
| Responses API | untouched — does not use the backend abstraction |
Streaming is the one that matters. The resolver rewrites
`body["model"]`, so had `_stream_response_bedrock` kept reading
`self.anthropic_backend`, every streamed routed request would have sent
a foreign model id to Anthropic. Both streaming helpers now take an
optional `backend`, defaulting to the configured one.
## Details worth review
- **Validate the provider name before building.** `LiteLLMBackend`
accepts *any* provider string — the registry falls through to a generic
pass-through config — so a typo silently builds a backend that only
fails later, at request time, with an error pointing nowhere near the
typo. `_known_provider()` checks against `litellm.provider_list` first.
- **Cache per provider, and cache the failures too**, or a broken
provider name costs a construction attempt on every request. (Bedrock
construction calls out to AWS to enumerate inference profiles — it is
not free.)
- **`backend_owns_translation` now asks the per-request backend.** It
decides whether Headroom or the backend owns the `max_tokens` /
`max_completion_tokens` spelling; asking `self.anthropic_backend` would
answer "Headroom does" for a request about to be served by a translating
backend that does.
- **`_route_resolver` lives in `route_advice.py`, not on a handler
mixin.** Two mixins need it, and reaching across sibling mixins only
works by accident of how `HeadroomProxy` composes them.
## Tests
`tests/test_route_advice.py` — 20 tests, most of them asserting the
absent-means-unchanged property from a different angle.
Local runs: 20/20 on the new file; **1102 passed, 1 failed** on `-k
"openai or chat_completions or ccr"`, and **414 passed, 0 failed** on
`-k "stream or bedrock or route_advice"`. The single failure is
`test_realignment_live_multi_turn::test_ccr_marker_round_trip_live`,
which fails identically on this branch's merge-base — verified by
checking out `59314cff~1` and re-running it.
Note for anyone reproducing: `pytest-asyncio` is a declared dev
dependency but was missing from my venv, which made every `async def
test_` in the repo fail. Worth checking before diagnosing a large
failure count.
## Docs
`docs/content/docs/pipeline-extensions.mdx` gains a section on the
contract, next to the existing `x-headroom-base-url` one.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Tejas Chopra
|
0237cbffbb
|
fix(proxy): enable tool search by default and repair poisoned transcripts (#2807)
## Description Server-side tool search poisons the Claude Code transcript: once the proxy injects deferral and the model runs one search, Anthropic's `server_tool_use` + `tool_search_tool_result` pair lives in the message history forever. Upstream validates **every `tool_reference` in that history against the *current* request's `tools` array** — and Claude Code replays one transcript across requests with wildly different tools arrays (main loop: hundreds of tools; prompt-type Stop hook evaluator, `/compact`, other side-requests: a handful). Every one of those side-requests 400s with `Tool reference 'X' not found in available tools`. This PR keeps tool search **on** — it's the whole point of the feature, and the default `coding` savings profile already turned it on at proxy startup — and instead repairs the transcript per request, statelessly. The issue author's preferred fix (never inject for Claude Code clients) would disable the feature for its main audience. A session-sticky approach was also considered and rejected: it needs session state, it can't re-add ~500 tool definitions to a 5-tool side-request without erasing the savings, and it can't heal transcripts already poisoned before the upgrade. Closes #2805 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **`headroom/proxy/helpers.py`** — new `strip_unsupported_tool_search_blocks(messages, tools)`. Builds the set of names this request can resolve, drops any `tool_search_tool_result` whose `tool_reference` entries aren't all resolvable (or when no search tool is present at all), and drops the paired `server_tool_use` by `tool_use_id`. Other server tools (`web_search`, code execution) are untouched. Turns left with zero content blocks are removed rather than forwarded empty. Copy-on-write: returns the **original** `messages` object by identity when nothing was removed. - **`headroom/proxy/handlers/anthropic.py`** — runs the repair right after the injection block, so the tool just injected counts as present and the main loop is a no-op with a byte-identical prefix. Deliberately **not** gated on `HEADROOM_TOOL_SEARCH`, so transcripts poisoned before an upgrade (or before someone sets the flag to `0`) still recover. Logs and tags `router:tool_search_repair:Nblocks` when it fires. - **`headroom/proxy/handlers/anthropic.py`** — `HEADROOM_TOOL_SEARCH` now defaults to `1`. This matches the posture `seed_proxy_env_defaults()` already established for the default `coding` profile; the flip only affects entry points that never seeded. - **`docs/content/docs/proxy.mdx`** — documents on-by-default plus `HEADROOM_TOOL_SEARCH=0` as the opt-out. - **`tests/test_issue_746_tool_search.py`** — 6 tests covering the repair. ### Answering the issue's open question > we could not determine what enables it — `/proc/<pid>/environ` shows no `HEADROOM_TOOL_SEARCH` `seed_proxy_env_defaults()` calls `os.environ.setdefault("HEADROOM_TOOL_SEARCH", "1")` at proxy startup because the default savings profile is `coding`, which has `tool_search=True` (`headroom/agent_savings.py`). In-process mutation of `os.environ` never appears in the process's environ snapshot, which is why the flag looked unset. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_issue_746_tool_search.py -q 45 passed, 1 warning in 1.56s $ python -m pytest tests/test_*anthropic*.py tests/test_*tool*.py -q 4 failed, 459 passed, 2 skipped, 7 warnings in 27.40s # the 4 failures are in tests/test_bedrock_tool_result_cache_and_streaming_stats.py # and reproduce identically on this branch's merge-base with the changes stashed: # 4 failed, 9 passed, 5 warnings in 3.02s $ ruff check headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py tests/test_issue_746_tool_search.py All checks passed! $ ruff format --check <same three files> 3 files already formatted $ mypy --python-version 3.12 headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py Success: no issues found in 2 source files # --python-version 3.12 only to skip a pre-existing numpy-stub syntax error that # the repo's python_version = "3.10" setting triggers on this machine. ``` New tests: | Test | Asserts | |---|---| | `test_repair_drops_blocks_the_hook_evaluator_cannot_resolve` | small tools array → both blocks dropped, surrounding assistant text survives | | `test_repair_is_noop_on_the_main_loop` | search tool + referenced tool present → `removed == 0` and `messages is transcript` (prefix cache untouched) | | `test_repair_drops_a_turn_left_with_no_blocks` | a turn that was *only* the search round-trip is removed, not forwarded empty | | `test_repair_leaves_other_server_tools_alone` | `web_search` `server_tool_use` blocks survive | | `test_repair_is_idempotent` | second pass over a repaired transcript removes nothing | | `test_repair_strips_search_history_when_only_the_tool_is_missing` | references resolvable but no search tool in the array → still stripped | ## Real Behavior Proof - **Environment:** macOS 25.4.0, Python 3.12 venv, live `api.anthropic.com`, `claude-sonnet-4-6`, local proxy on `127.0.0.1:8799` built from this branch. - **Exact command / steps:** one request body — a poisoned transcript (`server_tool_use` + `tool_search_tool_result` referencing `AskUserQuestion`) with a **1-tool** `tools` array (`Read`), exactly the shape a Claude Code side-request replays — sent twice: once straight to `https://api.anthropic.com`, once to the proxy. ```text $ python /tmp/hr-2805-repro.py https://api.anthropic.com HTTP 400 {"type": "invalid_request_error", "message": "Tool reference 'AskUserQuestion' not found in available tools"} $ python /tmp/hr-2805-repro.py http://127.0.0.1:8799 HTTP 200 content: [{"type": "text", "text": "OK"}] ``` - **Observed result:** the exact 400 from the issue reproduces against upstream; the identical body through the proxy returns 200. The proxy's savings event for that request records `before: 133, after: 32, saved: 101` tokens — the two dropped blocks. The one-tool array is below `_TOOL_SEARCH_MIN_TOOLS = 12`, so no injection ran; the repair alone is what made the request valid. - **Not tested:** a full end-to-end Claude Code session with a real Stop hook (the synthetic replay above is the same request shape the hook evaluator produces); non-Anthropic providers, which don't have server-side tool search. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Screenshots (if applicable) N/A — proxy-side behavior, covered by the command output above. ## Additional Notes - **Cache cost is zero on the hot path.** The repair only rewrites requests whose transcripts reference tools they don't carry — request families that were 400ing anyway. The main loop takes the identity path and its prefix stays byte-identical. - **Out of scope, spotted while here:** `run-all-plugins.sh` exports `HEADROOM_TOOL_SEARCH_MIN_TOOLS=5`, but nothing in Python reads it — `_TOOL_SEARCH_MIN_TOOLS` is a hardcoded `12`. Worth a follow-up. |
||
|
Tejas Chopra
|
303e0522c4
|
fix(opencode): don't preload a missing transport shim into child processes (#2806)
## Description `headroom wrap opencode` broke third-party MCP servers in pip/wheel installs. The wrap transport plugin appended `NODE_OPTIONS=--import=<plugin dir>/../hook-shim/handler.js` to its own env (and injected it into every child it spawns), but that path only resolves in a repo checkout. Wheel installs load the standalone bundle from `headroom/providers/opencode/_dist/`, which has no `hook-shim/` sibling — the shim lives under `plugins/` and maturin only ships files under `headroom/` (pyproject.toml `python-source`/package-dir behavior). Every Node child then aborted with `ERR_MODULE_NOT_FOUND` before executing a line, including OpenCode's stdio MCP servers. OpenCode reports that as `<server> MCP error -32000: Connection closed`. Headroom's own MCP server is a Python process, so it stayed connected — which is why the breakage looked selective, and why nothing appeared in the proxy logs (the failure is entirely inside OpenCode's child process). Docker and `--no-proxy` are incidental: the plugin installs the transport on load in every wrap mode. Fix: resolve the shim only when it exists on disk, and skip the `NODE_OPTIONS` mutation otherwise. Children go direct instead of dying. Checkout builds still get child-process transport hooking, unchanged. Closes #2798 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `plugins/opencode/src/transport.ts`: `shimImportSpecifier()` returns `string | undefined`, gated on `fs.existsSync`; `installProcessEnv()` and `withShimEnv()` leave `NODE_OPTIONS` untouched when the shim is absent. - `plugins/opencode/src/transport.test.ts`: new regression test — with the shim missing, the parent's `NODE_OPTIONS` is unmodified and a spawned `npx -y firecrawl-mcp` receives no `--import`. - `headroom/providers/opencode/_dist/entry.opencode.js`: regenerated via `npm run build:standalone` (the bundle that wheel installs actually load). ## Testing - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed No Python source changed, so `pytest` / `ruff` / `mypy` are N/A here; the TypeScript equivalents were run instead. ### Test Output ```text $ npm run typecheck > tsc --noEmit (no output) $ npm test RUN v4.1.9 /private/tmp/hr-pr-2798/plugins/opencode Test Files 2 passed (2) Tests 14 passed (14) Duration 416ms # The new test is not vacuous — reverting the guard to `return shim.href` reddens it: $ npx vitest run -t "#2798" Test Files 1 failed | 1 skipped (2) Tests 1 failed | 13 skipped (14) ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0), Node v24, Bun present; both bundles loaded directly from disk. - Exact command / steps: load each built bundle, invoke the default plugin export, print `process.env.NODE_OPTIONS`, then `spawnSync(process.execPath, ["-e", "console.log('mcp server handshake ok')"])` — the same way OpenCode launches a stdio MCP server. ```text ### BEFORE (wheel layout, shim missing) ### NODE_OPTIONS: "--import=file:///…/headroom/providers/opencode/hook-shim/handler.js" child: Error [ERR_MODULE_NOT_FOUND]: Cannot find module '…/headroom/providers/opencode/hook-shim/handler.js' <-- becomes MCP -32000 ### AFTER — wheel layout (headroom/providers/opencode/_dist/) ### NODE_OPTIONS after plugin load: undefined child status: 0 | stdout: mcp server handshake ok ### AFTER — checkout layout (plugins/opencode/dist/, shim present) ### NODE_OPTIONS after plugin load: "--import=file:///…/plugins/opencode/hook-shim/handler.js" child status: 0 | stdout: mcp server handshake ok ``` - Observed result: wheel installs no longer poison child env, so Node MCP servers start; checkout builds keep the preload and still start children cleanly. - Not tested: no reproduction against a live `opencode` + codegraph/firecrawl session on Ubuntu (no OpenCode install on this machine); the child-process failure was reproduced directly instead, which is the exact mechanism behind the reported `-32000`. Docker proxy path not re-tested — it is unrelated to the fix. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Docs unchanged: this is an internal packaging/runtime bug with no documented behavior attached. Follow-up (deliberately not in this PR): wheel installs now lose child-process transport hooking rather than crashing — the same coverage they effectively had, since the preload never once loaded from a wheel. Restoring it means a standalone shim build emitted into `_dist/` plus exporting `installHeadroomTransport` from that bundle; `hook-shim/handler.js` also imports `../dist/index.js`, which does not exist in the wheel layout, so copying the file alone would not be enough. Worth doing only if something needs a subprocess's LLM traffic proxied. |
||
|
Tejas Chopra
|
6ec3e3478a
|
feat(cli,pricing): add CLI extension seam and prompt-cache TTL pricing (#2802)
## Description
Two small, independent additions. Both exist because an out-of-tree
package needed
them and neither had a home in the current API.
1. **`headroom.cli_extension`** — an entry-point group so a package can
add a
`headroom` subcommand. `headroom.proxy_extension` requires a running
FastAPI
app, so it cannot carry a read-only CLI tool, and `_register_commands()`
was a
hardcoded import list with no discovery.
2. **`headroom/pricing/cache_ttl.py`** — the prompt-cache TTL price
structure
(read `0.10x`, 5m write `1.25x`, 1h write `2.00x` of base input) plus
the
break-even share above which the 1h TTL is cheaper. `ModelPricing`
carries
`cached_input_per_1m` for reads but has no field for the *write* side.
Closes #
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/cli/extensions.py` (new) — `register_all(main)` discovers
the
`headroom.cli_extension` group. Contract: `register(main: click.Group)
-> None`.
Invoked from `_register_commands()` **last**, so built-ins are already
attached.
- Deliberately **not** opt-in gated, unlike proxy extensions: installing
the
package is the opt-in, because adding a subcommand cannot silently
change what
an existing command does. What *would* be a silent change is shadowing a
built-in, so that is detected and rolled back — a stale plugin can never
quietly
take over `headroom proxy`. Load failures and partial registrations roll
back
too, and one bad plugin never blocks another.
- `headroom/pricing/cache_ttl.py` (new) — `CACHE_READ_MULTIPLIER`,
`CACHE_WRITE_MULTIPLIERS`, `cache_write_multiplier()`,
`cache_rates_per_1m()`,
`ttl_breakeven_share()`. Ratios are derived from base input rather than
transcribed into a per-model table that would triple its columns and
drift.
- `cache_write_multiplier()` raises on an unknown TTL rather than
falling back to
the cheaper 5m rate, which would understate cost.
- `headroom/pricing/litellm_pricing.py` — three additive optional fields
on
`LiteLLMModelPricing` exposing LiteLLM's own
`cache_read_input_token_cost`,
`cache_creation_input_token_cost` and
`cache_creation_input_token_cost_above_1hr`
(present for 212 and 123 models respectively). Published rates should
win over
derived ones. All default to `None`, and `None` means "not published" —
distinct
from `0.0` meaning "free" — so every existing caller is unaffected.
- `headroom/pricing/__init__.py` — re-exports.
### Why `ttl_breakeven_share()` exists
The TTL trade has two terms and both must be counted: moving to 1h turns
idle-gap
rewrites into cheap reads **and** raises the price of every write that
still
happens. Modelling only the recovery overstates the saving. On a real
531-transcript corpus that error was **1.9x** — $1,021 claimed against
$538 real.
`test_write_premium_is_not_forgotten` pins those exact figures so the
mistake
cannot be reintroduced quietly.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`) — see note below
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_cli_extension_seam.py tests/test_pricing_cache_ttl.py tests/test_pricing_from_litellm.py -q
tests/test_cli_extension_seam.py ....... [ 25%]
tests/test_pricing_cache_ttl.py .......... [ 62%]
tests/test_pricing_from_litellm.py .......... [100%]
======================== 27 passed, 1 warning in 3.54s =========================
$ .venv/bin/ruff check headroom/cli/extensions.py headroom/cli/main.py headroom/pricing/ tests/test_cli_extension_seam.py tests/test_pricing_cache_ttl.py
All checks passed!
$ .venv/bin/mypy --python-version 3.12 headroom/cli/extensions.py headroom/pricing/cache_ttl.py headroom/pricing/litellm_pricing.py
Success: no issues found in 3 source files
```
`tests/test_pricing_from_litellm.py` is the **pre-existing** pricing
suite, included
to show the `LiteLLMModelPricing` change is non-breaking.
**mypy note.** With the repo's configured `python_version = "3.10"`,
mypy fails on
numpy's own stubs for any file that transitively reaches numpy:
```text
$ .venv/bin/mypy headroom/pricing/cache_ttl.py
.venv/lib/python3.12/site-packages/numpy/__init__.pyi:737: error: Type statement is only supported in Python 3.12 and greater [syntax]
Found 1 error in 1 file (errors prevented further checking)
```
This is pre-existing and unrelated — untouched files reproduce it
identically
(`mypy headroom/cli/doctor.py`, `mypy headroom/pricing/registry.py`).
Hence the
`--python-version 3.12` run above, which matches the interpreter
actually in use.
Worth fixing separately; not addressed here.
## Real Behavior Proof
- **Environment:** macOS 15 (darwin 25.4.0), Python 3.12.6,
`headroom-ai` 0.34.0
working tree, branch off `upstream/main`.
- **Exact command / steps:**
1. Built a separate out-of-tree package declaring
`[project.entry-points."headroom.cli_extension"] fleet =
"headroom_fleet.cli:register"`.
2. `pip install --no-deps headroom_fleet-0.1.0-py3-none-any.whl`
3. `headroom econ --help`
- **Observed result:** the subcommand registers with no configuration
and appears
in `headroom --help`:
```text
$ python -c "import importlib.metadata as m; print([e.name+' ->
'+e.value for e in m.entry_points(group='headroom.cli_extension')])"
['fleet -> headroom_fleet.cli:register']
$ headroom --help | grep econ
econ Report where local AI-coding token spend goes, and what...
$ headroom econ --help
Usage: headroom econ [OPTIONS] [COMMAND] [ARGS]...
Commands:
fix Write the recommended cache-TTL and compaction settings.
unfix Restore every setting ``econ fix`` changed, exactly as it was.
```
`headroom --help` and every built-in still work with the plugin
installed and
after it is uninstalled. Verified per-model cache rates resolve from
LiteLLM for
`claude-opus-4-8`, `claude-opus-5`, `claude-sonnet-5` and
`claude-haiku-4-5-20251001` — all exactly `1.250x` / `2.000x` / `0.100x`
of base
input, matching the derived fallback.
- **Not tested:** Windows; a plugin that raises at *import* time rather
than in
`register()` (covered by unit test with a stubbed entry point, not a
real
package); the `--python-version 3.10` mypy path, which is blocked by the
pre-existing numpy stub issue above.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
## Note for reviewers
This PR contains **only** the two additions above. Unrelated
`plugins/opencode/src/transport.ts` changes in my working tree are
deliberately
excluded and will follow separately.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
JD Davis
|
64e203931b
|
fix(deps): enforce audited transitive dependency floors (#2791)
## Description Enforces patched minimum versions for the vulnerable transitive `aiohttp` and `cryptography` dependencies so future lockfile refreshes cannot reintroduce the pip-audit failures affecting open pull requests. Related to the shared Security / pip-audit failures across open PRs. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Enforces `aiohttp>=3.14.3` for PYSEC-2026-3545/3546/3547. - Enforces `cryptography>=50.0.0` for PYSEC-2026-3552/3553/3554. - Synchronizes the project version recorded in `uv.lock` with `pyproject.toml`. ## Testing - [x] Dependency audit passes (`pip-audit`) - [x] Lockfile validation passes (`uv lock --check`) - [ ] Unit tests pass (`pytest`) - [ ] Type checking passes (`mypy headroom`) - [x] Manual verification performed ### Test Output ```text $ uv lock --check Resolved 269 packages $ uv export --frozen --no-dev --no-emit-project --no-hashes --extra all --format requirements-txt | uvx --python 3.12 pip-audit -r /dev/stdin No known vulnerabilities found ``` ## Real Behavior Proof - Environment: Local macOS worktree using CPython 3.12.13 and the frozen production dependency export. - Exact command / steps: Validated the lockfile, exported every production dependency with the `all` extra, and audited that exact export with pip-audit. - Observed result: The lockfile resolved successfully and pip-audit reported no known vulnerabilities. - Not tested: Publishing or deployment; the refreshed GitHub CI suite covers builds, wheels, containers, security scans, and platform tests. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project style guidelines - [x] I have performed a self-review of my changes - [x] No explanatory code comments are required beyond the PYSEC constraint annotations - [x] Documentation changes are not required for transitive security floors - [x] My changes generate no new local warnings - [x] The dependency audit proves the security fix is effective - [ ] Full repository tests are delegated to GitHub CI - [x] I did not edit `CHANGELOG.md`; release-please owns it ## Screenshots (if applicable) N/A — dependency metadata only. ## Additional Notes The earlier Docker-native failure was a transient Docker Hub HTTP 502 while resolving `python:3.13-slim`; the build did not reach project code. A fresh CI suite is running on the current head. Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net> |
||
|
JD Davis
|
f236ef2e31
|
test(ccr): cross SQLite max lifetime boundary (#2794)
## Description
Fixes the failing Rust test on `main` after #2669 made SQLite CCR
entries valid at the exact TTL boundary. The integration test waited
only 3.3 seconds for a three-second ceiling; unix-second truncation can
represent that as exactly three seconds, so the entry is correctly still
valid. The test now crosses a guaranteed four-second elapsed boundary.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Extend the max-lifetime test's access loop from four to five 700 ms
gaps.
- Document why four gaps can land on the valid equality boundary and why
five are deterministic.
- Leave production SQLite TTL behavior and defaults unchanged.
## Testing
- [x] Unit tests pass (`cargo test -p headroom-core --test
ccr_backends`)
- [x] Formatting passes (`cargo fmt --all -- --check`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
cargo test -p headroom-core --test ccr_backends
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
5 consecutive repetitions of the previously failing test:
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 11 filtered out
```
## Real Behavior Proof
- Environment: macOS, Rust workspace at
`
|
||
|
Tejas Chopra
|
e9a24f3ec1
|
fix(beacon): report all-layers savings, not context-compression only (#2796)
## Description
`rates.saved_pct` and `rates.yield_pct` in the beacon payload divide
`tokens_saved` by `original` / `attempted`. Tool-schema deferral never
lands in either denominator — `outcome.py` says so explicitly, and
`tokens.tool_saved` exists precisely because of it — so every beacon
rate silently reports context compression only.
On a tool-heavy fleet that is not a rounding difference. Across the
first 516 sessions in the corpus the beacon reads **2.80%** where the
dashboard headline for the same traffic reads **12.82%**: 157.6M context
tokens vs 803.9M all-layers, with 646.3M of tool-schema deferral missing
from the ratio.
`headroom/proxy/server.py` already resolved this for the dashboard in
#2737 — `savings_percent` is `all_layers_saved / (input +
all_layers_saved)` and `active_savings_percent` puts tool savings on
both sides of the ratio. The beacon was never brought along. This does
that.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/telemetry/session.py`: add `rates.all_layers_saved_pct` and
`rates.all_layers_yield_pct`, computed the way `server.py` builds
`savings_percent` / `active_savings_percent` — tool savings added to
**both** sides, since deferred schemas were attempted work that
succeeded whole.
- `headroom/telemetry/session.py`: extend the `demo()` self-test to
assert both new rates against the existing tool-heavy fixture.
- `deploy/beacon/query.sh`: add an `all_layers_pct` column to the fleet
summary, so the reader stops showing the understated number too.
**Kept alongside `saved_pct` rather than folded into it.** Every row
already in the corpus means context-only under that name; redefining it
would make old and new rows non-comparable with no field to tell them
apart.
**`SCHEMA_VERSION` deliberately stays at 1.** The change is purely
additive, nothing reads the field, and the query uses `union_by_name =
true`, so old and new rows mix cleanly. Happy to bump it if maintainers
want the marker.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m headroom.telemetry.session
ok
$ python -m pytest tests/test_savings_tool_search_aggregation.py tests/test_outcome_dual_ruler_funnel.py -q
FAILED tests/test_outcome_dual_ruler_funnel.py::test_ledger_delta_stays_on_the_local_ruler
FAILED tests/test_outcome_dual_ruler_funnel.py::test_ledger_falls_back_to_billed_when_local_omitted
2 failed, 5 passed, 3 warnings in 4.42s
$ ruff check headroom/
All checks passed!
$ mypy --python-version 3.12 headroom/telemetry/session.py
Success: no issues found in 1 source file
```
The two `test_outcome_dual_ruler_funnel.py` failures are **pre-existing
on `main`, not caused by this PR** — verified by `git stash`-ing the
change and re-running:
```text
$ git stash && python -m pytest tests/test_outcome_dual_ruler_funnel.py -q
FAILED tests/test_outcome_dual_ruler_funnel.py::test_ledger_delta_stays_on_the_local_ruler
FAILED tests/test_outcome_dual_ruler_funnel.py::test_ledger_falls_back_to_billed_when_local_omitted
2 failed, 3 passed, 3 warnings in 3.71s
```
## Real Behavior Proof
- **Environment:** macOS 25.4.0 arm64, Python 3.12.6, repo `.venv`,
branch rebased on `upstream/main` @ `
|
||
|
Parideboy
|
b6f9877c78
|
fix(tokenizer): coerce non-string tool_call fields before counting (#2801)
## Description
`/v1/compress` returned HTTP 503 with an unhandled `TypeError` when a
message carried a `tool_calls[].function.arguments` value that was not a
string. `arguments` is a JSON *string* per the OpenAI spec, but
OpenAI-compatible upstreams do emit `None` or a raw object there, and
every token counter passed the value straight to `tiktoken.encode()`.
Because the malformed message persists in conversation history, the
failure was sticky: every later request replaying that history failed
too, regardless of destination provider.
Reported in #2782. The exact repro in that issue (`arguments: null`) no
longer raises — `count_text` grew a falsy guard since 0.33.0 — but the
root cause is still live for any *truthy* non-string, which I reproduced
against all four counters on `main` before the fix.
## Type of Change
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Documentation update
- [ ] Refactor / internal cleanup
## Changes Made
- `headroom/tokenizers/base.py`: new `coerce_countable_text()`. Strings
pass through untouched, `None` counts as nothing, dict/list/tuple are
JSON-serialized, anything else falls back to `str()`. The serialized
form is capped at 200K chars so a malformed upstream can't turn a token
*estimate* into a multi-megabyte encode.
- Applied at the tool-call field sites (`function.name`,
`function.arguments`, `id`, and the legacy `function_call`) in
`tokenizers/base.py`, `tokenizers/tiktoken_counter.py`,
`providers/openai.py`, `providers/openai_compatible.py`,
`providers/anthropic.py`.
- Guarded `{"function": null}` / `{"id": null}`, which reach the same
encode path.
- New test file `tests/test_tool_call_arguments_not_a_string.py` (11
cases).
Serializing dicts rather than the one-liner suggested in the issue
(`str(func.get("arguments") or "")`) is deliberate: `str()` on a dict
yields Python repr with single quotes, which is not what the upstream
would have billed, and it is unbounded.
## Testing
- [x] Existing tests pass
- [x] New tests added for the fix
- [ ] Manual testing performed
New tests:
```
$ python -m pytest tests/test_tool_call_arguments_not_a_string.py -q
tests\test_tool_call_arguments_not_a_string.py ........... [100%]
============================= 11 passed in 0.40s ==============================
```
Surrounding tokenizer/provider suites, unchanged:
```
$ python -m pytest tests/test_tool_call_arguments_not_a_string.py tests/test_tokenizer.py \
tests/test_tokenizers.py tests/test_tokenizers \
tests/test_provider_counter_content_blocks.py tests/test_provider_tokenizer_one_ruler.py -q
tests\test_provider_counter_content_blocks.py ............ [ 91%]
tests\test_provider_tokenizer_one_ruler.py ......... [100%]
======================= 91 passed, 14 skipped in 1.50s ========================
```
Lint/format on the touched files:
```
$ ruff check <touched files> && ruff format --check <touched files>
All checks passed!
6 files already formatted
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.13.11, pytest 9.1.1, repo at
`upstream/main` (
|