mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
2682 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
JD Davis
|
36cc800162
|
fix(copilot): honor corporate TLS for token refresh (#3246)
## Description Copilot OAuth/device-auth, user-info, and short-lived token exchange requests used `urllib.request.urlopen` directly, bypassing the corporate CA and X.509 strictness configuration already applied to Headroom's upstream HTTP client. Reuse that TLS resolver for every Copilot GitHub request so token refresh works behind TLS inspection. Closes #3244 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added a `urlopen` adapter for Headroom's existing corporate TLS resolver. - Routed Copilot device authorization, user-info, and token exchange through it. - Added a regression test proving token exchange receives the configured TLS context. ## Testing - [ ] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text pytest tests/test_copilot_auth.py tests/test_ssl_context.py tests/test_copilot_vscode_completions_routing.py -q 202 passed in 2.45s ruff check . --exclude .codex-worktrees All checks passed! ruff format --check . --exclude .codex-worktrees 1449 files already formatted mypy headroom/copilot_auth.py headroom/proxy/ssl_context.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12.13, OpenSSL 3.5.0; local HTTPS server signed by a private test CA; `REQUESTS_CA_BUNDLE` set to that CA. The exercised request path is the same adapter used by Copilot token exchange. - Exact command / steps: generated a one-day localhost certificate, started an in-process TLS HTTP server, set only `REQUESTS_CA_BUNDLE` to the private CA, and called `headroom.copilot_auth._urlopen(Request(local_https_url), timeout=5)`. - Observed result: `corporate_ca_https_status=200` and `response_body=ok`. - Not tested: a real Cisco/Zscaler interception appliance, macOS, or a live GitHub Copilot Business token (no corporate network/account is available locally). ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: N/A. - Stable/default behavior changed: Only Copilot GitHub requests when a custom CA or `HEADROOM_TLS_STRICT=0` produces an explicit TLS context; default `urlopen` behavior remains unchanged otherwise. - Kill switch / disable path: Unset `SSL_CERT_FILE`, `REQUESTS_CA_BUNDLE`, or `NODE_EXTRA_CA_CERTS` and leave `HEADROOM_TLS_STRICT` enabled. - Unsafe override required: No. - Qualification impact: Restores existing documented corporate TLS settings for Copilot authentication traffic. - Rollback path: Revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing relevant unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A. ## Additional Notes The issue attributes token exchange to the Rust extension, but current `main` performs it in Python via `urllib`. The direct `urllib` path was the trust-configuration gap. Full-suite execution was also started locally; unrelated environment-dependent failures appeared outside the changed Copilot/TLS scope, while all focused tests pass. |
||
|
gglucass
|
c2fbb4eed0
|
test(agno): follow the metrics dataclass move in agno 3.0.0 (#3260)
## Description
agno 3.0.0 (released 2026-08-24) removed the `agno.models.metrics`
module; the per-message usage dataclass now lives at `agno.metrics`
under the name `MessageMetrics`. The mock fixtures in
`tests/test_integrations/agno/test_model.py` import the old path inline,
and the `test-agno` CI job installs `wheel[dev,agno]` with an unpinned
`agno>=1.0.0`, so it now resolves agno 3.0.0 and fails on every branch -
including `main` (see the CI run for #3239's merge commit) and
currently-open PRs.
This resolves the class once at module level: prefer the pre-3 location,
fall back to `MessageMetrics` on agno >= 3. `MessageMetrics` exists
under both names in 2.x and the constructor kwargs the fixtures use
(`input_tokens`, `output_tokens`, `total_tokens`) are unchanged, so both
major versions stay green. Tests-only change; the runtime integration
(`headroom/integrations/agno/`) never imported the removed module - the
other 76 agno tests already pass on 3.0.0.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Replace the two inline `from agno.models.metrics import Metrics`
imports in the `mock_agno_model` fixture with one module-level compat
resolution that tries `agno.models.metrics.Metrics` (agno < 3) and falls
back to `agno.metrics.MessageMetrics as Metrics` (agno >= 3).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ uv run --frozen --extra dev --extra agno --with agno==3.0.0 pytest tests/test_integrations/agno/ -q
================== 79 passed, 5 skipped, 1 warning in 19.87s ===================
$ uv run --frozen --extra dev --extra agno --with agno==2.9.0 pytest tests/test_integrations/agno/ -q
================== 79 passed, 5 skipped, 1 warning in 11.60s ===================
$ ruff check tests/test_integrations/agno/test_model.py
All checks passed!
$ ruff format --check tests/test_integrations/agno/test_model.py
1 file already formatted
```
## Real Behavior Proof
- Environment: macOS 15 (arm64), CPython 3.12, uv-managed venv; branch =
upstream/main `
|
||
|
Abhay Singh
|
6262c28a48
|
fix(memory/graph): skip a corrupt row instead of aborting a whole graph scan (#3239)
## Description
`SQLiteGraphStore._row_to_entity` and `_row_to_relationship` parse
stored text back into objects with no error handling:
```python
properties=json.loads(row["properties"]),
created_at=datetime.fromisoformat(row["created_at"]),
metadata=json.loads(row["metadata"]),
```
These run inside row loops in the multi-row scans — `get_relationships`
and `query_subgraph` (both the relationship loop and neighbour-entity
expansion). A single unparseable row — from a partial write, a manual
edit, or a bad migration — raises `ValueError` (`JSONDecodeError`/bad
ISO timestamp) *inside the loop*, aborting the **entire** query and
taking unrelated, perfectly good edges/nodes down with it.
Reproduction (A→B and A→C both valid; corrupt only A→B's `properties`):
```python
# corrupt one row out-of-band
con.execute("UPDATE relationships SET properties='{oops' WHERE target_id=?", (b.id,))
# BEFORE: both of these raise JSONDecodeError, even though A->C is fine:
await store.get_relationships(a.id)
await store.query_subgraph([a.id], max_hops=1, direction=OUTGOING)
```
This is the same "one bad row breaks the whole scan" robustness gap
already fixed for the CCR store (`cache/backends/sqlite.py`) and the
vector adapter (`memory/adapters/sqlite_vector.py`); the graph adapter
was the remaining store with unguarded row parsing.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/memory/adapters/sqlite_graph.py`:
- `_row_to_entity` / `_row_to_relationship` now return `... | None`,
wrapping construction in `except (ValueError, TypeError, KeyError)` and
returning `None` (with a `logger.warning`) on a corrupt row.
- Multi-row call sites skip `None`: `get_relationships`,
`query_subgraph` (initial entities, relationship loop, neighbour
expansion), and the per-user entity listing. The single-row `get_entity`
/ `get_entity_by_name` already return `Entity | None`, so a corrupt row
now reads as "not found" rather than raising.
- Added a module `logger`.
- `tests/test_sqlite_graph_store.py`: added
`test_one_corrupt_row_does_not_abort_a_multi_row_scan` — corrupts one
relationship row out-of-band and asserts `get_relationships` returns the
one good edge and `query_subgraph` completes with `{A, C}`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
### Test Output
```text
tests/test_sqlite_graph_store.py::...one_corrupt_row_does_not_abort_a_multi_row_scan -> passes with fix, FAILS without it (verified via git stash)
uvx ruff@0.16.2 check headroom/memory/adapters/sqlite_graph.py tests/test_sqlite_graph_store.py -> All checks passed!
uvx mypy@1.20.2 headroom/memory/adapters/sqlite_graph.py -> Success: no issues found in 1 source file
```
(Note: this test file has pre-existing, unrelated failures/errors on
`main` on Windows — `TestSQLiteGraphStoreMemoryTrackerIntegration` plus
temp-file teardown `WinError 32` in the `NamedTemporaryFile`-based
fixtures. Verified identical counts before and after this change; my new
test uses `tmp_path` and is unaffected.)
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: built A→B and A→C edges, corrupted A→B's
`properties` to invalid JSON via a direct sqlite connection, then called
`get_relationships(A)` and `query_subgraph([A], OUTGOING)`. Before the
fix both raised `JSONDecodeError`; after the fix `get_relationships`
returns just the A→C edge and `query_subgraph` returns entities `{A, C}`
with one relationship, skipping the corrupt row.
- Observed result: corrupt rows are skipped (with a warning log); valid
rows in the same scan are returned normally.
- Not tested: no corruption occurs in normal operation; the corrupt row
is produced out-of-band to exercise the guard (matching the real
triggers: partial write, manual edit, migration).
## Runtime Rollout Safety
- Rollout-managed feature(s): none. This is a SQLite graph-store read
path in the memory subsystem, not a rollout-channel-gated runtime
feature.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no for well-formed data — every valid
row parses and is returned exactly as before. Only the
previously-crashing corrupt-row case changes, from an aborted query to a
skipped row.
- Kill switch / disable path: N/A.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this PR.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal behavior)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
|
||
|
Raúl
|
4408e88106
|
fix(proxy): protect file reads from lossy compression on the Responses API path (Copilot view + HEADROOM_PROTECT_READS) (#3238)
## Description On the OpenAI Responses API path (used by `headroom wrap copilot` and Codex), fresh file reads were lossy-compressed one turn after production, so the model saw its own just-read file content garbled (Kompress word-dropping) and had to re-read it — the exact turn inflation `HEADROOM_PROTECT_READS` was built to prevent on the chat/Anthropic path. Two gaps combined: 1. Copilot CLI's `view` tool (its file-read tool) was not in `DEFAULT_EXCLUDE_TOOLS` — the set only covered Claude-Code names (`Read`, `Write`, …). 2. `_compress_openai_responses_live_text_units_with_router` (`headroom/proxy/handlers/openai.py`) protected only excluded tool *names* and never implemented the `HEADROOM_PROTECT_READS` read-command detection that `ContentRouter.apply()` has — so `bash` reads like `nl -ba FILE | sed -n '1,75p'` were lossy-compressed even with the `coding` profile's `protect_reads=True`. Fix (design adversarially reviewed with gpt-5.6-sol before implementation; verdict "correct with modifications" — all modifications adopted): - `view` added to **both** `DEFAULT_EXCLUDE_TOOLS` and `DEFAULT_VERBATIM_EXCLUDE_TOOLS` — byte-exact contract: no lossy compression, no lossless JSON rewrite, no cross-turn dedup fold. - Responses units path now ports the read-command guard: the producing command is normalized from both wire shapes (`function_call.arguments`, `local_shell_call.action` argv/string) via the shared `_tool_call_command_text`; each output is content-gated by `_read_output_should_be_protected` (lockfiles/JSON/logs/search stay compressible); protected ids are unioned into the dedup protection set. - Shared `read_protection_enabled()` env helper extracted in `content_router.py`, used by both paths. - Latent debug-path defect fixed (unbound `fold` when an excluded tool's output is a content-part list and debug logging is enabled). Follow-up (not in scope): Rust Responses path (`crates/headroom-core/src/transforms/live_zone.rs`) currently only protects `headroom_retrieve` — needs parity before that runtime becomes default. Closes #3237 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/config.py` — `view` in both exclusion sets. - `headroom/proxy/handlers/openai.py` — read-command protection for the Responses units path; dedup shield; debug-path fix. - `headroom/transforms/content_router.py` — shared `read_protection_enabled()` helper (both paths). - `tests/test_openai_responses_read_protection.py` — 16 regression tests. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # Before fix (first commit on this branch, repro-only): 2 failed, 1 passed # view read compressed; bash nl|sed read compressed despite HEADROOM_PROTECT_READS=1 # After fix: $ uv run pytest tests/test_openai_responses_read_protection.py -q 16 passed (incl. content-gate release, string-form local_shell_call, debug paths, scan robustness) $ uv run pytest tests/test_openai_responses_compression_units.py tests/test_responses_cross_turn_dedup.py \ tests/test_lossless_excluded_compaction.py tests/test_observed_wire_shapes.py \ tests/test_content_router_exclude_tools.py tests/test_content_router_compact_json.py -q 69 passed in 3.85s $ uv run ruff check <changed files> && uv run ruff format --check <changed files> All checks passed! 4 files already formatted $ uv run mypy headroom/config.py headroom/transforms/content_router.py headroom/proxy/handlers/openai.py Success: no issues found ``` ## Real Behavior Proof - Environment: macOS (Darwin), Python 3.13.7, headroom proxy 0.37.0-dev, `HEADROOM_STACK=wrap_copilot`, savings profile `coding` (effective per proxy banner), model `gpt-5.6-luna` via GitHub Copilot API. - Exact command / steps: incident forensics on Copilot CLI session `5487d36f-3e7d-48b0-a56a-a92e4969c17b` (events.jsonl tool results byte-matched to proxy log compression units), then the failing→passing repro above. - Observed result: (pre-fix proxy log `~/.headroom/logs/proxy-8794.log`) ```text 08:46:34 [hr_1787553986_000014] WS /v1/responses slow compression unit … strategy=text … bytes=3857 … tokens_saved=235 08:46:34 [hr_1787553986_000014] … strategy=text … bytes=6852 … tokens_saved=413 08:46:34 [hr_1787553986_000014] … strategy=text … bytes=3079 … tokens_saved=178 08:46:47 [hr_1787554001_000015] … strategy=text … bytes=6924 … tokens_saved=478 08:46:47 [hr_1787554001_000015] … strategy=text … bytes=4418 … tokens_saved=305 ``` Byte sizes match the session's `view` (3857/6852/3079) and `nl|sed` (6924/4418) tool results exactly. Post-fix, those payload shapes are byte-exact through `_compress_openai_responses_live_text_units_with_router` (asserted by the regression tests over the same wire shapes). - Not tested: full `pytest tests/` run (upstream suite has pre-existing order-dependent failures — 7 failed on clean `main` under `-k "content_router or read or protect"` — and a pre-existing `litellm` import error in `tests/test_memory_eval.py`; the 5 additional failures in that selection with my branch pass in isolation and also fail on clean main under the same selection); live end-to-end with a running Copilot wrap (unit-level wire-shape coverage instead); Rust core path (follow-up). ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: N/A - Stable/default behavior changed: yes — `view` outputs and `HEADROOM_PROTECT_READS`-covered bash read outputs stay verbatim on the Responses path (fidelity improvement; slightly fewer tokens saved) - Kill switch / disable path: `HEADROOM_PROTECT_READS=0` restores old bash-read behavior; `HEADROOM_EXCLUDE_TOOLS` overrides tool exclusion - Unsafe override required: no - Qualification impact: none - Rollback path: revert the commit; no state/migration ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (N/A, no user-facing docs for this internal guard) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` |
||
|
inix
|
b9d7dcc3da
|
fix(proxy): make output-savings flush atomic and keep it off the event loop (#3231)
## Description
The output-shaper's periodic savings-ledger flush ran synchronously on
the asyncio event loop: every 25th shaped request,
`emit_request_outcome` performed a full ledger reload (file read +
`json.loads`) followed by a `json.dumps` + in-place `write_text`, with
no await or executor. The write was also non-atomic, so a crash
mid-write truncated the existing ledger, and `SavingsLedger.load()`
silently swallowed the resulting decode error — corrupted history was
indistinguishable from no history yet.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `emit_request_outcome` now runs `record_from_labels` +
`estimate_request_savings` together on a worker thread via one
`asyncio.to_thread` call — both take the recorder lock, and the periodic
flush holds that lock across disk I/O, so nothing touches it from the
event loop anymore.
- `SavingsLedger.save()` writes through the existing
`headroom.fsutil.write_text` helper (temp file in the target directory,
fsync, atomic `os.replace`, temp cleanup on failure) instead of a
truncating in-place write.
- `SavingsLedger.load()` logs a warning naming the unreadable ledger
file and still fails open with an empty ledger.
- Added `TestFlushDurability` to `tests/test_output_savings.py`:
failed-save intactness (+ no temp residue), corrupt-file warning, and
off-loop-thread assertions.
## Testing
<!-- Check what you actually ran, then paste the real command output
below. -->
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_output_savings.py tests/test_output_shaping_rollup.py tests/test_output_savings_cli.py -q
============================== 49 passed in 1.38s ==============================
$ ruff check headroom/proxy/output_savings.py headroom/proxy/outcome.py tests/test_output_savings.py
All checks passed!
$ ruff format --check headroom/proxy/output_savings.py headroom/proxy/outcome.py tests/test_output_savings.py
3 files already formatted
$ mypy headroom/proxy/output_savings.py headroom/proxy/outcome.py
Success: no issues found in 2 source files
Fail-before evidence (the three files checked out at upstream/main, fix reverted):
$ python -m pytest tests/test_output_savings.py::TestFlushDurability -q
FAILED tests/test_output_savings.py::TestFlushDurability::test_crash_mid_write_leaves_previous_ledger_intact - KeyError: 'opus|code|m|tools'
FAILED tests/test_output_savings.py::TestFlushDurability::test_corrupt_ledger_warns_and_starts_empty - AssertionError: corrupt ledger was swallowed silently
FAILED tests/test_output_savings.py::TestFlushDurability::test_emit_request_outcome_flushes_off_the_loop_thread - assert False
3 failed in 0.49s
```
## Real Behavior Proof
- Environment: macOS 15 (arm64), CPython 3.13.13, project venv; branch
`fix/output-savings-atomic-offload` = upstream/main `
|
||
|
Tejas Chopra
|
f27f235032
|
fix(wrap): stop concurrent wrap sessions clobbering settings.local.json (#3232)
## Description Several `headroom wrap` sessions in one project each write the proxy URL into `.claude/settings.local.json` and restore it on exit. That read-modify-write was unsynchronised. The write itself is atomic so the file never tears, but the updates were still lost against each other: - **Live sessions were silently unrouted.** The first session to exit deleted the key while its siblings were still running. They kept working, but their traffic stopped going through the proxy — no error, no warning, no savings. - **A dead proxy was written back into the project.** A session that started second captured the *first* session's proxy URL as "the original", so its exit restored a URL pointing at a port that was already gone. Every later session in that project then failed to connect. - **SIGTERM/SIGHUP never ran the restore at all.** `cleanup` was registered as the handler, but a Python signal handler that returns normally does not unwind the stack — under PEP 475 the interrupted `waitpid` is simply retried. The `finally` block that restores `settings.local.json` never ran, while the handler had already terminated the proxy underneath a child that was still alive. Closes #3205 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - **`_wrap_settings_lock`** — an exclusive OS lock (flock / `msvcrt.locking`) held across the settings read-modify-write. A workspace that cannot hold lock state degrades to the previous behaviour rather than failing, matching `_proxy_start_lock`. - **`.headroom_wrap_owners.json`** — a sidecar recording, per env key, the true pre-wrap `original` plus the live sessions holding it. The first writer records the original; later writers inherit it and are flagged `inherited`, so no session restores a value it did not observe first-hand. A session exits without restoring while a sibling still holds the key. Dead holders are pruned with the same conservative PID+identity liveness the proxy-client markers use, so a SIGKILLed session cannot wedge the key. - **`unwrap` passes `force=True`** — unwrap is the user explicitly asking for their settings back, so it drops every claim instead of deferring to a live sibling and silently printing success while leaving the proxy URL in the file. - **The #2221 self-heal passes `dead_ports`** — a wrapper process can outlive its proxy (proxy alone SIGKILLed). Its claim would otherwise veto the self-heal and leave `ANTHROPIC_BASE_URL` pointing at a port just proven dead. - **`_rehome_wrap_marker`** — the wrap marker has one slot, won by the last writer. When that writer exits while a sibling still owns the key, the marker is rewritten to describe the survivor (carrying the record's true original), so the survivor keeps its #2221 self-heal record instead of being left with a marker describing a dead process. - **`_exit_on_signal`** replaces `cleanup` as the SIGTERM/SIGHUP handler. Raising `SystemExit` unwinds, so the settings restore actually runs and cleanup happens exactly once from `finally`. - **`_proxy_start_lock` now shares `_locked_file`** with the new settings lock rather than carrying a second verbatim copy of the platform branches. ## Testing - [x] Unit tests pass (`pytest`) — full suite, 11518 passed / 588 skipped - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed `tests/test_wrap_concurrent_settings.py` (14 tests) covers: a sibling exit leaving survivors routed, the last session out restoring the true original, a pre-existing user URL surviving the whole cycle, three sessions in every exit order, a crashed session not wedging the key, forced unwrap past a live session, a holder that outlived its proxy not vetoing the self-heal, marker rehoming, and the signal-handler unwind. ### Test Output ```text $ uv run pytest tests/test_cli/test_unwrap_claude.py tests/test_cli/test_wrap_claude_base_url.py \ tests/test_cli/test_wrap_claude_finally_unbound.py tests/test_cli/test_wrap_claude_vertex_proxy_env.py \ tests/test_cli/test_wrap_claude.py tests/test_cli/test_wrap_dead_marker_selfheal.py \ tests/test_cli/test_wrap_helpers.py tests/test_cli/test_wrap_stale_marker.py \ tests/test_cli/test_wrap_persistent.py tests/test_wrap_concurrent_settings.py tests/test_cli_doctor.py -q tests/test_wrap_concurrent_settings.py .............. [ 72%] tests/test_cli_doctor.py ............................................... [ 89%] ............................... [100%] ============================= 285 passed in 3.01s ============================== $ uv run pytest tests/ -q ======== 11518 passed, 588 skipped, 6036 warnings in 1831.34s (0:30:31) ======== $ uv run ruff check . All checks passed! $ uv run mypy headroom Success: no issues found in 527 source files ``` ## Real Behavior Proof - **Environment:** macOS 15 (Darwin 25.4.0), Python 3.12.13, repo venv, Claude provider path (`ANTHROPIC_BASE_URL` in `.claude/settings.local.json`). - **Exact command / steps:** a script spawning **two real OS processes** — no mocks, real PIDs, real files — that call the same `_write_claude_wrap_base_url` / `_restore_claude_wrap_base_url` helpers `wrap claude` uses. The project starts with a real user gateway already set. Session A (port 8787) starts, session B (port 8788) starts 0.7s later, A exits while B is still running, then B exits. Run identically on `main` and on this branch. **Before (on `main`) — both bugs visible:** ```text start : {"ANTHROPIC_BASE_URL": "https://my-gateway.example.com"} session port=8787 started, remembers previous='https://my-gateway.example.com' session port=8788 started, remembers previous='http://127.0.0.1:8787' both sessions running : {"ANTHROPIC_BASE_URL": "http://127.0.0.1:8788"} session port=8787 exited after FIRST session exits : {"ANTHROPIC_BASE_URL": "https://my-gateway.example.com"} session port=8788 exited after LAST session exits : {"ANTHROPIC_BASE_URL": "http://127.0.0.1:8787"} ``` Session B is still running, but after A exits the proxy URL is gone from under it — B is unrouted with no error. And the final state is `http://127.0.0.1:8787`: a dead proxy left permanently in the user's project, with their real gateway lost. **After (this branch):** ```text start : {"ANTHROPIC_BASE_URL": "https://my-gateway.example.com"} session port=8787 started, remembers previous='https://my-gateway.example.com' session port=8788 started, remembers previous='http://127.0.0.1:8787' both sessions running : {"ANTHROPIC_BASE_URL": "http://127.0.0.1:8788"} session port=8787 exited after FIRST session exits : {"ANTHROPIC_BASE_URL": "http://127.0.0.1:8788"} session port=8788 exited after LAST session exits : {"ANTHROPIC_BASE_URL": "https://my-gateway.example.com"} ``` B stays routed after A exits, and the last session out restores the user's real gateway. - **Observed result:** matches the intent on both counts — no unrouting, no dead proxy residue, user's pre-existing URL preserved. - **Not tested:** Windows (`msvcrt.locking`) — the lock and dead-holder pruning are exercised on POSIX only; the Windows branch is the same code path `_proxy_start_lock` has shipped with. No live end-to-end run against a real Anthropic endpoint with two concurrent `claude` CLIs; the proof above drives the same helpers out of two real processes instead. Foundry/Vertex key variants are covered by unit tests, not by a live run. Real SIGTERM/SIGHUP delivery to a running `wrap claude` was not exercised end to end — the handler's unwind is covered by a unit test, and full signal delivery would need a spawned and killed subprocess, which the existing #1768 test also declined to do. ## Runtime Rollout Safety - **Rollout-managed feature(s):** none — this is an unconditional correctness fix on the wrap settings path. - **Minimum rollout channel:** n/a. - **Stable/default behavior changed:** yes, three ways. (1) A wrap session exiting while a sibling holds the key now leaves the key in place instead of removing it. (2) SIGTERM/SIGHUP now unwinds, so the child CLI is terminated by `subprocess.run`'s cleanup rather than being left running against a torn-down proxy. (3) Two new sidecar files appear next to `settings.local.json`: `.headroom_wrap_owners.json` (removed when the last holder exits) and `.headroom_wrap_settings.lock` (retained by design — deleting a live lock file creates an inode-replacement race). - **Kill switch / disable path:** none. A workspace where the lock file cannot be created degrades to the previous unsynchronised behaviour automatically. - **Unsafe override required:** no. - **Qualification impact:** none beyond the wrap settings path. - **Rollback path:** revert the commit; the sidecar files are ignored by older versions and can be deleted safely. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes - The ownership record is keyed per env key, so `ANTHROPIC_BASE_URL`, the Foundry/Vertex variants and the tool-search entry are tracked independently. - Documentation: the behaviour is documented in the helper docstrings rather than user-facing docs — the sidecar files are internal state a user never configures. - Follow-up worth considering: `.headroom_wrap_settings.lock` is intentionally never deleted (matching `_proxy_start_lock`'s retention rationale), so it stays in `.claude/` after `unwrap`. Removing it safely needs a separate think about the inode-replacement race. Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Rod Boev
|
701e4616d9
|
fix(kimi): route managed Kimi Code through the proxy (#3223)
## Description Managed Kimi Code reads KIMI_CODE_BASE_URL while headroom wrap kimi previously supplied only KIMI_BASE_URL. The managed client can therefore keep its direct endpoint while the wrapper appears healthy. Emit both provider-owned keys and recompute them through the existing launch callback at the proxy's actual port. Preserve the legacy route and unrelated wrappers. Closes #3207 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (feature that would cause existing behavior to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Set KIMI_CODE_BASE_URL and KIMI_BASE_URL from one project-aware proxy URL. - Recompute both values and their display lines through the Kimi configure_launch callback after port fallback. - Remove the generic display rewrite from _launch_tool so other wrappers retain their base behavior. - Add production-boundary child, fallback-port, legacy-preservation, and non-Kimi negative-space tests. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_cli/test_wrap_kimi.py -q`) - [x] Linting passes (`uv run ruff check .`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for the regression - [x] Manual testing performed through the production subprocess boundary ### Test Output ```text uv run pytest tests/test_cli/test_wrap_kimi.py -q 10 passed in 0.40s uv run pytest tests/test_cli/test_wrap_grok.py -q 2 passed uv run ruff check . All checks passed! uv run ruff format . --check 1534 files already formatted git diff --check ``` ## Real Behavior Proof - Environment: Windows, isolated Kimi wrapper subprocess harness. - Exact command / steps: launch a contract-compatible child through the Kimi wrapper; exercise requested and fallback ports, project prefixes, legacy selection, and a non-Kimi wrapper. - Observed result: the child receives the effective project-aware proxy URL in both Kimi keys; the displayed URL matches it after fallback; legacy and non-Kimi behavior remain unchanged. - Not tested: live authenticated Kimi Code managed request ## Runtime Rollout Safety - Rollout-managed feature(s): None; managed Kimi Code routing is selected by the existing wrapper mode. - Minimum rollout channel: Stable; no staged rollout mechanism exists for this wrapper path. - Stable/default behavior changed: Yes, managed Kimi Code launches now receive the effective proxy URL in both provider-owned keys. - Kill switch / disable path: Stop using the managed Kimi wrapper path or revert the release commit. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert the release commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes Kimi Code owns OAuth credentials and the /login flow. Headroom does not read or modify Kimi config or credential files. The changelog is generated by the release pipeline. |
||
|
Raúl
|
7784bb1846
|
fix(transforms): stop folding datetime-prefixed user messages as search results (#3221)
## Description Interactive `headroom wrap copilot` sessions intermittently lose the user's message: the model answers "How can I help you today?" to a real task prompt. Root cause: Copilot CLI prepends `<current_datetime>…</current_datetime>` to every interactive user turn; the ISO-8601 timestamp matches the grep `file:line:` detector, so a datetime + one-line prompt (1 match / 2 non-empty lines = 50% ≥ 30%) classifies as `SEARCH_RESULTS`, and `SearchCompressor` — which keeps only detector-matching lines — deletes the prompt before upstream. On the OpenAI chat streaming path there is no retrieval tool, so the loss is unrecoverable. Fix: `_try_detect_search` now (a) requires the pre-colon segment to look like a file path (no `<`, `>`, `=`), and (b) requires at least two matching lines, so one coincidental `word:digits:` line can no longer classify a whole payload. A genuine one-line grep result loses nothing: all its lines match, so the compressor would have kept it verbatim anyway. Closes #3220 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/transforms/content_detector.py`: new `_is_search_result_line` helper (path-like prefix gate); `_try_detect_search` gains a two-matching-line absolute floor. - `tests/test_transforms_content_detection.py`: regression tests — datetime-prefixed one-liner not search; two-line floor; tag-like / `key=value` prefixes rejected; genuine grep output still detected. - `tests/test_transforms_content_router.py`: router-level regression — the incident payload never routes to SEARCH and the prose survives `ContentRouter().compress()`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/python -m pytest tests/test_transforms_content_detection.py tests/test_transforms_content_router.py tests/test_mixed_content_sections.py tests/test_text_compressors.py tests/test_transforms_tabular.py -q 135 passed in 21.06s $ .venv/bin/ruff check headroom/transforms/content_detector.py tests/test_transforms_content_detection.py tests/test_transforms_content_router.py All checks passed! $ .venv/bin/mypy headroom/transforms/content_detector.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS 26.5 arm64, Python 3.13, editable source build 0.37.0-dev; upstream `api.githubcopilot.com`, cheapest subscription model `kimi-k2.7-code`. - Exact command / steps: standalone copilot-routed proxy (`OPENAI_TARGET_API_URL=https://api.githubcopilot.com headroom proxy --port 8899`) + `.overlay/e2e-copilot-content-probe.sh --port 8899 --model kimi-k2.7-code`, which sends the real interactive wire shape (`<current_datetime>…` + one-line sentinel prompt, streaming) and a multi-line control. - Observed result: BEFORE the fix, probe 1 FAIL — model replied "Hello! I see the current datetime is … How can I assist you today?" with proxy log `transforms=router:search:0.50` (prompt deleted). AFTER the fix, both probes PASS — the sentinel echoes verbatim, proving the user message reached upstream intact. - Not tested: other harnesses' interactive wrappers (claude/droid/auggie send different shapes; the detector fix is generic); the mixed-content section splitter has its own grep pattern (out of scope — its 1-line "search" sections are kept verbatim, no data loss). ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: N/A (no flag). - Stable/default behavior changed: content with exactly one `path:line:`-shaped line no longer classifies as search results (stays uncompressed instead — safe direction; compression only ever engages on ≥2 matching lines now). - Kill switch / disable path: N/A. - Unsafe override required: no. - Qualification impact: none. - Rollback path: revert; prior behavior restores (with the bug). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — proxy transform change; no UI. ## Additional Notes Detection-precision tradeoff is documented in code comments: single-line genuine grep output is no longer folded (no data loss either way — the compressor keeps all-matching content verbatim). A residual edge (prose with ≥2 coincidental `x:1:` lines in ≤6 lines) is accepted and documented in the issue. |
||
|
Rod Boev
|
7550efb68f
|
fix(mcp): add explicit Serena reconciliation (#3222)
## Description Headroom repeatedly warns about user-managed Serena drift but has no scoped remediation command. Add a Claude-only read-only mcp reconcile command with explicit --adopt consent, using the canonical Serena spec and existing Claude registrar. Adoption validates every relevant ledger and Claude config root before mutation, writes only the Serena entry, and records ownership after the config write succeeds. Automatic wrap migration and ordinary install remain unchanged. Closes #3054 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (feature that would cause existing behavior to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add Claude-only `headroom mcp reconcile`, read-only by default, with `--adopt` as its only mutation action. - Reuse the shared `CLAUDE_SERENA_CONTEXT` and canonical Claude Serena spec builder. - Fail closed on malformed or unreadable ledger/config state before adoption. - Preserve automatic wrap recovery, user-managed warnings, ordinary `mcp install --force`, unrelated Claude config, and corrupt-ledger tolerance outside explicit adoption. - Record Headroom ownership only after a successful registrar write. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_cli/test_mcp_reconcile.py tests/test_cli/test_serena_reconcile.py tests/test_mcp_registry/test_ledger.py`) - [x] Linting passes (`uv run ruff check .`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed through the file-backed Claude registrar ### Test Output ```text uv run pytest tests/test_cli/test_mcp_reconcile.py tests/test_mcp_registry/test_ledger.py tests/test_cli/test_serena_reconcile.py tests/test_mcp_registry/test_claude_registrar.py tests/test_mcp_registry/test_install.py -q 102 passed in 0.70s uv run ruff check headroom/mcp_registry/ledger.py headroom/cli/wrap.py tests/test_mcp_registry/test_ledger.py tests/test_cli/test_serena_reconcile.py tests/test_cli/test_mcp_reconcile.py All checks passed! uv run ruff format --check headroom/mcp_registry/ledger.py headroom/cli/wrap.py tests/test_mcp_registry/test_ledger.py tests/test_cli/test_serena_reconcile.py tests/test_cli/test_mcp_reconcile.py 5 files already formatted git diff --check ``` ## Real Behavior Proof - Environment: Windows, file-backed Claude configuration and isolated MCP ledger. - Exact command / steps: run the stale user-managed Serena fixture from `tests/fixtures/headroom-issue-3054.json`; run read-only reconcile; run `mcp reconcile --adopt`; rerun wrap and ordinary `mcp install --force`; exercise malformed JSON, non-dict `mcpServers`, null ledger agents, and unreadable-ledger adoption. - Observed result: read-only reconciliation leaves config and ledger bytes unchanged; adoption updates only Claude Serena and records ownership after a successful write; automatic wrap remains lenient; unsafe adoption inputs leave all files unchanged; ordinary install does not adopt Serena. - Not tested: live Claude CLI acceptance and Serena stdio handshake ## Runtime Rollout Safety - Rollout-managed feature(s): None; explicit `mcp reconcile --adopt` is the only mutation path. - Minimum rollout channel: Stable; no staged rollout mechanism exists for this command. - Stable/default behavior changed: No, read-only reconcile is the default and automatic wrap plus ordinary install remain unchanged. - Kill switch / disable path: Do not invoke `--adopt` or revert the release commit. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert the release commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes The changelog is generated by the release pipeline. This change is limited to Claude Serena reconciliation and does not add a new persistent acknowledgement state or a multi-provider adoption route. |
||
|
Abhay Singh
|
455f4f263c
|
fix(cache/semantic): don't semantic-match an empty query across contexts (#3226)
## Description
`SemanticCache.get()` matches on the **embedding of the last user
message** whenever an `embedding_fn` is wired. That query is empty
(`""`) for the overwhelming majority of agent/tool turns — a
`tool_result` continuation carries no text block, so
`SemanticCacheLayer._extract_query` returns `""`. A real sentence
embedder maps `""` to a fixed **non-zero** vector, so every empty-query
turn is ~identical to every other in embedding space. The exact
`messages_hash` guard (correctly chosen so `"continue"`/`"yes"` turns in
different contexts don't collide) is then bypassed by the semantic path:
an empty-query request misses on its unique hash, falls through to
embedding matching, and hits a **different conversation's** stored
response.
Reproduction (realistic embedder, non-zero for `""`):
```python
c = SemanticCache(embedding_fn=embed)
c.put(query="", response={"answer": "A"}, messages_hash="ctxA") # conversation A
c.get(query="", messages_hash="ctxB") # conversation B, different context
# -> returned A's response (cross-context false hit)
```
Measured on 330 real Claude Code transcripts (28,441 requests): **95.7%
have an empty extracted query**, so this is the dominant case, not a
corner case. The exact-hash path is unaffected; only the
embedding-similarity path is.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/cache/semantic.py`:
- `get()`: gate the semantic-similarity branch on `query.strip()` — an
empty/blank query can only ever hit via its exact `messages_hash`
(context-complete), never via embedding similarity.
- `put()`: store no embedding for an empty/blank query, so such an entry
is skipped by `_find_similar` (which ignores entries with no embedding)
and can never be a match target.
- `tests/test_cache/test_semantic.py`: added
`test_empty_query_never_semantic_matches` (cross-context empty-query
miss, exact-hash still hits, whitespace treated as empty).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
### Test Output
```text
tests/test_cache/test_semantic.py -> 22 passed in 2.39s
uvx ruff@0.16.2 check headroom/cache/semantic.py tests/test_cache/test_semantic.py -> All checks passed!
uvx mypy@1.20.2 headroom/cache/semantic.py -> Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: before the fix, two different-context
empty-query requests (`ctxA` then `ctxB`) returned `ctxA`'s response via
the embedding path. After the fix, the second returns `None`, while
`ctxA`'s own exact-hash lookup still returns its response, and a
legitimate non-empty semantic hit (`"What is the weather today?"` ->
`"How is the weather?"`) still works.
- Observed result: empty/blank queries no longer semantic-match across
contexts; exact-hash and non-empty semantic matching are unchanged.
- Not tested: no live embedder model wired (the current client wires
none — the embedding path is exercised with an injected `embedding_fn`,
which is the documented usage).
## Runtime Rollout Safety
- Rollout-managed feature(s): none. `SemanticCache` is an SDK-side cache
(`headroom.cache`), not a rollout-channel-gated runtime feature;
semantic matching only runs when a caller injects an `embedding_fn`.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no. Exact-hash matching and non-empty
semantic matching are unchanged; only empty/blank-query semantic
matching (a false-hit source) is removed.
- Kill switch / disable path: N/A.
- Unsafe override required: no.
- Qualification impact: none; correctness-only.
- Rollback path: revert this PR.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal behavior)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
|
||
|
Parideboy
|
cc484864b2
|
docs(troubleshooting): note server-managed settings skip custom ANTHROPIC_BASE_URL (#3118)
## Description Documents a known Claude Code client-side limitation: server-managed settings (delivered from the claude.ai admin console) are silently skipped whenever `ANTHROPIC_BASE_URL` is non-default — which is exactly the condition Headroom wrapping creates. Fixes #3074 by explaining the root cause is upstream, not a Headroom bug, and pointing affected users at the endpoint-managed alternative. ## Type of Change - [x] Documentation - [ ] Bug fix - [ ] New feature - [ ] Breaking change - [ ] Refactor / chore ## Changes Made - Added a new "Server-managed settings unavailable through custom ANTHROPIC_BASE_URL" section to `docs/content/docs/troubleshooting.mdx`, immediately after the existing "Remote Control unavailable through custom ANTHROPIC_BASE_URL" section (same class of Claude-side gate, same Symptom/Cause/Fix format). - Explains why Headroom has no endpoint to implement here (per Anthropic's docs, Claude Code skips the fetch client-side before any request is sent) and distinguishes this from the unrelated, unaffected OS-level `managed-settings.json` file. - Links to Anthropic's official docs and to #3074. ## Testing - [x] Verified locally - [ ] Added/updated automated tests - [ ] N/A ``` $ python3 -c " import re text = open('docs/content/docs/troubleshooting.mdx', encoding='utf-8').read() headings = re.findall(r'^##\s+.*$', text, re.MULTILINE) start = text.index('## Server-managed settings') end = text.index('## Compression Too Aggressive') section = text[start:end] print('backticks even:', section.count(chr(96)) % 2 == 0) print('brackets balanced:', section.count('[') == section.count(']')) print('parens balanced:', section.count('(') == section.count(')')) " backticks even: True brackets balanced: True parens balanced: True ``` ## Real Behavior Proof - Environment: Docs-only change (MDX prose, no code path). `docs/` npm deps are not installed in this sandbox, so the Next.js docs build (`npm run build`) was not run. - Exact command / steps: Diffed the new section against the file's existing neighboring section (git diff), and ran a Python script validating heading structure and backtick/bracket/paren balance within the new section (shown above). - Observed result: New `##` heading inserted cleanly between the two existing sections with no structural changes elsewhere in the file; markdown syntax (bold labels, inline code, links) mirrors the adjacent "Remote Control" section exactly, and is balanced/well-formed. - Not tested: The actual Next.js docs site build/render (`npm run build` in `docs/`) — no network/npm install available in this sandbox. No functional/runtime behavior is affected by this change. ## Runtime Rollout Safety - Rollout-managed feature(s): None - Minimum rollout channel: N/A - Stable/default behavior changed: No - Kill switch / disable path: N/A - Unsafe override required: No - Qualification impact: None - Rollback path: Revert the commit; no state or config is introduced ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
JD Davis
|
34a5517562
|
chore: release 0.36.5 (#3214)
🤖 I have created a release *beep* *boop* --- ## [0.36.5](https://github.com/headroomlabs-ai/headroom/compare/v0.36.4...v0.36.5) (2026-08-22) ### Bug Fixes * **codex:** detect ChatGPT auth from id_token claims so wrap/init emit requires_openai_auth ([#3212](https://github.com/headroomlabs-ai/headroom/issues/3212)) ([ |
||
|
Tejas Chopra
|
2f81fa5931
|
fix(codex): detect ChatGPT auth from id_token claims so wrap/init emit requires_openai_auth (#3212)
Fixes #3206. ## The report `headroom wrap codex` / `init codex` write a provider block without `requires_openai_auth = true`. Codex then attaches **no `Authorization` header**, and every request through the proxy 401s: ``` unexpected status 401 Unauthorized: Missing bearer or basic authentication in header ``` Silently — `headroom doctor` reported green throughout. The reporter lost ~15h of scheduled Codex automation before bisecting it. ## Not the fix the issue suggested The issue proposes adding the line unconditionally. **That would re-break API-key users**, which is the regression `requires_openai_auth` was made conditional for in the first place (#406) — the flag forces Codex to demand an OpenAI OAuth login. All three writers (install provider-scope, `init codex`, `wrap codex`) *already* call `codex_uses_chatgpt_auth()` and emit the key when it returns True. **The bug is in the detection, not the writers.** ## Root cause `codex_uses_chatgpt_auth` recognised two shapes: 1. `auth_mode == "chatgpt"` 2. a top-level `tokens.account_id` Newer Codex can write an `auth.json` with **neither** — the account identity lives only in the `id_token` claims, under `https://api.openai.com/auth.chatgpt_account_id`. That config reads as API-key mode, the flag is omitted, and every request 401s. Verified against a real `auth.json`: the JWT claim carries the *same* account id as the top-level key, so it is a faithful signal for the shape that lacks it. ## Fix A third detection tier, consulted only when the first two are absent: | Shape | Before | After | |---|---|---| | `auth_mode = "chatgpt"` | True | True | | legacy `tokens.account_id` | True | True | | **only the `id_token` claim** | **False** ← the bug | **True** | | `auth_mode = "apikey"` + ChatGPT id_token | False | **False** (#406 stays closed) | | API key, no tokens | False | False | | id_token without the claim / malformed / blank id | False | False | The payload is **decoded, not verified**. It is a local file the user already owns, and the result only chooses which key we write into their own `config.toml` — nothing is authenticated or authorised on the strength of it. An API-key user has no ChatGPT id_token, so this cannot resurrect #406, and an explicit `auth_mode` still wins outright (pinned by test). ## Doctor stops reporting a false green This failure is invisible from every other signal — proxy up, provider block present. So the codex check now WARNs when the config is routed, the user is on ChatGPT auth, **and** the block lacks the flag, naming the re-run that repairs it. It only runs when the flag is already missing, and the keyring fallback it can reach is bounded by an existing 3s timeout, so `doctor` stays fast. API-key users are never nagged. ## Existing configs Self-healing — all three writers strip and regenerate the managed block on every run, so re-running `wrap`/`init` emits the key now that detection is correct. No separate migration needed. ## Testing 99 passing across the two suites. Confirmed **discriminating**: 3 of the new tests fail against unfixed source and pass after — - `test_chatgpt_auth_detected_from_id_token_claims_alone` - `test_provider_block_emits_requires_openai_auth_for_the_new_shape` - `TestCodexRouting::test_chatgpt_auth_without_requires_openai_auth_warns` plus explicit coverage for the #406 guard, malformed tokens, blank account ids, and the API-key-not-nagged case. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tejas Chopra
|
8f3e33a00e
|
fix(doctor): report project-scoped Claude routing instead of a false negative (#3213)
Refs #3205 — **issue 2 of 2**. The `wrap`-session crashes reported in that issue are *not* addressed here; see the note at the bottom. ## The report A session routed via `headroom init claude` was reported by `headroom doctor` as **not routed**, while it demonstrably was: - `ps eww` on the live `claude` process showed `ANTHROPIC_BASE_URL=http://127.0.0.1:8787` - the `mcp__headroom__*` tools were present and firing - `headroom_stats` showed **164 of 174 requests compressed** on that very session The cost wasn't cosmetic. The team believed 3 of 4 sessions were unrouted on doctor's word, and hand-checked `ps eww` plus MCP tool presence on each one to find the real state. ## Root cause — a scope mismatch | | Path | |---|---| | `init claude` (non-global) **writes** | `./.claude/settings.local.json` | | `doctor` **read** | `~/.claude/settings.json` only | Claude Code layers project settings over user settings, so project-scoped routing — what `init` writes by default — was invisible to the check. ## Fix `check_claude_routing` now takes the project-scoped candidates and consults them in **Claude's own precedence order** (project-local, project, then user), reporting the first that carries `ANTHROPIC_BASE_URL`. The summary names the file that supplied it, so which scope is in effect is never ambiguous — that ambiguity is what made this expensive to diagnose. Reading more files must not turn a routed session into a crash or a silent skip: - a per-file parse failure is surfaced verbatim (`could not parse …`) rather than swallowed into the misleading "not routed" - the non-dict guard is preserved **per file** — a hand-edited settings file containing `[]` or `null` would otherwise raise `AttributeError` inside the very command run to diagnose it - a missing project file is skipped, not fatal The third argument is optional and defaults to the previous single-file behaviour, so existing callers and tests are unaffected. ## Not scraping `ps` The reporter suggested inspecting live `claude` process environments. That isn't needed and would be platform-specific — the routing is written to a file whose path we already know. The gap was that we read the wrong scope, so that's what this fixes. ## Testing 86 passing. Confirmed **discriminating** — all 7 new tests fail against unfixed `doctor.py` and pass after: | Test | Covers | |---|---| | project-local counts as routed | the reported bug | | project `settings.json` counts as routed | the other project file | | project takes precedence over user | Claude's layering | | falls back to user when project has no base URL | no false positive | | still warns when nothing routes | no blanket pass | | missing project file skipped | not fatal | | unparseable project file surfaces | not silently "not routed" | | no project paths → original behaviour | backward compatibility | 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
JD Davis
|
91186b40d8
|
chore: release 0.36.4 (#3189)
🤖 I have created a release *beep* *boop* --- ## [0.36.4](https://github.com/headroomlabs-ai/headroom/compare/v0.36.3...v0.36.4) (2026-08-22) ### Bug Fixes * **dashboard:** pin MIME types for the vendored static assets ([#3193](https://github.com/headroomlabs-ai/headroom/issues/3193)) ([ |
||
|
Tejas Chopra
|
5d25abd356
|
test: repair three suite failures that are red on main (#3196)
## Summary
Three tests fail on a clean `main` full-suite run. None is a product
defect — all three are tests that stopped describing reality, and they
will noise up or block the 0.36.4 release.
| Test | Why it fails | Fix |
|---|---|---|
| `test_release_workflows::test_no_native_tls_in_wheel_build_tree` |
Shells out to `cargo`; raises `FileNotFoundError` wherever the Rust
toolchain is absent | Copied the skip guards its own dual already had |
|
`test_learn/test_integration::TestCodexIntegration::test_full_pipeline`
| Asserts `"Bash" in all_tools` against **real local Codex data**; Codex
renamed its shell tool | Assert what the test is for, across Codex
versions |
|
`test_graceful_shutdown::test_run_server_installs_cancelled_error_filter`
| Counts installs on the **process-global** `uvicorn.error` logger;
order-dependent | Isolate the global state; assert the real contract |
## 1. native-tls / cargo
The `openssl-sys` gate 30 lines above is described in-code as this
test's dual. It already skips when `cargo` is missing, **and** when
cargo fails for a reason other than `"package did not match"` (the Linux
wheel target not being installed locally). The native-tls test never
copied either guard.
Not disabled: CI installs the toolchain via `dtolnay/rust-toolchain`, so
the check still executes there. The skip only applies where cargo is
genuinely absent.
## 2. Codex tool vocabulary
This test runs against whatever Codex sessions the machine actually has
(gated by `HAS_CODEX_DATA`), and asserted:
```python
# Codex has only Bash tool (shell)
assert "Bash" in all_tools
```
Codex has since renamed its shell tool (`Bash` → `shell` → `exec`), and
0.149.0 added agent tools (`spawn_agent`, `send_message`, `wait`) beside
it. The assertion pinned one release's vocabulary, so it fails on any
current install.
It now asserts what the pipeline is actually being tested for — that
tool calls were extracted, including a shell-execution tool under any of
its known names — and names the remedy in the failure message for the
next rename.
**Still discriminating** (verified, not assumed):
| Scenario | Result |
|---|---|
| pipeline parsed nothing | fails ✓ |
| tool names garbled | fails ✓ |
| agent tools only, no shell tool | fails ✓ |
| real current Codex data | passes ✓ |
## 3. Global logger state
```python
if not any(isinstance(item, _SuppressCancelledErrorFilter) for item in uvicorn_error_logger.filters):
uvicorn_error_logger.addFilter(_SuppressCancelledErrorFilter())
```
`run_server` is deliberately idempotent and `uvicorn.error` is a
process-global logger, so any earlier test in the session that reached
`run_server` leaves the filter attached — and this test then observes
**zero** installs against its `== 1` assertion. It passes alone and
fails in a full run, which is exactly the symptom.
The test now clears and restores that global state around itself, and
additionally asserts the idempotence guard that is the real contract:
calling `run_server` twice must not stack a duplicate filter. The test
got stronger, not just quieter.
## Scope
Tests only — no product code is touched.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Tejas Chopra
|
3e3c409436
|
fix(security): validate caller-supplied upstreams on every resolution path (#3195)
## Summary CVE-2026-77775 (SSRF via `x-headroom-base-url`) is **not fully fixed on current `main`**. The advisory lists 0.36.1 as the last affected version; one route still forwards to any destination a caller names. `upstream_guard.is_safe_upstream_url` was added and wired into `/v1/messages` and the catch-all passthrough. But `select_passthrough_base_url` moved from `providers/proxy_routes.py` to `providers/proxy_targets.py`, and the guard did not follow it. Its Azure branch returns the header verbatim whenever an `api-key` header is present — **both values are caller-supplied** — and `POST /v1/alpha/search` resolves its upstream through that helper without checking the header itself. ## Verified, not inferred Against the current tree, with a listener on loopback standing in for an internal service: ``` proxy status : 200 internal service hit : 1 time(s) Authorization it received : 'Bearer SECRET-CLIENT-TOKEN' internal body relayed back : True ``` The caller's credentials are forwarded to the attacker-named host and the internal response is relayed back. After this change: `400`, zero hits, nothing relayed. A sweep of all 99 routes isolates exactly one leak on unfixed code — `POST /v1/alpha/search` with `api-key` — and zero after. ## 1. The missing enforcement **Guarded at the chokepoint, not just the route.** `select_passthrough_base_url` now validates before returning, in `proxy_targets.py` and in the parallel copy in `providers/registry.py`, so a future caller that forgets the header check cannot reopen this. `/v1/alpha/search` also rejects explicitly with 400, matching its sibling routes. ## 2. A second gap in the address policy RFC 6598 shared address space (`100.64.0.0/10`) is not `is_private`, so it passed the guard — while routing to ISP and cloud-internal infrastructure. `_is_internal_address` now also rejects anything not globally routable. Verified over a 27-vector battery — 0 bypasses, public control unaffected: | Vector | Before | After | |---|---|---| | `100.64.0.0/10` shared address space | **allowed** | blocked | | `198.18/15`, TEST-NET, `240/4` | **allowed** | blocked | | 6to4 / Teredo embedding internal IPv4 | **allowed** | blocked | | NAT64 `64:ff9b::/96` embedding loopback | **allowed** | blocked | | loopback, RFC1918, link-local, metadata, IPv4-mapped, userinfo tricks | blocked | blocked | | multicast `224.0.0.1` | blocked | blocked | | public `8.8.8.8` | allowed | allowed | The category checks are **kept alongside** `is_global` rather than replaced — `is_global` is `True` for multicast, so a replacement would have regressed. NAT64 also reports as global, so its embedded IPv4 is extracted and judged on its own. ## 3. Unauthenticated stall via the resolver `socket.getaddrinfo` takes no timeout and runs on the calling thread — the event loop. Since the hostname is caller-supplied, a deliberately slow-resolving name stalled every other in-flight request; a handful of concurrent requests made the proxy unresponsive, unauthenticated. Resolution now runs in a small dedicated pool with a budget (`HEADROOM_UPSTREAM_RESOLVE_TIMEOUT_S`, default 3s) and fails closed on overrun, which bounds every caller including the synchronous chokepoint. `is_safe_upstream_url_async` runs the lookup off the loop, and the three route handlers that validate a caller-supplied upstream now await it. Caching was deliberately avoided: a TTL cache in front of a security decision invites poisoning, and would widen the rebinding window rather than narrow it. ## Why this survived The existing tests unit-tested the guard's *logic* but never asserted it was *reached*. Added enforcement tests at the sinks plus a **sweep over the whole route table** that fails if any route forwards to a loopback address — so the next unguarded upstream resolution fails in CI rather than in a CVE. All new tests were confirmed failing against the unfixed tree and passing after. ## Known residual — deliberately not addressed **DNS rebinding.** Validation and connection resolve the host separately, so a low-TTL answer can differ between them. Closing this needs connection-time pinning in the shared `http_client` transport, which carries every request in the proxy — too broad to fold into this patch. It should not be described as fixed. ## Compatibility An endpoint that does not resolve publicly (split-horizon, on-prem) is now rejected where it previously passed unvalidated. `HEADROOM_ALLOWED_BASE_URLS` is the documented opt-in, covered by test. Three existing tests used fictional hostnames and legitimately began failing; DNS is pinned in them so they keep testing target precedence rather than depending on the missing guard. Separately: `docker-compose.yml` has already been hardened since the advisory — `HEADROOM_PROXY_TOKEN` is now mandatory and ports are loopback-only — so the "exposed by default" multiplier the advisory cites no longer applies to the shipped compose. Full suite: the 3 failures outside this area (`test_learn/test_integration`, `test_release_workflows::test_no_native_tls_in_wheel_build_tree`, and a `test_graceful_shutdown` ordering flake) reproduce on clean `main` and are unrelated. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tejas Chopra
|
1617f839a1
|
fix(proxy/responses): keep the Codex additional_tools carrier on the wire (#3194)
## Description 0.36.3 regressed Codex tool access. A user reproduced it cleanly: Codex CLI 0.149.0 + Codex TUI/app-server, terminal tools available at first (`pwd` executes), then **all shell/filesystem access disappears for the rest of the session**. The same setup on 0.36.2 works. The only functional change in 0.36.3 was #3186. ## Root cause #3186 lifted `additional_tools` definitions into top-level `tools` so the tools consumers (schema compaction, output shaper, token accounting) would engage, and dropped the carrier item. That changed the definitions' **lifetime**, not just their location: - `tools` is a **per-request parameter**, scoped to one response. - `additional_tools` is an **`input` item** — part of the conversation transcript. A stateful session declares its tools once. Codex over WebSocket sends the carrier on turn one and relies on the transcript afterwards. Forwarding the lifted shape leaves that transcript tool-less, so turn one works and every turn after it has no tool surface at all. Stateless HTTP hid this in review — it re-sends the carrier on every request, so the lift refires each turn and nothing is ever lost. That is why the original manual verification passed. ## Fix The lift stays; the savings fix it shipped for is real. It is now **symmetric**: - `_lift_codex_additional_tools` records where each carrier came from (`restore_plan`). - `_restore_codex_additional_tools` puts the post-compaction definitions back into that carrier before the payload is forwarded. Consumers still see a classic top-level array. The client still sees the shape it sent. Compaction's savings survive the round trip, because it is the *compacted* schemas that go back into the carrier. Restoration is conservative: | Situation | Behaviour | |---|---| | Compaction preserved the definition count | original per-carrier split rebuilt exactly | | A consumer rewrote the array (deferral, injection) | whole set rides the first carrier | | Array came back empty | definitions Codex sent are restored, never a tool-less forward | | Carrier cannot be put back at all | logged, never a silent lifted-shape forward | | Called twice | idempotent, no duplication | Wired into `_compress_openai_responses_payload_in_executor`, so all five call sites — HTTP, both WebSocket sites, and passthrough — are covered by construction. `HEADROOM_CODEX_ADDITIONAL_TOOLS_LIFT=0` still disables the lift entirely and remains the immediate unblock for anyone on 0.36.3 right now. ## Testing The gap in #3186 was that all nine of its tests were single-turn. These are not. - **Multi-turn regression test** — a turn-one payload is driven through the real compression entry point, and turn two is built from what was actually forwarded. On shipped `main` that turn-two transcript carries **zero** tool definitions; with this change it carries both. - **Exhaustive round trip** — 363 arrangements of messages, carriers, empty carriers, adjacent/leading/trailing carriers. Zero mismatches. This is what pins the insert-offset arithmetic. - Round-trip shape preservation, carrier position, multiple carriers, count-change fallback, emptied-array recovery, extra carrier keys, idempotence, the unrestorable-warning path, the kill switch, and untouched classic-encoding clients are each asserted. 22 tests in the file; 112 across the related suites (proxy, codex routing, passthrough, compaction); full suite 3740 passed / 156 skipped. `ruff check` and `ruff format` clean. Before/after against shipped `main`, same scenario: | | 0.36.3 (`main`) | this PR | |---|---|---| | forwarded top-level `tools` | present | absent | | carrier surviving in `input` | **0** | 1 | | tools visible to turn 2 | **none — tool loss** | `shell`, `update_plan` | ## Validation gap — please read This proves the **forwarded shape now matches what the client sent**, which is the invariant that matters regardless of the exact upstream mechanism. What is *not* directly observed here is the transcript-persistence mechanism itself — that is inferred from Responses API semantics, because there is no Codex 0.149.0 stateful WebSocket backend in CI. That is the same gap that let #3186 ship broken, so it should not be waved through twice. The reporter has a reliable reproduction and should confirm this build before it tags. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Ayush Kumar Jha
|
b4857685ff
|
fix(dashboard): pin MIME types for the vendored static assets (#3193)
## Description
The dashboard's vendored scripts can be served as `text/plain`, and the
proxy's own
`X-Content-Type-Options: nosniff` then stops the browser executing them
— the dashboard
loads unstyled and dataless.
`StaticFiles` types every response from `mimetypes.guess_type`, and
Python seeds that
database from the host: the Windows registry (`HKCR\<ext>\Content Type`)
and, elsewhere,
files like `/etc/mime.types`. headroom never calls `mimetypes.add_type`
anywhere, so it
inherits whatever the host says. On a host that maps `.js` to
`text/plain` — a stale
registry entry, or a minimal container image with no mime database at
all — the three
vendored assets go out as plain text.
Neither half is wrong on its own. `nosniff` at `_apply_security_headers`
is correct and
should stay; the mislabel is the bug. Together they break the dashboard
completely.
Closes #3179
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Added `register_static_mime_types()` and the `_STATIC_MIME_TYPES`
table to `headroom/dashboard/__init__.py`, next to the `STATIC_DIR` it
describes.
- `create_app` calls it immediately before mounting `/dashboard/static`,
so the served type no longer depends on the host mime database.
- Registered `.js`/`.mjs` as `text/javascript`, `.css` as `text/css`,
and `.json`/`.map` as `application/json`.
- Added `tests/test_dashboard_static_mime_types.py` (11 tests) covering
a deliberately broken host database, each registered extension,
idempotency, and a guard that fails if a future vendored asset arrives
with an unregistered extension.
### Design notes
`mimetypes.add_type` is strict by default, so these registrations
replace a bad host
entry rather than losing to it. They are the current IANA/WHATWG values,
so this only
ever repairs a host database — it never invents a mapping.
Registration runs from `create_app` rather than at module import.
Mutating the
process-wide table is right for the proxy that serves these files, but
it should not be
a side effect of `import headroom` for someone using the library.
Two deliberate departures from the fix sketched in the issue:
`text/javascript` rather
than `application/javascript` (the current registration, and what Python
3.12+ returns
natively, so the fix converges with the stdlib instead of diverging from
it — both
execute in every browser), and `.map` as `application/json` rather than
`application/javascript`, since a source map is a JSON document.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_dashboard_static_mime_types.py -q
11 passed, 1 warning in 0.94s
# against the unpatched tree the same file cannot even import:
ERROR tests/test_dashboard_static_mime_types.py
ImportError: cannot import name 'register_static_mime_types' from 'headroom.dashboard'
$ python -m pytest tests/*dashboard* -q --continue-on-collection-errors
2 failed, 18 passed, 5 skipped, 2 errors in 11.75s
# baseline on the same tree with the fix stashed:
2 failed, 7 passed, 5 skipped, 2 errors in 6.98s
# identical failures/errors either way (they need the Rust _core extension, which is
# not built on this machine); the fix adds the 11 passing tests and breaks nothing.
$ python -m ruff check headroom/dashboard/__init__.py headroom/proxy/server.py tests/test_dashboard_static_mime_types.py
All checks passed!
$ python -m ruff format --check ...
3 files already formatted
$ python -m mypy headroom/dashboard/__init__.py headroom/proxy/server.py
Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: Windows 11 Home 26200, Python 3.11.9, clone of
`upstream/main` at `
|
||
|
Raúl
|
9c30b62962
|
fix: skip cross-turn dedup pointers on OpenAI chat streaming (#3191)
## Description
Cross-turn dedup (`HEADROOM_DEDUPE` / `enable_cross_turn_dedup`, plus
the cold-prefix recompaction router) folds a repeated tool-output span
into a one-line in-context pointer, `[↑NL same as msg M: 'anchor']`.
That pointer is only recoverable where the model can resolve the
reference. On the OpenAI chat-completions STREAMING path (what `headroom
wrap copilot` serves) it cannot, for two independent reasons:
1. The proxy itself logs `CCR: skipping retrieval-tool injection for
OpenAI chat streaming; this path cannot intercept tool calls`, so no
`headroom_retrieve` tool exists on this path and nothing can
mechanically resolve a fold.
2. The pointer names its source as `msg M`, Headroom's internal message
index. OpenAI-compatible chat clients never show the model numbered
messages, so the reference is unresolvable even though the original
bytes are technically still earlier in the same request.
Observed with Kimi k2.7-code / k3 via `wrap copilot`: the model treats
the pointer as deleted output, reports "the renderer is
deduplicating/compressing", and retry-loops near-identical reads (one
session burned ~200 turns; a folded conflicted-files listing hid 4 of 5
conflicted files and the agent committed unresolved `<<<<<<<` markers).
The router already keeps unrecoverable LOSSY output verbatim
(`lossy_unrecoverable_skipped`). Dedup folds are lossless in theory but
unrecoverable in practice on this path; this PR gives them the same
recoverability gate.
Closes #3190
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/transforms/content_router.py`: `ContentRouter.apply()`
accepts a per-request `cross_turn_dedup_recoverable` kwarg (default
`True`, so every existing caller is byte-identical). When `False`, the
cross-turn dedup pass is skipped and repeated spans stay verbatim,
mirroring the recoverability posture of the lossy
`lossy_unrecoverable_skipped` guard. Config comment on
`enable_cross_turn_dedup` documents the gate.
- `headroom/proxy/handlers/openai.py`: `handle_openai_chat` computes the
gate from the same predicate that already gates CCR retrieval-tool
injection, `_should_inject_openai_chat_ccr_tool(ccr_inject_tool,
stream)`, and threads it into both `openai_pipeline.apply(...)` call
sites (token-mode and non-token-mode branches). Streaming chat requests
skip the fold; buffered (non-streaming) chat, which can inject and
redeem the retrieval tool, keeps folding.
- `headroom/transforms/cold_prefix.py`: `cold_recompact_messages` no
longer hardcodes pointer emission; new keyword-only
`cross_turn_dedup_recoverable: bool = True` is forwarded to the router
gate. The only caller (Anthropic cache-mode cold turn) keeps the default
and is unchanged.
- `tests/test_cross_turn_dedup.py`: router-gate regression tests
(unrecoverable path keeps verbatim bytes for both the OpenAI `role:tool`
string shape and the Anthropic `tool_result` block shape;
default/explicit-`True` still folds).
- `tests/test_cold_prefix.py` (new): recompaction folds by default
(Anthropic path unchanged) and keeps verbatim bytes with
`cross_turn_dedup_recoverable=False`.
- `tests/test_openai_chat_dedup_recoverability.py` (new): end-to-end
through the real `/v1/chat/completions` handler with
`HEADROOM_DEDUPE=1`, capturing the exact upstream request body:
`stream=True` keeps both copies byte-verbatim with no `[↑` pointer;
`stream=False` still folds; `stream=False` under `--lossless` (which
forces `ccr_inject_tool=False`) also keeps verbatim bytes, locking the
intended coupling of "no retrieval tool" to "no bare pointer".
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
# BEFORE (branch base, fix reverted): the streaming regression test fails,
# the upstream body carries the unresolvable pointer and drops the bytes.
$ git stash push headroom/ && uv run pytest -q \
tests/test_openai_chat_dedup_recoverability.py::test_streaming_chat_keeps_verbatim_bytes_no_dedup_pointer
E assert '[↑' not in "fix the ove...t merge.py']"
E '[↑' is contained here:
E [↑14L same as msg 2: '$ cat merge.py']
FAILED tests/test_openai_chat_dedup_recoverability.py::test_streaming_chat_keeps_verbatim_bytes_no_dedup_pointer
(same run: test_cold_recompact_unrecoverable_path_keeps_verbatim_bytes also fails pre-fix;
both recoverable-path legs pass before and after)
# AFTER (full diff applied):
$ uv run pytest tests/test_cross_turn_dedup.py tests/test_cold_prefix.py \
tests/test_openai_chat_dedup_recoverability.py \
tests/test_proxy/test_openai_chat_ccr_injection.py tests/test_no_ccr_lossy.py \
tests/test_openai_chat_turn_hooks.py tests/test_openai_chat_tool_desc_compaction.py \
tests/test_responses_cross_turn_dedup.py -q
45 passed, 2 warnings in 8.75s
$ uv run pytest tests/test_proxy/ tests/test_openai_codex_routing.py \
tests/test_openai_chat_turn_hooks.py tests/test_openai_chat_tool_desc_compaction.py \
tests/test_openai_beta_session_sticky.py tests/test_openai_max_completion_tokens.py \
tests/test_no_ccr_lossy.py tests/test_netcost_gate.py tests/test_agent_savings.py \
tests/test_cross_turn_dedup.py tests/test_cold_prefix.py \
tests/test_openai_chat_dedup_recoverability.py -q
424 passed, 2 warnings in 73.52s
$ uv run ruff format --check <touched files> && uv run ruff check <touched files>
All checks passed!
$ uv run mypy headroom/transforms/cold_prefix.py headroom/transforms/content_router.py headroom/proxy/handlers/openai.py
Success: no issues found in 3 source files
$ cargo fmt --all -- --check # FMT_OK
$ cargo clippy --all-targets # 2 pre-existing warnings in untouched lib-test code, no errors
$ cargo test # all targets green; see Additional Notes for the one environmental exception
```
## Real Behavior Proof
- Environment: macOS (Darwin), Python 3.13, repo tip `upstream/main`
|
||
|
Ayush Kumar Jha
|
202c1895e1
|
fix(wrap): make the Serena pre-index stall budget configurable (#3183)
## Description `headroom wrap` blocks the agent launch on a synchronous Serena pre-index whose 300-second ceiling is a hardcoded module constant. When indexing exceeds it the user waits the full five minutes, the work is discarded (`Serena: pre-index timed out (will index on demand)`), and nothing — env var, flag, or config — can shrink that budget. Closes #3093 ### Why this is still open after #2938 `_serena_project_skip_reason` keeps the pre-index off non-project roots, which covers the reporter's two repro directories. But it **defers the stall by one wrap rather than removing it**: as that function's own docstring notes, Serena's MCP server generates `project.yml` itself on first start, "so the pre-index simply resumes from the next wrap onwards." A parent-of-many-repos directory therefore gets claimed during the first session and pays the full 300s budget on every wrap after that. The reporter's remaining ask — "I'd also like the pre-index timeout to be configurable" — is the unfixed half. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Added `HEADROOM_SERENA_INDEX_TIMEOUT` and `_resolve_serena_index_timeout_seconds()`, modelled on the existing `_resolve_wrap_proxy_timeout_seconds()` in the same module. - `_index_serena_project` resolves the budget after the `uvx` guard and passes it to `communicate()` instead of the bare constant. - `_SERENA_INDEX_TIMEOUT = 300` stays as the default, so unset behavior is unchanged. - Added 19 tests covering the resolver and the pre-index call path. ### Deliberate divergence from the proxy-timeout precedent `_resolve_wrap_proxy_timeout_seconds` raises `RuntimeError` on a bad value, which is right for a subsystem the wrap cannot proceed without. The pre-index is documented as best-effort and non-fatal, so raising there would let a typo'd env var abort a launch that would otherwise succeed. An unusable value instead warns and falls back to 300s. The warning is unconditional (not gated on `--verbose`) because a knob that looks applied but is not is the failure this issue reports. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_cli/test_wrap_serena_boost.py -q 43 passed, 1 skipped, 1 warning in 1.13s # 24 pre-existing + 19 new # the same 19 tests against the unpatched tree: 18 failed, 1 passed, 24 deselected # the 1 passer is a pre-existing test caught by -k $ python -m pytest tests/test_cli/ -q 3 failed, 696 passed, 2 skipped in 57.80s # the 3 are pre-existing Windows failures (symlink handling in test_recover_codex.py # and test_unwrap_claude.py); they fail identically on an unpatched tree. $ python -m ruff check headroom/cli/wrap.py tests/test_cli/test_wrap_serena_boost.py All checks passed! $ python -m ruff format --check headroom/cli/wrap.py tests/test_cli/test_wrap_serena_boost.py 2 files already formatted $ python -m mypy headroom/cli/wrap.py Success: no issues found in 1 source file ``` Regression check across all 42 test modules that import `headroom.cli.wrap`, run in both states with the working tree md5-verified before each run: identical 81-line failure/error set, +19 passing with the fix. ## Real Behavior Proof - Environment: Windows 11 Home 26200, Python 3.11.9, headroom at 0.36.2 (`5e0ce24`). Serena/`uvx` are not installed on this machine and the Rust `_core` extension is not built (no Rust toolchain), so a full `headroom wrap claude` could not be launched — see `Not tested`. - Exact command / steps: drove the real `_index_serena_project()` with a real child process, a real process group, real `communicate(timeout=...)`, real `TimeoutExpired`, and the real `_kill_serena_index_tree`, timing each phase with a monotonic clock at `HEADROOM_SERENA_INDEX_TIMEOUT=2` and `=4`. Only *which* binary runs was substituted (a 120s sleeper in place of `serena project index`), since the timeout logic is indifferent to the callee. - Observed result: the configured budget controls the wait exactly — a 2s budget waits 2.02s and a 4s budget waits 4.02s, where before the change the same harness reports 300s regardless of any env var set. Full output below. - Not tested: an end-to-end `headroom wrap claude/opencode` against a real `serena project index` (uvx/serena unavailable here); non-Windows platforms; the interaction with a genuinely large monorepo index. ```text budget=2s | waited 2.02s for timeout | teardown 10.02s | total 12.03s budget=4s | waited 4.02s for timeout | teardown 10.02s | total 14.03s misconfigured value: Serena: ignoring HEADROOM_SERENA_INDEX_TIMEOUT='30s' (want a positive integer number of seconds) - using 300s -> resolved to 300s, no exception raised ``` ### Incidental finding (not addressed here) On Windows, `_kill_serena_index_tree` adds a constant ~10s after any timed-out pre-index — one of its two 10s bounds (`taskkill` / `proc.wait`) is hit every time. So a 2s budget still costs ~12s wall clock. That is pre-existing #2938 code untouched by this PR, but it caps how small the stall can usefully get and may deserve its own issue. ## Runtime Rollout Safety - Rollout-managed feature(s): none — this is a plain env var, not a rollout-channel feature. - Minimum rollout channel: n/a — available on every channel, inert unless set. - Stable/default behavior changed: no — unset resolves to the existing 300s constant. - Kill switch / disable path: unset `HEADROOM_SERENA_INDEX_TIMEOUT`; skipping the pre-index entirely remains `--no-serena`. - Unsafe override required: no. - Qualification impact: none — no change to compression, proxy, or provider behavior. - Rollback path: revert the commit; no persisted state, no migration, no config to clean up. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes **Alternatives considered.** A CLI flag (`--serena-index-timeout`) is more discoverable but has to be threaded through four `wrap` subcommands, adds CLI surface that CONTRIBUTING gates behind maintainer sign-off, and would not reach `wrap ... -- agents` sessions. Making the pre-index asynchronous removes the stall outright and is arguably the better end state, but it is an architectural change and would reopen the orphaned-grandchild failure mode #2938 just closed. Auto-scaling the budget by project size reintroduces the kind of hand-maintained heuristic #2938 deliberately removed. **What this does not solve.** The default is still 300s, so a user who never sets the variable still stalls; the reporter's third point (using Serena in background agent sessions launched from a parent directory) is a Serena-semantics question rather than a headroom defect; and an in-flight pre-index is still not interruptible. **Open questions for maintainers.** 1. Should `0` mean "skip the pre-index" instead of being rejected? I kept the proxy-timeout precedent (reject `<= 0`) since `--no-serena` already covers disabling, but the other reading is defensible. 2. `HEADROOM_WRAP_PROXY_TIMEOUT` — the closest precedent — is not in `docs/content/docs/configuration.mdx`, so I matched it and left docs alone. Happy to add a row if you would rather document it. 3. If you consider a new env knob a feature rather than part of this bug, say so and I will hold for a maintainer sign-off before you spend review time. Documentation: no `CHANGELOG.md` edit (release-please generates it from the PR title). |
||
|
JD Davis
|
87e71dd100
|
chore: release 0.36.3 (#3188)
🤖 I have created a release *beep* *boop* --- ## [0.36.3](https://github.com/headroomlabs-ai/headroom/compare/v0.36.2...v0.36.3) (2026-08-21) ### Bug Fixes * **proxy/responses:** lift Codex >= 0.149.0 additional_tools into top-level tools ([#3186](https://github.com/headroomlabs-ai/headroom/issues/3186)) ([ |
||
|
gglucass
|
25ca580825
|
fix(proxy/responses): lift Codex >= 0.149.0 additional_tools into top-level tools (#3186)
## Description
Codex CLI 0.149.0 (npm `latest` since 2026-08-20 21:09 UTC) stopped
sending a top-level `tools` array on `/v1/responses` for models its
server-fetched capability cache flags (`gpt-5.6-sol`, its new default).
Tool definitions now ride inside `input` as items of a new type:
```json
{"type": "additional_tools", "tools": [ {...}, {...} ]}
```
Every tools consumer in the proxy - `tool_schema_compaction`, the
output-shaper stratum, the tools token accounting - reads only
`payload["tools"]`, so these requests classify `notools` and record
exactly zero tool-schema savings while forwarding and streaming
normally. Users on Codex <= 0.148 are unaffected; users silently lose
savings the moment their CLI updates. On our fleet the day after the
Codex release, 42 of 54 codex-primary users active in a 12h window had
savings frozen, and 0 of that day's codex new signups recorded any
savings.
This PR normalizes the new encoding to the classic one before
compression: `_lift_codex_additional_tools(payload)` concatenates the
carrier items' `tools` arrays into `payload["tools"]` and drops the
carriers from `input`, in place, once per compression pass - at the top
of `_compress_openai_responses_payload_in_executor`, the single funnel
every responses call site goes through (HTTP `/v1/responses`, WS first
and subsequent frames, passthrough). It no-ops when top-level `tools` is
already present, so classic-encoding clients pay nothing and a future
Codex reverting the change costs nothing. Normalizing (rather than
compacting inside the items and preserving the new wire shape) keeps
every downstream consumer working without touching their accounting; the
alternative shape is discussed in #3185.
Closes #3185
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/openai.py`: new module function
`_lift_codex_additional_tools(payload, *, request_id=None)` plus
`_codex_additional_tools_lift_enabled()` (env gate via
`runtime_env.getenv`, hot-reloadable); called defensively at the top of
`_compress_openai_responses_payload_in_executor` so a lift failure can
never break forwarding.
- `tests/test_openai_responses_additional_tools.py`: 8 tests - lift
shape, multi-carrier concatenation, no-op on classic encoding, no-op
without carriers / non-dict / non-list input, kill switch, logging,
empty-carrier preservation, and lift-then-compaction integration
reproducing the exact production failure (compaction returns unmodified
without the lift).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ uv run --frozen --extra dev pytest tests/test_openai_responses_additional_tools.py tests/test_openai_responses_context_compaction.py -q
==== 18 passed in 2.71s ====
$ uv run --frozen --extra dev pytest tests/test_proxy_openai.py -q # adjacent handler suite
==== 31 passed, 1 warning in 26.36s ====
$ uv run --frozen ruff check headroom/proxy/handlers/openai.py tests/test_openai_responses_additional_tools.py
All checks passed!
$ uv run --frozen ruff format --check headroom/proxy/handlers/openai.py tests/test_openai_responses_additional_tools.py
2 files already formatted
$ uv run --frozen mypy headroom/proxy/handlers/openai.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: macOS 15 (arm64), headroom-ai 0.35.0 wheel in a fresh
venv with empty state (`HOME` pointed at an empty dir), `headroom proxy
--port 6799 --no-http2 --log-messages --no-ccr`; Codex CLI 0.149.0
(standalone npm install) and 0.142.4, ChatGPT-plan OAuth, routed via a
`[model_providers]` block in `config.toml`.
- Exact command / steps: `CODEX_HOME=<test home> codex exec
--skip-git-repo-check "Run the shell command: echo headroom-test-123.
Then reply with exactly the output it printed."` against the proxy,
before and after injecting the lift (via a sitecustomize carrying the
same function); cross-checked Codex 0.142.4 default (gpt-5.5), 0.142.4
`-m gpt-5.6-sol`, and 0.149.0 `-m gpt-5.5`.
- Observed result: before - `/v1/responses compressed 59425->59425 bytes
(0 tokens saved,
transforms=['output_shaper:stratum:gpt|new_user_ask|m|notools',
'output_shaper:verbosity:L2'])` despite ~12k tokens of tool schemas in
the request (Codex's own `tool_token_count` log field). After -
`/v1/responses compressed 59437->58716 bytes (608 tokens saved,
transforms=['output_shaper:stratum:gpt|new_user_ask|m|tools',
'output_shaper:verbosity:L2',
'openai:responses:tool_schema_compaction'])`; the shell tool call
executed against the live ChatGPT Codex backend and returned its output,
the follow-up turn classified `mechanical_continuation|m|tools`, and the
prefix cache stayed hot (cache_hit_pct=100 on turn 2). The three
cross-check matrix cells all compress, confirming the backend accepts
the classic top-level encoding for these models and that the regression
is 0.149.0's default-model path specifically.
- Not tested: Codex over the WebSocket transport (the verified setups
pin `supports_websockets = false`; the lift sits in the shared executor
those frames also funnel through, and unit tests cover the per-frame
payload shapes); non-ChatGPT (API-key) Codex auth; models other than
gpt-5.5/gpt-5.6-sol.
## Runtime Rollout Safety
- Rollout-managed feature(s): none - not wired to the rollout system.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: only for requests carrying
`additional_tools` input items with no top-level `tools` (the Codex >=
0.149.0 default-model encoding, which today gets zero compression); all
other traffic is byte-identical.
- Kill switch / disable path: `HEADROOM_CODEX_ADDITIONAL_TOOLS_LIFT=0`
(read through `runtime_env.getenv`, so hot-reload overrides apply
without a restart).
- Unsafe override required: no.
- Qualification impact: none known.
- Rollback path: set the kill switch, or revert this single commit - the
lift is self-contained (one function + one guarded call site).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
n/a - proxy log lines quoted under Real Behavior Proof.
## Additional Notes
- Documentation checklist item is unchecked because no user-facing docs
describe the responses tools handling; happy to add a line wherever you
track client-compat notes if you have a preferred spot.
- If you would rather preserve the new wire shape upstream (compact
inside the carrier items instead of normalizing), I am happy to rework -
trade-offs are laid out in #3185.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
JD Davis
|
5e0ce242e9
|
chore: release 0.36.2 (#3157)
🤖 I have created a release *beep* *boop* --- ## [0.36.2](https://github.com/headroomlabs-ai/headroom/compare/v0.36.1...v0.36.2) (2026-08-21) ### Bug Fixes * **copilot:** bind the minted token to the integration ID we forward ([#3164](https://github.com/headroomlabs-ai/headroom/issues/3164)) ([ |
||
|
Tejas Chopra
|
4006964a03
|
fix(proxy): count output tokens from the stream's text, not its wire size (#3163)
## Description
From a user's proxy log (Copilot Chat, 0.36.x), on every streamed turn:
```
WARNING Could not parse output_tokens from SSE, estimating 8 from 334 bytes
```
When an upstream sends no usage chunk, output tokens were estimated as
`total_bytes // 40` over the **raw SSE wire** — every `data:` prefix,
JSON envelope, `role` / `finish_reason` / `id` / `model` field and
blank-line framing included.
The divisor is a fudge for "bytes per token *including framing
overhead*", so its error tracks **how chattily the answer was chunked**
rather than how long the answer was. The same text split into more
deltas scores higher purely for being split.
GitHub's Copilot CAPI is one of the upstreams that omits the usage
chunk, so this was every Copilot turn's output number — and output
tokens feed both the output-shaping savings estimate and the cost model.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- New pure module `headroom/proxy/stream_output_tokens.py`. The stream's
own text is already in the buffer at the estimation site
(`_finalize_stream_response` receives `full_sse_data`), so extract it
and count that instead of the wire.
- Handles all three forwarded surfaces: OpenAI chat
`choices[].delta.content`, OpenAI responses `*.delta`, Anthropic
`content_block_delta`.
- Counts **reasoning deltas and tool-call arguments** too — the provider
bills those as output, so omitting them would under-count exactly the
most expensive turns.
- `bytes // 40` survives only as the last resort for a stream whose text
could not be recovered. That is the upstream-error path, which reaches
the finalizer with no stream text and has no generated text to count —
so it keeps its previous behavior exactly.
- The log line named the wrong basis (it always said "from N bytes"), so
it now reports which rung produced the number.
- Parsing is I/O-free and hardened against malformed input — it runs on
the response path, where an exception would break a turn that had
already succeeded.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest tests/test_stream_output_tokens.py -q
21 passed in 0.23s
$ pytest tests/ -q -k stream
502 passed, 19 skipped
$ pytest tests/ -q # this branch
6 failed, 11394 passed, 587 skipped in 426.13s
All 6 also fail on clean origin/main, same machine — pre-existing, not regressions:
test_graceful_shutdown.py::test_run_server_installs_cancelled_error_filter
test_learn/test_integration.py::TestCodexIntegration::test_full_pipeline
test_release_workflows.py::test_no_native_tls_in_wheel_build_tree
test_providers/test_deepseek.py::... (3 litellm pricing tests)
$ ruff check headroom/
All checks passed!
$ mypy headroom/proxy/stream_output_tokens.py
Success
```
Coverage includes: per-surface extraction; reasoning/tool-argument
deltas; multi-line `data:` fields (per the SSE spec); 10 malformed-input
shapes that must yield `""` rather than raise; and the two properties
that motivated the change —
- **chunk-invariance**: the same text split one-delta vs per-character
now yields the same count, where the wire estimator disagreed wildly;
- **a short answer is never recorded as zero** (integer division would
report 0 tokens for `"OK"`).
## Real Behavior Proof
- **Environment:** macOS, Python 3.12.13, branch on `origin/main` @
`
|
||
|
Tejas Chopra
|
397803a942
|
fix(copilot): bind the minted token to the integration ID we forward (#3164)
## Description
Reported from a Copilot CLI session:
```
[CopilotCLISession] Failed to fetch models: Error: 401 "unauthorized:
unable to validate HMAC for the given Copilot-Integration-ID"
[CopilotCLISession] Proxy URL configured (authType=hmac), skipping
client-side token validation
```
GitHub **binds a Copilot API token to the `Copilot-Integration-Id` it
was minted under** and verifies the pairing with an HMAC. Present a
token minted for integration A alongside a header naming integration B,
and you get exactly this error.
`apply_copilot_api_auth` applied the integration ID with *set-default*
semantics — `_set_header_default` returns early when the header is
already present — **before** deciding whose token to use:
```python
for name, value in _copilot_chat_header_defaults().items():
_set_header_default(resolved, name, value) # ← never overwrites
...
if incoming_auth and _is_forwardable_copilot_bearer_token(...):
return resolved # client's token kept
...
token = await get_copilot_token_provider().get_api_token() # ← REPLACED
```
The client always sends an ID, so when Headroom replaced the token — the
common case, logged as `incoming token not suitable (kind=unknown), will
replace` — the request left carrying **the client's integration ID next
to Headroom's token**, minted under `vscode-chat` via
`_copilot_token_exchange_headers`. A Copilot CLI session does not
identify as `vscode-chat`.
The second log line is why nothing caught it sooner: seeing a proxy URL,
the Copilot client reports `authType=hmac` and **skips its own token
validation**, deferring to the proxy. Nobody validates the pairing until
GitHub rejects it.
**Why this matters beyond one 401:** the failing call is *model
discovery*. When it fails the client falls back to its built-in model
list — which is why a user's selected model never appeared in telemetry
and all traffic surfaced as `gpt-4o-mini`.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
Restores one invariant: **the credential and the integration ID leave
together.**
- **Mint under the client's ID** rather than the proxy's default, so
GitHub's usage attribution keeps pointing at the surface that actually
made the call.
- **Overwrite the forwarded header to match what we minted** — but only
on the replace path. The pass-through branch returns earlier and keeps
the client's own ID beside the client's own token, which is equally a
matched pair.
- **Key the token cache by integration ID.** A single slot would hand a
`vscode-chat` token to a CLI session and reproduce the same 401 straight
from cache.
Two existing contracts deliberately preserved:
- Resolution order is **client header > `GITHUB_COPILOT_INTEGRATION_ID`
> built-in default**. The env var configures the *default* this proxy
sends; it does not override a client that stated its own identity.
Pinned by the existing
`test_apply_copilot_api_auth_preserves_existing_copilot_headers` (whose
fixture literally names the value `should-not-override`).
- The overwrite writes through the client's **existing key**, so a
lowercase `copilot-integration-id` does not gain a second capitalised
variant beside it — pinned by the existing
`..._preserves_existing_headers_case_insensitively`.
Existing test stubs for `get_api_token` gained the new keyword — the
same signature-drift hazard this repo just hit in
`RemoteKompressCompressor` (#3162).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest tests/ -q -k copilot
338 passed, 8 skipped
$ pytest tests/ -q # this branch
6 failed, 11386 passed, 587 skipped in 425.40s
All 6 also fail on clean origin/main, same machine — pre-existing, not regressions:
test_graceful_shutdown.py::test_run_server_installs_cancelled_error_filter
test_learn/test_integration.py::TestCodexIntegration::test_full_pipeline
test_release_workflows.py::test_no_native_tls_in_wheel_build_tree
test_providers/test_deepseek.py::... (3 litellm pricing tests)
$ ruff check headroom/
All checks passed!
$ mypy headroom/copilot_auth.py
0 errors
```
12 new tests: the mint/forward pairing, the pass-through branch keeping
the client's pair untouched, no duplicate case-variant header,
resolution order in both directions, blank/absent client values,
non-Copilot upstreams untouched, and per-integration cache isolation.
## Real Behavior Proof
- **Environment:** macOS, Python 3.12.13, branch on `origin/main` @
`
|
||
|
Tejas Chopra
|
45cb1b9c48
|
fix(kompress): accept ccr_original on the remote compressor (#3162)
## Description
From a user's proxy log (Copilot Chat 0.61.0 on Windows, VS Code
1.133.0, Headroom 0.36.x). This appears on **every single request**:
```
WARNING Kompress failed: RemoteKompressCompressor.compress() got an
unexpected keyword argument 'ccr_original'
INFO [router] route_counts={'ratio_too_high': 1, 'cache_miss': 1} compressed=0 frozen=1 msgs=2
INFO Transform content_router: 1611 -> 1611 tokens (saved 0) [48.3ms]
INFO PERF model=... tok_before=1623 tok_after=1623 tok_saved=0 tool_saved=0 savings=none
```
`RemoteKompressCompressor`'s module docstring promises the class
"mirrors `KompressCompressor`'s public surface (`is_ready` / `preload` /
`ensure_background_load` / `compress`), so it is a drop-in at the
ContentRouter seam". That promise lapsed — the local `compress` gained a
`ccr_original` keyword and the remote one did not.
`ContentRouter._try_ml_compressor` passes `ccr_original` whenever custom
tags are protected. The comment there reads:
> Only set it when tags were protected so callers/compressors that don't
accept the kwarg are unaffected on the common path.
That assumption is wrong. The remote compressor **is** affected: the
call raises `TypeError`, which the surrounding broad `except Exception`
catches and downgrades to `logger.warning("Kompress failed: %s", e)`.
The request then forwards uncompressed and the proxy reports success.
**The blast radius is the entire deployment, not one request.**
`_get_kompress` returns the remote compressor *ahead of* every local
path, so on any install with `HEADROOM_KOMPRESS_ENDPOINT` set —
precisely the sandboxed/enterprise deployment this class exists to serve
— ML compression was silently disabled while every dashboard read
"working, 0 tokens saved".
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
Two parts, because fixing only the crash would leave the bug
`ccr_original` exists to prevent:
- **Accept the keyword** on `RemoteKompressCompressor.compress`, so the
seam contract actually holds.
- **Honor it** — store the pre-protection text in CCR rather than the
placeholder intermediate, so a later full retrieval returns the real
block instead of `{{HEADROOM_TAG_N}}`. The endpoint's own
`original_tokens` describes `content`, so when an override is supplied
the stored text is counted locally; the common path (no override) keeps
the endpoint's count exactly as before.
- **A signature-compatibility test** over the two `compress` methods, so
this drift cannot recur silently. It compares *public* keywords only —
`_deadline_started_at` is underscore-prefixed and only ever passed by
`kompress_compressor` to itself on its recursive batch path, never
across the seam.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest tests/test_remote_kompress_dropin.py -q
8 passed in 0.25s
# Same file against pre-fix code (git stash) — reproduces the reported error:
3 failed, 5 passed
FAILED test_remote_compress_accepts_every_local_keyword
FAILED test_passing_ccr_original_no_longer_raises
FAILED test_ccr_stores_the_pre_protection_text_not_the_placeholder
E TypeError: RemoteKompressCompressor.compress() got an unexpected
keyword argument 'ccr_original'
$ pytest tests/ -q -k "kompress or content_router"
411 passed, 9 skipped
$ pytest tests/ -q # this branch
6 failed, 11381 passed, 587 skipped in 446.31s
All 6 also fail on clean origin/main, same machine — pre-existing, not regressions:
test_graceful_shutdown.py::test_run_server_installs_cancelled_error_filter
test_learn/test_integration.py::TestCodexIntegration::test_full_pipeline
test_release_workflows.py::test_no_native_tls_in_wheel_build_tree
test_providers/test_deepseek.py::...v4_flash_litellm_pricing
test_providers/test_deepseek.py::...v4_pro_litellm_pricing
test_providers/test_deepseek.py::...cost_per_token_resolves_deepseek_v4_flash
(verified by stashing this branch and running test_deepseek.py: 3 failed, 17 passed)
$ ruff check headroom/
All checks passed!
$ mypy headroom/transforms/kompress_remote.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- **Environment:** macOS, Python 3.12.13, branch on `origin/main` @
`
|
||
|
JD Davis
|
1bea0ea31a
|
test: track active LiteLLM DeepSeek pricing (#3161)
## Description Keep the LiteLLM DeepSeek V4 integration tests compatible with upstream-owned pricing entries. LiteLLM now publishes these models directly, so Headroom correctly preserves upstream values instead of installing its fallback values; the tests must validate the active entry rather than require fallback prices. Related: #3157 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Validate that active upstream DeepSeek V4 price entries contain positive input and output prices. - Compare `cost_per_token` results with the active LiteLLM model-cost entry. - Preserve the existing fallback-price and non-overwrite coverage. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text python -m pytest tests/test_providers/test_deepseek.py -q 20 passed in 4.63s ruff check tests/test_providers/test_deepseek.py All checks passed! ruff format --check tests/test_providers/test_deepseek.py 1 file already formatted pre-commit: Ruff alignment, merge-conflict check, Ruff, Ruff format, and mypy all passed ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.13, LiteLLM model-cost data available. - Exact command / steps: `python -m pytest tests/test_providers/test_deepseek.py -q` - Observed result: all 20 DeepSeek provider and pricing tests pass against the active LiteLLM entries. - Not tested: provider API calls; this change only concerns local pricing metadata assertions. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: N/A. - Stable/default behavior changed: No runtime behavior changes. - Kill switch / disable path: N/A. - Unsafe override required: No. - Qualification impact: Restores deterministic CI coverage for upstream-owned pricing entries. - Rollback path: Revert this test-only commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code where needed - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] Existing tests prove the fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — test-only change. ## Additional Notes Documentation changes are not applicable because runtime behavior and public APIs are unchanged. |
||
|
dependabot[bot]
|
a382137844
|
deps: bump typescript from 5.9.3 to 7.0.2 in /plugins/opencode (#2280)
Bumps [typescript](https://github.com/microsoft/TypeScript) from 5.9.3 to 7.0.2. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/microsoft/TypeScript/releases">typescript's releases</a>.</em></p> <blockquote> <h2>TypeScript 6.0.3</h2> <p>For release notes, check out the <a href="https://devblogs.microsoft.com/typescript/announcing-typescript-6-0/">release announcement blog post</a>.</p> <ul> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.0%22">fixed issues query for TypeScript 6.0.0 (Beta)</a>.</li> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.1%22">fixed issues query for TypeScript 6.0.1 (RC)</a>.</li> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.2%22">fixed issues query for TypeScript 6.0.2 (Stable)</a>.</li> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.3%22">fixed issues query for TypeScript 6.0.3 (Stable)</a>.</li> </ul> <p>Downloads are available on:</p> <ul> <li><a href="https://www.npmjs.com/package/typescript">npm</a></li> </ul> <h2>TypeScript 6.0</h2> <p>For release notes, check out the <a href="https://devblogs.microsoft.com/typescript/announcing-typescript-6-0/">release announcement blog post</a>.</p> <ul> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.0%22">fixed issues query for TypeScript 6.0.0 (Beta)</a>.</li> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.1%22">fixed issues query for TypeScript 6.0.1 (RC)</a>.</li> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.2%22">fixed issues query for TypeScript 6.0.2 (Stable)</a>.</li> </ul> <p>Downloads are available on:</p> <ul> <li><a href="https://www.npmjs.com/package/typescript">npm</a></li> </ul> <h2>TypeScript 6.0.1 RC</h2> <p>For release notes, check out the <a href="https://devblogs.microsoft.com/typescript/announcing-typescript-6-0-rc/">release announcement blog post</a>.</p> <ul> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.0%22">fixed issues query for TypeScript 6.0.0 (Beta)</a>.</li> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.1%22">fixed issues query for TypeScript 6.0.1 (RC)</a>.</li> </ul> <p>Downloads are available on:</p> <ul> <li><a href="https://www.npmjs.com/package/typescript">npm</a></li> </ul> <h2>TypeScript 6.0 Beta</h2> <p>For release notes, check out the <a href="https://devblogs.microsoft.com/typescript/announcing-typescript-6-0-beta/">release announcement</a>.</p> <ul> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.0%22+is%3Aclosed+">fixed issues query for Typescript 6.0.0 (Beta)</a>.</li> </ul> <p>Downloads are available on:</p> <ul> <li><a href="https://www.npmjs.com/package/typescript">npm</a></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/microsoft/TypeScript/commits">compare view</a></li> </ul> </details> <details> <summary>Maintainer changes</summary> <p>This version was pushed to npm by <a href="https://www.npmjs.com/~microsoft1es">microsoft1es</a>, a new releaser for typescript since your current version.</p> </details> <br /> > **Note** > Automatic rebases have been disabled on this pull request as it has been open for over 30 days. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
dependabot[bot]
|
85774fcb70
|
deps: bump typescript from 5.9.3 to 7.0.2 in /plugins/openclaw (#2279)
Bumps [typescript](https://github.com/microsoft/TypeScript) from 5.9.3 to 7.0.2. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/microsoft/TypeScript/releases">typescript's releases</a>.</em></p> <blockquote> <h2>TypeScript 6.0.3</h2> <p>For release notes, check out the <a href="https://devblogs.microsoft.com/typescript/announcing-typescript-6-0/">release announcement blog post</a>.</p> <ul> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.0%22">fixed issues query for TypeScript 6.0.0 (Beta)</a>.</li> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.1%22">fixed issues query for TypeScript 6.0.1 (RC)</a>.</li> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.2%22">fixed issues query for TypeScript 6.0.2 (Stable)</a>.</li> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.3%22">fixed issues query for TypeScript 6.0.3 (Stable)</a>.</li> </ul> <p>Downloads are available on:</p> <ul> <li><a href="https://www.npmjs.com/package/typescript">npm</a></li> </ul> <h2>TypeScript 6.0</h2> <p>For release notes, check out the <a href="https://devblogs.microsoft.com/typescript/announcing-typescript-6-0/">release announcement blog post</a>.</p> <ul> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.0%22">fixed issues query for TypeScript 6.0.0 (Beta)</a>.</li> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.1%22">fixed issues query for TypeScript 6.0.1 (RC)</a>.</li> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.2%22">fixed issues query for TypeScript 6.0.2 (Stable)</a>.</li> </ul> <p>Downloads are available on:</p> <ul> <li><a href="https://www.npmjs.com/package/typescript">npm</a></li> </ul> <h2>TypeScript 6.0.1 RC</h2> <p>For release notes, check out the <a href="https://devblogs.microsoft.com/typescript/announcing-typescript-6-0-rc/">release announcement blog post</a>.</p> <ul> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.0%22">fixed issues query for TypeScript 6.0.0 (Beta)</a>.</li> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.1%22">fixed issues query for TypeScript 6.0.1 (RC)</a>.</li> </ul> <p>Downloads are available on:</p> <ul> <li><a href="https://www.npmjs.com/package/typescript">npm</a></li> </ul> <h2>TypeScript 6.0 Beta</h2> <p>For release notes, check out the <a href="https://devblogs.microsoft.com/typescript/announcing-typescript-6-0-beta/">release announcement</a>.</p> <ul> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.0%22+is%3Aclosed+">fixed issues query for Typescript 6.0.0 (Beta)</a>.</li> </ul> <p>Downloads are available on:</p> <ul> <li><a href="https://www.npmjs.com/package/typescript">npm</a></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/microsoft/TypeScript/commits">compare view</a></li> </ul> </details> <details> <summary>Maintainer changes</summary> <p>This version was pushed to npm by <a href="https://www.npmjs.com/~microsoft1es">microsoft1es</a>, a new releaser for typescript since your current version.</p> </details> <br /> > **Note** > Automatic rebases have been disabled on this pull request as it has been open for over 30 days. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
dependabot[bot]
|
f7e5d37f52
|
deps: bump ai from 6.0.149 to 7.0.59 in /docs (#2277)
Bumps [ai](https://github.com/vercel/ai/tree/HEAD/packages/ai) from 6.0.149 to 7.0.59. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/vercel/ai/releases">ai's releases</a>.</em></p> <blockquote> <h2>ai@6.0.253</h2> <h3>Patch Changes</h3> <ul> <li>d91d30b: Preserve reasoning block IDs from UI message streams on reasoning UI parts.</li> <li>Updated dependencies [0ec239b] <ul> <li><code>@ai-sdk/gateway</code><a href="https://github.com/3"><code>@3</code></a>.0.172</li> </ul> </li> </ul> <h2>ai@6.0.252</h2> <h3>Patch Changes</h3> <ul> <li>2f96d3f: Allow providers without reranking model support to satisfy the <code>Provider</code> type.</li> <li>afb1965: Propagate errors thrown by the Chat <code>onFinish</code> callback to the initiating request.</li> <li>Updated dependencies [18b0965]</li> <li>Updated dependencies [451d2c3] <ul> <li><code>@ai-sdk/gateway</code><a href="https://github.com/3"><code>@3</code></a>.0.171</li> </ul> </li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/vercel/ai/blob/main/packages/ai/CHANGELOG.md">ai's changelog</a>.</em></p> <blockquote> <h2>7.0.59</h2> <h3>Patch Changes</h3> <ul> <li>Updated dependencies [401a4ba]</li> <li>Updated dependencies [7af9646] <ul> <li><code>@ai-sdk/provider-utils</code><a href="https://github.com/5"><code>@5</code></a>.0.26</li> <li><code>@ai-sdk/gateway</code><a href="https://github.com/4"><code>@4</code></a>.0.47</li> </ul> </li> </ul> <h2>7.0.58</h2> <h3>Patch Changes</h3> <ul> <li> <p>72ad23f: Respect ToolLoopAgent timeouts configured in agent settings.</p> </li> <li> <p>ad6a650: feat(video): allow <code>aspectRatio: 'adaptive'</code> on <code>generateVideo</code></p> <p>Some video models derive the output ratio from the input and reject explicit <code>{width}:{height}</code> values — BytePlus Seedance 2.5 does this for first-frame, first-and-last-frame, editing, and extension tasks. <code>aspectRatio</code> on <code>VideoModelV3CallOptions</code>, <code>VideoModelV4CallOptions</code>, and <code>experimental_generateVideo</code> is now <code>`${number}:${number}` | 'adaptive'</code>, so those calls no longer need a type assertion. Support is provider-specific.</p> </li> <li> <p>81cd026: Reduce bundle size by making internal Zod v4 imports tree-shakeable.</p> </li> <li> <p>Updated dependencies [c477556]</p> </li> <li> <p>Updated dependencies [ad6a650]</p> </li> <li> <p>Updated dependencies [81cd026]</p> <ul> <li><code>@ai-sdk/gateway</code><a href="https://github.com/4"><code>@4</code></a>.0.46</li> <li><code>@ai-sdk/provider</code><a href="https://github.com/4"><code>@4</code></a>.0.7</li> <li><code>@ai-sdk/provider-utils</code><a href="https://github.com/5"><code>@5</code></a>.0.25</li> </ul> </li> </ul> <h2>7.0.57</h2> <h3>Patch Changes</h3> <ul> <li>Updated dependencies [1937bef] <ul> <li><code>@ai-sdk/provider-utils</code><a href="https://github.com/5"><code>@5</code></a>.0.24</li> <li><code>@ai-sdk/gateway</code><a href="https://github.com/4"><code>@4</code></a>.0.45</li> </ul> </li> </ul> <h2>7.0.56</h2> <h3>Patch Changes</h3> <ul> <li> <p>25c9120: Expose provider metadata on language-model-call end callbacks and telemetry spans.</p> </li> <li> <p>89080c8: fix (ai/gateway): make retried <code>doStart</code> calls idempotent</p> <p><code>generateVideo</code> retries <code>doStart</code>, which creates a billable generation, so a retry after a lost response could start a second one. It now mints one idempotency token per logical start — outside the retry closure — and forwards it as an <code>idempotency-key</code> header, so a provider that deduplicates (the Vercel AI</p> </li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
c6dd823384
|
deps: bump md-5 from 0.10.6 to 0.11.0 (#3146)
Bumps [md-5](https://github.com/RustCrypto/hashes) from 0.10.6 to 0.11.0. <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
c8db13d5ad
|
deps: bump ruff from 0.16.2 to 0.16.3 in the pip-minor-patch group (#3143)
Bumps the pip-minor-patch group with 1 update: [ruff](https://github.com/astral-sh/ruff). Updates `ruff` from 0.16.2 to 0.16.3 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/astral-sh/ruff/releases">ruff's releases</a>.</em></p> <blockquote> <h2>0.16.3</h2> <h2>Release Notes</h2> <p>Released on 2026-08-13.</p> <h3>Preview features</h3> <ul> <li>[<code>pylint</code>] Fix false negatives on negative numbers (<code>PLR6104</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/27251">#27251</a>)</li> <li>[<code>pyupgrade</code>] Add rule to replace <code>while 1</code> with <code>while True</code> (<code>UP048</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/27190">#27190</a>)</li> </ul> <h3>Bug fixes</h3> <ul> <li>[<code>flake8-bandit</code>] Also check keyword arguments (<code>S602</code>, <code>S603</code>, <code>S607</code>, <code>S609</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/27687">#27687</a>)</li> <li>[<code>pylint</code>] Allow <code>continue</code> in <code>finally</code> on Python 3.8 (<a href="https://redirect.github.com/astral-sh/ruff/pull/27626">#27626</a>)</li> <li>[<code>pylint</code>] Fix <code>PLE1307</code> false positive with bools (<a href="https://redirect.github.com/astral-sh/ruff/pull/27651">#27651</a>)</li> <li>[<code>pylint</code>] Fix false positives and negatives with <code>%b</code> format character (<code>PLE1300</code>, <code>PLE1307</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/27560">#27560</a>)</li> <li>[<code>pylint</code>] Improve handling of concatenated strings (<code>PLE1300</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/27659">#27659</a>)</li> </ul> <h3>Rule changes</h3> <ul> <li>[<code>numpy</code>] Make <code>np.chararray</code> autofix backwards-compatible (<code>NPY201</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/27527">#27527</a>)</li> </ul> <h3>Performance</h3> <ul> <li>Enable PGO for Linux x86-64 Ruff releases (<a href="https://redirect.github.com/astral-sh/ruff/pull/27570">#27570</a>)</li> <li>Enable PGO for Linux ARM64 Ruff releases (<a href="https://redirect.github.com/astral-sh/ruff/pull/27574">#27574</a>)</li> <li>Enable PGO for Windows x86-64 Ruff releases (<a href="https://redirect.github.com/astral-sh/ruff/pull/27573">#27573</a>)</li> <li>Enable PGO for macOS ARM64 Ruff releases (<a href="https://redirect.github.com/astral-sh/ruff/pull/27572">#27572</a>)</li> <li>Reduce <code>Expr</code> size to 64 bytes (<a href="https://redirect.github.com/astral-sh/ruff/pull/27591">#27591</a>)</li> </ul> <h3>CLI</h3> <ul> <li>Hyperlink rule codes in <code>ruff check --statistics</code> output (<a href="https://redirect.github.com/astral-sh/ruff/pull/27646">#27646</a>)</li> </ul> <h3>Documentation</h3> <ul> <li>[<code>ruff</code>] Also suggest <code>asyncio.TaskGroup</code> (<code>RUF006</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/27461">#27461</a>)</li> </ul> <h3>Other changes</h3> <ul> <li>Use mimalloc v3 (<a href="https://redirect.github.com/astral-sh/ruff/pull/27586">#27586</a>)</li> </ul> <h3>Contributors</h3> <ul> <li><a href="https://github.com/Andrej730"><code>@Andrej730</code></a></li> <li><a href="https://github.com/alonfaraj"><code>@alonfaraj</code></a></li> <li><a href="https://github.com/romero-deshaw"><code>@romero-deshaw</code></a></li> <li><a href="https://github.com/Avasam"><code>@Avasam</code></a></li> <li><a href="https://github.com/tjkuson"><code>@tjkuson</code></a></li> <li><a href="https://github.com/charliermarsh"><code>@charliermarsh</code></a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md">ruff's changelog</a>.</em></p> <blockquote> <h2>0.16.3</h2> <p>Released on 2026-08-13.</p> <h3>Preview features</h3> <ul> <li>[<code>pylint</code>] Fix false negatives on negative numbers (<code>PLR6104</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/27251">#27251</a>)</li> <li>[<code>pyupgrade</code>] Add rule to replace <code>while 1</code> with <code>while True</code> (<code>UP048</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/27190">#27190</a>)</li> </ul> <h3>Bug fixes</h3> <ul> <li>[<code>flake8-bandit</code>] Also check keyword arguments (<code>S602</code>, <code>S603</code>, <code>S607</code>, <code>S609</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/27687">#27687</a>)</li> <li>[<code>pylint</code>] Allow <code>continue</code> in <code>finally</code> on Python 3.8 (<a href="https://redirect.github.com/astral-sh/ruff/pull/27626">#27626</a>)</li> <li>[<code>pylint</code>] Fix <code>PLE1307</code> false positive with bools (<a href="https://redirect.github.com/astral-sh/ruff/pull/27651">#27651</a>)</li> <li>[<code>pylint</code>] Fix false positives and negatives with <code>%b</code> format character (<code>PLE1300</code>, <code>PLE1307</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/27560">#27560</a>)</li> <li>[<code>pylint</code>] Improve handling of concatenated strings (<code>PLE1300</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/27659">#27659</a>)</li> </ul> <h3>Rule changes</h3> <ul> <li>[<code>numpy</code>] Make <code>np.chararray</code> autofix backwards-compatible (<code>NPY201</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/27527">#27527</a>)</li> </ul> <h3>Performance</h3> <ul> <li>Enable PGO for Linux x86-64 Ruff releases (<a href="https://redirect.github.com/astral-sh/ruff/pull/27570">#27570</a>)</li> <li>Enable PGO for Linux ARM64 Ruff releases (<a href="https://redirect.github.com/astral-sh/ruff/pull/27574">#27574</a>)</li> <li>Enable PGO for Windows x86-64 Ruff releases (<a href="https://redirect.github.com/astral-sh/ruff/pull/27573">#27573</a>)</li> <li>Enable PGO for macOS ARM64 Ruff releases (<a href="https://redirect.github.com/astral-sh/ruff/pull/27572">#27572</a>)</li> <li>Reduce <code>Expr</code> size to 64 bytes (<a href="https://redirect.github.com/astral-sh/ruff/pull/27591">#27591</a>)</li> </ul> <h3>CLI</h3> <ul> <li>Hyperlink rule codes in <code>ruff check --statistics</code> output (<a href="https://redirect.github.com/astral-sh/ruff/pull/27646">#27646</a>)</li> </ul> <h3>Documentation</h3> <ul> <li>[<code>ruff</code>] Also suggest <code>asyncio.TaskGroup</code> (<code>RUF006</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/27461">#27461</a>)</li> </ul> <h3>Other changes</h3> <ul> <li>Use mimalloc v3 (<a href="https://redirect.github.com/astral-sh/ruff/pull/27586">#27586</a>)</li> </ul> <h3>Contributors</h3> <ul> <li><a href="https://github.com/Andrej730"><code>@Andrej730</code></a></li> <li><a href="https://github.com/alonfaraj"><code>@alonfaraj</code></a></li> <li><a href="https://github.com/romero-deshaw"><code>@romero-deshaw</code></a></li> <li><a href="https://github.com/Avasam"><code>@Avasam</code></a></li> <li><a href="https://github.com/tjkuson"><code>@tjkuson</code></a></li> <li><a href="https://github.com/charliermarsh"><code>@charliermarsh</code></a></li> <li><a href="https://github.com/chirizxc"><code>@chirizxc</code></a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
08910624fb
|
deps: bump ai from 6.0.138 to 7.0.59 in /sdk/typescript (#2281)
Bumps [ai](https://github.com/vercel/ai/tree/HEAD/packages/ai) from 6.0.138 to 7.0.59. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/vercel/ai/releases">ai's releases</a>.</em></p> <blockquote> <h2>ai@6.0.253</h2> <h3>Patch Changes</h3> <ul> <li>d91d30b: Preserve reasoning block IDs from UI message streams on reasoning UI parts.</li> <li>Updated dependencies [0ec239b] <ul> <li><code>@ai-sdk/gateway</code><a href="https://github.com/3"><code>@3</code></a>.0.172</li> </ul> </li> </ul> <h2>ai@6.0.252</h2> <h3>Patch Changes</h3> <ul> <li>2f96d3f: Allow providers without reranking model support to satisfy the <code>Provider</code> type.</li> <li>afb1965: Propagate errors thrown by the Chat <code>onFinish</code> callback to the initiating request.</li> <li>Updated dependencies [18b0965]</li> <li>Updated dependencies [451d2c3] <ul> <li><code>@ai-sdk/gateway</code><a href="https://github.com/3"><code>@3</code></a>.0.171</li> </ul> </li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/vercel/ai/blob/main/packages/ai/CHANGELOG.md">ai's changelog</a>.</em></p> <blockquote> <h2>7.0.59</h2> <h3>Patch Changes</h3> <ul> <li>Updated dependencies [401a4ba]</li> <li>Updated dependencies [7af9646] <ul> <li><code>@ai-sdk/provider-utils</code><a href="https://github.com/5"><code>@5</code></a>.0.26</li> <li><code>@ai-sdk/gateway</code><a href="https://github.com/4"><code>@4</code></a>.0.47</li> </ul> </li> </ul> <h2>7.0.58</h2> <h3>Patch Changes</h3> <ul> <li> <p>72ad23f: Respect ToolLoopAgent timeouts configured in agent settings.</p> </li> <li> <p>ad6a650: feat(video): allow <code>aspectRatio: 'adaptive'</code> on <code>generateVideo</code></p> <p>Some video models derive the output ratio from the input and reject explicit <code>{width}:{height}</code> values — BytePlus Seedance 2.5 does this for first-frame, first-and-last-frame, editing, and extension tasks. <code>aspectRatio</code> on <code>VideoModelV3CallOptions</code>, <code>VideoModelV4CallOptions</code>, and <code>experimental_generateVideo</code> is now <code>`${number}:${number}` | 'adaptive'</code>, so those calls no longer need a type assertion. Support is provider-specific.</p> </li> <li> <p>81cd026: Reduce bundle size by making internal Zod v4 imports tree-shakeable.</p> </li> <li> <p>Updated dependencies [c477556]</p> </li> <li> <p>Updated dependencies [ad6a650]</p> </li> <li> <p>Updated dependencies [81cd026]</p> <ul> <li><code>@ai-sdk/gateway</code><a href="https://github.com/4"><code>@4</code></a>.0.46</li> <li><code>@ai-sdk/provider</code><a href="https://github.com/4"><code>@4</code></a>.0.7</li> <li><code>@ai-sdk/provider-utils</code><a href="https://github.com/5"><code>@5</code></a>.0.25</li> </ul> </li> </ul> <h2>7.0.57</h2> <h3>Patch Changes</h3> <ul> <li>Updated dependencies [1937bef] <ul> <li><code>@ai-sdk/provider-utils</code><a href="https://github.com/5"><code>@5</code></a>.0.24</li> <li><code>@ai-sdk/gateway</code><a href="https://github.com/4"><code>@4</code></a>.0.45</li> </ul> </li> </ul> <h2>7.0.56</h2> <h3>Patch Changes</h3> <ul> <li> <p>25c9120: Expose provider metadata on language-model-call end callbacks and telemetry spans.</p> </li> <li> <p>89080c8: fix (ai/gateway): make retried <code>doStart</code> calls idempotent</p> <p><code>generateVideo</code> retries <code>doStart</code>, which creates a billable generation, so a retry after a lost response could start a second one. It now mints one idempotency token per logical start — outside the retry closure — and forwards it as an <code>idempotency-key</code> header, so a provider that deduplicates (the Vercel AI</p> </li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
6928d1932c
|
deps: update mcp requirement from <2.0.0,>=1.28.1 to >=1.28.1,<3.0.0 (#3144)
Updates the requirements on [mcp](https://github.com/modelcontextprotocol/python-sdk) to permit the latest version. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/modelcontextprotocol/python-sdk/releases">mcp's releases</a>.</em></p> <blockquote> <h2>v2.0.0</h2> <h1>MCP Python SDK v2 Stable Release</h1> <p>This is v2.0.0, the stable v2 release of the MCP Python SDK. It supports the 2026-07-28 revision of the Model Context Protocol and serves every earlier revision from the same server. <code>pip install mcp</code> now installs 2.x.</p> <pre lang="bash"><code>pip install "mcp[cli]" # or uv add "mcp[cli]" </code></pre> <h3>Documentation Rewrite</h3> <p>The <a href="https://py.sdk.modelcontextprotocol.io/">documentation</a> has the full tutorial and API reference. Coming from v1? <a href="https://py.sdk.modelcontextprotocol.io/whats-new/">What's new in v2</a> is the tour of what changed and why, and the <a href="https://py.sdk.modelcontextprotocol.io/migration/">migration guide</a> lists every breaking change with before-and-after code.</p> <h3>V1 Maintenance mode</h3> <p><strong>v1.x is in maintenance mode and will only receive security fixes from now on</strong> The 1.x line lives on the <a href="https://github.com/modelcontextprotocol/python-sdk/tree/v1.x"><code>v1.x</code> branch</a>, continues to receive critical bug fixes and security patches, and is documented at <a href="https://py.sdk.modelcontextprotocol.io/v1/">https://py.sdk.modelcontextprotocol.io/v1/</a>. If your project is not ready to migrate, keep a <code><2</code> upper bound on your requirement (for example <code>mcp>=1.28,<2</code>).</p> <h2>Highlights</h2> <h3>One SDK, both protocol eras</h3> <p>v2 speaks the 2026-07-28 revision (stateless requests with no handshake, <code>server/discover</code>, <code>subscriptions/listen</code>, multi-round-trip requests) and still serves every 2025-era client from the same <code>MCPServer</code>, over Streamable HTTP and stdio, with nothing to configure. <code>Client(target)</code> negotiates the version automatically.</p> <h3><code>FastMCP</code> is now <code>MCPServer</code>, and there is a first-class <code>Client</code></h3> <p>The decorator API is unchanged; the low-level <code>Server</code> is rebuilt around a shared dispatcher engine, and one <code>Client</code> object replaces v1's transport-plus-<code>ClientSession</code>-plus-<code>initialize()</code> layering. It connects to a URL, a stdio subprocess, a custom transport, or straight to a server object in memory for tests.</p> <h3>Multi-round-trip requests and resolver dependency injection</h3> <p>At 2026-07-28 the server can no longer call the client, so tools return the question instead. A <code>Resolve(fn)</code> parameter is filled by your function invisibly to the model and can put a question to the user; one tool body serves both eras.</p> <h3>Extension APIs, OpenTelemetry, and a standalone types package</h3> <p>Servers and clients compose protocol extensions through pluggable extension APIs (MCP Apps built in); OpenTelemetry tracing ships on by default; every protocol type is its own package, <code>mcp-types</code> (imported as <code>mcp_types</code>), published in lock-step with <code>mcp</code>.</p> <h3>Hardened stdio and auth</h3> <p>stdio servers keep handler subprocesses and stray prints off the wire, and stdout is diverted to stderr while serving. OAuth adds RFC 9207 issuer validation, the SEP-990 identity-assertion flow, and the client-credentials extension.</p> <h2>Coming from a v2 pre-release</h2> <p>Since the last release candidate: the per-version wire packages are private (<code>mcp_types._v*</code>), <code>mcp.types</code> is a permanent alias for <code>mcp_types</code>, the auth registration request model is split from the registered-client record, cancelled requests are no longer answered, and log notifications are gated on the per-request log-level opt-in at 2026-07-28. Since the betas: <code>Client(cache=False)</code> is now <code>cache=None</code> with <code>CacheConfig()</code> the default; <code>Context.client_id</code>, <code>RFC7523OAuthClientProvider</code>, and <code>OAuthClientProvider(timeout=)</code> are removed; the client-credentials providers take <code>scope=</code>; <code>message_handler</code> receives notifications and exceptions only; <code>FileResource(is_binary=)</code> becomes <code>encoding</code>; <code>MCP_*</code> env vars are gone with <code>pydantic-settings</code>; Streamable HTTP servers reject bodies over 4 MiB with HTTP 413. The migration guide covers all of it.</p> <h2>Known gaps</h2> <p>The tasks extension (SEP-2663) is not part of this release. On the client, the DPoP proof binding (SEP-1932) and the workload-identity <code>jwt-bearer</code> grant are not implemented; both are additive and can land in 2.x.</p> <h2>Feedback</h2> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
9c14e3aa95
|
deps: bump the cargo-minor-patch group with 8 updates (#3145)
Bumps the cargo-minor-patch group with 8 updates: | Package | From | To | | --- | --- | --- | | [aws-config](https://github.com/smithy-lang/smithy-rs) | `1.10.0` | `1.10.1` | | [rusqlite](https://github.com/rusqlite/rusqlite) | `0.40.1` | `0.40.2` | | [uuid](https://github.com/uuid-rs/uuid) | `1.24.0` | `1.24.1` | | [futures](https://github.com/rust-lang/futures-rs) | `0.3.33` | `0.3.34` | | [futures-util](https://github.com/rust-lang/futures-rs) | `0.3.33` | `0.3.34` | | [http-body-util](https://github.com/hyperium/http-body) | `0.1.4` | `0.1.5` | | [async-trait](https://github.com/dtolnay/async-trait) | `0.1.91` | `0.1.92` | | [cc](https://github.com/rust-lang/cc-rs) | `1.4.1` | `1.4.3` | Updates `aws-config` from 1.10.0 to 1.10.1 <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/smithy-lang/smithy-rs/commits">compare view</a></li> </ul> </details> <br /> Updates `rusqlite` from 0.40.1 to 0.40.2 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/rusqlite/rusqlite/releases">rusqlite's releases</a>.</em></p> <blockquote> <h2>0.40.2</h2> <h2>What's Changed</h2> <ul> <li>Lower MSRV to 1.88.0</li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/rusqlite/rusqlite/compare/v0.40.1...v0.40.2">https://github.com/rusqlite/rusqlite/compare/v0.40.1...v0.40.2</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
a307c11109
|
deps: bump tiktoken-rs from 0.11.0 to 0.12.0 (#3147)
Bumps [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs) from 0.11.0 to 0.12.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/zurawiki/tiktoken-rs/releases">tiktoken-rs's releases</a>.</em></p> <blockquote> <h2>v0.12.0</h2> <h2>Summary</h2> <p>This release backports OpenAI <code>tiktoken</code> 0.13.0 into <code>tiktoken-rs</code>. The main reason to upgrade is better alignment with upstream tokenization behavior, especially the upstream Rust core changes for large BPE pieces and error-aware encoding.</p> <p>For most users who call the high-level model/token counting helpers, this should behave the same aside from the new Rust compiler requirement. Users who call lower-level <code>CoreBPE</code> encoding methods directly should review the breaking changes below.</p> <h2>What Changed</h2> <ul> <li>Backported the vendored OpenAI <code>tiktoken</code> Rust core from 0.9.0 to 0.13.0.</li> <li>Added the upstream large-piece BPE merge path. Functionally, this improves behavior for very large or repetitive inputs that previously stressed the merge algorithm.</li> <li>Changed <code>CoreBPE::encode</code> to return <code>Result<(Vec<Rank>, usize), EncodeError></code>, matching upstream. Regex/tokenization failures can now be reported instead of being hidden behind infallible APIs.</li> <li>Updated <code>encode_as</code> and <code>count</code> to return <code>Result</code> because they call <code>encode</code>.</li> <li>Re-exported <code>EncodeError</code> so callers can handle encode failures directly.</li> <li>Aligned the vendored core with Rust 2024 and raised the crate MSRV to Rust 1.85.</li> <li>Synced model-to-tokenizer mappings with upstream <code>tiktoken</code> 0.13.0 while keeping local extra prefixes isolated.</li> <li>Hardened asset downloads with SHA-256 checks and a repo-root-aware asset path.</li> </ul> <h2>Breaking Changes</h2> <p>If your code calls <code>CoreBPE::encode</code>, unwrap or propagate the result before using the tokens:</p> <pre lang="rust"><code>let allowed = bpe.special_tokens(); let (tokens, last_piece_token_len) = bpe.encode("hello <|endoftext|>", &allowed)?; </code></pre> <p>The generic helpers changed similarly:</p> <pre lang="rust"><code>let (tokens, last_piece_token_len) = bpe.encode_as::<usize>(text, &allowed)?; let token_count = bpe.count(text, &allowed)?; </code></pre> <p><code>encode_ordinary</code>, <code>encode_ordinary_as</code>, <code>encode_with_special_tokens</code>, and <code>count_ordinary</code> remain infallible.</p> <p>Projects must now build with Rust 1.85 or newer.</p> <h2>Practical Impact</h2> <ul> <li>Applications processing long repeated text should see more robust tokenization behavior.</li> <li>Code that only uses helpers like <code>get_chat_completion_max_tokens</code>, <code>get_text_completion_max_tokens</code>, <code>bpe_for_model</code>, or singleton tokenizer constructors should not need call-site changes.</li> <li>Code using low-level <code>CoreBPE::encode</code>, <code>encode_as</code>, or <code>count</code> needs a small migration to handle <code>Result</code>.</li> </ul> <h2>Links</h2> <ul> <li>PR: <a href="https://redirect.github.com/zurawiki/tiktoken-rs/pull/164">zurawiki/tiktoken-rs#164</a></li> <li>Upstream <code>tiktoken</code> 0.13.0: <a href="https://github.com/openai/tiktoken/releases/tag/0.13.0">https://github.com/openai/tiktoken/releases/tag/0.13.0</a></li> <li>Full changelog: <a href="https://github.com/zurawiki/tiktoken-rs/compare/v0.11.0...v0.12.0">https://github.com/zurawiki/tiktoken-rs/compare/v0.11.0...v0.12.0</a></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
6e2e10f67a
|
deps: bump tokenizers from 0.22.2 to 0.23.1 (#3149)
Bumps [tokenizers](https://github.com/huggingface/tokenizers) from 0.22.2 to 0.23.1. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/huggingface/tokenizers/releases">tokenizers's releases</a>.</em></p> <blockquote> <h2>Release v0.23.1</h2> <h2>TL;DR</h2> <p><code>tokenizers 0.23.1</code> is the first proper stable release in the <code>0.23</code> line — <code>0.23.0</code> only ever shipped as <code>rc0</code> because the release pipeline itself was broken (Node side hadn't shipped multi-platform binaries since 2023, Python side was on <code>pyo3 0.27</code> without free-threaded support). <code>0.23.1</code> is the version where everything actually goes out the door together: full Node multi-platform wheels for the first time in years, Python 3.14 (regular <strong>and</strong> free-threaded <code>3.14t</code>), full type hints for every Python class, and a stack of measurable perf wins on the BPE / added-vocab hot paths.</p> <p>There is no functional <code>0.23.0</code> published — we tag <code>0.23.1</code> directly so users don't accidentally pull a never-shipped version.</p> <hr /> <h2>🚨 Breaking changes</h2> <ul> <li><strong>Drop Python 3.9</strong> (<a href="https://redirect.github.com/huggingface/tokenizers/issues/1952">#1952</a>) — <code>requires-python = ">=3.10"</code>; 3.9 users stay on <code>0.22.x</code>.</li> <li><strong><code>add_tokens</code> normalizes <code>content</code> at insertion</strong> (<a href="https://redirect.github.com/huggingface/tokenizers/issues/1995">#1995</a>) — re-saved <code>tokenizer.json</code> may differ in the <code>added_tokens</code> block. Existing files load unchanged.</li> <li><strong>Type stubs are precise</strong> (<a href="https://redirect.github.com/huggingface/tokenizers/issues/1928">#1928</a>, <a href="https://redirect.github.com/huggingface/tokenizers/issues/1997">#1997</a>) — methods that returned <code>Any</code> now return real types; <code>mypy --strict</code> may surface previously-hidden errors. Stub layout also moved from <code>tokenizers/<sub>/__init__.pyi</code> to <code>tokenizers/<sub>.pyi</code>. This breaks the surface of some of the processors like <code>RobertaProcessign</code>'s <code>__init__</code> .</li> <li><strong>3.14t-only</strong>: setters/getters return <code>PyResult<T></code> because of <code>Arc<RwLock<Tokenizer>></code>; a poisoned lock surfaces as <code>PyException</code> instead of a panic.</li> </ul> <hr /> <h2>⚡ Performance — measured locally on this Mac, not lifted from PRs</h2> <p>Run with <code>cargo bench --bench <name> -- --save-baseline v0_22_2</code> on <code>v0.22.2</code>, then <code>--baseline v0_22_2</code> on <code>v0.23.1</code>. Numbers are point-in-time wall clock on a single laptop; relative deltas are what matters, absolute numbers will differ on CI hardware.</p> <h3>Added-vocabulary deserialize — the headline win (<a href="https://redirect.github.com/huggingface/tokenizers/issues/1995">#1995</a>, <a href="https://redirect.github.com/huggingface/tokenizers/issues/1999">#1999</a>)</h3> <p><code>bench: improve added_vocab_deserialize to reflect real-world workloads</code> (<a href="https://redirect.github.com/huggingface/tokenizers/issues/2000">#2000</a>) is now representative of how transformers actually loads tokenizer.json files. The combined effect of <code>daachorse</code> for the matching automaton plus the normalize-on-insert refactor is enormous on this workload:</p> <table> <thead> <tr> <th>benchmark</th> <th align="right">v0.22.2</th> <th align="right">v0.23.1</th> <th align="right">change</th> </tr> </thead> <tbody> <tr> <td>100k tokens, special, no norm</td> <td align="right">~410 ms</td> <td align="right">248 ms</td> <td align="right"><strong>−40%</strong></td> </tr> <tr> <td>100k tokens, non-special, no norm</td> <td align="right">~7.1 s</td> <td align="right">273 ms</td> <td align="right"><strong>−96%</strong></td> </tr> <tr> <td>100k tokens, special, NFKC</td> <td align="right">~395 ms</td> <td align="right">235 ms</td> <td align="right"><strong>−40%</strong></td> </tr> <tr> <td>100k tokens, non-special, NFKC</td> <td align="right">~7.4 s</td> <td align="right">290 ms</td> <td align="right"><strong>−96%</strong></td> </tr> <tr> <td>400k tokens, special, no norm</td> <td align="right">~15 s</td> <td align="right">980 ms</td> <td align="right"><strong>−94%</strong></td> </tr> </tbody> </table> <p>Real-world impact: loading a Llama-3-style tokenizer with a large set of added tokens dropped from "noticeable pause" to "instant".</p> <h3>BPE encode</h3> <table> <thead> <tr> <th>benchmark</th> <th align="right">v0.22.2</th> <th align="right">v0.23.1</th> <th align="right">change</th> </tr> </thead> <tbody> <tr> <td><code>BPE GPT2 encode batch, no cache</code></td> <td align="right">530 ms</td> <td align="right">446 ms</td> <td align="right"><strong>−16%</strong></td> </tr> <tr> <td><code>BPE GPT2 encode batch</code> (cached)</td> <td align="right">690 ms</td> <td align="right">685 ms</td> <td align="right">noise</td> </tr> <tr> <td><code>BPE GPT2 encode</code> (single)</td> <td align="right">1.95 s</td> <td align="right">1.94 s</td> <td align="right">noise</td> </tr> <tr> <td><code>BPE Train (small)</code></td> <td align="right">32.6 ms</td> <td align="right">31.5 ms</td> <td align="right">−3%</td> </tr> <tr> <td><code>BPE Train (big)</code></td> <td align="right">1.01 s</td> <td align="right">988 ms</td> <td align="right">−2%</td> </tr> </tbody> </table> <p>The BPE per-thread cache PR (<a href="https://redirect.github.com/huggingface/tokenizers/issues/2028">#2028</a>) shows much larger wins on highly-parallel workloads (+47–62% at 88+ threads on a server box, per the PR's own measurements on Vera). Single-thread batch numbers above are flat or slightly improved because cache-hit overhead was already low without contention.</p> <h3>Llama-3 encode</h3> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
JD Davis
|
37faf2f247
|
chore: release 0.36.1 (#3152)
## Description Release 0.36.1, generated by Release Please, containing the security fixes from #2207 (WEB-01–07). This updates the changelog and keeps Python, TypeScript SDK, plugin package, marketplace, server, and release metadata versions aligned at 0.36.1. ## Type of Change - [x] Release / version metadata ## Changes Made - Updated the release manifest and generated changelog for 0.36.1. - Synchronized `pyproject.toml`, TypeScript SDK, OpenClaw, OpenCode, agent-hook plugin, marketplace, server, and release metadata versions. - Included the 0.36.1 changelog entry for the security assessment fixes merged in #2207. ## Testing - [x] CI and release validation pass ### Test Output All current required checks are complete and passing, including version sync, package builds, wheel smoke imports, security scans, Python test shards, native wrapper checks, and devcontainer validation. ## Real Behavior Proof - Environment: GitHub Actions release and CI workflows for commit `52c0a0c61dce0af81af3ff73a34efe8b451501cb`. - Observed result: all generated version-bearing files report 0.36.1; build and smoke-import jobs produced and validated the release artifacts. - Not exercised: publishing jobs are intentionally skipped for a pull request and run only after the release receives final human approval and is merged. ## Runtime Rollout Safety - Rollout-managed features: none; this PR packages already-merged behavior. - Stable/default behavior changed: no additional runtime behavior beyond the included, already-reviewed security fixes. - Kill switch / disable path: not applicable to generated release metadata. - Qualification impact: release artifact construction and smoke-import validation are green. - Rollback path: do not merge the release PR, or revert the release commit before publishing. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Release Notes ### Bug Fixes - **security:** address u9up assessment findings (WEB-01–07) (#2207) This PR was generated with Release Please and then its description was expanded to document review and qualification evidence. It still requires final human review; no publishing or merge has been performed. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Tejas Chopra
|
81fe9d5345
|
fix(metrics): attribute tool-schema savings per model, not just compression (#3155)
## Description
Reported against 0.36.0 (VS Code + Copilot + Claude Code): the per-model
breakdown disagreed with the headline printed four lines above it.
```
Tokens saved: 625,277
· messages 36,071
· tool schemas 589,206
Per-Model Breakdown
<a>: 35,907 tokens saved
<b>: 0 tokens saved
<c>: 164 tokens saved
<d>: 0 tokens saved
```
The rows sum to **36,071** — the *messages* line exactly. All 589,206
tokens of tool-schema deferral, 94% of the headline, had no row to land
in, so every tool-heavy model reported "0 tokens saved" while real
dollars were credited to it.
Deferral is disjoint from message compression by construction: deferred
schemas never enter the message token counts, so they move neither
`tokens_saved` nor `tokens_sent`. The headline, the PERF line, and the
savings ledger (#2795) all already fold the two together. Three
per-model surfaces did not.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **`perf/analyzer.py`** — the per-model loop summed `tokens_saved`
while its own headline summed `tokens_saved + tool_saved`. Now uses the
same all-layers construction (`headline_before = before + tool_saved`),
and prints a `· messages / · tool schemas` split line only when there is
a split to show.
- **`proxy/savings_tracker.py`** — added a `tool_tokens_saved` bucket to
`_empty_by_model_entry()`, normalization, and
`_record_by_model_locked()`; `record_request()` gained a
`tool_search_saved` parameter. `_by_model_snapshot_locked()` ranks and
computes `savings_percent` off the combined figure and exposes
`headline_tokens_saved`.
- **`proxy/prometheus_metrics.py`** — **the seam.** `record_request`
already accepted `tool_search_saved` and already folded it into the
per-model *dollars*, but never passed it to
`savings_tracker.record_request`. Tokens and money therefore disagreed
on the same row.
- **`proxy/cost.py`** (feeds the dashboard's "Per-Model Token Savings"
table) — added `_tool_saved_by_model`, a `tool_schema_saved` kwarg, and
`compression_tokens_saved` / `tool_tokens_saved` alongside a combined
`tokens_saved`. The `stats()` loop now iterates the **union** of both
dicts: keying off compression alone dropped a deferral-only model from
the table entirely rather than merely under-reporting it.
- **`proxy/outcome.py`** — forwards the figure it already computed for
`metrics.record_request` to `cost_tracker.record_tokens`.
- **`dashboard.html`** — the "Tokens Saved" cell gains a `title` showing
the compression/deferral split.
Design notes:
- Components stay separately addressable rather than widening an
existing field's meaning in place, so persisted state remains readable
by older readers.
- Percentages use the all-layers numerator over `saved + sent` —
deferred schemas were never in `sent`, so that is still the pre-Headroom
volume.
- `CostTracker.stats()["savings_usd"]` is deliberately **not** widened:
deferral is already priced by `SavingsTracker`, and this tracker's
dollars feed budget enforcement, where counting it twice would
double-book the saving.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest tests/test_per_model_tool_savings.py -q
11 passed in 0.94s
# Same file against pre-fix code (git stash), proving the tests bite:
5 failed, 1 passed
FAILED test_per_model_rows_reconcile_with_the_headline
FAILED test_a_tool_only_model_no_longer_reads_zero
FAILED test_tracker_attributes_deferral_to_the_model
FAILED test_tracker_default_is_unchanged_without_deferral
FAILED test_state_written_before_this_field_existed_still_loads
(the one that passes pre-fix is the "compression-only model is unchanged" guard)
$ pytest tests/ -q # this branch
3 failed, 11374 passed, 587 skipped in 343.55s
$ pytest tests/ -q # clean origin/main, same machine
3 failed, 11364 passed, 587 skipped in 352.64s
Identical 3 failures on both — pre-existing and environmental, not regressions:
test_learn/test_integration.py::TestCodexIntegration::test_full_pipeline
test_release_workflows.py::test_no_native_tls_in_wheel_build_tree (FileNotFoundError: 'cargo')
test_graceful_shutdown.py::test_run_server_installs_cancelled_error_filter
(whole-suite ordering flake; tests/test_graceful_shutdown.py passes 11/11 in isolation on this branch)
$ ruff check headroom/
All checks passed!
$ mypy headroom/proxy/cost.py headroom/proxy/savings_tracker.py \
headroom/proxy/prometheus_metrics.py headroom/proxy/outcome.py \
headroom/perf/analyzer.py
Success: no issues found in 5 source files
```
## Real Behavior Proof
- Environment: macOS, Python 3.12.13, this branch rebased on
`origin/main` @ `
|
||
|
Tejas Chopra
|
bf651c3dc1
|
fix(docker): give :latest exactly one writer (#3154)
## Description Closes #3150. `ghcr.io/headroomlabs-ai/headroom:latest` resolved to the distroless `code-slim` build, whose `import onnxruntime` segfaults on arm64. The proxy imports onnxruntime at startup in cache mode, so the container never bound its port and `headroom deploy` crash-looped (exit 139) on Apple Silicon. @ricwo's report is exceptionally good — it isolates the base image with a copy-`site-packages`-onto-`debian:trixie-slim` experiment, and explicitly retracts an earlier wrong theory about the `cpuid_info` line. I verified the tagging half independently against the live registry: ``` latest sha256:6b34905489e3... <- identical 0.36.0-code-slim sha256:6b34905489e3... <- identical 0.36.0 sha256:bb8e77d01b54... ``` **Root cause, proven from the job log rather than inferred.** `docker/metadata-action` defaults to `latest=auto`, which appends a bare `latest` for any semver release — and its own log line reads `suffixLatest=false`, meaning the per-tag `suffix=` that keeps every other tag variant-scoped never reaches it. All eight variant cells therefore emitted `:latest`, and the last to finish won. From the 0.36.0 `code-slim` cell: ``` latest=auto suffixLatest=false tags: [..."ghcr.io/headroomlabs-ai/headroom:code-slim", "ghcr.io/headroomlabs-ai/headroom:latest"] pushing sha256:fbcbb68... to ghcr.io/headroomlabs-ai/headroom:latest ``` It landed on `code-slim` by scheduling luck. Any of the eight could have won on any release. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - **`flavor: latest=false`** on the `docker-manifest` metadata-action. Stops the tag being generated at all, leaving the root-cell promotion step as the single writer of `:latest`. - **A runtime guard** in `Create multi-arch manifest`: if a suffixed variant reaches the push carrying a bare `latest`, the job fails instead of publishing. `VARIANT_NAME` is passed via `env:` rather than spliced inline. - **A test that encodes the missing half of the contract.** `test_docker_latest_promotion_is_owned_by_root_manifest_cell` already existed and passed throughout — it asserted the *intended* writer was the root cell but never the *absence of unintended ones*. The new test asserts exclusivity: `latest=false` is set, no tag rule reintroduces `value=latest`, and the guard runs before anything is pushed. ## Testing - [x] Unit tests pass (`pytest`) ### Test Output ```text tests/test_release_workflows.py 48 passed, 1 skipped, 1 failed The failure is test_no_native_tls_in_wheel_build_tree: FileNotFoundError: [Errno 2] No such file or directory: 'cargo' Pre-existing and environmental — cargo is not installed on this machine; it fails identically on a clean main checkout. ruff check: All checks passed ruff format --check: 1 file already formatted YAML parses; flavor='latest=false', env keys ['IMAGE','DIGEST_DIR','VARIANT_NAME']. ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0), worktree off `main`. Live registry queried anonymously via the GHCR token endpoint. - Exact command / steps: (1) resolved `latest`, `0.36.0` and all four variant tags to manifest digests directly from `ghcr.io/v2/.../manifests/*` to confirm the aliasing; (2) pulled the `docker-manifest (code-slim)` job log from the 0.36.0 release run to see which tags that cell actually pushed; (3) applied the fix and ran the workflow test suite; (4) **removed `latest=false` again and re-ran the new test** to confirm it reproduces the bug. - Observed result: `:latest` and `:0.36.0-code-slim` share digest `sha256:6b34905489e3...` while `:0.36.0` is `sha256:bb8e77d01b54...`, exactly as reported. The code-slim job log shows `latest=auto` / `suffixLatest=false` and `pushing ... to ghcr.io/headroomlabs-ai/headroom:latest`. With the fix removed the new test fails on `assert 'latest=false' in ''`; with it restored, it passes. - Not tested: I could not exercise the arm64 segfault or a real multi-arch push from here — no ghcr write credential and no arm64 runner. The tagging fix is verified at the config layer plus the registry evidence above; the end-to-end proof is the re-run described below. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: n/a - Stable/default behavior changed: Yes, and that is the fix — `:latest` will track the plain Debian-based build instead of whichever variant cell happened to finish last. - Kill switch / disable path: n/a (CI tagging policy). - Unsafe override required: No. - Qualification impact: A variant cell that would publish a bare `latest` now fails the Docker job loudly rather than silently repointing the default tag. - Rollback path: Revert the commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes **The live `:latest` is still wrong until the images are re-tagged.** Merging this fixes future releases but does not touch the registry. Once merged, run `docker.yml` via `workflow_dispatch` with `version=0.36.0` to rebuild and repoint `:latest` at the plain build. I don't hold a `write:packages` credential, so that step needs a maintainer. **Not fixed here, and it outlives this PR:** the distroless arm64 segfault itself. After this change `:latest` points at the Debian build that works, but `0.36.0-slim` and `0.36.0-code-slim` remain broken on arm64 for anyone selecting them explicitly. @ricwo's evidence points squarely at the distroless base — same wheel, same numpy 2.5.2, same Python 3.13.5, works on `debian:trixie-slim` and segfaults on distroless. That deserves its own issue; the two failures are independent and this one is a release-tagging bug, exactly as the report says. Related but separate, from an earlier audit of this same file: the four bare variants set `RUNTIME_USER = "root"` in `docker-bake.hcl` while `Dockerfile:162` defaults to `nonroot`, and the `runtime-default` (nonroot) bake target is referenced by the docs but by no workflow. Worth its own change. Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> |
||
|
Tejas Chopra
|
1f96dabc19
|
fix(security): address u9up assessment findings (WEB-01–07) (#2207)
Hardens client-selected upstreams, memory identity resolution, downloaded binary integrity, telemetry import, Docker defaults, Neo4j credentials, and archive extraction. Refreshes the branch against current main and preserves newer same-origin and loopback protections. |
||
|
Serge ARADJ
|
a3d9424de9
|
fix(proxy): return 502, not 200, when upstream connect retries are exhausted (#3083)
## Description When every connect retry to the upstream API fails, `_stream_response_inner` synthesizes its own SSE error response (added in #1639, so an h2 `StreamReset` wouldn't surface as an unhandled 502). It was built without a `status_code`, so Starlette defaulted it to **200**. A 200 carrying a lone `event: error` frame and no `message_start` is indistinguishable, to every Anthropic/OpenAI SDK, from a successful stream that produced no events. Claude Code reports: ``` API Error: API returned an empty or malformed response (HTTP 200) - check for a proxy or gateway intercepting the request ``` The client also cannot recover, because 200 is not a retryable status. **It does not self-heal.** Compression fails open on timeout, so the proxy forwards the full uncompressed body; the client retries, re-sends the same oversized payload, hits the same transport failure, and gets another 200. The session is stuck until the client is pointed away from the proxy. Related — same *symptom*, different root cause, so this closes none of them: #3040, #3055, #3019, #2952 (CCR buffered-stream conversion), #3071, #3017. Worth noting that #3040 ("first messages succeed, fails after several turns", closed `NOT_PLANNED`) matches this failure's shape exactly. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) Marked breaking because the status code on this path changes 200 to 502. See **Runtime Rollout Safety**. ## Changes Made - `handlers/streaming.py` — the synthesized transport-error response now returns **502**. The structured SSE body is unchanged for clients that read it. No body byte has been forwarded at that point, so the status line is still ours to set. - `prometheus_metrics.py` — new `headroom_upstream_connection_errors_total{provider}`. This path forwards no upstream status, so there was nothing to attribute the failure to in `/metrics`; it survived only as a log line. Mirrors `record_compression_failed` and takes the same `_obs_counter_lock`. - `server.py` — `HEADROOM_LOG_LEVEL` for uvicorn's level, previously hardcoded to `"warning"` with no env var and no CLI flag. Default unchanged. An unrecognized value warns and falls back rather than raising (uvicorn raises `KeyError` on unknown levels). - `docs/content/docs/proxy.mdx` — documents the new env var in the Observability table. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed `test_stream_reset_exhaustion_yields_sse_error_not_crash` asserted the SSE body but never the status — which is how the 200 survived. Added a test that pins the status specifically, a happy-path guard, and coverage for the counter and the env-var resolver. ### Test Output ```text $ python -m pytest tests/test_h2_stream_reset_retry.py tests/test_prometheus_obs_counters.py tests/test_uvicorn_log_level_env.py -q 29 passed in 5.26s $ python -m ruff check . All checks passed! $ python -m ruff format --check . 1506 files already formatted $ python -m mypy headroom/proxy/handlers/streaming.py headroom/proxy/prometheus_metrics.py headroom/proxy/server.py Success: no issues found in 3 source files # Fails before the fix (status_code=502 line removed, nothing else changed): $ python -m pytest tests/test_h2_stream_reset_retry.py -k status_is_not_200 assert result.status_code == 502 E assert 200 == 502 FAILED tests/test_h2_stream_reset_retry.py::test_stream_reset_exhaustion_status_is_not_200 1 failed, 5 deselected in 1.28s ``` Broader regression run (181 passed): `test_h2_stream_reset_retry`, `test_prometheus_obs_counters`, `test_uvicorn_log_level_env`, `test_prometheus_label_escaping`, `test_observability_metrics`, `test_prometheus_stage_timing_concurrency`, `test_proxy_streaming_ratelimit_headers`, `test_proxy_retry_429`, `test_proxy_byte_faithful_forwarding`, `test_ws_http_fallback`, `test_mid_turn_steering`, `test_proxy_anthropic_cache_stability`. ## Real Behavior Proof - Environment: Windows 11, Python 3.13.15, headroom @ this branch. Genuine `create_app()` FastAPI app under real uvicorn — no mocks, no TestClient. Upstream pinned to `http://127.0.0.1:59999` (a closed port), so every connect attempt is a real TCP refusal, producing a real `httpx.ConnectError` (an `httpx.TransportError`) into the branch under test. `retry_max_attempts=2`. - Exact command / steps: boot the real app with `HEADROOM_LOG_LEVEL=info` and `ProxyConfig(anthropic_api_url="http://127.0.0.1:59999")`, POST a `stream:true` request to `/v1/messages`, then scrape `/metrics`. Verbatim commands below. - Observed result: `HTTP_STATUS=502` (previously 200), structured SSE error body intact, `headroom_upstream_connection_errors_total{provider="anthropic"} 1`, and a uvicorn access line present only because `HEADROOM_LOG_LEVEL=info` was honored. Verbatim output below. - Not tested: the h2 `StreamReset` variant specifically — reproduced via `ConnectError`, a sibling `httpx.TransportError` travelling the identical code path (the existing `test_stream_reset_exhaustion_*` tests cover `RemoteProtocolError` at unit level). Not exercised against the OpenAI, Gemini, or Bedrock streaming handlers, which have their own error paths. No load or concurrency testing. Commands run after the patch: ```bash # boot the real app with a dead upstream and the new env var set HEADROOM_LOG_LEVEL=info python run_proxy_proof.py # ProxyConfig(anthropic_api_url="http://127.0.0.1:59999") curl -s -o resp.txt -w "HTTP_STATUS=%{http_code}\ncontent_type=%{content_type}\n" \ http://127.0.0.1:8799/v1/messages \ -H "content-type: application/json" \ -H "x-api-key: proof-key" \ -H "anthropic-version: 2023-06-01" \ -d @request.json # {"model":"claude-opus-5","max_tokens":64,"stream":true,"messages":[...]} ``` After-fix evidence: ```text PROOF: HEADROOM_LOG_LEVEL='info' -> uvicorn log_level='info' PROOF: upstream pinned to http://127.0.0.1:59999 (closed port) HTTP_STATUS=502 content_type=text/event-stream; charset=utf-8 event: error data: {"type": "error", "error": {"type": "connection_error", "message": "Failed to connect to upstream API: All connection attempts failed"}} ``` ```text $ curl -s http://127.0.0.1:8799/metrics | grep upstream_connection_errors # HELP headroom_upstream_connection_errors_total Exhausted-retries upstream transport failures by provider; the proxy answered 502 itself because no upstream response arrived # TYPE headroom_upstream_connection_errors_total counter headroom_upstream_connection_errors_total{provider="anthropic"} 1 ``` ```text # uvicorn access log — present only because HEADROOM_LOG_LEVEL=info was honored: INFO: 127.0.0.1:62472 - "POST /v1/messages HTTP/1.1" 502 Bad Gateway INFO: 127.0.0.1:62479 - "GET /metrics HTTP/1.1" 200 OK ``` All three changes are exercised end to end: the status is 502, the structured body survives, the counter increments, and the env var takes effect. Separately, this ran against a real deployment: the fix is live on a self-hosted proxy at `0.35.1-alpha.3` (Azure Container Apps, Cloudflare in front), where the original HTTP 200 was first observed against `0.35.1-alpha.1`. ## Runtime Rollout Safety - Rollout-managed feature(s): none — unconditional bug fix, no flag. - Minimum rollout channel: n/a — ships with the change. - Stable/default behavior changed: yes. This path returns 502 instead of 200. `HEADROOM_LOG_LEVEL` and the new counter both default to current behavior (`warning`; the counter is absent from `/metrics` until the first occurrence). - Kill switch / disable path: none. Happy to add an env guard if you would prefer it staged, though a 200 on this path is never correct. - Unsafe override required: no. - Qualification impact: any client treating the synthesized 200 as success now sees a 5xx. That is the fix — such a client was silently accepting a truncated response. Retry-on-5xx logic in the Anthropic and OpenAI SDKs will now retry a transient transport failure, which is the intended behavior. - Rollback path: revert the commit; single and self-contained. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — terminal output above. ## Additional Notes **Scope.** Three changes in one PR, against the "one logical change" guidance. They share a single root cause: this bug was only findable by reading `/metrics`, because the failing path emitted no status, no counter, and (see below) no usable log line. The counter and the env var are the observability that should have made it a five-minute diagnosis instead of a forensic exercise. Happy to split the `HEADROOM_LOG_LEVEL` change into its own PR if you would rather keep the fix minimal — just say so. **Related defect, filed separately as #3087.** While producing the proof above I found that the proxy's own `logger.error("Connection error to upstream API: ...")` never reaches stdout: that run produced **zero** `headroom.proxy` logger lines, only uvicorn's own. Root cause is `_setup_file_logging()` setting `propagate = False` on the `headroom` logger (`helpers.py:1536`), which sends every application record to `~/.headroom/logs/proxy.log` and nowhere else — invisible in any container, where stdout is the log channel. That is precisely why this PR adds a counter rather than trusting a log line. Not fixed here: the right remedy is a maintainer call, so it is written up in #3087 with a repro rather than folded into this PR. **No dependency changes.** The dead-upstream harness used for the proof above is ~25 lines (`ProxyConfig(anthropic_api_url="http://127.0.0.1:59999")` + `uvicorn.run(create_app(config))`); happy to contribute it as an e2e test if that is useful. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
JD Davis
|
b88b9078d8
|
chore: release 0.36.0 (#3067)
🤖 I have created a release *beep* *boop* --- ## [0.36.0](https://github.com/headroomlabs-ai/headroom/compare/v0.35.0...v0.36.0) (2026-08-20) ### Features * add deterministic runtime rollout controls ([#1490](https://github.com/headroomlabs-ai/headroom/issues/1490)) ([ |
||
|
Tejas Chopra
|
0e26fb80de
|
fix(proxy/anthropic): stop answering a non-streaming turn with an event stream (#3142)
## Description Closes #3130. Unifies #3131 (@Joaovsales) and #3132 (@taiseii), which landed within hours of each other on the same bug. Neither is redundant — **#3131 contributed the clearest statement of the contract; #3132 contributed the reconstruction that can actually be trusted to satisfy it.** This takes both. A caller that sent `stream: false` was handed a `text/event-stream` body at HTTP 200. The reply was complete — 8756 bytes, a valid upstream `request-id` — it was simply wearing a wire format the SDK cannot parse, so the turn was lost. **On root cause.** #3130 says outright: *"I could not pin down why the upstream answered a `stream`-less request with an event stream."* I think this does. At `v0.35.0` the CCR path flips the body to `stream: false` and never touches the client's `Accept` header — I checked the tag and the count of Accept rewrites at that site is **zero**. So upstream receives a self-contradicting request: *"answer as JSON"* in the body, *"I only accept SSE"* in the headers. Both reporters (#3130, #3140) show `server: cloudflare` / `cf-ray`, and both describe it as intermittent — consistent with an edge honouring `Accept` under retry. #3102 fixed that for the CCR flip; this PR moves the rewrite to the buffered boundary **every** non-streaming request reaches, so the client's own non-streaming retry is covered too. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made **From #3131 — the contract.** `headroom/proxy/nonstream_sse_policy.py`: a pure module with a behaviour matrix and `should_recover_sse_reply` as a single predicate. The three negative arms are deliberate — a streaming caller wants SSE, a JSON content-type is already correct, a non-200 carries an upstream error the client should see verbatim. **From #3132 — the reconstruction.** `require_complete=True` demands `message_start`, a terminal `message_stop`, every opened block closed, no in-band `error` event, and no delta the reconstructor cannot replay. Anything short of that is a 502. Three things only #3132 had, each load-bearing: - **`index` is stripped from rebuilt content blocks.** The parser writes it (`streaming.py:425`) and a client persists the reconstructed turn and echoes it back — at which point Anthropic 400s with `content.0.text.index: Extra inputs are not permitted`. `_strip_streaming_only_content_fields` (`anthropic.py:185`) already documents this exact failure. That inbound stripper would mask it *while the proxy is in the path*, but the client's stored history is still polluted. - **SSE framing is normalized and `data:` no longer requires the optional space.** The old `startswith("data: ")` skipped a spec-valid stream **entirely** — zero events parsed, which is literally what the report describes (*"0 stream events received"*). - **Detection sniffs the body**, so a mislabeled or absent content-type is still caught. **Reconciled where they disagreed:** - *Headers.* #3131 hand-rolled a framing list; this uses the established `sanitize_forwarded_response_headers`. That already strips `connection`, `keep-alive` and `server` alongside the content-* family — and per the comment at `helpers.py:325`, leaving `transfer-encoding` on a rebuilt body is what produced an empty HTTP 200 in #3019. #3131's list would have left three of those on. #3132's `cf-*` filter is kept. - *Detection.* The body sniff arrives as `body_is_event_stream`, so the policy module stays pure — the sniff needs the response object and the handler owns that. - Dropped #3131's `json_reply_headers` and its test class; everything else from both PRs is retained. ## Testing - [x] Unit tests pass (`pytest`) - [x] Integration tests pass ### Test Output ```text tests/test_nonstream_sse_policy.py 18 passed (from #3131) tests/test_anthropic_buffered_sse.py 18 passed (from #3132) 36 passed Regression sweep (-k "stream or sse or ccr or anthropic or proxy or buffered or usage"): 2982 passed, 181 skipped, 0 failed in 153.41s ruff check: All checks passed ruff format --check: 527 files already formatted ``` Both contributors' suites are kept whole and both pass unmodified against the merged implementation, which is the useful signal here — they were written independently against different implementations. ## Real Behavior Proof - Environment: macOS (darwin 25.4.0), Python 3.12.13, worktree off `main`, `_core.abi3.so` copied in. - Exact command / steps: applied #3132 as the engine, layered #3131's policy module over it, rewired the decision site to the predicate, then ran both suites and a 2982-test sweep concentrated on everything touching the shared SSE parser. - Observed result: 36/36 across both contributed suites, 2982 passed / 0 failed on the sweep. The sweep matters more than usual here — `_parse_sse_to_response` is shared with the streaming path's usage accounting, and `require_complete` defaults to `False` specifically so existing callers keep the lenient reconstruction they were written against. Nothing regressed. - Not tested: no live upstream. I could not reproduce the upstream answering a `stream`-less request with SSE against real `api.anthropic.com` — that is the condition #3130 reports as intermittent and load-dependent, and the Accept explanation above remains a well-supported hypothesis rather than something I observed. The fix does not depend on it: whatever the upstream returns, a caller that did not ask for streaming is no longer handed SSE. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: n/a - Stable/default behavior changed: Yes, deliberately, in two places. A non-streaming turn answered with SSE is now reconstructed as JSON instead of relayed; an SSE reply that cannot be faithfully reconstructed is now a 502 instead of an unparseable 200. Both are the point. `require_complete` defaults to `False`, so streaming callers of the shared parser are untouched. - Kill switch / disable path: none by design — relaying a body the client cannot parse has no legitimate mode. - Unsafe override required: No. - Qualification impact: A truncated upstream stream now surfaces as an explicit 502 rather than a short-but-successful turn. More visible failures, fewer silent ones. - Rollback path: Revert the commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes If this lands, #3131 and #3132 should be closed as superseded — both authors are credited via `Co-authored-by:` and their tests ship intact. I would not close either before a maintainer agrees this unification is the direction, since it discards a design decision from each. **Wider context, not fixed here:** #3130 and #3140 both report against **0.35.0**, and `main` already carries a stack of fixes for this symptom class that has never shipped — #3102 (Accept), #3092, #3091, #3094, #3101, #3069, #3084, #3124, #3134. All of them are gated behind #3067 `chore: release 0.36.0`. Every closed lookalike (#3019, #3055, #3071, #3040, #2952) was fixed into that same unreleased window. Merging this PR does not help either reporter until 0.36.0 ships; **cutting that release is the higher-leverage action.** The interim workaround for anyone on 0.35.0 is `HEADROOM_NO_CCR=1` — the buffered flip is gated on `_has_headroom_retrieve_tool`, and `no_ccr` stops the tool being injected, so the flip never engages. Note `headroom wrap` has no `--no-ccr` flag in 0.35.0, so it has to be the env var. Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> Co-authored-by: João Souto <73318835+Joaovsales@users.noreply.github.com> Co-authored-by: taiseii <37083727+taiseii@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tejas Chopra
|
93c474e84b
|
fix(deps): clear the two Rust advisories and make cargo audit blocking (#3121)
## Description
An independent OSV sweep of every locked package in the repo (1,463
across PyPI, crates.io and npm) surfaced three RUSTSEC advisories that
**no gate was reporting**:
| advisory | package | status |
|---|---|---|
| RUSTSEC-2026-0258 (GHSA-q83h-524g-xf6h) | h2 0.4.15 | fixed here →
0.4.16 |
| RUSTSEC-2026-0204 | crossbeam-epoch 0.9.18 | fixed here → 0.9.20 |
| RUSTSEC-2024-0436 | paste 1.0.15 | unmaintained, **no patched version
exists** |
**h2 is the one that matters.** It accepted and queued empty DATA frames
without limit; a peer that never drains a stream drives unbounded memory
growth, or a panic when the length overflows. It is not a corner of the
tree — it reaches the published wheel (`hf-hub -> headroom-core ->
headroom-py`) and the entire axum/reqwest/aws-config surface of
`headroom-proxy`.
**Why none of this was visible** is the more important half of this PR.
The `audit` job was already correct in one respect I initially misread —
the `rust-changes` job reports `rust=true` for `schedule`, so it *does*
run nightly rather than only on Rust changes. The actual defect is that
`cargo audit` was `continue-on-error: true`. It has been faithfully
reporting findings into a green run that nobody looks at.
Two of the three are also invisible to Dependabot entirely:
`crossbeam-epoch` and `paste` are RUSTSEC-only with no GHSA, so the
advisory database GitHub scans does not contain them. This job is their
only possible coverage.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `Cargo.lock`: `h2` 0.4.15 → 0.4.16, `crossbeam-epoch` 0.9.18 → 0.9.20.
Version + checksum only, 4 lines each way.
- `.github/workflows/rust.yml`: dropped `continue-on-error: true` from
the `cargo audit` step. `cargo deny check licenses` is deliberately left
soft-fail — `deny.toml` documents itself as intentionally permissive for
now, and tightening license policy is a separate decision.
- `.cargo/audit.toml` (new): lists `RUSTSEC-2024-0436` as accepted, with
the reason. Path matters — cargo-audit reads `.cargo/audit.toml`; a
root-level `audit.toml` is silently ignored.
`paste` is unmaintained rather than vulnerable, and there is nothing to
move to. It arrives via `tokenizers -> paste` and `rav1e -> paste`, both
under `fastembed`, so it is not actionable at our layer. Worth
revisiting when `tokenizers` adopts `pastey`.
## Testing
- [x] Manual testing performed
### Test Output
```text
Checksums verified against the real crates.io tarballs, not just the API field:
OK h2 0.4.16 (173331 bytes)
lock : a9f37a958b41b3b19ee2707c06439c0e9e547e847223eb791ecb0cb821c65e27
real : a9f37a958b41b3b19ee2707c06439c0e9e547e847223eb791ecb0cb821c65e27
OK crossbeam-epoch 0.9.20 (47545 bytes)
lock : 2d6914041f254d6e9176c01941b21115dcfb7089e55135a35411081bd106ef3f
real : 2d6914041f254d6e9176c01941b21115dcfb7089e55135a35411081bd106ef3f
Dependency-set equality (crates.io API, kind=normal):
h2 0.4.15 -> 0.4.16 : 11 deps before, 11 after, identical
crossbeam-epoch 0.9.18 -> .20: 2 deps before, 2 after, identical
```
## Real Behavior Proof
- Environment: macOS (darwin 25.4.0), worktree off `main` @ `
|
||
|
Tejas Chopra
|
709d74cd78
|
test(proxy): pin down what Anthropic's thinking signature actually covers (#3135)
## Why #3124 relaxed the signed-thinking lock on the premise that **the signature seals the thinking block, not the request**. Nothing in Anthropic's public docs states the scope, so that premise was inference — and it shipped **on by default**. This measures it instead. ## Result Each test replays a turn holding a real signed thinking block, mutates exactly one part, and asserts the request is still accepted. **Identical on all five models tested** — `sonnet-4-5`, `opus-4-5`, `sonnet-4-6`, `sonnet-5`, `opus-5`: | mutation | status | |---|---| | exact replay (control) | 200 | | compress a `tool_result` in a later user message — *what we actually do* | 200 | | rewrite sibling `text`/`tool_use` blocks **inside the assistant message holding the thinking block** | 200 | | rewrite top-level `system` + tool descriptions (schema compaction, tool-search deferral) | 200 | | re-serialize the body with reordered keys (canonical encode) | 200 | | **forge the signature** | **400** invalid signature in thinking block | ## The two tests that matter **The sibling case** is the gap the fingerprint cannot close by inspection. `thinking_blocks_survived_mutation` proves the thinking blocks are byte-identical, but says nothing about their *neighbours in the same assistant message*. If the seal covered the whole assistant turn, a compressed sibling would break it and the fingerprint would wave it through. It doesn't. **The forged-signature test is the negative control**, and the load-bearing test in the file. Without it, a wall of green would be equally consistent with *"Anthropic never validates signatures on this request shape"* — which would make every other assertion here vacuous. It 400s, so validation is live and the acceptances carry information. This also disproves #2254's stated cause directly: a plain canonical re-encode changes the bytes and is accepted. Those 400s were real, but were never traced to their true trigger. ## Scope - Gated behind `pytest.mark.live`, skipped without a key. Verified it skips cleanly (`6 skipped`) and deselects under `-m "not live"`, so CI is unaffected. - Model override via `HEADROOM_LIVE_THINKING_MODEL`. - Also replaces the speculative risk note in `body_forwarding.py` with the measured finding. The relaxation still only forwards when every thinking block is byte-identical — narrower than this evidence permits — so these results are headroom, not the safety margin. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tejas Chopra
|
284ff31947
|
fix(proxy): stop a lone surrogate turning a thinking body into a 500 (#3134)
## What
`serialize_body_canonical` uses `ensure_ascii=False`, so a lone
surrogate anywhere in the body raises `UnicodeEncodeError` at
`.encode("utf-8")`.
This is reachable input, not a hypothetical:
- `"\ud800"` is **valid JSON** — `json.loads` accepts it happily
- a tool result carrying truncated UTF-16 or sliced binary produces one
Both forwarders resolve outbound bytes **outside** their
connection-retry loop (`streaming.py:1131`, `server.py:2170`), so the
exception escapes as an **unretried 500**.
## Why now
#3124 made this newly load-bearing. Before it, a mutated
thinking-bearing body returned the client's bytes verbatim and **never
reached canonical serialization at all**. Now it does — so the largest,
most tool-result-heavy population in Claude Code traffic depends on this
not raising.
Reproduced against `main`:
```
serialize_body_canonical RAISES: UnicodeEncodeError: 'utf-8' codec can't
encode character '\ud800' in position 91: surrogates not allowed
select_outbound_body RAISES: UnicodeEncodeError: ...
```
## The fix
Fall back to the escaped encoding on `UnicodeEncodeError`.
**Why this and not passthrough.** Falling back to the client's original
bytes would silently drop every mutation — including the handler's
`stream` flip — and diverge from `outbound_body_is_client_bytes`, which
cannot predict a serialization failure without doing the serialization.
That reintroduces the #2952 buffered/streamed mismatch. The escaped form
keeps all mutations on the wire.
It encodes the **identical parsed values**, so upstream reconstructs
exactly the same request and the signed thinking blocks round-trip
untouched (asserted in the test). Only the byte-level encoding differs,
costing one cache miss on a request that would otherwise have failed
outright. Normal bodies are unaffected — the fast path is unchanged and
still emits compact non-ASCII.
## Test
`test_lone_surrogate_in_thinking_body_serializes_instead_of_raising` —
asserts no raise, `source == "canonical"`, mutation preserved, and the
signed block round-tripping to exactly the client's values.
Local: 78 passed across `test_proxy_byte_faithful_forwarding.py` +
`test_ccr_buffered_stream_signed_thinking.py`; 191 passed across all
serialization-touching tests. ruff + mypy clean.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Tejas Chopra
|
df6ff6bd5b
|
fix(deps): bump datasets past PYSEC-2026-3716 (#3136)
## Why this is urgent
`datasets` 4.5.0 picked up **PYSEC-2026-3716** — path traversal in
folder-based dataset builders, where an unvalidated `file_name` metadata
field is joined to the dataset directory, so crafted traversal sequences
can read arbitrary local files into output on
`save_to_disk`/`push_to_hub`.
**The advisory was published today between 07:17 and 15:35 UTC.**
`main`'s audit passed at 07:17 on `
|
||
|
Tejas Chopra
|
17522fb0a1
|
fix(proxy): scope the signed-thinking lock to blocks that actually changed (#3124)
## Description Anthropic signs the thinking **block**, not the request — the signature covers that block's own content. #2254 responded to real 400s by freezing the **entire body** whenever any thinking block appeared anywhere in history. That protects bytes no signature covers, including top-level `tools` and `system`, which are not even inside `messages`. Measured on 227,777 lines of real proxy logs from a user reporting ~1% savings: - **618 of 1,802 requests (34.3%)** had every computed compression discarded. **100%** were `client=claude-code`; Codex/GPT traffic was untouched. - One session logged turn 1 saving 428 tokens, then **229 consecutive turns saving exactly 0**. - **491.9s — 34.2% of all optimization time** — was spent computing compressions that were then thrown away. One request paid 21.2s to compute a real 8.0% reduction that never shipped. - It orphaned the turn-1 cache prefix on **12 of 35** sessions, corroborated by Headroom's own `CACHE-MISS-ATTRIBUTION` events (21/21 are `reason=prefix_change`, **none** TTL expiry), with an exact token match: `expected_cached=27,541` equalling turn 1's write. ## Changes Made - Replace the presence test with a **positional, order-sensitive fingerprint** of every `thinking` / `redacted_thinking` block, compared against the client's original. Byte-equal blocks → forward the edits. Any difference (edited text, edited signature, dropped, reordered, moved) or any failure to prove equality → today's verbatim passthrough. Keys are sorted so a dict rebuilt in a different order is not mistaken for an edit. - `outbound_body_is_client_bytes` mirrors the relaxation exactly, or the CCR buffering probe and the forwarder would disagree and re-create #2952 in reverse. - The #2990/#3015 accounting reset now **recomputes** the lock immediately before use instead of reusing the probe taken before the CCR branch. The predicate tests block *content* now, and `enforce_cache_control_ttl_order` rewrites `body["messages"]` in between, so the early answer can go stale. (Latent before this PR; load-bearing after.) - **Perf:** parse the client body once per decision, plus a substring prescreen. A 9.3 MB body (the real production maximum) could otherwise be parsed four times per request on a stage that already carries a 30s timeout whose expiry quarantines compression process-wide. ## Rollout safety **On by default at the maintainer's explicit direction.** `HEADROOM_THINKING_PRESERVING_MUTATIONS=0` restores the previous blanket lock with no deploy. The risk is recorded in the module rather than smoothed over: #2254's stated cause — a plain canonical re-encode — cannot alter parsed values and therefore cannot by itself invalidate a signature, and that report's own log shows a transform (`tool_search_deferral`) firing on the failing turn. So the stated cause does not hold up, **but the failure was real and its true trigger was never isolated.** This relaxation is strictly narrower than what broke: it forwards edits only when every block is provably identical, which is the property the blanket rule was a crude proxy for. ## Testing ```text uv run pytest tests/test_proxy_byte_faithful_forwarding.py tests/test_ccr_buffered_stream_signed_thinking.py \ tests/test_proxy/test_anthropic_ccr_deferred_injection.py 92 passed uv run mypy headroom/proxy/body_forwarding.py headroom/proxy/handlers/anthropic.py # Success uv run ruff check . && ruff format --check . # clean ``` Existing tests that encoded the blanket lock were **re-pointed at the correct trigger, not deleted** — each now tampers with a thinking block so it still guards what it was written for. `test_signed_thinking_discarded_mutation_uses_wire_truth_for_all_accounting` (#3015) now runs under the kill switch, which proves both that the accounting neutralisation still works and that the env-var rollback is a complete restoration. ## Real behavior proof - **Setup:** macOS arm64, Python 3.12, this branch, byte-capturing transport. - **After-fix evidence** — end-to-end through `/v1/messages` with a signed thinking block in history and a compactable tool schema (`test_untouched_thinking_lets_tool_compaction_reach_the_wire`): the annotation keys the compaction strips (`$schema`, `title`) are **absent from the captured upstream bytes**, and `wire["messages"][1]["content"][0]` is **byte-identical to the client's signed block**. Under the kill switch the same request forwards the client's bytes unchanged with accounting zeroed. - **Parse-count measured, not assumed:** 7.2 MB thinking-bearing body → 2 parses became 1. 2 MB body with no thinking blocks (~2 of 3 requests) → 1 parse became **0**, i.e. faster than before this feature existed. - **Projected effect on the reporting user's traffic**, derived from their unlocked requests: Claude Code headline **2.27% → roughly 5–6%**. Their unlocked requests already achieve 5.62% overall and 7.2–7.4% in the 20K–150K band, which matches our fleet beacon (~8%); the 2.27% is a blend where 60% of tokens sat in requests that shipped nothing. - **NOT tested: live paid Anthropic traffic with a real signed thinking block.** This is the one thing that matters most and I could not do it here. The signature-verification behaviour is Anthropic's, and no local test can prove it accepts a re-serialized body carrying an untouched block. **Please validate on live traffic before relying on the default.** Watch for 400 `invalid_request_error` mentioning `thinking`, and `CACHE-MISS-ATTRIBUTION reason=prefix_change` rates. ## Known risk not eliminated Enabling this changes the wire bytes for in-flight sessions, so expect a **one-time prefix change** on the first affected turn of each live conversation. Supporting evidence that this is bounded: canonical serialization is already the norm for the ~66% of traffic without thinking blocks, and that traffic sustains a 94.3% cache hit rate. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |