mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
254 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
JD Davis
|
34a5517562
|
chore: release 0.36.5 (#3214)
🤖 I have created a release *beep* *boop* --- ## [0.36.5](https://github.com/headroomlabs-ai/headroom/compare/v0.36.4...v0.36.5) (2026-08-22) ### Bug Fixes * **codex:** detect ChatGPT auth from id_token claims so wrap/init emit requires_openai_auth ([#3212](https://github.com/headroomlabs-ai/headroom/issues/3212)) ([ |
||
|
JD Davis
|
91186b40d8
|
chore: release 0.36.4 (#3189)
🤖 I have created a release *beep* *boop* --- ## [0.36.4](https://github.com/headroomlabs-ai/headroom/compare/v0.36.3...v0.36.4) (2026-08-22) ### Bug Fixes * **dashboard:** pin MIME types for the vendored static assets ([#3193](https://github.com/headroomlabs-ai/headroom/issues/3193)) ([ |
||
|
JD Davis
|
87e71dd100
|
chore: release 0.36.3 (#3188)
🤖 I have created a release *beep* *boop* --- ## [0.36.3](https://github.com/headroomlabs-ai/headroom/compare/v0.36.2...v0.36.3) (2026-08-21) ### Bug Fixes * **proxy/responses:** lift Codex >= 0.149.0 additional_tools into top-level tools ([#3186](https://github.com/headroomlabs-ai/headroom/issues/3186)) ([ |
||
|
JD Davis
|
5e0ce242e9
|
chore: release 0.36.2 (#3157)
🤖 I have created a release *beep* *boop* --- ## [0.36.2](https://github.com/headroomlabs-ai/headroom/compare/v0.36.1...v0.36.2) (2026-08-21) ### Bug Fixes * **copilot:** bind the minted token to the integration ID we forward ([#3164](https://github.com/headroomlabs-ai/headroom/issues/3164)) ([ |
||
|
JD Davis
|
37faf2f247
|
chore: release 0.36.1 (#3152)
## Description Release 0.36.1, generated by Release Please, containing the security fixes from #2207 (WEB-01–07). This updates the changelog and keeps Python, TypeScript SDK, plugin package, marketplace, server, and release metadata versions aligned at 0.36.1. ## Type of Change - [x] Release / version metadata ## Changes Made - Updated the release manifest and generated changelog for 0.36.1. - Synchronized `pyproject.toml`, TypeScript SDK, OpenClaw, OpenCode, agent-hook plugin, marketplace, server, and release metadata versions. - Included the 0.36.1 changelog entry for the security assessment fixes merged in #2207. ## Testing - [x] CI and release validation pass ### Test Output All current required checks are complete and passing, including version sync, package builds, wheel smoke imports, security scans, Python test shards, native wrapper checks, and devcontainer validation. ## Real Behavior Proof - Environment: GitHub Actions release and CI workflows for commit `52c0a0c61dce0af81af3ff73a34efe8b451501cb`. - Observed result: all generated version-bearing files report 0.36.1; build and smoke-import jobs produced and validated the release artifacts. - Not exercised: publishing jobs are intentionally skipped for a pull request and run only after the release receives final human approval and is merged. ## Runtime Rollout Safety - Rollout-managed features: none; this PR packages already-merged behavior. - Stable/default behavior changed: no additional runtime behavior beyond the included, already-reviewed security fixes. - Kill switch / disable path: not applicable to generated release metadata. - Qualification impact: release artifact construction and smoke-import validation are green. - Rollback path: do not merge the release PR, or revert the release commit before publishing. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Release Notes ### Bug Fixes - **security:** address u9up assessment findings (WEB-01–07) (#2207) This PR was generated with Release Please and then its description was expanded to document review and qualification evidence. It still requires final human review; no publishing or merge has been performed. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
JD Davis
|
b88b9078d8
|
chore: release 0.36.0 (#3067)
🤖 I have created a release *beep* *boop* --- ## [0.36.0](https://github.com/headroomlabs-ai/headroom/compare/v0.35.0...v0.36.0) (2026-08-20) ### Features * add deterministic runtime rollout controls ([#1490](https://github.com/headroomlabs-ai/headroom/issues/1490)) ([ |
||
|
Tejas Chopra
|
93f2d7a2da
|
chore: release main (#2792)
🤖 I have created a release *beep* *boop* --- <details><summary>0.35.0</summary> ## [0.35.0](https://github.com/headroomlabs-ai/headroom/compare/v0.34.0...v0.35.0) (2026-08-12) ### Features * **beacon:** allowlist the routing summary key ([#2818](https://github.com/headroomlabs-ai/headroom/issues/2818)) ([ |
||
|
Tejas Chopra
|
9fd5ae3d53
|
chore: release main (#2679)
🤖 I have created a release *beep* *boop* --- <details><summary>0.34.0</summary> ## [0.34.0](https://github.com/headroomlabs-ai/headroom/compare/v0.33.0...v0.34.0) (2026-08-05) ### Features * **claude:** support Claude Code in VS Code ([#2752](https://github.com/headroomlabs-ai/headroom/issues/2752)) ([ |
||
|
Tejas Chopra
|
28aa53dc7c
|
chore: release main (#2339)
🤖 I have created a release *beep* *boop* --- <details><summary>0.33.0</summary> ## [0.33.0](https://github.com/headroomlabs-ai/headroom/compare/v0.32.0...v0.33.0) (2026-07-29) ### Features * **lossless:** factor shared directory prefix in the grep search fold ([#2547](https://github.com/headroomlabs-ai/headroom/issues/2547)) ([ |
||
|
Tejas Chopra
|
4381388d56
|
chore: release main (#1923)
## Description Release Please generated the 0.33.0 release PR for main. This updates release metadata, package versions, and the generated changelog for the 0.33.0 release. I also aligned the agent-hook plugin manifests, marketplace metadata, editable lockfile package version, and canonical MCP `server.json` descriptor to 0.33.0 so all package/plugin/registry version declarations match the Release Please version bump. ## Type of Change - [x] Documentation update - [x] Release / packaging metadata ## Changes Made - Updated `.release-please-manifest.json`, `pyproject.toml`, `plugins/openclaw/package.json`, and `sdk/typescript/package.json` to 0.33.0. - Updated the generated `CHANGELOG.md` release notes for 0.33.0. - Synced `plugins/headroom-agent-hooks` plugin manifests and marketplace metadata to 0.33.0. - Synced `uv.lock` editable `headroom-ai` package version to 0.33.0. - Regenerated the canonical MCP `server.json` descriptor to 0.33.0. ## Testing - [x] Version verification passes - [x] Version-sync tests pass - [x] MCP server descriptor test passes - [x] Whitespace check passes ### Test Output ```text uv run python scripts/verify-versions.py All versions aligned at 0.33.0 uv run pytest scripts/tests/test_version_sync.py scripts/tests/test_sync_plugin_versions.py -q 14 passed in 0.80s uv run pytest tests/test_mcp_registry/test_server_json.py -q 4 passed in 0.42s git diff --check # no output ``` ## Real Behavior Proof - Environment: Windows 11, local checkout of the Release Please branch. - Exact command / steps: Ran version verification and MCP descriptor tests after syncing release metadata, plugin marketplace versions, lockfile version, and `server.json`. - Observed result: All package, plugin manifest, marketplace, lockfile, and MCP descriptor release versions are aligned at 0.33.0. |
||
|
Gautam Sharma
|
5279c33b19
|
fix(memory): preserve semantically similar memories (#2303)
## Description
Prevent memory_save from automatically deleting semantically similar but
distinct memories. The previous fire-and-forget deduplication path
deleted existing memories at cosine similarity scores of 0.92 or higher
after the save had already returned success. Similarity remains
available as a consolidation hint, while supersession now requires an
explicit memory_update or memory_delete operation.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Removed the automatic background deletion scheduled by memory_save.
- Removed the automatic-dedup threshold and background coroutine that
were no longer needed.
- Preserved the existing similarity search and consolidation hint.
- Kept explicit memory_update and memory_delete behavior unchanged.
- Added a regression test proving that distinct memories survive even at
0.99 simulated similarity.
- Added an Unreleased changelog entry.
## Testing
- [x] Unit tests pass (pytest)
- [x] Linting passes (ruff check .)
- [x] Type checking passes (mypy headroom)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
$ uv run --extra dev --frozen pytest
tests/test_memory_handler_native_ops.py
33 passed
$ uv run --extra dev --frozen ruff check .
All checks passed!
$ uv run --extra dev --frozen ruff format --check
headroom/proxy/memory_handler.py tests/test_memory_handler_native_ops.py
2 files already formatted
$ uv run --extra dev --frozen mypy headroom --ignore-missing-imports
Success: no issues found in 504 source files
$ uv run --extra dev --frozen pytest
9361 passed, 565 skipped, 4 failed
The four full-suite failures are unrelated to this diff: the Anthropic
compaction test passed in isolation; the Codex recovery test exceeded
the macOS AF_UNIX path limit; the dashboard test expects text absent
from the existing implementation; and the content-router test expects a
fallback absent from the existing strategy chain.
The repository-wide format check also flags pre-existing formatting in
the untouched headroom/proxy/handlers/anthropic.py.
## Real Behavior Proof
- Environment: macOS on Apple Silicon, CPython 3.12.13, real
LocalBackend, temporary SQLite database, and the local
sentence-transformers
embedding backend; no external provider or model API.
- Exact command / steps: Ran uv run --extra dev --frozen python with a
temporary database, saved User's primary backend framework at work is
FastAPI., queried its similarity to User's primary backend framework at
home is FastAPI., saved the second fact through
MemoryHandler._execute_save, and listed the user's memories.
- Observed result: The real embedding similarity was 0.9387, above the
former 0.92 deletion threshold. The second save returned saved,
included the consolidation hint, retained the original memory, and left
both distinct facts in the database (memory_count: 2).
- Not tested: Live OpenAI or Anthropic provider calls, a deployed proxy
or MCP client session, and Qdrant or Neo4j memory backends. These
paths share the handler policy changed here; backend-specific explicit
update and delete behavior is unchanged.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The documentation and code-comment checklist items are not applicable
because this change removes unsafe behavior without introducing a new
public interface or complex implementation. The full-suite checkbox
remains unchecked because four unrelated tests failed locally, as
documented above.
|
||
|
Aashish Tamsya
|
420dc9077b
|
feat(grok-build): add Grok Build wrap command and MCP integration (#1629)
## Description
Adds first-class Grok Build support to Headroom so Grok CLI sessions can
route through the local proxy for context compression and savings
tracking.
This PR introduces `headroom wrap grok-build` / `headroom unwrap
grok-build`, a `grok_build` provider slice, Grok MCP registrar support,
and install/telemetry wiring so Grok traffic is attributed correctly in
the proxy and dashboard.
Review follow-up (`9368c413`): when users already own
`[model.grok-build]` in `~/.grok/config.toml`, wrap rewrites `base_url`
in that table in place instead of appending a duplicate header (invalid
TOML).
## Type of Change
- [x] New feature (non-breaking change that adds functionality)
## Changes Made
- Added `headroom/providers/grok_build/` with runtime helpers,
reversible `~/.grok/config.toml` injection, and install env builders.
- Added `headroom wrap grok-build` and `headroom unwrap grok-build` CLI
commands.
- Added `GrokRegistrar` for Headroom MCP registration in Grok config.
- Wired `grok_build` into install planner/registry, agent savings,
telemetry, and proxy client detection (`grok/` user agent).
- **Review fix:** rewrite `base_url` inside an existing user-owned
`[model.grok-build]` table in place (`# was: …` metadata).
- Added regression tests + docs (`grok-build.mdx`, `proxy.mdx`) and
CHANGELOG entry.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest -q tests/test_provider_grok_build.py tests/test_mcp_registry/test_grok_registrar.py
============================== 12 passed in 1.13s ==============================
```
See **Screenshots** below for terminal captures (pytest, review-fix
in-place rewrite, proxy `/readyz`, unwrap).
## Real Behavior Proof
- Environment: macOS, Python 3.11.12 venv, feat/grok-build @ `
|
||
|
monkeygold
|
02c77640a9
|
fix(transforms): guard Log fallback against invalid JSON + fix MIXED false-positive on source code (#1347)
## Summary Three related fixes in the content router/detector, addressing data-loss and misrouting bugs found via chaotic audit: - **SMART_CRUSHER → Log fallback guard (#1306):** Truncated/invalid JSON tool outputs were tagged `json_array` by the native magika detector (classifies by shape, not parseability), routed to SmartCrusher (no-op), Kompress (no-op), then collapsed by LogCompressor to a single CCR-retrieval marker — **99.9% data loss** when CCR retrieval isn't configured. A JSON-validity guard (`_content_is_valid_json`) now skips the Log fallback for content that fails `json.loads`; valid JSON arrays still reach it (LogCompressor is a no-op on them). - **MIXED false-positive on source code:** `is_mixed_content` regex heuristics misclassify Python with dict/list literals (`{`, `[` at line start → `has_json_blocks`) + docstrings (`has_prose`) as MIXED, wasting 1–1.4s latency with 0% compression. When the native detector confidently says `SOURCE_CODE` (confidence ≥ 0.8), `_determine_strategy` now trusts it over the regex heuristics. - **PASSTHROUGH for code when CodeAware disabled:** When `prefer_code_aware_for_code=False` (default), source code now uses `PASSTHROUGH` instead of `KOMPRESS`, honouring the config's "let code pass through unmangled" intent. KOMPRESS can destroy code semantics (98% compression, 11% fact recall on large blobs). - **RecursionError hardening:** Caught in both `_try_detect_json` and `_content_is_valid_json` so deeply nested JSON (`[[[[...]]]]` with 10k+ levels) no longer crashes the detector/router — also serves as a DoS mitigation. #### Test plan - [x] `tests/test_transforms_content_router.py` — 36 passed (8 new tests) - [x] `tests/test_transforms_content_detection.py` — 9 passed - [x] `tests/test_cache_aligner_detector_only.py` — 22 passed - [x] `tests/test_compression_decision.py`, `test_compression_policy.py`, `test_compress_api.py`, `test_compression_safety_rails.py` — 137 passed, 5 skipped - [x] `ruff check` on changed files — all checks passed - [x] `mypy` on changed files — no issues found New tests cover: - Invalid JSON skips Log fallback (content preserved verbatim) - Valid JSON arrays still reach Log fallback - MIXED false-positive overridden by high-confidence SOURCE_CODE detection - Low-confidence SOURCE_CODE does NOT override MIXED (safety) - Genuine mixed content (PLAIN_TEXT detection) still uses MIXED - PASSTHROUGH preserves code verbatim, never invokes Kompress - CodeAware explicitly enabled still uses CODE_AWARE #### Risks / rollback - Behaviour change: code blobs previously routed through MIXED→KOMPRESS now use PASSTHROUGH. This is the documented intent of `prefer_code_aware_for_code=False`; if a deployment relied on the accidental KOMPRESS compression of code, set `prefer_code_aware_for_code=True` to restore CODE_AWARE. - The JSON-validity guard adds one `json.loads` call in the narrow "no savings" fallback path only — negligible overhead. - Revert is a single-commit revert; no schema/migration changes. Generated with [Devin](https://devin.ai) Co-authored-by: monkeygold <monkeygold@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Sneha Roy
|
e8bff1cfe3
|
feat: add CrewAI and AutoGen tool compression integrations (#1384)
## Description Add CrewAI and AutoGen tool compression integrations, following the same patterns as the existing LangChain agent integration (`HeadroomToolWrapper` / `wrap_tools_with_headroom`). Both delegate compression to `compress_tool_result()` from the MCP integration, with per-tool metrics tracking via `ToolCompressionMetrics` / `ToolMetricsCollector`. Closes #1379 ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - Add `headroom/integrations/crewai/` — `HeadroomToolWrapper` subclasses CrewAI `BaseTool`, wraps `_run()` with compression - Add `headroom/integrations/autogen/` — `HeadroomToolWrapper` wraps AutoGen `FunctionTool` (sync and async) with compression - Wire both into `headroom/integrations/__init__.py` with aliased re-exports (avoids name collision with LangChain's `HeadroomToolWrapper`) - Add `[crewai]` and `[autogen]` optional dependency extras to `pyproject.toml` - Add 24 unit tests (12 per framework) under `tests/test_integrations/` - Add `.mdx` doc pages for both frameworks under `docs/content/docs/` - Update `CHANGELOG.md` with entries under `### Added` ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check headroom/integrations/crewai headroom/integrations/autogen tests/test_integrations/crewai tests/test_integrations/autogen All checks passed! $ pytest tests/test_integrations/autogen -v 12 passed $ pytest tests/test_integrations/crewai -v 12 passed ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.11, crewai 1.14.7, autogen-agentchat 0.7.5 - Exact command / steps: Ran standalone adapter demos and benchmark runner across 4 task types - Observed result: | Task | Tokens (raw) | Tokens (compressed) | Savings | |------|-------------|-------------------|---------| | Inventory JSON (80 items) | 5,044 | 1,532 | 69.6% | | Server logs (150 lines) | 8,712 | 314 | 96.4% | | Analytics query (100 rows) | 10,762 | 10,762 | 0% | | API docs (20 endpoints) | 8,043 | 8,043 | 0% | Compression results are identical across CrewAI and AutoGen — expected since both route through the same `compress_tool_result()` pipeline. - Not tested: Full end-to-end with a live LLM agent loop (demos test the compression pipeline standalone). LangGraph not included — headroom already has `headroom/integrations/langchain/langgraph.py`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - LangGraph integration is intentionally excluded — headroom already has one at `headroom/integrations/langchain/langgraph.py` - Re-exports in `__init__.py` are aliased (`CrewAIToolWrapper`, `AutoGenToolWrapper`) to avoid collision with the existing LangChain `HeadroomToolWrapper` - Both integrations follow the exact same conventions as the existing LangChain agents module: optional dep guard, `compress_tool_result()` delegation, metrics with 1000-entry cap, Google-style docstrings - `mypy` not checked due to Rust build dependency (`maturin`) that requires Application Control policy changes on this machine --------- Co-authored-by: Sneha27feb <sroy27.ai@gmail.com> |
||
|
Zhenjia ZHOU
|
4035c04187
|
feat(text-crusher): CJK-aware segmentation + relevance via ICU (#1504)
## Description `TextCrusher` (the native extractive prose compressor added in #1171) only handled ASCII: `split_segments` split on `.!?`+whitespace and `tokens` split on whitespace/alphanumeric runs. CJK (Chinese/Japanese/Korean) has neither spaces nor ASCII terminators, so a whole CJK paragraph collapsed into **one segment / one token** — it passed through at ~0% compression, and BM25 relevance + salience scored zero terms. This makes `TextCrusher` CJK-aware. CJK-bearing content takes an ICU (`icu_segmenter`, UAX#29 sentence + dictionary word) segmentation path, with a length fallback for terminator-sparse runs, a local BM25 over the ICU word tokens, and ICU-token salience. Dispatch is on **content only**, so pure-ASCII text is byte-identical to before — the shared `BM25Scorer` and the ASCII path are untouched. It also adds a committed, reproducible answer-retention eval (`benchmarks/i18n_compression_eval.py`) with a deterministic zh/ja/ko CI regression gate, so the improvement below is permanently verifiable rather than a one-off measurement. Extends #1171. ## Type of Change - [x] Bug fix (CJK passed through near-uncompressed) - [x] New feature (CJK segmentation / relevance support) - [x] Performance improvement (CJK now compresses; ICU segmenters cached, not rebuilt per call) ## Changes Made - `is_cjk` predicate gates a CJK path (ideographs, kana, Hangul, CJK punctuation, full/half-width forms). - `split_segments` → ICU `SentenceSegmenter` for CJK + a mandatory length fallback (whitespace / CJK punctuation / hard cap) for terminator-sparse runs; ASCII path unchanged. - `tokens` → ICU `WordSegmenter` (dictionary) for CJK; ASCII path unchanged. - `relevance_cjk`: a local BM25 over ICU word tokens — the shared ASCII `BM25Scorer` scores zero terms for CJK and is parity-locked, so this is an intentional separate scorer (documented in code). - CJK salience uses ICU tokens (whitespace-split gave one giant "word" → zero salience). - `count_tokens`: CJK-aware so `compression_ratio` isn't nonsense for space-free text. - ICU segmenters resolved once in `static LazyLock` (compiled_data is static) instead of rebuilt per call. - New dep `icu_segmenter` 2.2, `compiled_data` only (see Dependency below). - `benchmarks/i18n_compression_eval.py` + `tests/test_transforms/test_text_crusher_cjk_eval.py`: a zh/ja/ko answer-retention eval — a deterministic needle CI gate (always-runs, no external data), real-transcript fidelity with CJK-aware salient, and optional `multi-wiki-qa` natural-data retention (loaded via the `[evals]` `datasets` extra, skipped if absent; data never vendored — CC-BY-NC-SA). ## Testing - [x] Unit tests pass (`pytest` + `cargo test`) - [x] Linting passes (`ruff check`/`format` on the new eval + test — clean) - [ ] Type checking passes (`mypy headroom`) — N/A, the only Python added is a benchmark + test, not `headroom/` source - [x] New tests added for new functionality - [x] Manual testing performed (see Real Behavior Proof) ### Test Output ```text $ cargo test -p headroom-core --lib text_crusher running 12 tests test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 841 filtered out $ .venv/bin/python -m pytest tests/test_transforms/test_text_crusher*.py 15 passed $ .venv/bin/python -m pytest tests/test_transforms/test_text_crusher_cjk_eval.py 6 passed # deterministic zh/ja/ko needle CI gate $ cargo clippy -p headroom-core && ruff check benchmarks/i18n_compression_eval.py # both clean ``` ## Real Behavior Proof - Environment: macOS (Darwin 25.3.0), Python in a uv venv, `headroom-core` built via `uv pip install -e .` (maturin), branch `feat/cjk-text-compression`. - Exact command / steps: built `_core`, then ran a mixed Chinese+Japanese doc (no spaces, `。` terminators) through `TextCrusher().compress(doc, "认证令牌缓存策略", 0.3)`; separately evaluated answer-retention on the public CMRC2018 Chinese QA dev set (bury the gold-answer paragraph among 25 distractors, query = the question, compress to 30%, check the gold answer survives), and end-to-end through `ContentRouter`. - Observed result: a mixed Chinese+Japanese doc compressed 189 → 78 tokens (ratio 0.41, kept 3/8 segments) with the query-relevant sentence surviving — before this change the same doc was a single segment → 100% passthrough. On the public CMRC2018 Chinese QA dev set, answer-retention under 30% compression rose 34% → 93% (multiple seeds). End-to-end through `ContentRouter` on real CJK content, aggregate savings rose 16% → 40%. Pure-ASCII (English) output stayed byte-identical (the English parity fixtures did not move). Demo terminal output: ```text ORIGINAL tokens= 189 chars=189 COMPRESS tokens= 78 ratio=0.41 segments kept 3/8 QUERY-RELEVANT sentence survived: True --- compressed output (verbatim kept CJK sentences) --- 认证令牌的缓存策略采用最近最少使用淘汰算法来管理过期条目。 请求重试使用指数退避并设置最大次数上限。 数据备份每天凌晨执行并保留最近三十天的快照。 ``` The committed eval now demonstrates this across all three CJK languages. The deterministic needle gate (in CI via `tests/test_transforms/test_text_crusher_cjk_eval.py`, 6 passed) has TextCrusher keep the query-relevant needle while truncate/random drop it in zh, ja, and ko. On real `multi-wiki-qa` natural data (n=80/lang), query-aware answer-retention is **zh 74% / ja 70% / ko 50%** vs **25–41%** for the truncate/random baselines: ```text === Part A: multi-wiki-qa answer-retention (n=80/lang, target_ratio=0.3) === lang text_crusher truncate random zh-cn 74% 25% 38% ja 70% 31% 39% ko 50% 26% 41% ``` Korean is measurably weaker (ICU has no Korean dictionary and falls back to UAX#29 word-breaking) — still well above baselines, and scoped as a follow-up. - Not tested: the live proxy HTTP path (validated at the `ContentRouter` / `TextCrusher` layer, not via a running proxy); no-space Korean (standard Korean is space-delimited and is covered); non-CJK SE-Asian scripts (out of scope). ## Dependency (per CONTRIBUTING supply-chain policy) `icu_segmenter` 2.2 (ICU4X), `features = ["compiled_data"]`: - **Why this package (vs. ourselves / existing deps):** CJK needs dictionary/UAX#29 segmentation. A hand-rolled char-bigram scored slightly worse on real data (CMRC2018 answer-retention: 92.5% ICU vs 91% bigram, 4 seeds); jieba/lindera are ZH-only or 13–207 MB dicts. ICU4X covers zh/ja/ko in one crate. The existing `unicode-segmentation` does UAX#29 only (no CJK dictionary), so it can't word-segment space-free CJK. - **Who maintains it:** the official `unicode-org` ICU4X project; active release cadence (2.2 in 2025); no known CVEs. - **Install surface:** ~13 new pure-Rust crates, no build scripts, no native code, no build/runtime network. `compiled_data` bundles locale data at compile time (hermetic). `auto`/`lstm` deliberately NOT enabled — LSTM covers SE-Asian scripts (Thai/Lao), not CJK, and would pull in `libm` for nothing. - **Why this version:** 2.x is the stabilized ICU4X API (1.x used a different data-provider model); floored at 2.2 (Cargo.lock pins the patch) since segmenter boundaries are observable in output and bumps should be deliberate. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (CHANGELOG) - [x] My changes generate no new warnings (clippy + fmt clean) - [x] I have added tests that prove my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md ## Additional Notes - **Parity:** the shared `BM25Scorer` (byte-exact parity-locked with `headroom/relevance/bm25.py`) is untouched. `relevance_cjk` is a separate local scorer because the shared one's tokenizer is ASCII-only. The whole CJK path lives in Rust (`text_crusher.py` is a thin wrapper over `_core`), so there is no Python mirror to keep in sync; the parity fixtures stay green (only the CJK `unicode` fixture was re-recorded, intentionally; English fixtures unchanged). - **Known by-design gap (not a bug):** CJK content + a pure-ASCII query yields no token overlap, so relevance falls back to recency + salience (cross-script query matching is unsupported). - The Python added is a benchmark (`benchmarks/i18n_compression_eval.py`) plus its test, not `headroom/` runtime source — both are `ruff`-clean; `mypy headroom` is unaffected. - **License:** the optional Part A pulls `alexandrainst/multi-wiki-qa` (CC-BY-NC-SA-4.0) at run time via the `[evals]` extra and is skipped if absent — the dataset is never vendored into the repo, and the always-run CI gate (Part C) uses only our own deterministic data. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
gglucass
|
12a9710665
|
feat(stats): per-bucket output-shaping savings in /stats-history (#1819)
## Description Adds per-bucket **output-shaping savings** to `/stats-history`. Today output-shaping savings exist only as a single global aggregate (`savings.by_layer.output_shaping`), so downstream consumers can't chart them over time. This threads a per-request output-savings estimate into the existing rollup so every `series` bucket carries `output_tokens_saved_delta` + `output_savings_usd_delta`, symmetric with the existing `compression_savings_usd_delta`. Motivation: on Claude Code subscription traffic, input is ~99% cache-discounted (the compressible live zone is a fraction of a percent), while output shaping is a ~36% reduction on full-price output tokens — so it's the dominant, honestly-attributable saving, and currently the only one a dashboard can't render per day. Closes #1816 ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - `output_savings.py`: new read-only `SavingsRecorder.estimate_request_savings(labels, output_tokens)` → per-request synthetic-control estimate `max(0, baseline_mean(stratum) - output_tokens)` for treatment requests; 0 for control / unknown stratum / no label. Does **not** mutate the ledger, so it composes with `record_from_labels` without double-counting. `record_from_labels`'s `bool` contract is unchanged. - `outcome.py`: in the funnel, capture that estimate and pass it to `record_request(output_tokens_saved=...)`. - `savings_tracker.py`: `record_request` gains `output_tokens_saved`; accumulates lifetime cumulative `output_tokens_saved` / `output_savings_usd` (priced via new `_estimate_output_savings_usd`, output-rate), writes them into each checkpoint, and now checkpoints when **either** compression **or** output savings occurred (so output-only requests aren't dropped). `_build_rollup` diffs the cumulative into `output_tokens_saved_delta` / `output_savings_usd_delta` per bucket; `_normalize_history_entry` and the CSV export carry the fields. - Additive + backward-compatible: checkpoints predating the feature default the new fields to 0. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --extra dev pytest tests/test_output_shaping_rollup.py tests/test_output_savings.py \ tests/test_output_savings_cli.py tests/test_proxy_savings_history.py tests/test_request_outcome.py -q ... 103 passed $ uv run --extra dev ruff check headroom/proxy/savings_tracker.py headroom/proxy/output_savings.py \ headroom/proxy/prometheus_metrics.py headroom/proxy/outcome.py tests/test_output_shaping_rollup.py All checks passed! $ uv run --extra dev mypy headroom/proxy/savings_tracker.py headroom/proxy/output_savings.py Success: no issues found in 2 source files ``` New tests (`tests/test_output_shaping_rollup.py`): output savings bucket into the daily series; an output-only request (no compression) still checkpoints; pre-feature requests default to 0; `estimate_request_savings` returns the baseline-relative saving for treatment and 0 for control / unknown / over-baseline. ## Real Behavior Proof - Environment: macOS, CPython 3.10.18, this branch (rebased on latest `main`), litellm pricing available. - Exact command / steps: seed a baseline (as `learn --verbosity` would), then drive 3 requests through the real, unmocked chain `SavingsRecorder.estimate_request_savings` → `SavingsTracker.record_request` → `history_response()`, and print `series.daily`. Full script + raw output: ```text $ uv run python proof.py # seeds baseline ~1000 out-tok; 3 treatment requests (out=600/550/700), one with no compression [ { "timestamp": "2026-07-05T00:00:00Z", "tokens_saved": 120, "compression_savings_usd_delta": 0.0006, "output_tokens_saved_delta": 850, "output_savings_usd_delta": 0.02125 }, { "timestamp": "2026-07-06T00:00:00Z", "tokens_saved": 80, "compression_savings_usd_delta": 0.0004, "output_tokens_saved_delta": 300, "output_savings_usd_delta": 0.0075 } ] ``` - Observed result: output-shaping savings appear per day and independent of the compression axis. 2026-07-05 = 850 (400+450 saved by two treatment requests vs the ~1000-token baseline, including one request with zero compression — proving the output-only checkpoint path), 2026-07-06 = 300, each priced at the model's output rate. Matches expectations. - Not tested: the full live proxy over HTTP with a real learned baseline and organic traffic — I exercised the same code path minus the HTTP/streaming layer. The measured-vs-estimated `method` gating is unchanged by this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — backend-only change (no UI surface in this repo). The runtime effect is the `/stats-history` `series.daily` JSON with the new `output_tokens_saved_delta` / `output_savings_usd_delta` fields, shown under **Real Behavior Proof** above. The downstream chart that renders them lives in the separate Headroom desktop app. ## Additional Notes - Per CONTRIBUTING's issue-first policy for features, I opened #1816 first with the spec; happy to adjust the API surface (field names / gating) to whatever you prefer. A downstream consumer (Headroom desktop chart) is already implemented against this exact contract and stacks the segment only when `output_reduction.method == "measured"`. - Docs checkbox left unchecked: I didn't find a `/stats-history` schema doc to update; point me at one if it exists. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Krishna Chaitanya
|
57e8dcb425
|
feat(proxy): add opt-in cost-aware model router (#1706) (#2205)
## Description Adds an optional, configuration driven model router (closes #1706). With `HEADROOM_MODEL_ROUTER_ENABLED` set, ordered rules in `HEADROOM_MODEL_ROUTES` rewrite the upstream model by estimated input size and tool presence, complementary to content compression, for example sending small, tool-free requests to a cheaper model. First matching rule wins and every decision is logged with a reason. Off by default so behavior is unchanged, skipped under `x-headroom-bypass`/passthrough, and wired on the Anthropic `/v1/messages` path. Malformed rules fail open, so a bad rule is skipped rather than silently widened. Closes #1706 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/model_router.py`: new `ModelRouter` component (ordered rules, first-match decision with reason, fail-open env parsing, tokenizer-free input estimate). - `headroom/proxy/models.py` + `headroom/proxy/server.py`: `ProxyConfig.model_router` field, env loader (`HEADROOM_MODEL_ROUTER_ENABLED` / `HEADROOM_MODEL_ROUTES`), and proxy wiring. - `headroom/proxy/handlers/anthropic.py`: apply routing on `/v1/messages` after the bypass gate, tracked as a body mutation. - Tests, docs (`configuration.mdx`), and a CHANGELOG entry. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest -q tests/test_proxy/test_model_router.py tests/test_proxy/test_model_router_wiring.py 36 passed, 1 warning $ ruff check . All checks passed! $ mypy headroom --ignore-missing-imports Success: no issues found in 477 source files ``` ## Real Behavior Proof - Environment: local, macOS, Python 3.12, headroom `.venv`, upstream mocked (no live provider call). - Exact command / steps: enable the router via `ProxyConfig(model_router=...)`, POST `/v1/messages` through `TestClient` with a rule routing low-risk requests to a cheaper model; repeat with header `x-headroom-bypass: true`. - Observed result: the forwarded upstream body model is rewritten from `claude-sonnet-4-6` to `claude-haiku-4-5` when the router is enabled, and is left unchanged under bypass (see `tests/test_proxy/test_model_router_wiring.py`). - Not tested: the OpenAI and Gemini handler paths (this PR wires the Anthropic path only); no live provider request (upstream is mocked). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes Happy to adjust the interface or scope (for example OpenAI and Gemini parity) if you'd prefer a different shape. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: reneleonhardt <65483435+reneleonhardt@users.noreply.github.com> |
||
|
Abhay Singh
|
842d7e1ad1
|
fix(ccr): lowercase a retrieved hash so an uppercase echo still hits the store (#2236)
## Description
A CCR retrieval fails whenever the model echoes the content hash in
uppercase, even though the content is present in the store.
`parse_tool_call` extracts and validates the hash from a
`headroom_retrieve` tool call:
```python
# Validate hex characters only
if not all(c in "0123456789abcdef" for c in hash_key.lower()):
return None
return hash_key
```
The hex check is deliberately case-insensitive (`hash_key.lower()`), so
an uppercase hash passes validation — but the value is then returned
**verbatim**. The compression store, however, keys every entry by a
lowercase hash: writes use either a sha256 hexdigest
(`hashlib.sha256(...).hexdigest()[:24]`, always lowercase) or
`explicit_hash.lower()`, and `retrieve` / `get_entry_status` look the
key up as-is with no normalization.
So when a model reproduces the marker hash in uppercase (LLMs routinely
normalize hex casing when they copy tokens), the retrieve endpoint
validates it, calls `store.retrieve("ABC…")` against a store that only
holds `"abc…"`, and reports a miss — the original content is unreachable
even though it is right there. The case-insensitive validation shows the
intent was to accept either casing; only the return value was left
un-normalized.
## Fix
Return the canonical lowercase form so the whole pipeline is
consistently lowercase:
```python
return hash_key.lower()
```
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/ccr/tool_injection.py`: `parse_tool_call` returns
`hash_key.lower()`.
- `tests/test_ccr_tool_injection.py`: new test asserting an uppercase
hash is normalized to lowercase.
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/ccr/tool_injection.py tests/test_ccr_tool_injection.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/ccr/tool_injection.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx
mypy@1.20.2`. A full `pytest` OOM-kills this box (ML stack import), so I
reproduced the validate/return + a lowercase-keyed store with a
dependency-free script and left the full pytest to CI.
- Exact command / steps: put `"abc123def456abc123def456" -> content` in
a store, then looked it up with the uppercase echo
`"ABC123DEF456ABC123DEF456"` through the OLD (return verbatim) and NEW
(return `.lower()`) paths.
- Observed result: OLD returns the uppercase hash → store miss; NEW
returns the lowercase hash → store hit (original content recovered). A
lowercase hash resolves under both.
- Not tested: a live model round-trip that uppercases the marker; full
local `pytest` deferred to CI (OOM).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" box is unchecked because the full suite
imports the ML stack, which I can't run here. The new test lives
alongside the existing `parse_tool_call` tests in
`tests/test_ccr_tool_injection.py`, so it runs under the normal CI
pytest job; behaviour is additionally verified by the standalone proof
above.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
LunarECL
|
fcf455a7eb
|
feat(wrap): add omp target (Oh My Pi) with models.yml override and unwrap (#1811)
## Description Adds `headroom wrap omp` / `headroom unwrap omp` — a one-command wrap for [Oh My Pi](https://www.npmjs.com/package/@oh-my-pi/pi-coding-agent) (`omp`), the pi-mono-lineage coding agent, as proposed in #1149. One honest correction to the issue: #1149 proposed reusing the `ANTHROPIC_BASE_URL` redirect from `wrap claude`. During implementation I probed that empirically and it turned out to be wrong — omp only reads `ANTHROPIC_BASE_URL` in its web-search helper; its **chat** endpoint comes from the model registry (`providers.anthropic.baseUrl` in `~/.omp/agent/models.yml`). With the env var pointed at a local probe server, omp's chat traffic still went straight to the real endpoint (0 probe hits); with a `models.yml` same-ID override, every request arrived at the probe (9/9 hits on `/v1/messages`). A same-ID override keeps omp's bundled Anthropic model catalog and stored credentials (both keyed by provider id `anthropic`), so only the endpoint moves. The wrap therefore injects a marker-fenced `providers.anthropic.baseUrl` override into `models.yml`, snapshotting the pre-wrap file **byte-for-byte** first, and `headroom unwrap omp` restores it exactly (or removes the file when the wrap created it) — the same durable-wrap + backup + unwrap contract `wrap codex` uses for `config.toml`. Closes #1149 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/providers/omp/` (new provider slice): `models_yml_path()` (honors `PI_CODING_AGENT_DIR`), `inject_models_override()` (yaml-merge preserving user providers; pristine byte-for-byte backup, never re-snapshotted while managed), `restore_models_override()` (`restored` / `removed` / `noop`; never touches an unmanaged file), `build_launch_env()` - `headroom/cli/wrap.py`: `wrap omp` (mirrors the aider/vibe `_launch_tool` shape; rtk instructions into the project's `AGENTS.md`, which omp reads natively) and `unwrap omp` (restore models.yml + scrub rtk block + stop proxy) - `headroom/telemetry/context.py`: `omp` added to `_KNOWN_WRAP_AGENTS` so the stack slug reports `wrap_omp` instead of `unknown` - `README.md` (agent matrix row + unwrap list), `llms.txt`, `CHANGELOG.md` - `tests/test_cli/test_wrap_omp.py`: 16 tests (injection fresh/merge/re-inject, restore statuses incl. unmanaged-file safety, env passthrough, CLI wiring, unwrap flows) ## Testing - [ ] Unit tests pass (`pytest`) — all new + `test_cli` tests pass; the full suite carries **3 pre-existing failures** that reproduce identically on unmodified `origin/main` (same set, same asserts — see Test Output and the rebase-validation comment) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest -q # post-rebase, base |
||
|
Vsevolod Rychkov
|
7a5d8a7ace
|
fix(mcp): reap orphaned mcp serve on client death (#2226)
## Description `headroom mcp serve` processes survive after the launching MCP client (e.g. Claude Code) exits, get reparented to init/launchd (`ppid == 1`), and never terminate — piling up one pinned Python interpreter + tree-sitter grammars per dead session (observed 3+ simultaneously). An MCP stdio server is supposed to shut down on stdin EOF, but an abrupt client `SIGKILL` leaves the MCP SDK's blocking stdin-reader thread wedged, so `await self.server.run(...)` in `run_stdio()` never returns and the process orphans. Refs #2185 (its secondary "orphaned `mcp serve` pileup", left out of #2204's `Refs`-only Perl fix), #1761 (same symptom: "orphaned `headroom mcp serve` processes accumulate … even after quitting"). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made `headroom/ccr/mcp_server.py`: - Added `PARENT_DEATH_POLL_INTERVAL = 5.0` module constant. - Added `HeadroomMCPServer._await_parent_death(interval)`: captures the launch ppid and resolves once it changes. Watching for a *change* (not a hard `== 1`) is portable to Linux PID subreapers, which adopt the orphan with their own pid. - Reworked `run_stdio()` to run that watchdog concurrently with `server.run()`. On parent death it `os._exit(0)`s **from inside** the `stdio_server()` context manager — the wedged stdin reader would also hang the context-manager teardown and a cooperative `server.run` cancel, so a hard exit is the only reliable reaper. The normal stdin-EOF path is unchanged: `server.run` wins the race, the watchdog is cancelled, and the context manager unwinds cleanly. `tests/test_ccr_mcp_server.py`: 3 regression tests (below). `CHANGELOG.md`: entry under Unreleased → Fixed. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_ccr_mcp_server.py -q`) - [x] Linting passes (`uv run ruff check headroom/ccr/mcp_server.py tests/test_ccr_mcp_server.py`) - [x] Type checking passes (`uv run mypy headroom/ccr/mcp_server.py`) - [x] New tests added for new functionality - [x] Manual testing performed (see Real Behavior Proof) New tests: - `test_parent_death_watchdog_fires_when_reparented` — ppid change resolves the watchdog. - `test_parent_death_watchdog_stays_quiet_with_live_parent` — a stable ppid never trips it. - `test_run_stdio_reaps_process_on_parent_death` — on reparent, `run_stdio` cleans up and hits `os._exit(0)` even though the (stubbed) `server.run` never returns. ### Test Output ```text $ uv run pytest tests/test_ccr_mcp_server.py -q collected 21 items tests/test_ccr_mcp_server.py ..................... [100%] ============================== 21 passed in 0.57s ============================== $ uv run ruff check headroom/ccr/mcp_server.py tests/test_ccr_mcp_server.py All checks passed! $ uv run mypy headroom/ccr/mcp_server.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS 26.5 arm64, Python 3.14.6, headroom built from this branch via `uv sync --all-extras` (Rust extension compiled). No provider call. - Exact command / steps: launch a real `HeadroomMCPServer.run_stdio()` as a child of a throwaway parent, with stdin wired to a FIFO whose write end is held open by a separate process (so stdin **never** reaches EOF — this isolates the watchdog as the only possible reaper). Then `kill -9` the parent to reparent the server to `pid 1`, and watch. The watchdog poll interval is passed via `run_stdio(parent_death_poll_interval=…)` to A/B the exact same shipped code path: ```text ### interval=9999s (watchdog effectively OFF — reproduces the bug) ### ppid(pre-kill)=43438 -> STILL ALIVE after 8s (orphan lingers) ### interval=0.5s (watchdog ON — the fix) ### ppid(pre-kill)=43461 -> REAPED at ~2s ``` And with the default flow (`headroom mcp serve`, default 5s interval), the watchdog logs before the process exits: ```text headroom.ccr.mcp - INFO - Headroom MCP Server starting (proxy: http://127.0.0.1:8787) headroom.ccr.mcp - WARNING - parent process gone (ppid 41956 -> 1); shutting down MCP server ``` - Observed result: with the watchdog disabled the orphaned server lingers indefinitely (reproduces the reported pileup); with it enabled the orphan is reaped within one poll interval of the parent dying. - Not tested: Linux/systemd and Windows spawn paths (the change is POSIX-portable via ppid-change detection, but I only exercised macOS); the reporters' desktop-app menu-bar quit path. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md ## Additional Notes - Deliberately `os._exit(0)`, not a cooperative shutdown: the failure mode is a wedged native stdin-reader thread, so both `server.run` cancellation and the `stdio_server` context-manager exit can block forever. Exiting from inside the context manager is the only path that reliably reaps the orphan; the normal EOF path never reaches it. - A Linux-only `prctl(PR_SET_PDEATHSIG)` fast-path could cut reap latency to ~0, but it is racy (must re-check `getppid()` after arming) and non-portable, so the portable poll is the primary mechanism. Happy to add prctl as a follow-up optimization if wanted. - Watchdog latency is bounded by `PARENT_DEATH_POLL_INTERVAL` (5s default); trivial to make env-configurable if a tighter bound is preferred. --- 🤖 This PR was created with [Claude Code](https://claude.com/claude-code) but checked by the author Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Andrei Boldyrev
|
7bfb1d7f38
|
fix(cache): stable session identity and per-conversation prefix trackers under agentic clients (#2193)
## Description Running headroom as the proxy for Claude Code destroys Anthropic prompt-cache reuse (#2085: ~4.4x cache-creation inflation, 2.5–3x net cost). Tracing live Claude Code traffic through the proxy shows **two independent session-identity defects**, both of which orphan or thrash the frozen-prefix state; this PR fixes both. ### Defect 1: `<system-reminder>` turns rotate the fallback session id mid-conversation Claude Code interleaves reminder turns into the history as actual `role:"system"` messages (hook output, skills lists, file-truncation notices). `compute_session_id` hashed **every** system message, so the id rotated each time a reminder landed. Live trace (subagent reading two 80KB files; sid changes exactly when the truncation reminder appears, and the tracker restarts at turn 0): ``` REQ#2 sid=68d4ee666990 nmsg=3 [0]SYSTEM<<top-level system>> [1]user [2]SYSTEM<<skills reminder>> REQ#3 sid=6944948c9fb2 nmsg=6 ... [5]SYSTEM<<Truncated: PARTIAL view ...>> <- id rotated ``` Everything keyed on the session id is orphaned at that moment: the prefix tracker (freeze never survives past a reminder-bearing turn), beta-header stickiness, the CCR and memory-tool registries, and the compression cache. **Fix:** hash only the **leading run** of system messages (everything before the first non-system turn) — the top-level system prompt on the Anthropic path (folded in as the synthetic first message), the conventional leading system message(s) on the OpenAI path. Stable for the life of a conversation; mid-history system turns are content, not identity. ### Defect 2: conversations sharing a (now stable) id thrash one tracker With ids stable, the fallback tuple `model + system prompt` is identical across every same-type parallel subagent (and any sessions reusing one system prompt) — all of them collapse onto one `PrefixCacheTracker`, and their interleaved histories cross-contaminate the freeze state: the forwarded prefix is byte-unstable on nearly every turn and the provider cache is re-written instead of read. Reproduced against the real code paths (script below): ``` 1) fallback session ids: A=3dc639aaf4f48fa1 B=3dc639aaf4f48fa1 -> COLLIDE=True 2) single conversation, legacy : stable prefix on 4/4 later turns, trackers=1 2) interleaved (subagents), legacy : stable prefix on 0/9 later turns, trackers=1 2) interleaved, lineage resolution : stable prefix on 8/8 later turns, trackers=2 ``` **Fix:** `SessionTrackerStore.resolve_tracker` — within a session id, reuse the tracker whose previous request messages are a prefix of the incoming history (client histories are append-only, so a conversation's next request always extends its previous one); a diverging or rewritten history (client-side compaction) starts a fresh lineage. Matching uses the repo's existing canonical cross-turn equivalence (`_canonicalize_for_prefix_compare`, the same one the cache-stable delta path uses) on the **original client bytes**, so moved cache breakpoints, string<->block sugar, transport annotations, or a tail-mutating `pre_compress` hook never read as a rewrite. Byte-identical histories (templated fan-outs before they diverge) intentionally share a tracker — their provider cache line is identical too. ### Both fixes together, on live Claude Code traffic (sonnet, 2 parallel Explore agents) ``` main conversation: sid=5b7e245a... one tracker, turns 0->4, id stable across reminders agents (collide): sid=2bdffc9e... -> lineage bare (alpha) turns 0->1->2 -> lineage "~1" (beta) turns 0->1->2 ``` Before: the agents' ids rotated per reminder (every tracker stuck at turn 0), and whenever they did share an id they thrashed one tracker (`0/9` stable prefixes in the repro). ### Why not key the session id on conversation content? Draft #1912 folds the first user turn into the fallback id; this change composes with it, but identity-level keying alone can't close #2085: identical first turns (templated fan-outs) still collide, and everything keyed on the session id rotates with it when the client rewrites history. The "session" (client/workspace grouping) and the "conversation" (positional cache lineage) are different identities; only the tracker holds positional per-turn state that thrashes under collision — beta stickiness is a monotone union and the compression cache is content-addressed — so lineage resolution lives one level below the session id and leaves the id semantics (and every other consumer) untouched. ## Changes Made - `headroom/cache/prefix_tracker.py`: - `compute_session_id`: harvest only the leading system run (defect 1). - `SessionTrackerStore.resolve_tracker`: conversation-lineage resolution (defect 2). First lineage lives under the bare session id — single-conversation sessions behave byte-identically to before; degrades to `get_or_create` when messages are absent or prefix freeze is disabled. - Lineages are capped per session id (`PrefixFreezeConfig.max_lineages_per_session`, default 32). **Over-cap conversations share one overflow tracker instead of evicting an established lineage** — any eviction policy degrades every conversation once the working set exceeds the cap (under round-robin the victim is always the conversation about to arrive), while overflow sharing degrades only the over-cap tail, to exactly the pre-lineage shared behavior; `0` disables lineage splitting. Chains are stored as structural snapshots that normalize `NaN` (`json.loads` accepts bare NaN, and `NaN != NaN` would read a byte-identical resend as a rewrite). Synthetic lineage keys use a `\x00` separator, which cannot appear in an HTTP header value, so they can never collide with a client-supplied `x-headroom-session-id`. - `headroom/proxy/handlers/anthropic.py`, `openai.py`: the session id and the lineage both derive from the **same original client bytes** (a turn-dependent hook rewrite can no longer rotate one without the other); anthropic folds in its synthetic system message so explicit-header clients with different system prompts stay separate. Plus a docstring correction in `streaming.py` that falsely claimed its coarse mid-turn key "mirrors" `compute_session_id`. - `tests/test_cache/test_prefix_tracker.py`: 24 new test cases — reminder-rotation regression; interleaved isolation + per-conversation turn state; identical-first-turn share-then-split; cache_control movement (3 cases); representation churn (string<->block sugar / streaming `index` / Bedrock cachePoint); rewritten history → fresh lineage (compacted / middle-edited / truncated); legacy no-messages / freeze-disabled / empty-canonical fallbacks; NaN-in-tool-payload stability; overflow sharing, established-lineages-survive-cap, and a cap+1 round-robin no-cliff guard; TTL cleanup; session-id-not-rotated-by-lineage guard. One existing test renamed (`uses_all_system_messages` → `distinguishes_leading_system_run`) to match the new contract. - Three SimpleNamespace stub stores in existing tests gained a `resolve_tracker` field (handlers call it unconditionally — a silent `hasattr` fallback would degrade to the pre-fix behavior with no signal). One of them is the cold-start fast-pass suite (#2073), which landed while this branch was in review. - `CHANGELOG.md` entry. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Testing - [x] Unit tests pass (`pytest`) — 11 failed, 8652 passed, 528 skipped in 4:37 (the 11 are pre-existing on unmodified `main` — verified by rerunning the same node ids on a clean checkout: gh-CLI/onnx/PID-reuse/deadline flakes and order-dependent cases, none touching session/cache/proxy paths) - [x] Linting passes (`ruff check .`) — All checks passed (ruff 0.15.17, CI-pinned; `ruff format --check .` clean) - [x] Type checking passes (`mypy headroom`) — Success: no issues found in 471 source files - [x] New tests added for new functionality — 24 test cases; the rotation/isolation/no-cliff ones fail on `main` - [x] Manual testing performed — live Claude Code end-to-end, below ### Test Output ```text $ python -m pytest tests/ -q 11 failed, 8652 passed, 528 skipped, 5857 warnings in 276.68s (0:04:36) # same 11 fail on unmodified main (env/order-dependent: test_wrap_claude_base_url pid-reuse, # copilot_auth gh-cli fallback, image_compression onnx, content_router deadline, rtk/output-shaper/dedup order flakes) $ python -m pytest tests/test_cache/test_prefix_tracker.py -q 63 passed $ uvx ruff@0.15.17 check . && uvx ruff@0.15.17 format --check . All checks passed! / 1208 files already formatted $ mypy headroom Success: no issues found in 471 source files $ python repro_2085.py 1) fallback session ids: A=3dc639aaf4f48fa1 B=3dc639aaf4f48fa1 -> COLLIDE=True 2) single conversation, legacy : stable prefix on 4/4 later turns, trackers=1 2) interleaved (subagents), legacy : stable prefix on 0/9 later turns, trackers=1 2) interleaved, lineage resolution : stable prefix on 8/8 later turns, trackers=2 ``` ## Real Behavior Proof - Environment: macOS arm64, Python 3.13, `uv sync --extra dev --extra proxy`; real Claude Code CLI pointed at the proxy via `ANTHROPIC_BASE_URL=http://127.0.0.1:8790`, real Anthropic backend. - Exact command / steps: ran Claude Code sessions that launch 2–3 parallel Explore subagents (each reading multi-KB JSON files, several tool-loop turns each), with an observability wrapper printing each request's resolved session id, tracker identity, and turn counter inside the proxy. - Observed result: on `main`, subagent session ids rotate on reminder-bearing turns (trackers permanently stuck at turn 0); when conversations do share an id they share one tracker whose turn counter interleaves all of them. On this branch: ids stable for the life of each conversation; colliding subagents resolve to separate lineages (`bare`, `~1`) with clean per-conversation turn progressions (trace above). Unit-level repro shows forwarded-prefix stability going 0/9 → 8/8 for the interleaved shape. - Not tested: reporter-scale cache-economics (his 4.4x needs his long-session workload against a paid backend); happy to coordinate with @RomanAlexanderW on a before/after — the number to watch is the cache-read ratio in Claude Code transcripts recovering toward ~96%. <details> <summary>repro_2085.py</summary> ```python """Repro for #2085: concurrent conversations sharing a fallback session id (same model + system prompt — e.g. a Claude Code session and its parallel subagents) collapse onto one PrefixCacheTracker and thrash its frozen-prefix state -> byte-unstable forwarded prefixes -> the provider prompt cache is re-written on nearly every call. Uses headroom's real code paths. Run from the repo root: python ../repro_2085.py """ from headroom.cache.prefix_tracker import PrefixFreezeConfig, SessionTrackerStore MODEL = "claude-sonnet-5" # Claude Code system prompt: long, static, identical across the main session # and every parallel subagent of the same type. SYSTEM = ("You are Claude Code, Anthropic's official CLI for Claude. " * 40)[:2000] def convo(name: str, turns: int) -> list[dict]: msgs = [{"role": "system", "content": SYSTEM}] for t in range(turns): msgs.append({"role": "user", "content": f"[{name}] user turn {t}: " + ("x" * 800)}) msgs.append( {"role": "assistant", "content": f"[{name}] tool_result {t}: " + ('{"data": 1}' * 200)} ) return msgs class _Req: # request stub: no x-headroom-session-id header headers: dict = {} # --- Part 1: identity collision (real derivation) ---------------------------- store = SessionTrackerStore(PrefixFreezeConfig()) id_a = store.compute_session_id(_Req(), MODEL, convo("A", 3)) id_b = store.compute_session_id(_Req(), MODEL, convo("B", 5)) print(f"1) fallback session ids: A={id_a} B={id_b} -> COLLIDE={id_a == id_b}") # --- Part 2: interleaved conversations thrash the freeze state --------------- def run(interleave: bool, lineage_resolution: bool) -> tuple[int, int, int]: store = SessionTrackerStore(PrefixFreezeConfig()) stable_turns = 0 later_turns = 0 seq = [] for t in range(1, 6): seq.append(("A", convo("A", t))) if interleave: seq.append(("B", convo("B", t))) for _name, msgs in seq: sid = store.compute_session_id(_Req(), MODEL, msgs) if lineage_resolution: tracker = store.resolve_tracker(sid, "anthropic", messages=msgs) else: tracker = store.get_or_create(sid, "anthropic") if tracker._turn_number > 0: later_turns += 1 if tracker._forwarded_prefix_stable(msgs): stable_turns += 1 tracker.update_from_response( cache_read_tokens=5000 * len(msgs), cache_write_tokens=2000, messages=msgs, ) return stable_turns, later_turns, store.active_sessions for label, interleave, fixed in ( ("single conversation, legacy ", False, False), ("interleaved (subagents), legacy ", True, False), ("interleaved, lineage resolution ", True, True), ): stable, later, sessions = run(interleave, fixed) print(f"2) {label}: stable prefix on {stable}/{later} later turns, trackers={sessions}") ``` </details> ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (CHANGELOG only — no docs describe the tracker store) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - Addresses the session-identity mechanisms of #2085; intentionally does not `Closes` it — the reporter should confirm the cache-read ratio recovers on live traffic first. - Composes with draft #1912 (first-user-turn fallback id). - Known bounded tradeoffs (all strictly milder than the per-turn thrash this fixes): a fork-style branch that resends a parent's full history adopts the parent's lineage, costing the parent one cold restart at its next turn; a request that aborts before the response and is retried with different bytes starts a fresh lineage; history truncation/tail-edit starts a fresh lineage even though the shorter provider prefix may still be warm. - Hot-path cost, measured on a 199-message/2.1MB agentic history: canonical projection 0.21ms + structural snapshot 0.92ms + match loop 0.06ms with 32 candidate lineages (2.27ms absolute worst case) ≈ **1.3ms per request** — same order as the handler's existing request deepcopy (0.80ms) and below one `json.dumps` of the body (2.9ms). Chain memory is structure-only (~180-330KB per lineage; message strings are shared with state the tracker already retains). - Known semantic shift to flag: hashing only the leading system run means conversations distinguished ONLY by mid-list system messages (e.g. clients injecting a per-conversation system context late in the list) now share a fallback id. The tracker is protected by lineage resolution; the residual sharing concentrates in the CCR sticky-tool registry and the monotone beta union — the same pre-existing class as same-system-prompt conversations today. Happy to file the CCR-stickiness scoping as a follow-up. - Out of scope, observed while tracing: `SessionCcrTracker.has_done_ccr` mildly cross-contaminates conversations sharing an id (monotone, no thrash) — can file separately if useful. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Ingmar Krusch
|
a352fa0168
|
fix(proxy/bedrock): wire PrefixCacheTracker updates into Bedrock backend paths (#2196)
## Description `update_from_response()` was only called from the direct-Anthropic-API branch of `handle_anthropic_messages`. Both Bedrock backend branches (streaming and non-streaming) returned before ever reaching it, so `PrefixCacheTracker` state stayed permanently empty for the life of a session on any `--backend bedrock` deployment: `extract_cache_stable_delta()` always saw no previous turn, and `--mode cache` fell back to full unmodified passthrough on every turn instead of compressing the append-only delta. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/anthropic.py`: non-streaming Bedrock branch now mirrors the direct-API branch — builds `next_original_messages`/`next_forwarded_messages` from the response, runs cache-miss attribution, and calls `prefix_tracker.update_from_response()` before returning. - `headroom/proxy/handlers/streaming.py`: `_stream_response_bedrock` gains `prefix_tracker`/`optimized_messages` parameters (previously absent entirely), accumulates raw SSE bytes only when a tracker is present, reconstructs the assistant message via the existing `_parse_sse_to_response` helper in the `finally:` block, then updates the tracker. Mirrors `_finalize_stream_response` and the OpenAI-via-backend sibling (`_stream_openai_via_backend`), which already had this wiring. - `tests/test_bedrock_prefix_tracker_wiring.py` (new): drives real `PrefixCacheTracker` instances (via `session_tracker_store`, not a fake) through both the non-streaming and streaming Bedrock paths using `TestClient`, and asserts the tracker's turn counter and last-forwarded/-original messages actually advance after a Bedrock call. A second non-streaming test drives two turns and asserts turn 2 sees a nonzero `frozen_message_count` once the cached total clears `min_cached_tokens`. Verified these tests fail against the pre-fix `anthropic.py`/`streaming.py` (turn counter stuck at 0) and pass against the fix. - `CHANGELOG.md`: added a `### Fixed` entry under `Unreleased`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_bedrock_prefix_tracker_wiring.py tests/test_backend_nonstreaming_cache_metrics.py tests/test_backend_streaming_cache_metrics.py tests/test_bedrock_streaming_input_tokens.py tests/test_cache/test_prefix_tracker.py tests/test_cache_prefix_overlay.py tests/test_cross_turn_cache_safety.py tests/test_proxy_anthropic_cache_stability.py -q collected 91 items tests/test_bedrock_prefix_tracker_wiring.py ... [ 3%] tests/test_backend_nonstreaming_cache_metrics.py .... [ 7%] tests/test_backend_streaming_cache_metrics.py .... [ 12%] tests/test_bedrock_streaming_input_tokens.py .. [ 14%] tests/test_cache/test_prefix_tracker.py .................................. [ 49%] tests/test_cache_prefix_overlay.py ......... [ 69%] tests/test_cross_turn_cache_safety.py ... [ 72%] tests/test_proxy_anthropic_cache_stability.py ......................... [100%] ======================== 91 passed, 1 warning in 9.15s ========================= $ uv run ruff check headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/streaming.py tests/test_bedrock_prefix_tracker_wiring.py All checks passed! $ uv run mypy headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/streaming.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: personal fork deployed as a real proxy (macOS launchd service, `headroom install apply`) with `--backend bedrock --mode cache`, fronting a live Claude Code session. - Exact command / steps: ran a two-turn streaming conversation against the running Bedrock-backed proxy, then a third append-only turn, while temporarily adding debug logging around `prefix_tracker.get_frozen_message_count()` / `get_last_original_messages()` (removed before this commit; the automated tests above are the permanent record). - Observed result: before the fix, `prev_orig_len`/`prev_fwd_len` were always 0 on every turn including turn 2+ — the tracker never advanced past its cold-start state. After the fix, turn 2 shows `prev_orig_len`/`prev_fwd_len` populated from turn 1's response, and the append-only turn 3 correctly triggers the delta-compression path (`router:noop` transform, pipeline actually runs) instead of falling to the router-never-called passthrough. In a separate live session captured while validating this fix, one turn showed `cache_write=98242` in the PERF log, and the immediately following turn showed `cache_read=98242 cache_hit_pct=94` — direct proof that the Bedrock path is now feeding real cache-read/write data back into the tracker end-to-end on live traffic, not just synthetic test fixtures. - Not tested: the live full-suite run during development surfaced one pre-existing unrelated failure in `test_provider_model_fallback.py`, confirmed independently failing on the commit prior to this fix (i.e., not introduced by this change, not fixed by it either). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — backend logic change, no UI surface. ## Additional Notes - No linked issue number: found via independent investigation of a personal deployment, not filed as a `headroomlabs-ai/headroom` issue first. - This is the more consequential of two related fixes from the same investigation; the sibling PR (`fix(proxy/savings): append history point on cache-only savings too`) fixes a savings-history reporting gap that this same `--mode cache` + Bedrock deployment surfaced. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Abhay Singh
|
daca1dd756
|
fix(cli/init): fail clearly on a target settings file with invalid JSON (#2227)
## Description
`headroom init` crashes with a raw traceback when a target's settings
file contains invalid JSON.
`_json_file` reads the JSON config files that init read-merge-writes
(Claude's `settings.json`, Codex's `hooks.json`, etc.):
```python
def _json_file(path: Path) -> dict[str, Any]:
if not path.exists():
return {}
content = path.read_text(encoding="utf-8").strip()
if not content:
return {}
payload = json.loads(content) # unguarded
return payload if isinstance(payload, dict) else {}
```
These are user-owned files that people hand-edit, so a stray trailing
comma or an unquoted key is entirely plausible. When that happens
`json.loads` raises `json.JSONDecodeError` and it propagates all the way
out, so `headroom init` dies with a Python traceback instead of a usable
message.
Returning `{}` on the error would be worse, not better: every caller
does `payload = _json_file(path)` then `_write_json(path, payload)`, so
an empty dict would make init overwrite the user's real settings with
just the hooks/env block — silent data loss.
## Fix
Guard the parse and convert it into an actionable `ClickException` that
names the file and the parse error, leaving the file untouched:
```python
try:
payload = json.loads(content)
except json.JSONDecodeError as e:
raise click.ClickException(
f"{path} contains invalid JSON ({e}); fix it and re-run, or move it aside."
) from e
```
`click.ClickException` is already the project's convention for
user-facing init failures (e.g. the `'claude' not found in PATH`
messages). The user now gets a clear instruction, and their file is
never clobbered.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/cli/init.py`: wrap the `json.loads` in `_json_file` and
raise a `ClickException` on `JSONDecodeError`.
- `tests/test_cli/test_init_cli.py`: new test asserting a malformed file
raises `ClickException` (matching "invalid JSON") and is left
byte-for-byte untouched.
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/cli/init.py tests/test_cli/test_init_cli.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/cli/init.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx
mypy@1.20.2`. A full `pytest` OOM-kills this box (ML stack import), so I
reproduced the behavior with a dependency-free script mirroring
`_json_file` and left the full pytest to CI.
- Exact command / steps: wrote a settings file containing `{"env": {"A":
"B",}}` (trailing comma), then called the OLD unguarded reader and the
NEW guarded reader; also re-checked a valid file round-trips.
- Observed result: OLD raises a raw `json.JSONDecodeError` (the init
traceback); NEW raises a `ClickException` containing "invalid JSON" and
leaves the file byte-for-byte unchanged; valid JSON still parses to the
same dict.
- Not tested: a full `headroom init` end-to-end run; full local `pytest`
deferred to CI (OOM).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" box is unchecked because the full suite
imports the ML stack, which I can't run here. The new test uses the
existing `_load_init_module` harness (the same one the neighbouring
`test_json_file_*` tests use), so it runs under the normal CI pytest
job; behaviour is additionally verified by the standalone proof above.
|
||
|
JD Davis
|
ea3d5a86b7
|
fix(deps): clear Dependabot lockfile alerts (#2175)
## Description Clears the current dependency/security-audit blockers that are making unrelated PRs red: - `transformers 5.3.0` / `CVE-2026-5241`, fixed by requiring `transformers>=5.5.0` in the locked optional dependency set. - `sqlitedict <=2.1.0` via the optional `benchmark` extra's `lm-eval[api]` dependency. There is no patched `sqlitedict` release, so this PR removes the published/locked `benchmark` extra instead of shipping a known-vulnerable transitive dependency. - `esbuild >=0.27.3,<0.28.1` in the OpenCode plugin lockfile, fixed by forcing `esbuild@0.28.1` through the OpenCode npm override and regenerated lockfile. The benchmark code still invokes `python -m lm_eval`; researchers who need that harness should install `lm-eval[api]` in their benchmark environment until its transitive vulnerability has a patched release. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `pyproject.toml`: remove the `benchmark` optional extra, document external `lm-eval[api]` installation guidance, and require `transformers>=5.5.0`. - `uv.lock`: regenerate without the `benchmark` extra, removing `lm-eval` and `sqlitedict` lock entries and locking the patched transformers floor. - `plugins/opencode/package.json`: add an `overrides` entry for `esbuild@0.28.1`. - `plugins/opencode/package-lock.json`: regenerate the OpenCode lockfile with `esbuild@0.28.1`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv lock --check rg -n -F 'sqlitedict' uv.lock # no matches rg -n -F 'name = "lm-eval"' uv.lock # no matches rg -n -F "extra == 'benchmark'" uv.lock # no matches rg -n -F '0.27.7' plugins/opencode/package-lock.json plugins/opencode/package.json # no matches npm ls esbuild --package-lock-only npm audit --package-lock-only # found 0 vulnerabilities git diff --check ``` Previous GitHub checks were green. After merging current `main`, fresh GitHub checks are running again; local targeted validation still passes. ## Real Behavior Proof - Environment: Windows 11, Python 3.13.3, uv, npm in `plugins/opencode`, Dependabot/pip-audit alert metadata from the failing PR jobs. - Exact command / steps: inspected the regenerated Python and npm lockfiles with `rg`, checked the uv lock with `uv lock --check`, checked OpenCode's dependency tree with `npm ls esbuild --package-lock-only`, and ran `npm audit --package-lock-only`. - Observed result: `uv.lock` no longer contains `sqlitedict`, `lm-eval`, or a `benchmark` extra marker; `transformers` resolves at the patched `>=5.5.0` floor; OpenCode's lock resolves `esbuild@0.28.1`; `npm audit --package-lock-only` reports 0 vulnerabilities; GitHub `Dependency audit (pip-audit)` passes. - Not tested: running the external `lm-eval` harness after installing it separately. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A - dependency and lockfile security fix. ## Additional Notes The `benchmark` extra can be restored once the upstream `lm-eval[api]` dependency chain stops pulling a vulnerable `sqlitedict` release. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
David Wells
|
2a954b69b4
|
feat(wrap): add ZCode desktop app support (#1845)
## Description Add `headroom wrap zcode` and `headroom unwrap zcode` commands for the ZCode desktop app (zcode.z.ai). ZCode is a desktop Electron IDE built by Z.AI, optimized for GLM-5.2 models. It has no CLI binary, so this follows the Pattern-B (proxy-only, print instructions) approach — same as Cursor, Cline, and Continue. **Upstream auto-detection:** `headroom wrap zcode` now reads `~/.zcode/v2/config.json` to detect the enabled provider and automatically configures the proxy upstream — no manual flags needed. Closes #1844 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - New module: `headroom/providers/zcode/__init__.py` and `runtime.py` (ZCodeProxyTargets, ZCodeUpstream, build_proxy_targets, detect_upstream, upstream_to_proxy_urls, render_setup_lines) - New CLI command: `headroom wrap zcode` in `headroom/cli/wrap.py:4815` — starts proxy, injects RTK into AGENTS.md, prints Base URL setup instructions - New CLI command: `headroom unwrap zcode` in `headroom/cli/wrap.py:5960` — removes RTK markers, stops proxy - New helper: `zcode_config_dir()` in `headroom/install/paths.py` - Updated `_run_proxy_only_watcher` to accept `anthropic_api_url`/`openai_api_url` params - Updated README.md: ZCode row in compatibility matrix, unwrap list, wrap command list - Updated CHANGELOG.md: entry under [Unreleased] > Added ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) — pre-existing numpy type stubs issue prevents full mypy run - [x] New tests added for new functionality (24 tests in `tests/test_cli/test_wrap_zcode.py`) - [x] Manual testing performed ### Test Output ```text tests/test_cli/test_wrap_zcode.py ........................ [100%] 24 passed, 1 warning in 0.22s ``` ## Real Behavior Proof - Environment: macOS 15.5, Python 3.12.13, headroom installed via `pip install -e .[dev]` - Exact command / steps: `headroom wrap zcode --port 9000` then `headroom unwrap zcode --port 9000` - Observed result: Wrap detects provider from `~/.zcode/v2/config.json` (e.g. "Z.ai Coding"), starts proxy with correct upstream on port 9000, injects RTK into AGENTS.md, prints detected provider + upstream + Base URL setup instructions. Unwrap removes RTK markers, deletes empty AGENTS.md, stops proxy. - Not tested: Actual ZCode app integration (ZCode is a desktop Electron app; Base URL configuration is manual in Settings > Model Settings) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas — N/A: code follows existing patterns - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — CLI-only changes ## Additional Notes - **Pattern-B approach:** ZCode is a desktop Electron app with no CLI binary. Same pattern as Cursor, Cline, and Continue — proxy-only, print instructions. - **Upstream auto-detection:** Reads `~/.zcode/v2/config.json`, finds the enabled provider, and passes its `baseURL` to the proxy. Falls back to Z.ai Anthropic endpoint if no config found. - **httpProxy investigation:** ZCode has an `httpProxy` setting in `~/.zcode/v2/setting.json`, but it is an Electron-level forward proxy (CONNECT tunneling), incompatible with headroom reverse proxy. The Base URL approach in Model Settings is the correct integration point. - **No dependencies added:** This PR adds zero new dependencies. --------- Co-authored-by: Epicism <epicism@Epiphanie.local> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: Tejas Chopra <chopratejas@gmail.com> |
||
|
Connor Campbell
|
021a762bf8
|
feat(compress): expose frozen_message_count in library-mode compress() (#2178)
## Description `read_lifecycle.apply()` already supports a frozen message prefix (`frozen_message_count`) — stale-Read replacements inside the prefix are skipped so compression never rewrites messages the provider's prompt cache has anchored. But only the proxy handlers can pass it: `ContentRouter` reads it from transform kwargs, `CompressConfig` has no such field, and the public `compress()` never forwards it. Library-mode callers that manage their own conversation loop (SDK integrations, offline evaluation, sidecar scoring) therefore can't stop transforms from rewriting already-sent history. On cached Anthropic traffic that's expensive: every byte after the first rewritten one stops billing as a 0.1× cache read and re-bills as a cache write (1.25× at the 5-minute TTL, 2× at the 1-hour TTL) — measured on live coding-agent traffic, retroactive stale-Read rewrites were the dominant cache-bust source once tool injection went session-sticky (PR-B7). Relates to #809 (cache-bust economics discussion); does not close it. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `CompressConfig.frozen_message_count: int = 0` — documented field; default `0` preserves existing behavior exactly. - `compress()` forwards it through `pipeline.apply()` to the transforms, matching what the proxy handlers already do. - `compress()` docstring: added to the kwargs shorthand list. - CHANGELOG entry under Unreleased → Features. - Four tests in `tests/test_compress_api.py` (`TestFrozenMessageCount`). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_compress_api.py tests/test_transforms/test_read_lifecycle.py \ tests/test_compression_safety_rails.py tests/test_compress_failure.py -q 59 passed, 1 warning in 3.05s $ uv run ruff check headroom/compress.py tests/test_compress_api.py All checks passed! $ uv run mypy headroom Success: no issues found in 471 source files ``` ## Real Behavior Proof - Environment: Linux, Python 3.12.3, this branch installed via `uv sync --extra dev` - Exact command / steps: build an Anthropic-format conversation with a stale Read (file read at message 2, edited at message 3), then: ```python r0 = compress(msgs, model="claude-sonnet-4-5-20250929") r5 = compress(msgs, model="claude-sonnet-4-5-20250929", frozen_message_count=5) ``` - Observed result: without frozen prefix the stale Read is rewritten; with frozen_message_count=5 the Read remains byte-identical. ```text without frozen prefix: stale Read rewritten: True transforms: ['read_lifecycle:stale:/app/config.py'] with frozen_message_count=5: Read byte-identical: True transforms: [] ``` - Not tested: proxy-mode code paths (untouched — they already pass `frozen_message_count` their own way); Rust crates (untouched). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — library API change, no UI. ## Additional Notes Default `0` makes this a strict superset of current behavior — no caller sees any change without opting in. The motivation data comes from a proxy-side measurement tool that prices compression's cache effects on live Anthropic agent traffic (per-request cache-adjusted dollars); happy to share methodology in #809 if useful. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: Tejas Chopra <chopratejas@gmail.com> |
||
|
Ingmar Krusch
|
896454e978
|
feat(install): add apply flag parity, --env passthrough, and EIO retry (#2152)
## Description Three related gaps in `headroom install apply` and its supervisor lifecycle, found operating a real persistent deployment on this fork: 1. `install apply` only exposed a fixed subset of `headroom proxy`'s flags (`--backend`, `--region`, `--mode`, `--port`, `--memory`, `--telemetry`, `--no-http2`). Deployments that need code-aware compression, tool-result interception, per-tool lossy-compression protection, or a named AWS profile for Bedrock had no native way to configure them through `install apply` — the generated `manifest.json` would have to be hand-edited after the fact, which silently reverts on the next `install apply` and isn't tracked anywhere. 2. Supervised runners (macOS launchd, Linux systemd/cron, Windows services/tasks) all start their runner scripts with a bare environment and do not inherit the interactive shell's exports. In particular, a custom `HEADROOM_WORKSPACE_DIR` never reached the supervised process, so `headroom install agent run` looked for its manifest in the wrong location and failed outright with "No deployment profile named 'default' is installed" even though `install apply` itself had succeeded moments earlier. 3. `install_supervisor`'s macOS branch does an unconditional `launchctl bootout` followed by a bare `bootstrap` with no retry, unlike `start_supervisor` (already fixed by #1290), which rides out the ~15s EIO (error 5) window launchd exhibits for several seconds after a bootout. This left `install apply`'s own reinstall path exposed to the same race #1290 fixed elsewhere — requiring the exact manual recovery (bootout + remove the plist + reapply) #1290 was meant to eliminate. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cli/install.py`: `install apply` gains `--code-aware/--no-code-aware`, `--intercept-tool-results`, `--protect-tool-results <tool1,tool2>`, and `--bedrock-profile <profile>`, mirroring the equivalent flags already on `headroom proxy` (same names, same help text style). Also gains `--env KEY=VALUE` (repeatable). - `headroom/install/planner.py`: `build_manifest()` threads all five new parameters into `proxy_args`/`base_env`, following the exact pattern already used for `--region`/`--no-http2`. `--env` entries are merged into `base_env` last, so they can override auto-derived defaults. - `headroom/install/supervisors.py`: - `_render_unix_runner`/`_render_windows_runner` emit `export`/`$env:` lines for `base_env` before the `exec`, so `run-headroom.sh`/`ensure-headroom.sh` (and Windows equivalents) carry the environment forward to both the outer `install agent run` process and the proxy subprocess it spawns. The Docker runtime path already threaded `base_env` into `docker run --env`; this closes the same gap for the process-based runtime. - New `_bootstrap_with_retry()` helper extracted from `start_supervisor`'s existing retry loop (from #1290), now shared by both `start_supervisor` and `install_supervisor`. - `tests/test_install/test_planner.py`: new tests for all five flags (default-omitted and persisted cases), following the existing `--no-http2` test pattern. - `tests/test_install/test_supervisors.py`: new tests for `--env` propagation into rendered runner scripts, and for `install_supervisor`'s retry-until-success and raise-after-exhausted-retries paths (mirroring the existing `start_supervisor` coverage). Also fixes a pre-existing test's mock that returned `None` from a `subprocess.run` stub — this only worked before because the old bare `bootstrap` call site never inspected the return value; the new `_bootstrap_with_retry()` call does. - `CHANGELOG.md`: added `### Features` and `### Fixed` entries under `Unreleased`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_install/ tests/test_cli/test_wrap_persistent.py tests/test_cli/test_init_cli.py -q ============================= test session starts ============================== platform darwin -- Python 3.13.13, pytest-9.0.3, pluggy-1.6.0 collected 215 items tests/test_install/test_health.py ... [ 1%] tests/test_install/test_native_installers.py ss [ 2%] tests/test_install/test_paths.py ... [ 3%] tests/test_install/test_planner.py .................. [ 12%] tests/test_install/test_providers.py ................................... [ 28%] ...... [ 31%] tests/test_install/test_runtime.py .................... [ 40%] tests/test_install/test_state.py ..... [ 42%] tests/test_install/test_supervisors.py ......................... [ 54%] tests/test_cli/test_wrap_persistent.py ............................ [ 67%] tests/test_cli/test_init_cli.py ........................................ [ 86%] .............................. [100%] ======================== 213 passed, 2 skipped in 0.57s ======================== $ uv run ruff check headroom/cli/install.py headroom/install/planner.py headroom/install/supervisors.py tests/test_install/ All checks passed! $ uv run mypy headroom/cli/install.py headroom/install/planner.py headroom/install/supervisors.py Success: no issues found in 3 source files ``` ## Real Behavior Proof - Environment: personal fork deployed as a real proxy (macOS launchd service via `headroom install apply`), profile `default`, backend `bedrock` with a named AWS SSO profile. - Exact command / steps: (flags 1 & 2) ran `headroom install apply --backend bedrock --mode token --code-aware --protect-tool-results Bash --bedrock-profile sso-bedrock --env HEADROOM_WORKSPACE_DIR=/Users/<redacted>/.headroom-workspace --env AWS_PROFILE=sso-bedrock --env AWS_REGION=eu-west-1`, then inspected the generated `manifest.json`, the rendered `run-headroom.sh`, and the running launchd job. - Observed result: before this PR, none of `--code-aware`, `--protect-tool-results`, `--bedrock-profile`, or `--env` were accepted flags on `install apply` at all (`Error: No such option`). Reproduced the `--env` gap specifically by running the exact command a launchd job invokes with a stripped environment (no `HEADROOM_WORKSPACE_DIR`, no `AWS_PROFILE`) — it failed to find the manifest; with the interactive shell's env forwarded manually, it started fine. The generated plist had no `EnvironmentVariables` key and `run-headroom.sh` was a bare `exec`, confirming this wasn't a config mistake but a real gap between `install apply`'s flag surface and what a supervisor actually runs with. After this PR, `install apply` with all the flags above produces a launchd job that starts clean, reports healthy, and successfully proxies a real request to Bedrock (200, not just a green health check) using the named AWS profile with no `AWS_PROFILE` env var needed elsewhere. - Exact command / steps: (EIO retry, flag 3) triggered the same EIO race #1290 documents by running `headroom install apply` twice in quick succession against the same profile (the second run's `install_supervisor` bootout+bootstrap lands inside the first run's launchd settle window). - Observed result: before this PR, the second `install apply` occasionally failed outright with `CalledProcessError` from the bare `subprocess.run(..., check=True)` bootstrap call, requiring the manual bootout+`rm` plist+reapply recovery. After this PR (with `_bootstrap_with_retry` in place), the same back-to-back sequence completes successfully every time observed, riding out the EIO window instead of failing. - Not tested: Linux systemd/cron and Windows service/task supervisor paths for the `--env` propagation — verified via the new unit tests (which cover the runner-script rendering directly) but not against a live Linux or Windows machine, since this deployment is macOS-only. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — CLI/install logic change, no UI surface. ## Additional Notes - "I have made corresponding changes to the documentation" is unchecked: no file in `docs/`, `README.md`, or `CONTRIBUTING.md` documents `install apply`'s flag surface in detail (it's discoverable via `--help`), so there is no existing section to update for the new flags. - Re-derivation note: this PR's `install_supervisor` EIO-retry fix and its `_bootstrap_with_retry` extraction are written directly against current `upstream/main`'s post-#1290 shape of `start_supervisor` (inline retry loop with `_MACOS_BOOTSTRAP_RETRIES`/`_MACOS_BOOTSTRAP_RETRY_DELAY`), not cherry-picked from an older fork commit that predated #1290 — the diff here is intentionally different from what a naive cherry-pick would have produced. - No linked issue number: found via operating a real persistent deployment on a personal fork, not filed as a `headroomlabs-ai/headroom` issue first. Checked `gh pr list --search` for "install apply flags/env" and "bootstrap EIO"/"install_supervisor bootstrap retry" — no open or merged coverage found beyond #1290 (which fixes `start_supervisor` only, a different call site from the one this PR fixes). Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Carlos Duplar Mello
|
5d17e9addc
|
fix: check feature configuration before reusing persistent deployments (#1330)
## Description
A persistent proxy started for one use case (e.g. `--backend anthropic`)
would be silently reused for another (e.g. `--subscription
--provider-type openai`) causing 401 auth failures because
`_ensure_proxy()` only checked health + version, skipping the feature
configuration check (memory, openai_api_url, learn, code_graph).
Closes #N/A
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Added feature configuration check in the persistent deployment path of
`_ensure_proxy()` in `headroom/cli/wrap.py:1726-1752`. When features
mismatch, the code now falls through to the non-persistent path which
handles proxy restart with upgraded config.
- Added three new tests in `tests/test_cli/test_wrap_persistent.py`:
-
`test_ensure_proxy_restarts_persistent_deployment_for_feature_mismatch`
— verifies proxy restart when openai_api_url differs
- `test_ensure_proxy_restarts_persistent_deployment_for_memory_mismatch`
— verifies proxy restart when memory is requested but not enabled
- `test_ensure_proxy_reuses_persistent_deployment_when_features_match` —
verifies proxy reuse when all features match
- Updated `CHANGELOG.md` with fix description
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ ruff check headroom/cli/wrap.py tests/test_cli/test_wrap_persistent.py
All checks passed!
$ ruff format --check headroom/cli/wrap.py tests/test_cli/test_wrap_persistent.py
2 files already formatted
$ python -c "
from tests.test_cli.test_wrap_persistent import *
import pytest
test_ensure_proxy_restarts_persistent_deployment_for_feature_mismatch(pytest.MonkeyPatch())
print('Test 1 passed: feature mismatch restarts proxy')
test_ensure_proxy_restarts_persistent_deployment_for_memory_mismatch(pytest.MonkeyPatch())
print('Test 2 passed: memory mismatch restarts proxy')
test_ensure_proxy_reuses_persistent_deployment_when_features_match(pytest.MonkeyPatch())
print('Test 3 passed: matching features reuse proxy')
print('All tests passed!')
"
Test 1 passed: feature mismatch restarts proxy
Test 2 passed: memory mismatch restarts proxy
Test 3 passed: matching features reuse proxy
All tests passed!
$ .venv/bin/mypy headroom
headroom/proxy/server.py:1230: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked]
headroom/proxy/server.py:1301: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked]
headroom/proxy/server.py:1305: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked]
Success: no issues found in 397 source files
```
## Real Behavior Proof
- Environment: Linux (WSL2) Ubuntu 24.04, Python 3.12, persistent
deployment via `headroom install agent run --profile default` with
`--backend anthropic`
- Exact command / steps:
1. Started persistent deployment: `headroom install agent run --profile
default`
2. Ran copilot wrapper: `headroom wrap copilot --subscription
--provider-type openai --wire-api responses --memory -- --model
gpt-5.4-mini`
- Observed result:
- Before fix: Proxy forwarded to `api.openai.com` instead of
`api.githubcopilot.com`, causing 401 auth errors
- After fix: Proxy correctly forwards to `api.githubcopilot.com` and
copilot works as expected
- Not tested: macOS, Windows, enterprise/data-residency accounts, other
wrapped tools (claude, codex, aider)
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A
## Additional Notes
The fix is minimal and targeted. It only adds a feature configuration
check in the persistent deployment path without changing any other
behavior. The non-persistent proxy path already had this check; this PR
brings the same logic to persistent deployments.
Co-authored-by: carlosduplar <[email protected]>
|
||
|
Abhay Singh
|
a5d7e12c90
|
fix(proxy/batch): preserve sibling tool configs on Google batch requests (#2177)
## Description
When Headroom optimizes a Google/Gemini batch request, it silently drops
every tool config that isn't `functionDeclarations`.
In `handle_google_batch_create` the per-item optimizer extracts the
function declarations:
```python
tools = req_content.get("tools")
existing_funcs = None
if tools:
for tool in tools:
if "functionDeclarations" in tool:
existing_funcs = tool["functionDeclarations"]
break
```
and then rebuilds the forwarded request's tools as a single entry:
```python
if existing_funcs is not None:
compressed_req_content["tools"] = [{"functionDeclarations": existing_funcs}]
```
Gemini's `tools` array is a list of heterogeneous entries —
`{"functionDeclarations": [...]}` can sit alongside `{"googleSearch":
{}}` and `{"codeExecution": {}}`. Collapsing the array to one
`functionDeclarations` entry discards those siblings, so a batch request
that combines function calling with Google Search or code execution
reaches Google with those features stripped out. The request still
succeeds, so the loss is silent — the model just never grounds against
Search / never runs code.
The branch fires whenever the item had any `functionDeclarations` (or
CCR injected a retrieval tool), i.e. exactly the requests most likely to
also declare Search/code-execution.
## Fix
Rebuild the tools list from the original, replacing only the
`functionDeclarations` entry with the (possibly CCR-injected) funcs and
appending a new entry when the original had none:
```python
rebuilt_tools = []
replaced = False
for tool in tools or []:
if "functionDeclarations" in tool:
rebuilt_tools.append({**tool, "functionDeclarations": existing_funcs})
replaced = True
else:
rebuilt_tools.append(tool)
if not replaced:
rebuilt_tools.append({"functionDeclarations": existing_funcs})
compressed_req_content["tools"] = rebuilt_tools
```
Sibling entries (`googleSearch`, `codeExecution`, ...) are preserved in
place; the search/no-search behavior of the request is unchanged.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/batch.py`: preserve
non-`functionDeclarations` tool entries when rebuilding the optimized
Gemini batch request's tools array.
- `tests/test_proxy_handlers_batch.py`: new regression test asserting
`googleSearch` / `codeExecution` survive alongside
`functionDeclarations` in the forwarded body (uses the existing
`RealConvHandler` harness with the real converters).
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/proxy/handlers/batch.py tests/test_proxy_handlers_batch.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/handlers/batch.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx
mypy@1.20.2`. A full `pytest` OOM-kills this box (ML stack import), so I
reproduced the array rebuild with a dependency-free script and left the
full pytest to CI.
- Exact command / steps: fed a tools array of
`[{functionDeclarations:[get_weather]}, {googleSearch:{}},
{codeExecution:{}}]` (plus a CCR-injected retrieval function) through
the OLD single-entry rebuild and the NEW preserving rebuild; also the
search-only case where CCR injects the first `functionDeclarations`.
- Observed result: OLD → `[{functionDeclarations:[...]}]` only
(googleSearch and codeExecution gone); NEW → all three entries retained
with the injected retrieval function present in `functionDeclarations`;
the search-only case gains a `functionDeclarations` entry while keeping
`googleSearch`.
- Not tested: a live Gemini batch submission; full local `pytest`
deferred to CI (OOM).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" box is unchecked because the full suite
imports the ML stack, which I can't run here. The new test reuses the
in-file `RealConvHandler` harness (same one the existing
`..._preserves_functioncall_response_order` test uses) so it runs under
the normal CI pytest job; behaviour is additionally verified by the
standalone proof above.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Abhay Singh
|
195ed90ced
|
fix(savings): record pre-compression original as ledger before, not forwarded count (#2176)
## Description
`headroom savings` overstates the proxy reduction percentage because the
durable ledger is written with the wrong `before` value.
In `PrometheusMetrics.record_request` the proxy appends a savings event:
```python
if tokens_saved > 0 and not self._stateless:
savings_ledger.record_savings_event(
tokens_before=input_tokens,
tokens_after=max(input_tokens - tokens_saved, 0),
...
)
```
But `input_tokens` here is the optimized, **post-compression** count
that was actually forwarded, not the original. `emit_request_outcome`
(the single funnel that calls `record_request`) passes
`input_tokens=outcome.optimized_tokens`.
The ledger derives the reported reduction as `saved / before`
(`savings_ledger._Bucket.savings_percent`), with `saved = max(before -
after, 0)`. Passing the forwarded count as `before` (and `before -
saved` as `after`) keeps `saved` correct but understates `before` by
`tokens_saved`, so the percentage is inflated:
- original input 1000 tokens, forwarded 600, saved 400 → true reduction
40%.
- recorded as `before=600, after=200` → `400 / 600` = **66.7%** on the
dashboard.
So `headroom savings` (which aggregates this ledger across restarts and
processes) reports a reduction percent well above what actually happened
for all proxy traffic.
## Fix
Reconstruct the original as forwarded + saved:
```python
tokens_before=input_tokens + tokens_saved, # the pre-compression original
tokens_after=input_tokens, # what we forwarded
```
`saved` (= `before - after` = `tokens_saved`) and the stored `cost_usd`
(derived from `saved`) are unchanged; only the `before`/`after` labels
are corrected, so the reduction percent becomes honest.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/prometheus_metrics.py`: pass
`tokens_before=input_tokens + tokens_saved` and
`tokens_after=input_tokens` to `record_savings_event`, with a comment
explaining that `input_tokens` is the forwarded count.
- `tests/test_savings_ledger_before_forwarded.py`: new regression guard
on the call shape.
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/proxy/prometheus_metrics.py tests/test_savings_ledger_before_forwarded.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/prometheus_metrics.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx
mypy@1.20.2`. A full `pytest` OOM-kills this box (ML stack import), so I
reproduced the math with a dependency-free script modelling the ledger's
own `saved = before - after` and `saved / before * 100`, and left the
full pytest to CI.
- Exact command / steps: fed a request with original=1000,
forwarded=600, saved=400 through the OLD call shape
(`before=input_tokens`, `after=input_tokens-saved`) and the NEW shape
(`before=input_tokens+saved`, `after=input_tokens`).
- Observed result: OLD → `before=600, after=200`, reported 66.7%; NEW →
`before=1000, after=600`, reported 40.0% (the true reduction). `saved`
is 400 in both, so the cost figure is unaffected.
- Not tested: a live proxy end-to-end run; full local `pytest` deferred
to CI (OOM).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" box is unchecked because the full suite
imports the ML stack, which I can't run here. The regression test
asserts on the source of `record_request` (the enclosing module imports
the ML stack, so it executes the assertion against the source text
rather than calling the method); the behavioural verification is the
standalone proof above.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Abhay Singh
|
f723925be7
|
fix(proxy/gemini): forward a non-JSON upstream body with its real status (#2174)
## Description
`handle_gemini_generate_content` turns a non-JSON upstream error
response into a generic 502, hiding the real status and body.
After the upstream call it extracts usage from `response.json()`:
```python
try:
resp_json = response.json()
usage = resp_json.get("usageMetadata", {})
...
cache_read_tokens = usage.get("cachedContentTokenCount", 0)
except (KeyError, TypeError, AttributeError) as e: # <-- missing JSONDecodeError / ValueError
...
```
`response.json()` raises `json.JSONDecodeError` (a `ValueError`
subclass) for a non-JSON body. That isn't in the tuple, so it escapes to
the function's outer `except Exception`, which returns a synthetic 502
and discards the real `response.status_code` / `response.content` (which
the success path forwards verbatim). An overloaded Google/Vertex/Copilot
frontend commonly returns a 503/500/429 with an HTML or empty body, so
the client sees a generic 502 instead of the true status — defeating
retry/backoff and dropping the diagnostic.
The all-non-text early-exit branch in the same handler already handles
this correctly with the full tuple (`except (json.JSONDecodeError,
ValueError, KeyError, TypeError, AttributeError)`) and then forwards the
real status/content.
## Fix
Add `json.JSONDecodeError, ValueError` to the token-extraction `except`,
matching that sibling. On a non-JSON body the extraction is skipped
(token metrics keep their fallbacks) and the handler falls through to
`return Response(content=response.content,
status_code=response.status_code, ...)` — the real status and body.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/gemini.py`: broaden the token-extraction
`except` in `handle_gemini_generate_content` to include
`json.JSONDecodeError, ValueError`.
- `tests/test_gemini_nonjson_status.py`: new test asserting that except
clause catches the JSON/ValueError family (guards against the
regression).
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/proxy/handlers/gemini.py tests/test_gemini_nonjson_status.py
All checks passed!
$ python -m py_compile headroom/proxy/handlers/gemini.py tests/test_gemini_nonjson_status.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. A full
`pytest` OOM-kills this box (ML stack import), so I verified the
exception handling with a dependency-free script that models the
token-extraction try/except plus the handler's verbatim-forward return,
and left the full pytest to CI.
- Exact command / steps: sent a 503 response whose `.json()` raises
`JSONDecodeError` (non-JSON body) through the old tuple and the new
tuple, plus a normal JSON 200 as a control.
- Observed result: old lets `JSONDecodeError` escape (→ the outer
handler's synthetic 502); new catches it and forwards the real 503; the
JSON 200 still extracts tokens under both. The new test asserts the real
handler's except clause includes the JSON/ValueError family.
- Not tested: a live overloaded Gemini upstream; full local `pytest`
deferred to CI (OOM).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" / "type checking" boxes are unchecked
because the full suite imports the ML stack, which I can't run here; the
change adds two exception types matching an existing, tested sibling
branch, verified by the standalone proof and a source-level regression
guard (a full handler-integration harness for Gemini doesn't exist
in-tree, and the existing gemini integration tests hit a live API).
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Abhay Singh
|
8f867e4622
|
fix(install): guard non-dict health config in 'install status' (#2150)
## Description
`headroom install status` crashes with an `AttributeError` when the
probed health endpoint returns a non-dict `config`.
```python
if payload and isinstance(payload, dict):
click.echo(f"Health URL: {manifest.health_url.replace('/readyz', '/health')}")
click.echo(f"Backend: {payload.get('config', {}).get('backend', manifest.backend)}")
```
`payload` is guarded as a dict, but `payload['config']` is not.
`dict.get('config', {})` only substitutes the `{}` default when the key
is **absent** — a present-but-non-dict `config` (`null`, a string, a
list) is returned as-is, and the chained `.get('backend', ...)` then
raises `AttributeError`, crashing the command with a raw traceback.
Reachability: the Headroom proxy normally returns `config` as an object,
so this bites when `install status` probes a port that a different or
older service is occupying (which can emit `config: null` or a
non-object), or a build that emits `config: null`. The correctly-guarded
sibling already exists in the codebase — `wrap.py`'s
`_proxy_health_config` does `config = payload.get("config"); return
config if isinstance(config, dict) else None`.
## Fix
Guard the `config` value with `isinstance(config, dict)` before the
`.get('backend', ...)` lookup, mirroring `_proxy_health_config`. A
non-dict (or missing) `config` falls back to the manifest's backend.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/cli/install.py`: `install status` guards `config` with
`isinstance(config, dict)` before reading `backend`.
- `tests/test_cli/test_install_cli.py`: add
`test_install_status_survives_non_dict_config` (health payload with
`config: null` must not crash; backend falls back to the manifest).
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/cli/install.py tests/test_cli/test_install_cli.py
All checks passed!
$ python -m py_compile headroom/cli/install.py tests/test_cli/test_install_cli.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified the access with a
dependency-free script that replicates the old vs guarded lookup, and
left the full pytest (including the new CLI test) to CI.
- Exact command / steps: ran the old `payload.get('config',
{}).get('backend', ...)` and the new guarded lookup against `config`
values of `null`, a string, a list, a proper object, and a missing key.
- Observed result: the old lookup raises `AttributeError` for every
non-dict `config`; the new lookup falls back to the manifest backend for
those and returns the real backend for a proper object (and the
missing-key case is unchanged). The new CLI test drives `install status`
with `probe_json` returning `{"config": null}` and asserts a clean exit
with the manifest backend.
- Not tested: a live foreign service occupying the port; full local
`pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" and "type checking" boxes are unchecked
because the full suite imports the ML stack, which I can't run in this
environment; the change adds an `isinstance` guard mirroring an existing
sibling, verified by the standalone proof and a new CLI test that reuses
the file's existing `install status` mocking harness.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Abhay Singh
|
0cddac632d
|
fix(telemetry): only advance usage-report baseline after a 200 (#2149)
## Description
The usage reporter permanently drops a reporting window's usage whenever
the send to the cloud fails.
`UsageReporter._report_usage` computes usage as a **delta** against the
last snapshot, POSTs it, and then rebases the baseline:
```python
try:
resp = await client.post(f"{self._cloud_url}/v1/license/usage", json=payload, timeout=10.0)
if resp.status_code == 200:
...
else:
logger.warning("Usage report returned status %d", resp.status_code)
except Exception:
logger.warning("Failed to send usage report", exc_info=True)
# Update snapshot
self._snapshot_metrics() # runs on success, non-200, AND exception
self._last_report_time = now
```
`_snapshot_metrics()` rebases `_last_tokens_saved_by_model` /
`_last_tokens_sent_by_model` / `_last_requests_by_model` to the current
cumulative counters. Because it runs unconditionally after the POST, a
report that fails to send — non-200 or a raised exception — still
advances the baseline. The module is explicitly built to tolerate a
briefly-unreachable cloud (7-day grace, cached license), so this is a
normal, recurring situation.
The consequence: the failed window's requests and tokens are never
re-included. The next report is a delta from the advanced baseline, so
that window is silently and permanently lost from usage-based billing /
quota. Every transient network blip under-counts usage. (The
`total_requests == 0` early-return already gets this right — it advances
only `_last_report_time`, without snapshotting, since there's nothing to
lose.)
## Fix
Advance the baseline (`_snapshot_metrics()` and `_last_report_time`)
only inside the `resp.status_code == 200` branch. On a non-200 or an
exception, both baselines are left intact, so the next report covers the
full period since the last successful send and re-includes the
previously-failed window.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/telemetry/reporter.py`: move `_snapshot_metrics()` +
`_last_report_time = now` into the 200 branch of `_report_usage`.
- `tests/test_usage_reporter_snapshot.py`: new tests — baseline advances
on 200, and stays intact on a non-200 and on an exception (window
preserved).
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/telemetry/reporter.py tests/test_usage_reporter_snapshot.py
All checks passed!
$ python -m py_compile headroom/telemetry/reporter.py tests/test_usage_reporter_snapshot.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified the delta accounting with a
dependency-free script that models two windows across a failed then
successful send, and left the full pytest to CI.
- Exact command / steps: window 1 saves 100 tokens and the send fails;
window 2 saves another 50 (cumulative 150) and the send succeeds. Ran
under the old (unconditional snapshot) and new (snapshot-on-200) logic.
- Observed result: old delivers only 50 tokens total (window 1's 100
dropped when the baseline advanced on the failed send); new delivers the
full 150 (window 2's delta re-includes window 1). The new tests assert
the baseline advances on 200 and is untouched on a non-200 / exception,
driving the real `_report_usage` with a fake proxy + client.
- Not tested: a live cloud round-trip; full local `pytest` deferred to
CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" and "type checking" boxes are unchecked
because the full suite imports the ML stack, which I can't run in this
environment; the change moves two lines into the success branch,
verified by the standalone delta-accounting proof and new tests that
drive the real `_report_usage` via `object.__new__` with a fake proxy
and HTTP client (200, non-200, and exception).
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Abhay Singh
|
fb17156bfa
|
fix(savings): don't bill free models at the $3/M fallback in the ledger (#2147)
## Description
The durable savings ledger records phantom cost-avoided for free
(0-priced) models, billing them at the `$3/M` blended fallback.
`estimate_cost_usd` prices a known model via
`_estimate_compression_savings_usd`, but gates the result on `> 0`:
```python
if model and model != UNKNOWN:
priced = _estimate_compression_savings_usd(model, tokens_saved)
if priced > 0: # <-- the bug
return round(priced, 6)
return round(float(tokens_saved) * float(fallback_rate), 6) # ~$3/M
```
`_estimate_compression_savings_usd` deliberately distinguishes three
cases: a litellm-priced model (`> 0`), a model litellm can't price
(returns the blended fallback itself), and a model that is *legitimately
free* — litellm has an entry with `input_cost_per_token == 0.0`, so it
returns `tokens_saved * 0.0 == 0.0`. Its own comment calls this out:
"`if not ...` treated a real 0.0 as unavailable and billed the $3/M
fallback — phantom savings for a model that costs nothing."
The ledger's `if priced > 0` re-introduces exactly that defect: a free
model's `0.0` is treated as "unpriced" and the code falls through to the
`$3/M` fallback. Every saved token on a free/local/promo model is then
written into the durable JSONL ledger — and surfaced by `headroom
savings` / `aggregate_savings` — as cost-avoided that never existed.
## Fix
Trust `_estimate_compression_savings_usd`'s return verbatim for known
models. It already returns the blended fallback for models litellm can't
price and `0.0` for free ones, so the `> 0` gate is not needed — and is
the source of the double-fallback. The `UNKNOWN`/empty-model path still
uses `fallback_rate` as before.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/savings_ledger.py`: `estimate_cost_usd` returns
`_estimate_compression_savings_usd(...)` unconditionally for known
models instead of gating on `> 0`.
- `tests/test_savings_ledger.py`: add
`test_free_model_is_not_billed_at_fallback` (free model → $0) and
`test_priced_model_uses_litellm_estimate` (priced model → estimate),
both monkeypatching the helper.
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/savings_ledger.py tests/test_savings_ledger.py
All checks passed!
$ python -m py_compile headroom/savings_ledger.py tests/test_savings_ledger.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified the pricing with a
dependency-free script that replicates the gate and the helper's three
cases, and left the full pytest to CI.
- Exact command / steps: priced 1,000,000 saved tokens for a free model,
a priced model, a litellm-unknown named model, and the explicit
`UNKNOWN` sentinel, under the old (`> 0` gate) and new (unconditional)
logic.
- Observed result: the old logic bills the free model `$3.00` (phantom);
the new logic bills `$0.00`. The priced model (`$2.00`), the
litellm-unknown fallback (`$3.00`), and the `UNKNOWN`-path
`fallback_rate` are unchanged. The new tests assert the free-model `$0`
and the priced-model estimate via a monkeypatched helper.
- Not tested: a live litellm lookup for a real free model; full local
`pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" and "type checking" boxes are unchecked
because the full suite imports the ML stack, which I can't run in this
environment; the change removes a `> 0` gate in a pure pricing function,
verified by the standalone proof and the new tests. This is the same
category as the earlier zero-price-model fix, but at a distinct,
still-buggy call site (the durable ledger) — the earlier fix landed
inside `_estimate_compression_savings_usd`.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Ingmar Krusch
|
d125805589
|
fix(proxy/savings): append history point on cache-only savings too (#2194)
## Description `SavingsTracker.record_request()`'s history-append guard only fired when `tokens_saved > 0` (headroom's own lossy compression). In `--mode cache`, `tokens_saved` is near-always 0 by design — the frozen prefix is byte-replayed rather than compressed, to keep the provider's prompt cache warm. That silently dropped every history point on a cache-mode deployment even when `cache_read_tokens`/`cache_savings_usd` were large, making `headroom-monthly`-style tooling read as a total savings collapse. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/savings_tracker.py`: widen the history-append guard in `record_request()` to fire on `tokens_saved > 0` OR `cache_read_tokens > 0`, and carry `cache_read_tokens`/`cache_savings_usd` on the appended history entry so downstream consumers can show them. `_normalize_history_entry` defaults both fields to `0`/`0.0` for legacy entries that predate this change. - `tests/test_proxy_savings_history.py`: regression coverage that a cache-only request (zero `tokens_saved`, nonzero `cache_read_tokens`) still appends a history point, that the new fields round-trip through normalization, and that legacy history entries without the new keys still normalize cleanly. - `CHANGELOG.md`: added a `### Fixed` entry under `Unreleased`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_proxy_savings_history.py tests/test_savings_tracker_zero_price.py tests/test_proxy_project_savings.py -q collected 62 items tests/test_proxy_savings_history.py .................................... [ 58%] ...... [ 67%] tests/test_savings_tracker_zero_price.py .... [ 74%] tests/test_proxy_project_savings.py ................ [100%] ======================== 62 passed, 1 warning in 5.81s ========================= $ uv run ruff check headroom/proxy/savings_tracker.py tests/test_proxy_savings_history.py All checks passed! $ uv run mypy headroom/proxy/savings_tracker.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - **Environment:** personal fork deployed as a real proxy (macOS launchd service, `headroom install apply`) with `--backend bedrock --mode cache`, fronting a live Claude Code session where `cache_read_tokens`/`cache_savings_usd` are the dominant savings mechanism and headroom's own compression (`tokens_saved`) is near-always 0. - **Exact command / steps:** ran a multi-turn Claude Code session against this deployment, then inspected `proxy_savings.json`'s `history` array and the `headroom-monthly` savings-history rollup that reads it. - **Observed result:** before the fix, `history` stayed empty (or stopped growing) across the whole cache-mode session despite the lifetime `cache_read_tokens`/`cache_savings_usd` counters climbing turn over turn, because every request had `tokens_saved == 0` and never passed the append guard. `headroom-monthly` therefore rendered a flat/zero savings trend for a deployment that was, by its own lifetime counters, saving real money. After the fix, the same session appends a history point on every cache-hit turn, `cache_read_tokens`/`cache_savings_usd` populate on each new entry, and the monthly rollup tracks the lifetime counters instead of reading as a collapse. - **Not tested:** this deployment is `--mode cache`-only; the `tokens_saved > 0` branch of the widened guard (plain `--mode token` compression-driven savings) was not independently re-verified live post-fix, only via the existing/updated test suite, since it was already covered by pre-existing behavior. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — backend logic change, no UI surface. ## Additional Notes - No linked issue number: found via independent investigation of a personal deployment, not filed as a `headroomlabs-ai/headroom` issue first. - Also incidentally fixes a pre-existing test isolation gap in `test_savings_tracker_helpers_normalize_inputs_and_paths` (it unset `HEADROOM_SAVINGS_PATH` but not `HEADROOM_WORKSPACE_DIR`, so the default-path assertion could pick up whatever workspace a live deployment on the test machine had exported, instead of the library default). --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: Tejas Chopra <chopratejas@gmail.com> |
||
|
Abhay Singh
|
84f66da36f
|
fix(init/codex): merge into hooks.json instead of overwriting it (#2173)
## Description
`headroom init codex` destroys a user's existing Codex hooks.
`_ensure_codex_hooks` builds a fresh payload containing only Headroom's
two hooks and writes it wholesale:
```python
payload = { "hooks": { "SessionStart": [...headroom...], "PreToolUse": [...headroom...] } }
path.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
```
It never reads the existing file, so any user-managed hooks (and any
other top-level keys) in `~/.codex/hooks.json` are silently replaced
with just Headroom's entries — data loss on every `init codex` run.
The sibling registrars do it correctly: `_ensure_claude_hooks` and
`_ensure_copilot_hooks` both read via `_json_file`, then merge per event
and dedup on the Headroom marker, preserving unrelated user entries. The
codex path was the lone writer that overwrote.
## Fix
Read-merge-write, mirroring `_ensure_claude_hooks`: load the existing
payload, keep each event's entries that don't carry the
`headroom-init-codex` marker, append Headroom's, and write back. User
hooks and other top-level keys survive; Headroom's are deduped
(idempotent re-runs).
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/cli/init.py`: `_ensure_codex_hooks` reads via `_json_file`,
merges per event with marker dedup, and writes via `_write_json` (no
more wholesale overwrite).
- `tests/test_cli/test_init_cli.py`: add
`test_ensure_codex_hooks_preserves_user_hooks` — a user hook and an
unrelated top-level key survive; Headroom's hook is appended once.
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/cli/init.py tests/test_cli/test_init_cli.py
All checks passed!
$ python -m py_compile headroom/cli/init.py tests/test_cli/test_init_cli.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified the merge with a
dependency-free script that replicates the old (overwrite) vs new
(merge) logic, and left the full pytest to CI.
- Exact command / steps: gave a config with a user `SessionStart` hook
(`echo my-own-hook`) and an unrelated top-level key (`notify: true`),
then ran both the old and new logic.
- Observed result: old drops both the user hook and the top-level key;
new keeps both and appends Headroom's hook exactly once. The new test
asserts this against the real `_ensure_codex_hooks`.
- Not tested: a live `headroom init codex` end to end; full local
`pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" and "type checking" boxes are unchecked
because the full suite imports the ML stack, which I can't run in this
environment; the change mirrors the existing `_ensure_claude_hooks`
merge logic, verified by the standalone proof and the new test (which
reuses the file's existing hooks-test harness).
|
||
|
Abhay Singh
|
8da4384bfc
|
fix(init/codex): don't delete per-profile provider settings (#2146)
## Description `headroom init codex` silently deletes a user's per-profile provider settings. `_ensure_codex_provider` owns the root-level `model_provider` / `openai_base_url` keys, and to avoid emitting a duplicate top-level key it strips any prior assignment before re-inserting its block: ```python content = re.sub(r"(?m)^[ \t]*model_provider[ \t]*=.*\r?\n", "", content) content = re.sub(r"(?m)^[ \t]*openai_base_url[ \t]*=.*\r?\n", "", content) ``` Those multiline regexes match the keys at any indentation, **in any TOML table**. Codex supports per-profile overrides: ```toml [profiles.work] model_provider = "azure" [profiles.gpt5] model_provider = "openai" ``` So a user with named Codex profiles who runs `headroom init codex` has every `[profiles.*]` `model_provider` / `openai_base_url` line silently removed. Those profiles then fall through to the injected root `model_provider = "headroom"` default — their routing is quietly changed. That collateral deletion isn't needed to prevent the root-level duplicate the strip exists for (#260); the unwrap-side sibling `_strip_codex_init_block` proves the intent is precise (it only removes the Headroom-owned value). ## Fix Scope the strip to the document root — everything before the first table header. Root-level `model_provider` / `openai_base_url` are still replaced (init owns them), but keys inside `[profiles.*]` (or any other table) are left untouched. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cli/init.py`: `_ensure_codex_provider` splits the config at the first table header and strips `model_provider`/`openai_base_url` only from the root section. - `tests/test_cli/test_init_cli.py`: add `test_ensure_codex_provider_preserves_profile_overrides` — a `[profiles.work]` override survives init while the root key is replaced by `headroom`. - `CHANGELOG.md`: Bug Fixes entry. ## Testing - [ ] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 check headroom/cli/init.py tests/test_cli/test_init_cli.py All checks passed! $ python -m py_compile headroom/cli/init.py tests/test_cli/test_init_cli.py OK ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing `headroom` pulls in the torch/transformers stack and a full `pytest` gets OOM-killed on this box, so I verified the strip with a dependency-free script that replicates the old (whole-file) vs new (root-scoped) regex, and left the full pytest to CI. - Exact command / steps: ran both strippers on a config with a root `model_provider = "openai"` and a `[profiles.work]` block overriding `model_provider`/`openai_base_url`. - Observed result: the old strip deletes the `[profiles.work]` overrides too; the new strip keeps them and still removes the root assignment. The new test asserts the profile override survives and the root becomes `headroom`. - Not tested: a live `headroom init codex` end to end; full local `pytest` deferred to CI (OOM, per above). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes The "unit tests pass locally" and "type checking" boxes are unchecked because the full suite imports the ML stack, which I can't run in this environment; the change scopes an existing regex strip to the document root, verified by the standalone proof and the new test (the two existing `_ensure_codex_provider` tests only exercise root-level and block-placement behavior, both preserved). I kept the fix to root-scoping rather than also matching only the `"headroom"` value, since that preserves the #260 duplicate-key guard without the broad deletion. Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Abhay Singh
|
8a71947023
|
fix(proxy): reject rate_limit_requests_per_minute=0 when limiting is enabled (#2142)
## Description
A `rate_limit_requests_per_minute` of 0 makes the proxy return a 500 on
every rate-limited request instead of failing configuration early.
The token-bucket wait computation divides by the per-minute rate:
```python
def consume_from_bucket(*, available_tokens, requested_tokens, rate_per_minute):
if available_tokens >= requested_tokens:
return True, available_tokens - requested_tokens, 0.0
wait_seconds = (requested_tokens - available_tokens) * (60.0 / rate_per_minute)
return False, available_tokens, wait_seconds
```
With `rate_limit_requests_per_minute == 0`, the bucket initializes to 0
tokens, so the first request reaches the division and raises
`ZeroDivisionError`. The CLI guards `--rpm` with
`click.IntRange(min=1)`, but `HEADROOM_PROXY_CONFIG_JSON` and
programmatic `ProxyConfig(...)` construction bypass that guard.
## Fix
Validate `rate_limit_requests_per_minute >= 1` in
`ProxyConfig.__post_init__` when `rate_limit_enabled`, mirroring the
existing `retry_max_attempts` validation. Bad enabled configs now fail
fast with a clear message. When rate limiting is disabled, `rpm=0`
remains inert and is not rejected.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/models.py`: reject `rate_limit_requests_per_minute <
1` when `rate_limit_enabled`.
- `tests/test_proxy_config_rate_limit.py`: cover zero/negative enabled
values, disabled zero, and a valid enabled value.
- `CHANGELOG.md`: add a bug-fix entry.
- Merged current `main` to pick up the repository-wide memory factory
type annotation fix that was breaking the PR lint job.
## Testing
- [x] Unit tests pass (`pytest` focused locally; broader CI passed on
the pre-merge head and fresh CI is running on the main-merged head)
- [x] Linting passes (`ruff check` focused locally)
- [x] Type checking passes for the prior mypy blocker after merging
current `main`
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
uvx ruff@0.15.17 check headroom/proxy/models.py tests/test_proxy_config_rate_limit.py headroom/memory/factory.py
All checks passed!
uvx ruff@0.15.17 format --check headroom/proxy/models.py tests/test_proxy_config_rate_limit.py headroom/memory/factory.py
3 files already formatted
git diff --check headroomlabs/main...HEAD
# no output
uv run --extra dev python -m pytest tests/test_proxy_config_rate_limit.py -q
4 passed
```
## Real Behavior Proof
- Environment: Windows 11 review worktree, Python 3.13.3.
- Exact command / steps: ran the focused rate-limit config test file and
targeted lint/format checks.
- Observed result: enabled zero and negative rpm raise `ValueError`;
disabled zero is accepted; valid enabled rpm is accepted.
- Not tested: full suite; fresh CI is queued after the main merge.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The validation is intentionally at the config boundary to match the
CLI's `IntRange(min=1)` contract and the existing fail-fast
`retry_max_attempts` check.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
Abhay Singh
|
def2f9a728
|
fix(learn): don't desync verbosity pairing on empty assistant turns (#2123)
## Description
`verbosity._ordered_events` and `_parse_session` disagree about empty
assistant turns, which desyncs the response list and produces spurious
fast-skips.
`_parse_session` only creates a `_Response` when an assistant message
actually said something:
```python
if words > 0 or out_tok > 0:
responses.append(_Response(...))
```
But `_ordered_events` consumes one `responses[ri]` for **every**
assistant line, with no matching filter:
```python
if ltype == "assistant" and ri < len(responses):
out.append((responses[ri].ts, "assistant", responses[ri]))
ri += 1
```
So an assistant turn with no text and no output tokens — for example a
pure `tool_use` turn where `usage` is absent — creates no `_Response` at
parse time, yet still consumes a slot in `_ordered_events`. That slot
actually belongs to a *later* real response, so the two lists drift by
one. A human reply that follows the real answer is then paired with the
next answer's (future) timestamp, `ts - last_resp.ts` goes negative, and
since a negative gap is always below the read-fraction threshold, a
spurious `fast_skip` is recorded. That inflates `fast_skip_rate`, which
feeds `pressure`, which lowers the recommended verbosity level.
The user side of `_ordered_events` already replicates its parse-site
filter (`_human_text(...) is None -> continue`); only the assistant side
was missing the equivalent guard. That asymmetry is the bug.
## Fix
In `_ordered_events`, compute `words`/`out_tok` for the assistant line
the same way `_parse_session` does and only consume a response when
`words > 0 or out_tok > 0`, keeping the two functions in lockstep.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/learn/verbosity.py`: `_ordered_events` applies the `words >
0 or out_tok > 0` guard on the assistant branch before consuming a
response, with a comment explaining the desync.
- `tests/test_verbosity_learn.py`: add `_empty_assistant` helper and
`test_empty_assistant_message_does_not_desync_fast_skip` (an empty
assistant turn before a real answer + a slow reply must not record a
fast skip).
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [x] Unit tests pass (`uv run --extra dev pytest
tests/test_verbosity_learn.py::TestSignalExtraction::test_empty_assistant_message_does_not_desync_fast_skip
-q`)
- [x] Linting passes (`uvx ruff@0.15.17 check
headroom/learn/verbosity.py tests/test_verbosity_learn.py
headroom/memory/factory.py`)
- [x] Type checking passes (`uvx mypy==1.20.2
headroom/memory/factory.py`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/learn/verbosity.py tests/test_verbosity_learn.py
All checks passed!
$ python -m py_compile headroom/learn/verbosity.py tests/test_verbosity_learn.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified the alignment with a
dependency-free script that models the parse-site filter, the old vs new
`_ordered_events` consume, and the resulting human-to-response pairing,
and left the full pytest to CI.
- Exact command / steps: built an event stream `[empty assistant, real
answer #1, fast human reply, real answer #2, reply]`, computed the
response list from the parse filter, then walked the old (unfiltered)
and new (filtered) consume to find the gap between the first human and
the response paired before it.
- Observed result: old consume pairs the reply with answer #2 (a future
timestamp) -> gap `-8` (spurious fast_skip); new consume keeps alignment
and pairs it with answer #1 -> gap `+1`. The new test builds a session
with an empty assistant turn and a genuinely slow reply and asserts
`fast_skips == 0`.
- Not tested: a real Claude Code transcript end to end; full local
`pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
Merged current `main` to pick up the repository-wide mypy cache-key
annotation fix, then verified the focused regression locally. the change
adds the existing parse-site filter to one branch of a pure file-parsing
function, verified by the standalone alignment proof and the new
regression test for CI.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Abhay Singh
|
faed4dcfe7
|
fix(wrap/claude): bind _wrap_settings_path before the try (#2126)
## Description
`headroom wrap claude` crashes with an `UnboundLocalError` from its
cleanup `finally` whenever the proxy fails to start, which both hides
the real error and skips cleanup.
`claude()` initializes its cleanup state before the `try` so the
`finally` can always reference it — `proxy_holder`, `_saved_base_url`,
`_settings_foundry`, `port_holder`, `_settings_vertex` are all bound up
front. But `_wrap_settings_path` was the exception: it was assigned only
inside the `try`, after `_ensure_proxy`:
```python
try:
...
proxy_holder[0], actual_port = _ensure_proxy(port, ...) # can raise
...
_wrap_settings_path = Path.cwd() / ".claude" / "settings.local.json" # assigned here
...
finally:
_restore_claude_wrap_base_url(..., settings_path=_wrap_settings_path) # referenced here
cleanup()
```
`_ensure_proxy` raises when the requested port is unavailable and the
range is exhausted, or when the proxy subprocess fails to start. When it
does, control jumps to the `finally`, which evaluates
`settings_path=_wrap_settings_path` — a local that was never assigned —
and raises `UnboundLocalError`. That replaces the real failure with a
raw traceback, and because the `finally` aborts on that line,
`cleanup()` never runs, so proxy cleanup and wrap-marker clearing are
skipped too.
## Fix
Bind `_wrap_settings_path` before the `try`, next to the other cleanup
holders, so the `finally` can always reference it. The value is
unchanged (the in-`try` assignment is removed since it computed the same
path).
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/cli/wrap.py`: hoist the `_wrap_settings_path` initialization
to before the `try` (alongside `proxy_holder`/`_saved_base_url`/…) and
drop the redundant in-`try` assignment.
- `tests/test_cli/test_wrap_claude_finally_unbound.py`: new test — drive
`wrap claude` with `_ensure_proxy` patched to raise and assert the
`finally` completes (no `UnboundLocalError`, and both restore and
cleanup ran).
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [x] Unit tests pass (`uv run --extra dev pytest
tests/test_cli/test_wrap_claude_finally_unbound.py -q`)
- [x] Linting passes (`uvx ruff@0.15.17 check headroom/cli/wrap.py
tests/test_cli/test_wrap_claude_finally_unbound.py
headroom/memory/factory.py`)
- [x] Type checking passes (`uvx mypy==1.20.2
headroom/memory/factory.py`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/cli/wrap.py tests/test_cli/test_wrap_claude_finally_unbound.py
All checks passed!
$ python -m py_compile headroom/cli/wrap.py tests/test_cli/test_wrap_claude_finally_unbound.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and running the CLI
test locally OOM-kills this box, so I verified the control flow with a
dependency-free script that reproduces the try/finally with the variable
assigned inside vs before the try, and left the full pytest (including
the new CLI test) to CI.
- Exact command / steps: ran the flow with the variable bound inside the
try (old) and before the try (new), each with an early failure that
fires before the in-try assignment.
- Observed result: old raises `UnboundLocalError` from the finally and
skips restore/cleanup; new runs the finally cleanly and lets the real
`RuntimeError` propagate. The new CLI test drives `wrap claude` with
`_ensure_proxy` raising and asserts no `UnboundLocalError` and that
restore and cleanup both ran.
- Not tested: a live proxy port-exhaustion end to end; full local
`pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
Merged current `main` to pick up the repository-wide mypy cache-key
annotation fix, then verified the focused regression locally. the change
hoists one assignment to before the `try` (mirroring the four sibling
holders three lines above), verified by the control-flow proof and a new
CLI test that reuses the same mocking pattern the existing `wrap claude`
vertex tests use.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Abhay Singh
|
d236b27c60
|
fix(wrap/codex): export the detected custom upstream base URL (#2125)
## Description `headroom wrap codex` detects a user's custom upstream gateway but never tells Codex to use it, so the user's gateway key is sent to `api.openai.com`. `_inject_codex_provider_config` handles a Codex user who has an OpenAI-compatible gateway declared in `~/.codex/config.toml`, e.g. ```toml model_provider = "freemodel" [model_providers.freemodel] base_url = "https://api.freemodel.dev" ``` It injects the Headroom provider with `env_http_headers = { ... "X-Headroom-Base-Url" = "HEADROOM_CODEX_UPSTREAM_BASE_URL" }` and **returns the preserved upstream URL** so the caller can export it. Its docstring even says: *"Callers that go on to launch Codex should export this value into `HEADROOM_CODEX_UPSTREAM_BASE_URL`."* But `_prepare_codex_wrap_state` called it as a bare statement and discarded the return, and `_run_codex_wrap` / `_build_codex_launch_env` only ever set `OPENAI_BASE_URL`. A repo-wide grep confirms `HEADROOM_CODEX_UPSTREAM_BASE_URL` (`_UPSTREAM_BASE_URL_ENV_VAR`) is never assigned into any process env — it appears only at its definition and in that docstring. Since Codex only emits the `X-Headroom-Base-Url` header when the env var exists, the header is omitted, the proxy's OpenAI handler falls back to its hardcoded `https://api.openai.com`, and the user's `freemodel.dev` key is sent to OpenAI, which rejects it. This is a regression: the wiring existed in the original `#1614` fix (`_codex_custom_upstream = _inject_codex_provider_config(...)` then `env[_UPSTREAM_BASE_URL_ENV_VAR] = ...`) and was dropped by a later refactor that extracted `_prepare_codex_wrap_state`. ## Fix Restore the wiring: `_prepare_codex_wrap_state` now captures and returns `_inject_codex_provider_config`'s value, and `_run_codex_wrap` exports it into the launch env (`env[_UPSTREAM_BASE_URL_ENV_VAR] = custom_upstream`) when it is non-None and not already set, so a user-provided value still wins. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cli/wrap.py`: `_prepare_codex_wrap_state` returns the detected custom upstream URL; `_run_codex_wrap` exports it into the launch env (and its display list) when set. - `tests/test_cli/test_wrap_codex.py`: add `TestCodexLaunchExportsCustomUpstream` — drives `_run_codex_wrap` with mocked prepare/launch and asserts the launch env carries `HEADROOM_CODEX_UPSTREAM_BASE_URL` when a custom upstream is detected, and does not when there isn't one. - `CHANGELOG.md`: Bug Fixes entry. ## Testing - [x] Unit tests pass (`uv run --extra dev pytest tests/test_cli/test_wrap_codex.py::TestCodexLaunchExportsCustomUpstream -q`) - [x] Linting passes (`uvx ruff@0.15.17 check headroom/cli/wrap.py tests/test_cli/test_wrap_codex.py headroom/memory/factory.py`) - [x] Type checking passes (`uvx mypy==1.20.2 headroom/memory/factory.py`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 check headroom/cli/wrap.py tests/test_cli/test_wrap_codex.py All checks passed! $ python -m py_compile headroom/cli/wrap.py tests/test_cli/test_wrap_codex.py OK ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing `headroom` pulls in the torch/transformers stack and running the CLI test locally OOM-kills this box, so I verified the wiring with a dependency-free script that models prepare -> run -> the proxy's upstream fallback, and left the full pytest (including the new CLI test) to CI. - Exact command / steps: modelled the old flow (inject return discarded) and the new flow (return exported into the launch env), then applied the proxy's rule that a missing `HEADROOM_CODEX_UPSTREAM_BASE_URL` falls back to `api.openai.com`. - Observed result: old effective upstream is `https://api.openai.com` (the gateway key is misrouted); new effective upstream is `https://api.freemodel.dev` (the user's gateway). The new CLI test asserts the launch env carries the var when a custom upstream is present and omits it otherwise. - Not tested: a live Codex process reading the env and emitting the header; full local `pytest` deferred to CI (OOM, per above). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes Merged current `main` to pick up the repository-wide mypy cache-key annotation fix, then verified the focused regression locally. the change threads one return value through two functions and exports it, verified by the wiring proof and a new CLI test that drives `_run_codex_wrap` with the heavy prepare/launch steps mocked so only the env-export logic is exercised. Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: Tejas Chopra <chopratejas@gmail.com> |
||
|
Abhay Singh
|
f8eaaeb26a
|
fix(cache): normalize embeddings before the semantic similarity check (#2122)
## Description
The semantic tier of the dynamic-content detector compares an
unnormalized dot product against a cosine threshold, so it flags almost
everything as dynamic and strips the static content it is supposed to
protect.
`SemanticDetector` pre-computes exemplar embeddings and, per sentence,
scores similarity with `np.dot` and compares to `semantic_threshold`:
```python
self._exemplar_embeddings = self._model.encode(self.DYNAMIC_EXEMPLARS, convert_to_numpy=True)
...
sentence_embeddings = self._model.encode(sentence_texts, convert_to_numpy=True)
similarities = np.dot(sentence_embeddings, self._exemplar_embeddings.T)
...
if max_sim < self.config.semantic_threshold: # semantic_threshold defaults to 0.7
continue
```
`sentence_transformers.encode(..., convert_to_numpy=True)` does **not**
normalize by default. So `np.dot` here is an inner product whose
magnitude scales with the embedding norms (typically ~5-15 for MiniLM),
not a cosine similarity in [0, 1]. Comparing that against
`semantic_threshold=0.7` (documented and configured as a 0-1 similarity)
is a scale mismatch: nearly every sentence clears the threshold, so the
semantic tier classifies almost all text as dynamic, moves it into
`dynamic_content`, and empties `static_content` — busting the very cache
the detector exists to protect.
A standalone repro: an unrelated sentence with a true cosine of ~0.1 to
an exemplar produces a raw dot of ~9.1 (well over 0.7); normalized, it
correctly scores ~0.09 and stays static.
The correct behavior is used by the in-repo siblings:
`prediction/feature_extractor.py` passes `normalize_embeddings=True`,
and `memory/adapters/embedders.py` L2-normalizes before dot-product
similarity. This detector did neither.
## Fix
Pass `normalize_embeddings=True` to both `encode` calls (exemplars in
`__init__` and sentences in `detect`). Both sides of the dot product are
then unit vectors, so `np.dot` is a true cosine similarity in [-1, 1],
comparable to `semantic_threshold`.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/cache/dynamic_detector.py`: add `normalize_embeddings=True`
to the exemplar encode (`__init__`) and the sentence encode (`detect`),
with comments explaining the cosine requirement.
- `tests/test_cache/test_dynamic_detector.py`: add
`TestSemanticDetectorNormalization` — a recording fake model asserts
both encode calls pass `normalize_embeddings=True` (via `object.__new__`
for `detect`, and a monkeypatched registry for `__init__`). No model
download needed.
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [x] Unit tests pass (`uv run --extra dev pytest
tests/test_cache/test_dynamic_detector.py::TestSemanticDetectorNormalization
-q`)
- [x] Linting passes (`uvx ruff@0.15.17 check
headroom/cache/dynamic_detector.py
tests/test_cache/test_dynamic_detector.py headroom/memory/factory.py`)
- [x] Type checking passes (`uvx mypy==1.20.2
headroom/memory/factory.py`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/cache/dynamic_detector.py tests/test_cache/test_dynamic_detector.py
All checks passed!
$ python -m py_compile headroom/cache/dynamic_detector.py tests/test_cache/test_dynamic_detector.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`, numpy.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the scale mismatch
with a dependency-free numpy script (no sentence-transformers), and left
the full pytest to CI.
- Exact command / steps: built a MiniLM-dimension exemplar direction and
a sentence direction with a true cosine of ~0.1 (genuinely not dynamic),
gave them realistic un-normalized magnitudes (~9 and ~11), and computed
the old `np.dot` of the raw vectors versus the new `np.dot` of the
normalized vectors, against the 0.7 threshold.
- Observed result: old raw dot ~9.1 (far above 0.7 -> the unrelated
sentence is wrongly flagged dynamic); new cosine ~0.09 (below 0.7 ->
correctly kept static), and always within [-1, 1]. The new tests assert
both encode calls pass `normalize_embeddings=True`.
- Not tested: a real sentence-transformers model end to end; full local
`pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
Merged current `main` to pick up the repository-wide mypy cache-key
annotation fix, then verified the focused regression locally. the change
adds one keyword argument to two `encode` calls, verified by the numpy
proof and the new fake-model tests for CI. The fix brings this detector
in line with the two sibling call sites that already normalize.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Rod Boev
|
fb683e18d4
|
fix(litellm): forward chat_template_kwargs and other vendor top-level fields to OpenAI-compatible backends via extra_body (#2128) (#2163)
## Description
When Headroom forwards a `/v1/chat/completions` request to an
OpenAI-compatible backend (vLLM) via the LiteLLM backend, the
non-standard-but-OpenAI-compatible top-level field
`chat_template_kwargs` (e.g. `{"chat_template_kwargs":
{"enable_thinking": false}}`, used by vLLM to toggle Qwen3-family
"thinking" mode per request) never reaches the upstream model. A caller
that needs thinking *off* for a specific request has no way to disable
it through Headroom: the reasoning model spends its whole output-token
budget on hidden `<think>...</think>` content and returns
empty/truncated visible content.
Root cause: `LiteLLMBackend.send_openai_message`
(`headroom/backends/litellm.py:1101-1210`) and `stream_openai_message`
(`headroom/backends/litellm.py:1285+`) build the outgoing LiteLLM
`kwargs` from an explicit allowlist of recognized OpenAI params
(`headroom/backends/litellm.py:1129-1141`: `max_tokens`, `temperature`,
`top_p`, `stop`, `tools`, `tool_choice`, `response_format`, `seed`,
`n`). Only `model` and `messages` are copied unconditionally; anything
not in the list — including `chat_template_kwargs` — is dropped before
`acompletion(**kwargs)`. This is exactly the "litellm-backed forwarding
only passes fields it recognizes as standard OpenAI params" the reporter
suspected.
LiteLLM already forwards arbitrary vendor fields to an OpenAI-compatible
backend verbatim through its documented `extra_body` parameter — the
same mechanism vLLM users use directly. This change collects the
top-level body fields Headroom does not consume as standard params and
forwards them under `extra_body`, so `chat_template_kwargs` (and any
other vendor top-level field) reaches vLLM unchanged, on both the
buffered and streaming paths.
Closes #2128.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- In `LiteLLMBackend.send_openai_message` and `stream_openai_message`,
after populating the standard-param allowlist, collect top-level `body`
keys not consumed by Headroom/LiteLLM (everything outside the standard
allowlist plus `model`/`messages`/`stream`/`stream_options` and internal
markers) and forward them to the backend via LiteLLM's `extra_body`.
- `chat_template_kwargs` and other vendor-specific top-level fields now
reach the OpenAI-compatible upstream verbatim.
- Left the standard-param allowlist, region/profile config, API-key
forwarding, and the cache-stats usage block untouched; standard params
stay first-class LiteLLM kwargs (not moved into `extra_body`).
- Scoped to the OpenAI-format methods; the Anthropic-format
`send_message`/`stream_message` and the metadata-only
`OpenAICompatibleProvider` are unchanged.
## Testing
- [x] Unit tests pass (`uv run pytest
tests/test_litellm_openai_passthrough.py`)
- [x] Linting passes (`uv run ruff check .`)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality when applicable
- [x] Manual testing performed
### Test Output
```text
$ uv run pytest tests/test_litellm_openai_passthrough.py -q
.... [100%]
4 passed in 1.88s
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uv run`; `acompletion` mocked
(no live vLLM).
- Exact command / steps: `uv run pytest
tests/test_litellm_openai_passthrough.py -q`, which drives
`send_openai_message` and `stream_openai_message` with a body containing
`chat_template_kwargs: {"enable_thinking": false}` and inspects the
captured `acompletion` call kwargs.
- Observed result: on both the buffered and streaming paths
`acompletion` is now called with `extra_body={"chat_template_kwargs":
{"enable_thinking": false}}`; a standard-only body produces no
`extra_body`, and standard params (`max_tokens`, `temperature`, …)
remain first-class kwargs. Before the change the same body reaches
`acompletion` with `chat_template_kwargs` absent.
- Not tested: live vLLM run confirming Qwen3 thinking mode toggles off
end-to-end.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- This implements reporter option (a): pass unrecognized top-level body
fields through verbatim (via `extra_body`), which needs no new config
surface. Reporter option (b), an explicit allowlist/config on the
provider, is a deliberate non-goal here and can follow if maintainers
prefer it. The Anthropic-format `send_message`/`stream_message`
translation path and the direct-httpx passthrough path (which already
forwards the full body) are out of scope.
- Issue diagnosed by George Stephanis (`@georgestephanis`) with Claude
Code assistance, per the report's AI disclosure.
- `mypy` left unchecked: not part of the focused validation for this
change.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Rod Boev
|
cc072f0821
|
fix(cache-aligner): hash the frozen conversation prefix so Claude Code cache invalidation is detected (#2085) (#2161)
## Description Running Headroom as the API proxy for Claude Code, provider prompt-cache reuse collapsed: uncached input tokens went from ~755 to ~4.5M, cache-creation (write) tokens inflated ~4.4×, and one session burned ~36% of a weekly model cap. Roughly 96%-cached traffic became uncached+rewrite traffic — a net cost multiplier, not a saving. The `CacheAligner` owns the pipeline's "is the cacheable prefix byte-stable across requests?" signal (`stable_prefix_hash` / `prefix_changed` on `CachePrefixMetrics`). But `CacheAligner.apply()` computes that hash over **only `role == "system"` messages** (`headroom/transforms/cache_aligner.py:314-325`). Under Claude Code the system prompt is the stable part; what actually churns between requests is the conversation head — earlier user turns and tool-result blocks — the range Claude Code relies on for provider cache reads. `apply()` already receives the authoritative freeze boundary (`frozen_message_count`, produced by `PrefixCacheTracker`) and uses it to skip volatile-content detection, but the hash ignores it. So `prefix_changed` reports "prefix stable" even while the real cacheable prefix churns: the budget-burning regression is invisible and the byte-stability invariant the issue asks for is neither asserted nor enforced. This change scopes the aligner's stable-prefix hash to the actual frozen cacheable prefix (`messages[:frozen_message_count]` plus system messages), keyed on the authoritative `frozen_message_count`, so `prefix_changed` becomes a true cache-invalidation signal — and adds the replay regression test the issue specifies, locking the invariant "for `messages[0..k]` identical to the previous request, the emitted prefix bytes and hash are identical." `apply()` remains strictly detector-only and byte-equal; no rewrite is introduced. Closes #2085. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Scoped `CacheAligner.apply()`'s `stable_prefix_hash` to the frozen cacheable prefix: the byte content of `result_messages[:frozen_message_count]` (the frozen conversation head, in order) combined with the system messages, keyed on the authoritative `frozen_message_count` kwarg from `PrefixCacheTracker`. - `prefix_changed` now reflects churn in the true provider-cacheable prefix (a changed tool-result block that leaves the system prompt untouched is now detected), so Claude Code cache invalidation is observable via the existing `CachePrefixMetrics` and the `stable_prefix_hash:<hash>` marker. - Preserved first-turn behavior: when `frozen_message_count == 0` the hash falls back to the current system-only scope, so the first request in a session is byte-for-byte unchanged. - Kept `apply()` detector-only (deep copy, never mutates messages) and left the `should_apply` skip gate, volatile-content warning, token counts, and `TransformResult` shape unchanged. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_cache_aligner_prefix_stability.py`) - [x] Linting passes (`uv run ruff check .`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_cache_aligner_prefix_stability.py -q ..... [100%] 5 passed in 0.40s ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, `uv run`; no live provider. - Exact command / steps: `uv run pytest tests/test_cache_aligner_prefix_stability.py -q`, which replays consecutive `apply()` calls in the Claude Code shape (stable system prompt + accumulated tool-result prefix) with `frozen_message_count > 0`. - Observed result: when a frozen tool-result block changes between requests while the system prompt is byte-identical, `prefix_changed` is now `True` and `stable_prefix_hash` differs; when the frozen prefix + system are identical, `prefix_changed` is `False`; when only the live/unfrozen tail changes, `prefix_changed` stays `False`; `apply()` output stays byte-equal to input. Before the change the same frozen-prefix churn reports `prefix_changed=False` because the hash covers only the system prompt. - Not tested: live Anthropic prompt-cache accounting over a full Claude Code session. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - Scope: this slice corrects and locks the byte-stability invariant **at the CacheAligner boundary** — the exact acceptance criterion in the issue (a cache-preservation invariant over identical prefixes plus a replay regression test). It is distinct from, and does not touch, the upstream sources of prefix churn (ContentRouter per-block verdict flaps under `min_ratio` drift, #1619), the `headroom stats` cache-delta surfacing (#960), or parallel-subagent stream misclassification (#1949); those remain separate follow-ups. `PrefixCacheTracker`'s independent forwarded-prefix byte check (`headroom/cache/prefix_tracker.py`) is unchanged. - `mypy` left unchecked: not part of the focused validation for this change. Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: Tejas Chopra <chopratejas@gmail.com> |
||
|
Rod Boev
|
e6df6ea470
|
docs: qualify CCR auto-resolution support for Gemini (#2044)
## Description Headroom's CCR docs describe automatic response handling as universal, but the current code only wires that continuation path for Anthropic and OpenAI-compatible handlers. This updates the docs to describe the real Gemini behavior today, including the native Gemini gap and the reported `MALFORMED_FUNCTION_CALL` risk on Gemini OpenAI-compatible round-2 continuations. Refs #2041 ## Type of Change - [x] Documentation update - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Narrow CCR response-handler claims to the providers that currently implement them. - Add a Gemini-specific note covering native-handler limits and the reported round-2 continuation failure. ## Testing - [x] Unit tests pass - [ ] Linting passes - [ ] Type checking passes - [ ] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text uv run --no-sync pytest tests/test_ccr_response_handler.py tests/test_ccr_response_handler_extra.py -q ============================= test session starts ============================= platform win32 -- Python 3.12.13, pytest-9.0.3, pluggy-1.6.0 rootdir: D:\Repos\headroom-pr-2041-gemini-ccr-docs configfile: pyproject.toml plugins: anyio-4.12.1, langsmith-0.9.3, asyncio-1.3.0, cov-7.0.0 asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 42 items tests\test_ccr_response_handler.py ............................... [ 73%] tests\test_ccr_response_handler_extra.py ........... [100%] ============================= 42 passed in 0.91s ============================== ``` ## Real Behavior Proof - Environment: Windows, Python 3.12.13, docs-only change with no live Gemini provider call - Exact command / steps: `uv run --no-sync pytest tests/test_ccr_response_handler.py tests/test_ccr_response_handler_extra.py -q` - Observed result: All 42 CCR response-handler tests pass, confirming the existing Anthropic/OpenAI-compatible continuation behavior is unchanged by the docs update - Not tested: a live Gemini round-2 continuation request ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review Co-authored-by: Tejas Chopra <chopratejas@gmail.com> |
||
|
Abhay Singh
|
5fb449e90b
|
fix(subscription): keep efficiency_pct from exceeding 100% (#2121)
## Description
`HeadroomContribution.efficiency_pct` can report values above 100%
because its numerator and denominator disagree about cache-read tokens.
```python
def total_saved(self) -> int:
return (self.tokens_saved_compression + self.cli_filtering_saved()
+ self.tokens_saved_cache_reads) # includes cache reads
def raw_without_headroom(self) -> int:
return (self.tokens_submitted + self.tokens_saved_compression
+ self.cli_filtering_saved()) # excludes cache reads
def efficiency_pct(self) -> float:
raw = self.raw_without_headroom()
if raw == 0:
return 0.0
return round(self.total_saved() / raw * 100, 1)
```
`tokens_saved_cache_reads` are input tokens that were *forwarded* to the
provider and served from the prefix cache at a discount, so they already
live inside `tokens_submitted` (the "raw input tokens actually
forwarded"). They are added to the numerator via `total_saved()` but
never to the denominator, so with `tokens_submitted=100` and
`tokens_saved_cache_reads=1000` the method returns `1000.0%`, which the
dashboard renders verbatim. An efficiency percentage should never exceed
100%.
## Fix
Use the existing sibling `compression_saved()` (compression + CLI
filtering, which already excludes cache reads) as the numerator. Then
`efficiency_pct = compression_saved / (tokens_submitted +
compression_saved)`, which is bounded by its own denominator and is the
meaningful quantity here: the fraction of the pre-Headroom input that
compression and CLI filtering actually removed. Cache reads are a
provider-side discount on forwarded tokens, not tokens Headroom removed,
so they don't belong in a removal-efficiency ratio. `total_saved()` is
left unchanged for its other callers (`to_dict`, etc.).
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/subscription/models.py`: `efficiency_pct` now uses
`compression_saved()` instead of `total_saved()` as the numerator, with
a comment explaining the cache-read inconsistency.
- `tests/test_subscription_contribution.py`: new tests — cache reads
can't push efficiency over 100%, the ratio equals the
compression-removal fraction, and empty input yields 0.
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [x] Unit tests pass (`uv run --extra dev pytest
tests/test_subscription_contribution.py -q`)
- [x] Linting passes (`uvx ruff@0.15.17 check
headroom/subscription/models.py tests/test_subscription_contribution.py
headroom/memory/factory.py`)
- [x] Type checking passes (`uvx mypy==1.20.2
headroom/memory/factory.py`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/subscription/models.py tests/test_subscription_contribution.py
All checks passed!
$ python -m py_compile headroom/subscription/models.py tests/test_subscription_contribution.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified the ratio with a
dependency-free script that replicates the three methods, and left the
full pytest to CI.
- Exact command / steps: computed `efficiency_pct` under the old
numerator (`total_saved`) and the new numerator (`compression_saved`)
for `tokens_submitted=100, tokens_saved_cache_reads=1000` and for a real
compression case (`submitted=1000, compression=400, cache_reads=300`).
- Observed result: old returns `1000.0%` for the cache-read case
(impossible) and the new returns `0.0%`; for the compression case old
returns `50.0%` (inflated by cache reads) and new returns `28.6%` (= 400
/ 1400), always `<= 100%`. The new tests assert these.
- Not tested: the dashboard render path end to end; full local `pytest`
deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
Merged current `main` to pick up the repository-wide mypy cache-key
annotation fix and current dependency security floors, then verified the
focused regression locally. the change swaps one method call in a pure
dataclass method, verified by the standalone proof and the new tests for
CI.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
Abhay Singh
|
bc24e258b1
|
fix(mcp/claude): don't clobber an unparseable Claude config on register (#1660)
## Description
When the `claude` CLI isn't on PATH (or its `mcp add` fails),
`ClaudeRegistrar`
falls back to `_register_via_file`, which does a full-file
read-modify-write of
`~/.claude/.claude.json`:
```python
config = _read_json(target) # returns {} on JSONDecodeError
servers = config.setdefault("mcpServers", {})
servers[spec.name] = _spec_to_entry(spec)
_write_json(target, config) # overwrites the ENTIRE file
```
`_read_json` returns `{}` for a file that exists but doesn't parse. So
if
`~/.claude/.claude.json` is momentarily corrupt or hand-edited (a
trailing
comma, a crash mid-write), the register path silently rewrites it as
just
`{"mcpServers": {"headroom": {...}}}` — **destroying every other key
Claude Code
keeps there**: `projects`, `oauthAccount`, session history, etc. There's
no
backup. The existing `test_get_server_robust_to_bad_json` only covers
the *read*
path; the destructive *write* path was untested.
Closes: no issue filed — found while auditing the MCP registry
config-write paths.
## Fix
Keep `_read_json` (returning `{}`) for the read-only callers
(`get_server`,
removal), where it's harmless. Add `_read_json_for_write` for the
rewrite path:
it returns `{}` only when the file is **absent or empty** (safe to start
fresh)
and raises `_MalformedConfigError` when the file is present but not a
JSON
object. `_register_via_file` catches it and returns a `FAILED` result
with an
actionable message instead of overwriting.
Result: absent/empty file → registers fresh (unchanged); valid file →
merges,
all other keys preserved (unchanged); present-but-invalid file → refuses
to
touch it and tells the user to fix or remove it.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/mcp_registry/claude.py`: add `_read_json_for_write` +
`_MalformedConfigError`; `_register_via_file` uses it and returns
`FAILED` (without writing) when the target is present-but-unparseable.
`_read_json` is unchanged for read-only callers.
- `tests/test_mcp_registry/test_claude_registrar.py`: regression tests —
register against malformed configs leaves the bytes untouched and
returns `FAILED`; register against a valid config still merges and
preserves unrelated keys.
- `CHANGELOG.md`: Bug Fixes entry under Unreleased.
## Testing
- [x] New tests added for the fixed behavior
- [x] Linting passes (`ruff check`) and formatting is clean (`ruff
format --check`)
- [ ] Full `pytest` deferred to CI (local-OOM reason under Real Behavior
Proof).
```text
$ uv run ruff check headroom/mcp_registry/claude.py tests/test_mcp_registry/test_claude_registrar.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, headroom built from this
branch. Importing `headroom` loads the torch/transformers stack; a full
`pytest` gets OOM-killed on this box, so I verified the write-path logic
with a dependency-free script and left the full pytest to CI.
- Exact command / steps: replicated `_read_json_for_write` and the
`_register_via_file` read-modify-write flow in a standalone script (only
stdlib, no `headroom` import) against real temp files, and exercised:
absent, empty, four malformed variants (`not json`, `{`, `{"projects":
}`, `[]`), and a valid config carrying `projects`/`oauthAccount`.
- Observed result: absent/empty register fresh; every malformed variant
returns FAILED and the original bytes on disk are byte-for-byte
unchanged (no clobber); a valid config merges in `headroom` while
`projects`/`oauthAccount` survive:
```text
OK: absent -> fresh register
OK: empty -> fresh register
OK: malformed -> FAILED, original bytes preserved (no clobber)
OK: valid config -> merged, unrelated keys preserved
MCP CONFIG-WRITE LOGIC VERIFIED
```
- Not tested: driving the real `claude` CLI-absent path end-to-end on a
live `~/.claude/.claude.json` (didn't want to touch a real Claude
install); the file-fallback logic is exercised directly by the
regression tests. Full local `pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- No new dependencies. `headroom/mcp_registry/opencode.py` has the same
read-`{}`-then-clobber shape on its write path (an OpenCode
`opencode.json` with comments/JSONC would be wiped) — I scoped this PR
to the Claude registrar to keep it focused and because OpenCode config
handling is being touched in other open PRs; happy to send a follow-up
for opencode with the same guard if useful.
---------
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
Rod Boev
|
81ddbd47d5
|
docs: document Claude VSCode deferred-tool rendering caveat (#2045)
## Description Headroom already documents why `ENABLE_TOOL_SEARCH=true` matters for Claude Code through a custom `ANTHROPIC_BASE_URL`, but it does not document the current VSCode extension rendering failure on the deferred-tool content blocks that setting can surface. This adds a narrow docs warning and workaround for the VSCode path without changing the CLI default that still helps the main Claude Code flow. Refs #2028 ## Type of Change - [x] Documentation update - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Document the Claude Code VSCode extension `unsupported content type` failure mode. - Explain when to set `ENABLE_TOOL_SEARCH=false` as a workaround. - Keep the existing default guidance for Claude CLI users unchanged. ## Testing - [x] Unit tests pass - [ ] Linting passes - [ ] Type checking passes - [ ] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text uv run --no-sync pytest tests/test_cli_doctor.py -q ============================= test session starts ============================= platform win32 -- Python 3.12.13, pytest-9.0.3, pluggy-1.6.0 rootdir: D:\Repos\headroom-pr-2028-claude-vscode-tool-search-docs configfile: pyproject.toml plugins: anyio-4.12.1, langsmith-0.9.3, asyncio-1.3.0, cov-7.0.0 asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 51 items tests\test_cli_doctor.py ............................................... [ 92%] .... [100%] ============================= 51 passed in 0.67s ============================== ``` ## Real Behavior Proof - Environment: Windows, Python 3.12.13, docs-only change with no LLM provider involved - Exact command / steps: `uv run --no-sync pytest tests/test_cli_doctor.py -q` - Observed result: All 51 `test_cli_doctor.py` tests pass, confirming the existing `headroom doctor` CLI behavior is unchanged by the new VSCode troubleshooting docs - Not tested: live rendering in the Claude Code VSCode extension ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes The extension renderer bug is upstream. This PR only makes the current Headroom behavior explicit and gives users the supported workaround. |
||
|
Rod Boev
|
3a39cb99ad
|
install: couple Codex routing to persistent runtime readiness (#2043)
## Description Persistent install currently writes Codex routing before runtime readiness, and it leaves that routing in place while the runtime is stopped or after a failed start. This changes the lifecycle so routing is applied only after the persistent runtime is ready and reverted before stop or removal, which prevents Codex from getting stranded on a dead `127.0.0.1:8787` provider. Closes #2038 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Apply persistent provider mutations only after runtime readiness. - Revert persistent provider mutations before stop and remove. - Clear stale routing before recovery start paths, then reapply after readiness. - Add focused install lifecycle tests for the ordering change. ## Testing - [x] Unit tests pass - [x] Linting passes - [ ] Type checking passes - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text uv run pytest tests/test_cli/test_install_cli.py -q 23 passed, 1 warning in 0.27s uv run ruff check headroom/cli/install.py tests/test_cli/test_install_cli.py All checks passed! uv run ruff format --check headroom/cli/install.py tests/test_cli/test_install_cli.py 2 files already formatted ``` ## Real Behavior Proof - Environment: Windows, Python 3.12.13, no provider (persistent install lifecycle change; no live Codex or proxy call) - Exact command / steps: `uv run pytest tests/test_cli/test_install_cli.py -q`, then `uv run ruff check` and `uv run ruff format --check` on `headroom/cli/install.py` and `tests/test_cli/test_install_cli.py` - Observed result: 23 install-lifecycle tests pass and lint/format checks pass, confirming persistent Codex provider mutations are now applied only after runtime readiness and reverted before stop/remove - Not tested: a live persistent-runtime start/stop cycle with Codex routing ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Rod Boev
|
09e72125b4
|
fix(proxy): isolate image compression in a subprocess so a native SIGSEGV can't take down the proxy (#2107) (#2162)
## Description On Apple Silicon (arm64) macOS the proxy hard-crashes with SIGSEGV the moment it compresses an image, and because the proxy is the single API endpoint for every routed client (`ANTHROPIC_BASE_URL=http://127.0.0.1:8787`), one image request takes down every agent on the machine at once — they then fail with `ConnectionRefused` and retry into a closed port until the proxy is manually restarted. The faulting stack is inside OpenCV's KleidiCV ARM NEON resize (`kleidicv::neon::kleidicv_resize_generic_stripe_u8`), reached from the SigLIP ONNX image encoder during `ImageCompressor.compress()`. The proxy runs that call on a `ThreadPoolExecutor` (`headroom/proxy/server.py:946`), and both handler call sites (`headroom/proxy/handlers/anthropic.py:1148-1172`, `headroom/proxy/handlers/openai.py:2274-2296`) wrap it in `try/except Exception` intending to fail open. That guard cannot help: a native SIGSEGV is not a Python exception, and a segfault on any worker thread aborts the whole interpreter. Thread isolation is not crash isolation. The defect Headroom owns is that an optional, best-effort, native-heavy transform runs in-process with no crash boundary, so any native fault in it is fatal to the proxy and to every unrelated client it fronts. This change gives image compression a real crash boundary by executing it in a spawned subprocess, so a native crash degrades to "image forwarded uncompressed" instead of killing the proxy. Closes #2107. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `headroom/proxy/image_isolation.py` with `run_image_compression_isolated(messages, provider, *, timeout) -> tuple[list[dict], dict | None]`, which runs an `ImageCompressor.compress` worker inside a lazily-created module-level `ProcessPoolExecutor(max_workers=1)` on a **spawn** multiprocessing context (ONNX sessions are not fork-safe) and carries the compression result (technique, `savings_percent`, token counts) back across the process boundary. The native OpenCV/KleidiCV work now runs in the child address space. - Made the runner fail open for **any** child outcome: `BrokenProcessPool` (the class raised when the child is killed by a signal, i.e. SIGSEGV/SIGABRT), `TimeoutError`, or any other `Exception` all return `(messages, None)` — the original `messages` unchanged, no telemetry — and reset the pool so the next request re-spawns a fresh child. - Routed the two request-path image-compression sites (`handlers/anthropic.py`, `handlers/openai.py`) through the runner, keeping the existing `ImageCompressionDecision` gate and the `image_compression` mutation tag, and emitting the savings `INFO` log line from the runner's returned result on the success path. - Scoped strictly to crash containment: `config.image_optimize` default is unchanged, no dependency pins, no new env switches. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_image_compression_isolation.py tests/test_image_compression_offload.py`) - [x] Linting passes (`uv run ruff check .`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_image_compression_isolation.py tests/test_image_compression_offload.py -q ....... [100%] 7 passed in 1.73s ``` The reproduction test (`test_worker_sigsegv_fails_open_parent_survives`) spawns a real subprocess whose worker dies by signal (`os.abort()`), then asserts `run_image_compression_isolated` returns the original messages and that the parent test process is still alive and continues past the call. ## Real Behavior Proof - Environment: Windows 11, Python 3.12, `uv run`; no live provider. - Exact command / steps: `uv run pytest tests/test_image_compression_isolation.py tests/test_image_compression_offload.py -q`, which drives the runner against a real spawned subprocess that is killed by signal, one that raises, one that times out, and one that returns normally, and also locks the handler wiring to `run_image_compression_isolated(...)`. - Observed result: on a signal-killed child the runner returns the original message list and the parent survives; on a raising or timing-out child it fails open the same way; on a normal child the compressed messages are returned. The handler source-level regression keeps the savings log and mutation-tag path wired through the new isolation helper. Before the change, the handlers called `compressor.compress(...)` through `_run_compression_in_executor(...)`, so a native crash on that worker thread would abort the interpreter. - Not tested: live arm64 macOS run against a real `opencv-python` 5.x KleidiCV fault. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - Non-goals, kept out deliberately to keep this slice shippable now: pinning `opencv-python<5` (a transitive-dependency change that only masks this one fault and does not contain the next native crash), a `HEADROOM_IMAGE_OPTIMIZE=0` env off-switch (distinct config surface; `config.image_optimize=False` already disables the feature), the per-request ONNX model reload, and the negative-savings (`preserve` logged as `-100%`) reporting bug. The last two are independent defects noted in the same report and are better fixed on their own. - `mypy` left unchecked: not part of the focused validation for this change. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
