mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
1211 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
JerrettDavis
|
fe5176dbe7 | test(proxy): cover resilient extension loading | ||
|
Rod Boev
|
dbbef4bd41
|
fix(code-compressor): recover valid Python rewrites after local syntax rejection (#2202)
## Description A compile-invalid Python definition rewrite currently makes `CodeAwareCompressor` discard every otherwise valid rewrite in the file and return the original source at 0 percent reduction. The existing whole-file safety guard stays in place, while a Python-only recovery replay now preserves the rejected definition and keeps independent valid compression. The recovery reuses the current Python validation authority in `ast.parse()` plus `compile()`, runs only after the first assembled module already fails `_verify_syntax()`, and stays out of non-Python paths. Closes #1233 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added a Python-only recovery replay after the first assembled module fails syntax validation. - Preserved only the invalid function or class rewrite while allowing independent valid definitions to remain compressed. - Kept the existing whole-file syntax guard and original-source fallback as the terminal safety check. - Added focused invalid-node, valid-modern-syntax, and fail-safe coverage. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_python_invalid_node_falls_back_locally tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_python_recovery_does_not_block_valid_modern_syntax tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_python_recovery_still_returns_original_when_all_candidates_invalid tests/test_transforms/test_code_compressor.py::TestEdgeCases::test_syntax_errors_in_input tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_ast_preserves_structure_for_python -v`) - [x] Linting passes (`uv run ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text uv run pytest tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_python_invalid_node_falls_back_locally tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_python_recovery_does_not_block_valid_modern_syntax tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_python_recovery_still_returns_original_when_all_candidates_invalid tests/test_transforms/test_code_compressor.py::TestEdgeCases::test_syntax_errors_in_input tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_ast_preserves_structure_for_python -v 5 passed in 0.34s uv run ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py All checks passed! uv run ruff format headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py --check 2 files already formatted ``` ## Real Behavior Proof - Environment: Windows, synced `uv` environment with `dev` and `code` extras installed - Exact command / steps: run the focused invalid-node regression through public `compress(..., language="python")` - Observed result: `1 passed in 0.19s`; the invalid candidate stays original, the neighboring valid candidate remains compressed, and `headroom-PR-TARGET-1233-PROOF.md` records the base `ratio=1.0` whole-file rollback against the fixed head behavior. - Not tested: the stale future-import mismatch discussed in the old issue comment, already covered on current main ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - The stale future-import comment on #1233 is not the live slice here; current main already validates Python with `compile()` and already covers that ordering case. - This fix keeps the existing whole-file fail-safe and does not broaden into cross-language recovery or new syntax models. - `CHANGELOG.md` remains unchanged because Headroom generates release notes from conventional commits. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Rod Boev
|
8522fcbc40
|
fix(code): quarantine Perl parser from code-aware compression (#2204)
## Description `headroom proxy` can wedge when code-aware compression enters the tree-sitter Perl external scanner and the native scan keeps the GIL indefinitely. Current main still has two routes into that scanner: explicit `perl` or `pl` hints flow through `CodeAwareCompressor`, and `detect_language()` can nominate Perl on non-Perl code because its prefilter matches generic sigils such as decorators, JSDoc tags, and shell variables before phase 2 parses every surviving candidate grammar. This change quarantines Perl at the code-aware compression funnel without widening scope. Perl remains recognized at the input boundary, but live proxy compression no longer requests a Perl parser. Non-Perl code keeps its existing code-aware behavior. Real Perl falls back through the existing safe Kompress or passthrough contract instead of entering tree-sitter. The diff stays inside `headroom/transforms/code_compressor.py` plus focused parser-safety regressions. Refs #2185 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added a Perl quarantine list in `headroom/transforms/code_compressor.py` and used it to stop Perl candidate parsing during language detection. - Added a hard `_get_parser()` guard so no live code-aware path can construct a Perl parser. - Routed resolved explicit Perl hints through the existing safe fallback or passthrough contract before AST compression. - Added focused parser-safety regressions for non-Perl candidate bleed, explicit `perl` and `pl`, inferred real Perl, mixed fenced Perl content, fallback-disabled passthrough, and non-Perl negative space. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_perl_scanner_safety.py -q`) - [x] Linting passes (`uv run ruff check headroom/transforms/code_compressor.py tests/test_perl_scanner_safety.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text uv run pytest tests/test_perl_scanner_safety.py -q 8 passed in 1.93s uv run ruff check headroom/transforms/code_compressor.py tests/test_perl_scanner_safety.py All checks passed! uv run ruff format headroom/transforms/code_compressor.py tests/test_perl_scanner_safety.py --check 2 files already formatted ``` ## Real Behavior Proof - Environment: Windows, Python from the synced `uv` environment, `dev` and `code` extras installed, no provider call - Exact command / steps: Run `uv run pytest tests/test_perl_scanner_safety.py -q`. - Observed result: `8 passed in 1.93s`; the suite proves explicit `perl` and `pl`, inferred real Perl, mixed fenced Perl, and non-Perl negative-space routes all avoid Perl parser entry. - Not tested: the reporter's macOS payload and long-running concurrent workload ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - Use `Refs #2185`, not `Closes #2185`. The wedge surface is this PR's scope, but #2185 also carries a separate orphaned `headroom mcp serve` report that this slice does not address. - This is a reachability fix. It does not repair the upstream Perl scanner and it does not harden other grammars against the same class of native wedge. - `CHANGELOG.md` remains unchanged because Headroom generates release notes from conventional commits. - Merged PR https://github.com/headroomlabs-ai/headroom/pull/2114 addresses a different cooperative compression stall and stays separate from this native parser-entry slice. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
David Wells
|
2a954b69b4
|
feat(wrap): add ZCode desktop app support (#1845)
## Description Add `headroom wrap zcode` and `headroom unwrap zcode` commands for the ZCode desktop app (zcode.z.ai). ZCode is a desktop Electron IDE built by Z.AI, optimized for GLM-5.2 models. It has no CLI binary, so this follows the Pattern-B (proxy-only, print instructions) approach — same as Cursor, Cline, and Continue. **Upstream auto-detection:** `headroom wrap zcode` now reads `~/.zcode/v2/config.json` to detect the enabled provider and automatically configures the proxy upstream — no manual flags needed. Closes #1844 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - New module: `headroom/providers/zcode/__init__.py` and `runtime.py` (ZCodeProxyTargets, ZCodeUpstream, build_proxy_targets, detect_upstream, upstream_to_proxy_urls, render_setup_lines) - New CLI command: `headroom wrap zcode` in `headroom/cli/wrap.py:4815` — starts proxy, injects RTK into AGENTS.md, prints Base URL setup instructions - New CLI command: `headroom unwrap zcode` in `headroom/cli/wrap.py:5960` — removes RTK markers, stops proxy - New helper: `zcode_config_dir()` in `headroom/install/paths.py` - Updated `_run_proxy_only_watcher` to accept `anthropic_api_url`/`openai_api_url` params - Updated README.md: ZCode row in compatibility matrix, unwrap list, wrap command list - Updated CHANGELOG.md: entry under [Unreleased] > Added ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) — pre-existing numpy type stubs issue prevents full mypy run - [x] New tests added for new functionality (24 tests in `tests/test_cli/test_wrap_zcode.py`) - [x] Manual testing performed ### Test Output ```text tests/test_cli/test_wrap_zcode.py ........................ [100%] 24 passed, 1 warning in 0.22s ``` ## Real Behavior Proof - Environment: macOS 15.5, Python 3.12.13, headroom installed via `pip install -e .[dev]` - Exact command / steps: `headroom wrap zcode --port 9000` then `headroom unwrap zcode --port 9000` - Observed result: Wrap detects provider from `~/.zcode/v2/config.json` (e.g. "Z.ai Coding"), starts proxy with correct upstream on port 9000, injects RTK into AGENTS.md, prints detected provider + upstream + Base URL setup instructions. Unwrap removes RTK markers, deletes empty AGENTS.md, stops proxy. - Not tested: Actual ZCode app integration (ZCode is a desktop Electron app; Base URL configuration is manual in Settings > Model Settings) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas — N/A: code follows existing patterns - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — CLI-only changes ## Additional Notes - **Pattern-B approach:** ZCode is a desktop Electron app with no CLI binary. Same pattern as Cursor, Cline, and Continue — proxy-only, print instructions. - **Upstream auto-detection:** Reads `~/.zcode/v2/config.json`, finds the enabled provider, and passes its `baseURL` to the proxy. Falls back to Z.ai Anthropic endpoint if no config found. - **httpProxy investigation:** ZCode has an `httpProxy` setting in `~/.zcode/v2/setting.json`, but it is an Electron-level forward proxy (CONNECT tunneling), incompatible with headroom reverse proxy. The Base URL approach in Model Settings is the correct integration point. - **No dependencies added:** This PR adds zero new dependencies. --------- Co-authored-by: Epicism <epicism@Epiphanie.local> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: Tejas Chopra <chopratejas@gmail.com> |
||
|
ninosat00
|
5709291914
|
chore(release): harden local artifact smokes (#1824)
## Description
Harden the release artifact workflow so maintainers can reproduce npm
and Python release checks locally and so OpenClaw packaging does not
depend on a not-yet-published SDK version. This follow-up also keeps the
OpenClaw loader export contract explicit and updates the PR after the
branch was merged with current `headroomlabs/main`.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [x] Documentation update
## Changes Made
- Added reusable local release smoke scripts for npm assets, Python
wheel/sdist artifacts, and the combined release gate.
- Switched the release workflow npm asset build to the reusable npm
builder.
- Made OpenClaw release asset building install against the just-built
local SDK tarball before rewriting packed metadata to the release
dependency range.
- Kept the OpenClaw source dependency registry-installable at
`headroom-ai@^0.22.3` while the packed artifact still ships
`^<release-version>`.
- Added `registerHeadroomPlugin` as a named export while preserving the
default `{ register }` OpenClaw loader contract.
- Added Windows development bootstrap docs/script and regression tests
for version sync, npm asset ordering, OpenClaw source installability,
and Python wheel smoke import isolation.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
uv run --with pytest python -m pytest tests/test_release_workflows.py scripts/tests/test_version_sync.py -q
44 passed, 1 warning
node --check scripts/build_npm_release_assets.mjs
node --check scripts/verify_npm_release_assets.mjs
python -m py_compile scripts/build_python_release_smoke.py scripts/release_smoke_all.py scripts/version-sync.py
python scripts/verify-versions.py
All versions aligned at 0.31.0
npm ci (plugins/openclaw)
added 88 packages, audited 89 packages, found 0 vulnerabilities
python scripts/release_smoke_all.py --out release-assets-local/all-pr1824-postmerge-fixed-20260710-104739
Verified npm release assets for 0.31.0
wheel metadata OK: headroom_ai-0.31.0-cp310-abi3-win_amd64.whl contains headroom/_core.pyd
sdist License-File metadata OK: ['LICENSE', 'NOTICE']
smoke-import OK: version=0.31.0 hello=headroom-core
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.14.3, Node v24.14.0, npm 11.9.0, uv
0.11.15, Rust/Cargo already installed in the local development
environment.
- Exact command / steps: Ran `python scripts/release_smoke_all.py --out
release-assets-local/all-pr1824-postmerge-fixed-20260710-104739` after
syncing this branch with current `headroomlabs/main`.
- Observed result: The command built `headroom-ai-0.31.0.tgz`,
`headroom-openclaw-0.31.0.tgz`,
`headroom_ai-0.31.0-cp310-abi3-win_amd64.whl`, and
`headroom_ai-0.31.0.tar.gz`; npm verifier passed; the wheel installed
into a fresh venv and imported `headroom._core`.
- Not tested: Full GitHub Actions release matrix and publish jobs with
real PyPI/npm/GitHub Packages credentials.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A.
## Additional Notes
The post-merge full smoke initially failed because the OpenClaw source
package depended on `headroom-ai@^0.31.0`, which is not available on
public npm yet. The builder now installs OpenClaw against the just-built
local SDK tarball before build/pack, and then rewrites packed metadata
to `^0.31.0`.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Connor Campbell
|
021a762bf8
|
feat(compress): expose frozen_message_count in library-mode compress() (#2178)
## Description `read_lifecycle.apply()` already supports a frozen message prefix (`frozen_message_count`) — stale-Read replacements inside the prefix are skipped so compression never rewrites messages the provider's prompt cache has anchored. But only the proxy handlers can pass it: `ContentRouter` reads it from transform kwargs, `CompressConfig` has no such field, and the public `compress()` never forwards it. Library-mode callers that manage their own conversation loop (SDK integrations, offline evaluation, sidecar scoring) therefore can't stop transforms from rewriting already-sent history. On cached Anthropic traffic that's expensive: every byte after the first rewritten one stops billing as a 0.1× cache read and re-bills as a cache write (1.25× at the 5-minute TTL, 2× at the 1-hour TTL) — measured on live coding-agent traffic, retroactive stale-Read rewrites were the dominant cache-bust source once tool injection went session-sticky (PR-B7). Relates to #809 (cache-bust economics discussion); does not close it. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `CompressConfig.frozen_message_count: int = 0` — documented field; default `0` preserves existing behavior exactly. - `compress()` forwards it through `pipeline.apply()` to the transforms, matching what the proxy handlers already do. - `compress()` docstring: added to the kwargs shorthand list. - CHANGELOG entry under Unreleased → Features. - Four tests in `tests/test_compress_api.py` (`TestFrozenMessageCount`). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_compress_api.py tests/test_transforms/test_read_lifecycle.py \ tests/test_compression_safety_rails.py tests/test_compress_failure.py -q 59 passed, 1 warning in 3.05s $ uv run ruff check headroom/compress.py tests/test_compress_api.py All checks passed! $ uv run mypy headroom Success: no issues found in 471 source files ``` ## Real Behavior Proof - Environment: Linux, Python 3.12.3, this branch installed via `uv sync --extra dev` - Exact command / steps: build an Anthropic-format conversation with a stale Read (file read at message 2, edited at message 3), then: ```python r0 = compress(msgs, model="claude-sonnet-4-5-20250929") r5 = compress(msgs, model="claude-sonnet-4-5-20250929", frozen_message_count=5) ``` - Observed result: without frozen prefix the stale Read is rewritten; with frozen_message_count=5 the Read remains byte-identical. ```text without frozen prefix: stale Read rewritten: True transforms: ['read_lifecycle:stale:/app/config.py'] with frozen_message_count=5: Read byte-identical: True transforms: [] ``` - Not tested: proxy-mode code paths (untouched — they already pass `frozen_message_count` their own way); Rust crates (untouched). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — library API change, no UI. ## Additional Notes Default `0` makes this a strict superset of current behavior — no caller sees any change without opting in. The motivation data comes from a proxy-side measurement tool that prices compression's cache effects on live Anthropic agent traffic (per-request cache-adjusted dollars); happy to share methodology in #809 if useful. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: Tejas Chopra <chopratejas@gmail.com> |
||
|
Noam Asor
|
5dbe3314a1
|
fix(auth): support GitHub Enterprise Copilot OAuth domain (#2192)
## Description Adds GitHub Enterprise OAuth domain support for Copilot auth. When `GITHUB_COPILOT_ENTERPRISE_URL` is set, the default OAuth domain resolves to that enterprise host; explicit `--domain` values still take precedence. Closes #1152 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/copilot_auth.py`: derive the default OAuth domain from `GITHUB_COPILOT_ENTERPRISE_URL` when present, falling back to `github.com` for unset or blank values. - `headroom/cli/copilot_auth.py`: keep explicit CLI domain overrides authoritative even when the enterprise env var is set. - `README.md`: document the enterprise OAuth environment setting and precedence. - Added regression tests for enterprise URL handling, blank/unset fallback, and explicit CLI override precedence. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run pytest tests/test_copilot_auth.py tests/test_cli/test_copilot_auth.py -q 69 passed in 1.05s uvx ruff@0.15.17 check headroom/cli/copilot_auth.py headroom/copilot_auth.py tests/test_cli/test_copilot_auth.py tests/test_copilot_auth.py All checks passed! uvx ruff@0.15.17 format --check headroom/cli/copilot_auth.py headroom/copilot_auth.py tests/test_cli/test_copilot_auth.py tests/test_copilot_auth.py 4 files already formatted ``` ## Real Behavior Proof - Environment: Windows 11 development checkout, Python 3.13.3, with focused Copilot auth tests using monkeypatched enterprise env vars. - Exact command / steps: ran the Copilot auth unit/CLI tests plus ruff check and format-check against the touched auth files and tests. - Observed result: `GITHUB_COPILOT_ENTERPRISE_URL=https://ghe.example.com` resolves `default_oauth_domain()` to `ghe.example.com`; unset/blank enterprise env vars fall back to `github.com`; and `headroom copilot-auth login --domain github.com` still honors the explicit override when enterprise env vars are set. - Not tested: live OAuth against a real GitHub Enterprise Server instance. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A - CLI/auth behavior and README update. ## Additional Notes `GITHUB_COPILOT_ENTERPRISE_URL` takes precedence over `GITHUB_COPILOT_ENTERPRISE_DOMAIN`; explicit `--domain` remains authoritative for the login command. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: Tejas Chopra <chopratejas@gmail.com> |
||
|
Waldo Zins
|
a61f534426
|
fix(ccr): store pre-protection original, not tag placeholder, in CCR (#1208)
## Description
When `ContentRouter` protects custom tags (e.g. `<system-reminder>`)
into `{{HEADROOM_TAG_N}}` placeholders before invoking Kompress, CCR can
persist the protected **placeholder intermediate** as the entry's
`original_content` instead of the pre-protection source text. A later
**full retrieve** (or proactive expansion / model-initiated retrieve) of
such an entry then returns `{{HEADROOM_TAG_0}}` and the real protected
block is lost from the retrieval path. The immediate upstream request is
unaffected — `restore_tags` correctly restores the compressed output
before it goes upstream; the confirmed corruption is in CCR storage and
only surfaces on later retrieval/expansion.
This threads the pre-protection `content` through as `ccr_original` so
CCR stores the real source text while the model still sees the
placeholdered text.
Closes #1209
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/content_router.py`: `_try_ml_compressor` passes
`ccr_original=content` to `compressor.compress(...)` **only when tags
were actually protected** (untagged callers keep the historic call shape
— backward compatible).
- `headroom/transforms/kompress_compressor.py`: `compress()` gains a
`ccr_original` kwarg; `compress_batch()` gains a per-item
`ccr_originals` list (validated against `len(contents)`).
- All four CCR store sites store `ccr_original` when present, else
`content`: inline `compress()`, single-content
`compress()`→`compress_batch` delegation, `compress_batch` sequential
fallback, and `compress_batch` batched/GPU path. The stored original's
token count is recomputed from the stored text.
- `tests/test_ccr_tag_placeholder_regression.py` (new, 5 tests): router
boundary forwarding, untagged backward-compat, `ccr_originals` length
validation, and two store-site tests driving the real `compress()` /
batched `compress_batch()` all the way to `_store_in_ccr` (a tiny fake
model stands in for the 274MB ModernBERT).
## Testing
<!-- Check what you actually ran, then paste the real command output
below. -->
- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ .venv/bin/python -m pytest tests/test_ccr_tag_placeholder_regression.py -q
============================= test session starts ==============================
platform darwin -- Python 3.12.12, pytest-9.1.1, pluggy-1.6.0
rootdir: /Users/.../headroom.worktrees/ccr-tag-placeholder
configfile: pyproject.toml
plugins: anyio-4.14.0
collected 5 items
tests/test_ccr_tag_placeholder_regression.py ..... [100%]
========================= 5 passed, 1 warning in 0.15s =========================
```
Fail-before / pass-after was confirmed against a freshly built Rust
`_core`: with the fix reverted the new tests fail (router forwards no
`ccr_original` → `None`/placeholder reaches the store; `compress_batch`
rejects the unknown `ccr_originals` kwarg with `TypeError`); with the
fix applied all 5 pass. The surrounding kompress/ccr/router suites stay
green (8 unrelated failures are pre-existing — identical with the patch
stashed — from missing optional test deps such as `pytest-asyncio`, not
caused by this change).
## Real Behavior Proof
- Environment: macOS (darwin), Python 3.12.12, locally built Rust
`_core` via `maturin develop`, pytest 9.1.1.
- Exact command / steps: `maturin develop` to build `_core`, then
`python -m pytest tests/test_ccr_tag_placeholder_regression.py -q`.
- Observed result: 5 passed with the fix applied; the same suite fails
before the fix (placeholder/`None` reaches `_store_in_ccr`;
`compress_batch` rejects `ccr_originals`).
- Not tested: end-to-end live proxy full-retrieve against a 274MB
ModernBERT model (tests use a fake model to keep them deterministic and
offline); `ruff`/`mypy` not run locally.
> Note: this fixes new CCR writes. Pre-existing entries written before
the fix keep their placeholder `original_content` until they expire.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
Docs/CHANGELOG unchanged: this is an internal CCR correctness fix with
no public API or user-facing behavior change beyond correct
full-retrieve content. `ruff`/`mypy` were not run in the local build
environment.
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Rocker Zhang
|
a069979466
|
fix(content_router): pin FREEZE_BLOCK_DECISION verdict to stop cache-write churn (#1620)
## Description The per-block freeze decision was inert. On the cached-block re-check, a later tighter `min_ratio` (context pressure rises within a session) could downgrade an earlier "compress" verdict to skip, restore the original block, and bust the prefix cache — a self-inflicted cache-write churn that costs the very tokens compression saved. This pins the decision instead. A frozen "compress" verdict re-accepts (`accept_threshold = 1.0`) rather than re-running the per-turn `min_ratio` gate, so a block that was accepted stays accepted. First-sighting still uses the live `min_ratio` gate (`accept_threshold = min_ratio`): the freeze only pins past accepts, it never loosens the first decision (that would be a silent ratio bet). Gated behind `HEADROOM_FREEZE_BLOCK_DECISION`, default off, byte-identical to today when unset. Composes with the #1307 reversibility guard on the compress path: a frozen accept still defers to the lossy-unrecoverable skip. Closes #1619 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `content_router.py`: on the cached-block hit path and the first-sighting path, compute `accept_threshold` (1.0 when a "compress" verdict is frozen for the block, else the live `min_ratio`) and gate accept on it; record the pin when the legacy re-check would have downgraded. - Frozen verdicts are stored per content-block key and only ever hold "compress" (a "skip" never warms the result cache). - No change when `HEADROOM_FREEZE_BLOCK_DECISION` is unset. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom` — no new errors) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_transforms_content_router.py -q 38 passed in 0.23s # 11 freeze/pin/churn cases + 27 existing; run against a real 0.28.0 _core. # - test_freeze_off_is_byte_identical_flapping_baseline: freeze-off == baseline byte-for-byte # - test_freeze_on_pins_compress_verdict_across_turns: pin fires; downgrade prevented $ ruff format --check . && ruff check . -> clean ``` ## Real Behavior Proof - Environment: isolated git worktree on latest `main`, real `_core.abi3.so` built for this tree via maturin (not a stale symlink), scratch venv. - Exact command: `pytest tests/test_transforms_content_router.py -q` - Observed: freeze-off path is byte-identical to the flapping baseline; with freeze on, the verdict is pinned across turns (pin-count assertion passes) so the block is not downgraded/restored and no cache-write churn occurs. - Not tested: end-to-end proxy A/B token-savings delta (follow-up; feature ships default-off). ## Screenshots (if applicable) N/A ## Review Readiness Ready for review. Default-off, composes with #1307, self-contained to the block-decision path. ## Checklist - [x] My code follows the style guidelines of this project - [x] I have performed a self-review of my code - [x] I have commented my code in hard-to-understand areas - [x] Documentation intentionally deferred until the default-off approach is confirmed - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] CHANGELOG intentionally deferred until the default-off approach is confirmed ## Additional Notes Neighbour of #625 (prefix-stability). Docs/CHANGELOG intentionally deferred until the approach is confirmed. The end-to-end A/B is a follow-up; the churn-prevention is proven at the router unit level here. Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Ingmar Krusch
|
896454e978
|
feat(install): add apply flag parity, --env passthrough, and EIO retry (#2152)
## Description Three related gaps in `headroom install apply` and its supervisor lifecycle, found operating a real persistent deployment on this fork: 1. `install apply` only exposed a fixed subset of `headroom proxy`'s flags (`--backend`, `--region`, `--mode`, `--port`, `--memory`, `--telemetry`, `--no-http2`). Deployments that need code-aware compression, tool-result interception, per-tool lossy-compression protection, or a named AWS profile for Bedrock had no native way to configure them through `install apply` — the generated `manifest.json` would have to be hand-edited after the fact, which silently reverts on the next `install apply` and isn't tracked anywhere. 2. Supervised runners (macOS launchd, Linux systemd/cron, Windows services/tasks) all start their runner scripts with a bare environment and do not inherit the interactive shell's exports. In particular, a custom `HEADROOM_WORKSPACE_DIR` never reached the supervised process, so `headroom install agent run` looked for its manifest in the wrong location and failed outright with "No deployment profile named 'default' is installed" even though `install apply` itself had succeeded moments earlier. 3. `install_supervisor`'s macOS branch does an unconditional `launchctl bootout` followed by a bare `bootstrap` with no retry, unlike `start_supervisor` (already fixed by #1290), which rides out the ~15s EIO (error 5) window launchd exhibits for several seconds after a bootout. This left `install apply`'s own reinstall path exposed to the same race #1290 fixed elsewhere — requiring the exact manual recovery (bootout + remove the plist + reapply) #1290 was meant to eliminate. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cli/install.py`: `install apply` gains `--code-aware/--no-code-aware`, `--intercept-tool-results`, `--protect-tool-results <tool1,tool2>`, and `--bedrock-profile <profile>`, mirroring the equivalent flags already on `headroom proxy` (same names, same help text style). Also gains `--env KEY=VALUE` (repeatable). - `headroom/install/planner.py`: `build_manifest()` threads all five new parameters into `proxy_args`/`base_env`, following the exact pattern already used for `--region`/`--no-http2`. `--env` entries are merged into `base_env` last, so they can override auto-derived defaults. - `headroom/install/supervisors.py`: - `_render_unix_runner`/`_render_windows_runner` emit `export`/`$env:` lines for `base_env` before the `exec`, so `run-headroom.sh`/`ensure-headroom.sh` (and Windows equivalents) carry the environment forward to both the outer `install agent run` process and the proxy subprocess it spawns. The Docker runtime path already threaded `base_env` into `docker run --env`; this closes the same gap for the process-based runtime. - New `_bootstrap_with_retry()` helper extracted from `start_supervisor`'s existing retry loop (from #1290), now shared by both `start_supervisor` and `install_supervisor`. - `tests/test_install/test_planner.py`: new tests for all five flags (default-omitted and persisted cases), following the existing `--no-http2` test pattern. - `tests/test_install/test_supervisors.py`: new tests for `--env` propagation into rendered runner scripts, and for `install_supervisor`'s retry-until-success and raise-after-exhausted-retries paths (mirroring the existing `start_supervisor` coverage). Also fixes a pre-existing test's mock that returned `None` from a `subprocess.run` stub — this only worked before because the old bare `bootstrap` call site never inspected the return value; the new `_bootstrap_with_retry()` call does. - `CHANGELOG.md`: added `### Features` and `### Fixed` entries under `Unreleased`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_install/ tests/test_cli/test_wrap_persistent.py tests/test_cli/test_init_cli.py -q ============================= test session starts ============================== platform darwin -- Python 3.13.13, pytest-9.0.3, pluggy-1.6.0 collected 215 items tests/test_install/test_health.py ... [ 1%] tests/test_install/test_native_installers.py ss [ 2%] tests/test_install/test_paths.py ... [ 3%] tests/test_install/test_planner.py .................. [ 12%] tests/test_install/test_providers.py ................................... [ 28%] ...... [ 31%] tests/test_install/test_runtime.py .................... [ 40%] tests/test_install/test_state.py ..... [ 42%] tests/test_install/test_supervisors.py ......................... [ 54%] tests/test_cli/test_wrap_persistent.py ............................ [ 67%] tests/test_cli/test_init_cli.py ........................................ [ 86%] .............................. [100%] ======================== 213 passed, 2 skipped in 0.57s ======================== $ uv run ruff check headroom/cli/install.py headroom/install/planner.py headroom/install/supervisors.py tests/test_install/ All checks passed! $ uv run mypy headroom/cli/install.py headroom/install/planner.py headroom/install/supervisors.py Success: no issues found in 3 source files ``` ## Real Behavior Proof - Environment: personal fork deployed as a real proxy (macOS launchd service via `headroom install apply`), profile `default`, backend `bedrock` with a named AWS SSO profile. - Exact command / steps: (flags 1 & 2) ran `headroom install apply --backend bedrock --mode token --code-aware --protect-tool-results Bash --bedrock-profile sso-bedrock --env HEADROOM_WORKSPACE_DIR=/Users/<redacted>/.headroom-workspace --env AWS_PROFILE=sso-bedrock --env AWS_REGION=eu-west-1`, then inspected the generated `manifest.json`, the rendered `run-headroom.sh`, and the running launchd job. - Observed result: before this PR, none of `--code-aware`, `--protect-tool-results`, `--bedrock-profile`, or `--env` were accepted flags on `install apply` at all (`Error: No such option`). Reproduced the `--env` gap specifically by running the exact command a launchd job invokes with a stripped environment (no `HEADROOM_WORKSPACE_DIR`, no `AWS_PROFILE`) — it failed to find the manifest; with the interactive shell's env forwarded manually, it started fine. The generated plist had no `EnvironmentVariables` key and `run-headroom.sh` was a bare `exec`, confirming this wasn't a config mistake but a real gap between `install apply`'s flag surface and what a supervisor actually runs with. After this PR, `install apply` with all the flags above produces a launchd job that starts clean, reports healthy, and successfully proxies a real request to Bedrock (200, not just a green health check) using the named AWS profile with no `AWS_PROFILE` env var needed elsewhere. - Exact command / steps: (EIO retry, flag 3) triggered the same EIO race #1290 documents by running `headroom install apply` twice in quick succession against the same profile (the second run's `install_supervisor` bootout+bootstrap lands inside the first run's launchd settle window). - Observed result: before this PR, the second `install apply` occasionally failed outright with `CalledProcessError` from the bare `subprocess.run(..., check=True)` bootstrap call, requiring the manual bootout+`rm` plist+reapply recovery. After this PR (with `_bootstrap_with_retry` in place), the same back-to-back sequence completes successfully every time observed, riding out the EIO window instead of failing. - Not tested: Linux systemd/cron and Windows service/task supervisor paths for the `--env` propagation — verified via the new unit tests (which cover the runner-script rendering directly) but not against a live Linux or Windows machine, since this deployment is macOS-only. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — CLI/install logic change, no UI surface. ## Additional Notes - "I have made corresponding changes to the documentation" is unchecked: no file in `docs/`, `README.md`, or `CONTRIBUTING.md` documents `install apply`'s flag surface in detail (it's discoverable via `--help`), so there is no existing section to update for the new flags. - Re-derivation note: this PR's `install_supervisor` EIO-retry fix and its `_bootstrap_with_retry` extraction are written directly against current `upstream/main`'s post-#1290 shape of `start_supervisor` (inline retry loop with `_MACOS_BOOTSTRAP_RETRIES`/`_MACOS_BOOTSTRAP_RETRY_DELAY`), not cherry-picked from an older fork commit that predated #1290 — the diff here is intentionally different from what a naive cherry-pick would have produced. - No linked issue number: found via operating a real persistent deployment on a personal fork, not filed as a `headroomlabs-ai/headroom` issue first. Checked `gh pr list --search` for "install apply flags/env" and "bootstrap EIO"/"install_supervisor bootstrap retry" — no open or merged coverage found beyond #1290 (which fixes `start_supervisor` only, a different call site from the one this PR fixes). Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Eyal Mizrachi
|
14011b42dd
|
fix(wrap): drop -p short flag from wrap claude so claude's own -p/--print passes through (#2048)
## Description `headroom wrap claude` declares `--port/-p`, and click parses wrapper options anywhere in the argv before unknown options fall through to `CLAUDE_ARGS`. So a user running claude's headless print mode through the wrapper — `headroom wrap claude -p "some prompt"` — fails with `Invalid value for '--port' / '-p': 'some prompt' is not a valid integer range`, and claude's own `-p`/`--print` can never reach claude. This bites hardest when `claude` is shell-aliased to `headroom wrap claude ...`: every `claude -p` invocation breaks. This PR drops the `-p` short alias from `wrap claude`'s `--port` option (long form stays; other subcommands' `-p` are untouched), so `-p` now falls through to `CLAUDE_ARGS` like any other claude flag. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cli/wrap.py`: removed `"-p"` from the `wrap claude` command's `--port` option; added a comment stating why the short alias must not exist there. ## Testing - [x] Unit tests pass (`pytest`) — targeted CLI suites, see output - [x] Linting passes (`ruff check .`) — on the touched file - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_cli/test_wrap_codex.py tests/test_cli/test_wrap_claude_base_url.py tests/test_cli/test_unwrap_claude.py -q 125 passed in 6.68s $ ruff check headroom/cli/wrap.py All checks passed! Full tests/test_cli run: 549 passed, 2 failed — test_wrap_copilot_auto_detects_running_proxy_backend fails identically on a clean upstream/main checkout (pre-existing, environment-sensitive), and test_wrap_codex_prepare_only_registers_serena_when_uvx_exists passes in isolation on this branch (full-suite ordering interaction, not this change). ``` ## Real Behavior Proof - Environment: Fedora 44, Python 3.14 editable install, `claude` aliased to `systemd-run --user --scope ... headroom wrap claude --no-context-tool` via terminal shell integration - Exact command / steps: `claude --model sonnet -p "Say only: ALIAS-P-FIXED"` in a fresh interactive shell (alias → wrapper → proxy → claude) - Observed result: before the fix — `Error: Invalid value for '--port' / '-p': ... is not a valid integer range` (exit 2, claude never spawns). After — headroom banner, proxy attach, claude prints `ALIAS-P-FIXED`, exit 0; `Extra args: --model sonnet -p Say only: ALIAS-P-FIXED` shows the passthrough. - Not tested: Windows; other wrapped tools' `-p` flags (left untouched by design); mypy (not run) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — CLI flag parsing. ## Additional Notes Docs/CHANGELOG: no user-facing docs mention `-p` as a `wrap claude` port alias, so no doc change; happy to add a CHANGELOG entry if maintainers want one. No new test added because the passthrough behavior is covered by the manual end-to-end proof above; can add a click-runner test asserting `-p` lands in `CLAUDE_ARGS` if preferred. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Carlos Duplar Mello
|
5d17e9addc
|
fix: check feature configuration before reusing persistent deployments (#1330)
## Description
A persistent proxy started for one use case (e.g. `--backend anthropic`)
would be silently reused for another (e.g. `--subscription
--provider-type openai`) causing 401 auth failures because
`_ensure_proxy()` only checked health + version, skipping the feature
configuration check (memory, openai_api_url, learn, code_graph).
Closes #N/A
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Added feature configuration check in the persistent deployment path of
`_ensure_proxy()` in `headroom/cli/wrap.py:1726-1752`. When features
mismatch, the code now falls through to the non-persistent path which
handles proxy restart with upgraded config.
- Added three new tests in `tests/test_cli/test_wrap_persistent.py`:
-
`test_ensure_proxy_restarts_persistent_deployment_for_feature_mismatch`
— verifies proxy restart when openai_api_url differs
- `test_ensure_proxy_restarts_persistent_deployment_for_memory_mismatch`
— verifies proxy restart when memory is requested but not enabled
- `test_ensure_proxy_reuses_persistent_deployment_when_features_match` —
verifies proxy reuse when all features match
- Updated `CHANGELOG.md` with fix description
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ ruff check headroom/cli/wrap.py tests/test_cli/test_wrap_persistent.py
All checks passed!
$ ruff format --check headroom/cli/wrap.py tests/test_cli/test_wrap_persistent.py
2 files already formatted
$ python -c "
from tests.test_cli.test_wrap_persistent import *
import pytest
test_ensure_proxy_restarts_persistent_deployment_for_feature_mismatch(pytest.MonkeyPatch())
print('Test 1 passed: feature mismatch restarts proxy')
test_ensure_proxy_restarts_persistent_deployment_for_memory_mismatch(pytest.MonkeyPatch())
print('Test 2 passed: memory mismatch restarts proxy')
test_ensure_proxy_reuses_persistent_deployment_when_features_match(pytest.MonkeyPatch())
print('Test 3 passed: matching features reuse proxy')
print('All tests passed!')
"
Test 1 passed: feature mismatch restarts proxy
Test 2 passed: memory mismatch restarts proxy
Test 3 passed: matching features reuse proxy
All tests passed!
$ .venv/bin/mypy headroom
headroom/proxy/server.py:1230: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked]
headroom/proxy/server.py:1301: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked]
headroom/proxy/server.py:1305: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked]
Success: no issues found in 397 source files
```
## Real Behavior Proof
- Environment: Linux (WSL2) Ubuntu 24.04, Python 3.12, persistent
deployment via `headroom install agent run --profile default` with
`--backend anthropic`
- Exact command / steps:
1. Started persistent deployment: `headroom install agent run --profile
default`
2. Ran copilot wrapper: `headroom wrap copilot --subscription
--provider-type openai --wire-api responses --memory -- --model
gpt-5.4-mini`
- Observed result:
- Before fix: Proxy forwarded to `api.openai.com` instead of
`api.githubcopilot.com`, causing 401 auth errors
- After fix: Proxy correctly forwards to `api.githubcopilot.com` and
copilot works as expected
- Not tested: macOS, Windows, enterprise/data-residency accounts, other
wrapped tools (claude, codex, aider)
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A
## Additional Notes
The fix is minimal and targeted. It only adds a feature configuration
check in the persistent deployment path without changing any other
behavior. The non-persistent proxy path already had this check; this PR
brings the same logic to persistent deployments.
Co-authored-by: carlosduplar <[email protected]>
|
||
|
Shubham Srivastava
|
f9f3162d38
|
docs(proxy): document HEADROOM_SAVINGS_PROFILE and correct --mode default (#2031) (#2040)
## Description
`HEADROOM_SAVINGS_PROFILE` is an implemented env var
(`headroom/agent_savings.py`) that selects a named profile bundling
Headroom's whole compression posture (proxy mode, keep-ratio, which
messages are compressed, `force_kompress`, etc.) at proxy startup. It
was entirely undocumented — `grep` over `docs/` found zero mentions.
Related, the proxy docs were **misleading about the default optimization
mode**: `docs/content/docs/proxy.mdx` stated `--mode` defaults to
`token`, but the code default is `cache`:
```python
# headroom/cli/proxy.py — the Click option has no default
@click.option("--mode", default=None, ...)
# ... mode resolution (default is CACHE):
effective_mode = normalize_proxy_mode(mode or os.environ.get("HEADROOM_MODE") or PROXY_MODE_CACHE)
```
A bare `headroom proxy` (no `--mode`, no `HEADROOM_MODE`) runs in
**cache** mode, and the default `coding` savings profile also sets
`proxy_mode="cache"` — which is exactly what the issue reporter found
confusing.
This documents `HEADROOM_SAVINGS_PROFILE` and corrects the `--mode`
default rows so the doc is accurate and internally consistent.
Closes #2031
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
`docs/content/docs/proxy.mdx` only:
- Corrected the `--mode` default in the Core-options table and the
Context-management table (`token` → `cache`), each pointing to the new
Savings profiles section for the reason.
- Added a `### Savings profiles` section documenting: the
`HEADROOM_SAVINGS_PROFILE` env var; a table of the four built-in
profiles (`coding` default, `balanced` fallback, `agent-90`, `general`)
with target savings, mode, and `force_kompress`; the unset→`coding`
default; the unknown-value→`balanced` warning-and-fallback (proxy never
fails to start); and the mode precedence (explicit `--mode` >
`HEADROOM_MODE` seeded by a profile > `cache` default), with an example.
No code change. Every documented value is pinned to
`headroom/agent_savings.py` (profile definitions) and
`headroom/cli/proxy.py` (default-mode resolution).
## Testing
- [x] Unit tests not run; docs-only source verification performed
- [x] Linting not run; docs-only MDX/source verification performed
- [x] Type checking not applicable; no Python code changed
- [x] New tests not applicable; documentation-only correction
- [x] Manual testing performed
### Test Output
Docs-only change; verification is cross-checking every documented value
against the source of truth:
```text
$ grep -n "DEFAULT_PROFILE = \|FALLBACK_PROFILE = " headroom/agent_savings.py
14:FALLBACK_PROFILE = "balanced"
18:DEFAULT_PROFILE = "coding"
# profile modes / knobs (agent_savings.py):
# coding → proxy_mode="cache", force_kompress=False, target_ratio=None (emergent)
# balanced → proxy_mode="token", force_kompress=False, target_ratio=0.30
# agent-90 → proxy_mode="token", force_kompress=True, target_ratio=0.10
# general → proxy_mode="token", force_kompress=False, target_ratio=None (emergent)
$ grep -n "effective_mode\|PROXY_MODE_CACHE" headroom/cli/proxy.py
# effective_mode = normalize_proxy_mode(mode or os.environ.get("HEADROOM_MODE") or PROXY_MODE_CACHE)
# → confirms the real default optimization mode is cache, not token
```
MDX sanity: code fences balance (even count) and the `### Savings
profiles` heading slugifies to `#savings-profiles`, matching the two
in-page anchor links added to the mode rows.
## Real Behavior Proof
- **Environment:** Windows 11; docs source inspected against the working
tree at the current `main` base.
- **Exact command / steps:** Each documented fact is grounded in code —
profile names, modes, `force_kompress`, and target ratios come from
`headroom/agent_savings.py:_PROFILES`; the default profile (`coding`)
from the `os.environ.get("HEADROOM_SAVINGS_PROFILE") or "coding"` reads
in `headroom/cli/proxy.py` and `headroom/proxy/server.py`; the `cache`
default mode from `headroom/cli/proxy.py`'s `mode or HEADROOM_MODE or
PROXY_MODE_CACHE`; the unknown-value fallback from
`get_agent_savings_profile` (`agent_savings.py`).
- **Observed result:** The new section's table and prose match those
sources exactly, and the previously-wrong `--mode` default rows now
state `cache`.
- **Not tested:** A live render of the Fumadocs/Next.js docs site (no
local docs build run here) — the change is MDX-syntax-valid (balanced
fences, well-formed table, standard heading-anchor slug).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] Code comments not applicable; documentation-only change
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] Tests not applicable; docs-only facts verified against source
- [x] New and existing unit tests pass locally with my changes
- [x] CHANGELOG not applicable; documentation-only correction
## Screenshots (if applicable)
N/A (docs prose/table addition; a rendered screenshot can be added if
the docs site is built for preview).
## Additional Notes
- Test/tests-added checklist items are N/A — this is a
documentation-only change.
- Out of scope (intentionally): the `--mode` Click **help text** in
`headroom/cli/proxy.py` also says "default: token" and is likewise
inaccurate, but correcting Python help text is a code change beyond this
docs issue — noted as a possible follow-up.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
Ashish Patel
|
7ab83c5107
|
fix(router): stop protecting passing build/test output as error traces (#1740)
## Description  `content_has_strong_error_indicators()` (`headroom/transforms/error_detection.py`) protects any message/content-block from compression when it contains 2+ distinct indicator keywords (`error`, `fail`, `exception`, `traceback`, `fatal`, `panic`, `crash`). That heuristic false-positives on **passing** build/test tool output: `tsc`'s `"Found 0 errors"` plus a passing test run's `"0 failures"` trips both `error` and `fail` — 2 distinct hits — despite nothing failing. In a long JS/TS coding session this fired on nearly every request (confirmed against the `stats.json` attached to #1696), permanently protecting legitimate tool output from ever being compressed and explaining the reported 0.3% savings vs. the advertised 60-95%. Closes #1696 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/error_detection.py`: strip common zero-result phrases before the 2-keyword scan in `content_has_strong_error_indicators()` — both `"N word"`/`"word N"` forms (`"0 errors"`, `"no failures"`) and `"label:value"`/`"label=value"` forms (`"Failures: 0"`, `"errors=0"`), covering `error(s)` and `fail`/`failed`/`failing`/`failure(s)` (the scan matches `fail` by substring, so all inflections needed covering). - `tests/test_error_detection.py` (new file — no prior coverage existed): 7 tests covering real error/traceback detection, single-keyword safety, `tsc`/`eslint` passing summaries, the `"0 failed"` regression a reviewer caught, label:value formats, and that a genuine second indicator elsewhere in the same blob still triggers protection. - `.github/pr-images/issue-1696-error-protection-fix.svg`: diagram explaining the mechanism (embedded above). ## Testing - [x] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) — not run locally, CI `lint` check is green - [ ] Type checking passes (`mypy headroom`) — not run locally, CI is green - [x] New tests added for new functionality - [x] Manual testing performed (see Real Behavior Proof) ### Test Output ```text $ .venv/Scripts/python -m pytest tests/test_error_detection.py tests/test_transforms/test_content_router.py tests/test_transforms_content_router.py -q tests\test_error_detection.py ....... [ 7%] tests\test_transforms\test_content_router.py ........................... [ 34%] ........................... [ 62%] tests\test_transforms_content_router.py ................................ [ 94%] ..... [100%] ============================= 98 passed in 1.21s ============================== ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.14.5, local venv, `headroom._core` built via `maturin develop --release` (not prebuilt in a fresh checkout) - Exact command / steps: ran the reporter's exact scenario patterns (`"Found 0 errors\nTests: 0 failures, 42 passed"`, eslint's `"0 problems (0 errors, 0 warnings)"`) through `content_has_strong_error_indicators()` directly, before and after the fix - Observed result: before → `True` (wrongly protected); after → `False` (correctly compressible). Real failure text (`Traceback... fatal error`) still returns `True` after the fix. - Not tested: have not reproduced the full KiloCode/proxy session end-to-end locally (no access to the reporter's actual traffic) — root cause was confirmed via the `stats.json` they attached to the issue, which shows `router:protected:error_output` firing on nearly every request in their session. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (N/A — internal heuristic, no user-facing docs reference it) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable (release-please generates this automatically from commit messages) ## Screenshots (if applicable) See the diagram embedded in Description above. ## Additional Notes **Investigation trail**: ruled out `protect_recent_reads_fraction` (proxy already overrides its `0.0` dataclass default to `0.3` in token mode, the default) before confirming the error-protection false-positive via the reporter's `stats.json`. **Response to review comments** (@AbelVM, @sparkbugz): prose mentions like `"Fix the errors in the code."` or `"console.error(...)"` contain only 1 distinct indicator keyword and were already safe under the pre-existing 2-keyword threshold — not something this PR changes. The broader concern about other CI summary formats is addressed above (label:value forms). A case like genuine prose that happens to mention *two* distinct keywords together (e.g. "there are errors and it failed") is a known limitation of a keyword-substring heuristic in general, predates this PR, and is out of scope here — downstream compressors (LogCompressor) still preserve real error lines even when a block isn't gate-protected, so the failure mode there is "slightly stricter than ideal," not data loss. **Response to @JerrettDavis's CHANGES_REQUESTED**: fixed in the follow-up commit — `"0 failed"` is now stripped (previously only `failing`/`failure(s)` were), with a regression test for the exact reproduction given. |
||
|
Rod Boev
|
4ea96a417c
|
feat(mcp): add streamable HTTP MCP transport (#1773)
## Description `headroom mcp serve` only exposed stdio, which blocked MCP clients that require a Streamable HTTP endpoint. This PR adds an explicit HTTP transport mode around the existing Headroom MCP server while keeping stdio as the default and keeping tool registration single-sourced. Closes #1346. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `headroom mcp serve --transport http` with host, port, and path options. - Serve `headroom_compress`, `headroom_retrieve`, and `headroom_stats` through the same MCP server instance used by stdio. - Keep `headroom mcp serve` defaulting to stdio for current Claude Code and local MCP host configs. - Update MCP docs for stdio and HTTP setup without implying the proxy automatically owns `/mcp`. - Keep the scope clean, rebased, and covered by focused tests. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_ccr_mcp_http.py tests/test_cli/test_mcp.py -q`) - [x] Linting passes (`uv run ruff check headroom/cli/mcp.py headroom/ccr/mcp_server.py headroom/ccr/mcp_http.py tests/test_ccr_mcp_http.py tests/test_cli/test_mcp.py`) - [x] Type checking passes (`uv run mypy headroom --ignore-missing-imports`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text uv run pytest tests/test_ccr_mcp_http.py tests/test_cli/test_mcp.py -q 20 passed in 0.53s uv run ruff check headroom/cli/mcp.py headroom/ccr/mcp_server.py headroom/ccr/mcp_http.py tests/test_ccr_mcp_http.py tests/test_cli/test_mcp.py All checks passed! uv run ruff format headroom/cli/mcp.py headroom/ccr/mcp_server.py headroom/ccr/mcp_http.py tests/test_ccr_mcp_http.py tests/test_cli/test_mcp.py --check 5 files already formatted uv run mypy headroom --ignore-missing-imports Success: no issues found in 407 source files ``` ## Real Behavior Proof - Environment: Local Python environment with Headroom dev dependencies and MCP extra installed. - Exact command / steps: Start `headroom mcp serve --transport http --host 127.0.0.1 --port <test-port> --path /mcp`, then perform an MCP SDK Streamable HTTP initialize/list-tools exchange. - Observed result: The HTTP transport initializes and lists the existing Headroom MCP tools; `headroom mcp serve` without `--transport` still selects stdio, and mixed-case `--transport HTTP` routes to the HTTP transport. - Not tested: live validation against external MCP hosts ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes `CHANGELOG.md` is not edited because this repository generates changelog entries from conventional commits. Full-suite validation is left to CI. |
||
|
Parideboy
|
c46cd8f950
|
fix(core): load ONNX Runtime dynamically so headroom._core imports on non-AVX2 x86-64 (#1715)
## Description
`import headroom._core` dies with SIGILL (`Illegal instruction`) on
x86-64 CPUs without AVX2 (Pentium N4200, Celeron N4500, AMD FX 8350 —
all reported on the issue). The repo sets no `RUSTFLAGS`/`target-cpu`
anywhere, so first-party Rust code is baseline x86-64; the AVX2 code
comes from Microsoft's prebuilt ONNX Runtime, statically linked into the
extension by fastembed's `ort-download-binaries-rustls-tls` feature on
non-Windows targets. Because it is statically linked, its code is mapped
and initialized when the extension module loads — **before** the runtime
AVX2 guard from #1162 can run, which is why that fix helped Magika init
but not the import-time crash.
Fix, mirroring what Windows already does for its own reasons (DirectML
link libs): build with `ort-load-dynamic` on every platform, so ONNX
Runtime is only `dlopen`'d at first use, where the #1162 AVX2 guard
falls back to the non-ONNX detection tiers on unsupported CPUs. Since
both target blocks became identical, they are collapsed into one
platform-independent `fastembed` dependency.
To keep Magika/fastembed working out of the box on Linux/macOS, the
existing `ORT_DYLIB_PATH` auto-pin (`headroom/_ort.py`, previously
Windows-only) now resolves the pip `onnxruntime` package's shared
library on all platforms (`onnxruntime.dll` / `libonnxruntime.so*` /
`libonnxruntime*.dylib`). The pip `onnxruntime` CPU wheels use runtime
CPU dispatch, so they also work on pre-AVX2 machines — non-AVX2 users
get working ML detection instead of a crash. Without the `onnxruntime`
package, ML detection degrades gracefully to the non-ONNX tiers exactly
as it already does on Windows.
Fixes #1278
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `crates/headroom-core/Cargo.toml`: replaced the per-target `fastembed`
blocks (`ort-download-binaries-rustls-tls` on non-Windows,
`ort-load-dynamic` on Windows) with a single platform-independent
dependency on `ort-load-dynamic`, with a comment documenting both the
DirectML and the AVX2/#1278 rationale.
- `Cargo.lock`: regenerated — `ort-sys` drops its static-download
dependencies (`hmac-sha256`, `lzma-rust2`, `ureq`); no version bumps.
- `headroom/_ort.py`: `ORT_DYLIB_PATH` auto-pin extended from
Windows-only to all platforms via a small `_find_dylib` helper that
resolves the platform's shared-library name inside the pip `onnxruntime`
package.
- `tests/test_transforms/test_ort_dylib.py`: replaced the obsolete
`test_noop_on_non_windows` with Linux (versioned `.so`) and macOS
(`.dylib`) pin tests; module docstring updated.
- `docs/content/docs/configuration.mdx`: `ORT_DYLIB_PATH` row updated
from Windows-only wording to the cross-platform behavior.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ cargo fmt --all -- --check && cargo clippy --workspace -- -D warnings && cargo test -p headroom-core --lib
clean
test result: 844 passed; 0 failed; 1 ignored
$ python -m pytest tests/test_transforms/test_ort_dylib.py -q
8 passed
$ ruff check headroom/_ort.py tests/test_transforms/test_ort_dylib.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11 (AVX2-capable — the SIGILL itself is not
reproducible on this machine), Python 3.13, Rust 1.95.0, local checkout
branched from `upstream/main` (
|
||
|
Ben Younes
|
541500811f
|
feat(cli): add wrap openclaude for OpenClaude CLI (#1416)
## Description
Adds `headroom wrap openclaude`, a Click subcommand that launches the
prose-format OpenClaude CLI through the local Headroom proxy using the
same OpenAI/Anthropic base URL environment shape as `wrap aider`.
Fixes #1411.
## Type of Change
- [x] Bug fix
- [x] New feature
- [ ] Breaking change
- [ ] Documentation update
- [x] Tests
## Changes Made
- Added the `wrap openclaude` command path for OpenClaude CLI launch env
routing.
- Kept `--no-context-tool` / `--no-rtk` support for proxy-only launch
behavior.
- Fixed the default RTK setup path requested in review: when RTK is
selected and installed, `wrap openclaude` now injects the RTK
instruction marker block into `CONVENTIONS.md` at the project root
instead of only downloading the binary.
- Added a regression test for the default RTK path so the PR fails if
OpenClaude stops receiving RTK instructions.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
RED, with the production RTK injection path temporarily reverted while
keeping the new regression test:
```text
.venv/bin/python -m pytest tests/test_cli/test_wrap_openclaude.py::test_wrap_openclaude_default_rtk_injects_instructions -q
FAILED tests/test_cli/test_wrap_openclaude.py::test_wrap_openclaude_default_rtk_injects_instructions
E AssertionError: assert False
E + where False = exists()
E + where exists = PosixPath('/tmp/pytest-of-ousama/pytest-1/test_wrap_openclaude_default_r0/CONVENTIONS.md').exists
```
GREEN, after restoring the fix:
```text
.venv/bin/python -m pytest tests/test_cli/test_wrap_openclaude.py -q
3 passed in 0.46s
```
Additional validation on the pushed commit
`
|
||
|
Rod Boev
|
e9e9cd55b7
|
feat(mcp): publish canonical server.json (#1510)
## Description Headroom can launch its MCP server, but did not publish a canonical `server.json` that registries and MCP hosts can consume directly. This PR adds a shared descriptor builder, commits a root `server.json`, parity-tests that artifact against the builder and existing runtime spec, and updates docs so registry authors do not need to reconstruct `headroom mcp serve` from prose. Closes #929. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added a shared `server_json.py` descriptor builder for Headroom MCP publication metadata. - Published a canonical root `server.json` and parity-tested it against the builder. - Encoded the publishable uvx contract as `headroom-ai[mcp]` plus `headroom mcp serve`. - Updated README and MCP docs to point registry authors at the canonical descriptor. - Added the README ownership marker used by MCP Registry verification. - Kept existing registrars and `headroom mcp install` behavior unchanged. ## Testing - [x] Unit tests pass - [x] Linting passes - [x] Type checking passes - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text Focused registry/server-json tests and reviewer approval were completed on this PR before the governance body cleanup. The current body update is documentation-only metadata for PR governance. ``` ## Real Behavior Proof - Environment: Headroom development checkout with MCP test dependencies. - Exact command / steps: Inspected the generated `server.json` contract and parity coverage against the descriptor builder and runtime MCP spec. - Observed result: The committed descriptor matches the builder/runtime contract and advertises the intended `headroom-ai[mcp]` / `headroom mcp serve` launch path. - Not tested: live publication to third-party registries ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes This body was normalized by a maintainer after approval so the governance parser reflects the already-reviewed PR state. |
||
|
Rocker Zhang
|
d7283387ac
|
feat(metrics): export compression-failed and kompress size-gate counters (#1569)
## What
Two Prometheus counters that make previously-invisible compression
behavior measurable on `/metrics`.
- `headroom_compression_failed_total{reason=timeout|error}` —
incremented at both Anthropic fail-open sites (single-message and
batch-create), where an optimization exception forwards the request
uncompressed. Before this, ratio could bleed at these sites with nothing
in `/metrics`; only a response header recorded the single-message case.
The timeout/error split separates "compression budget too tight" from
"real bug".
- `headroom_kompress_size_gate_total{outcome=within|exceeded}` — the
size gate (#1171) routes oversized blocks off ModernBERT. The
within/exceeded split proves whether the gate ever fires on real
traffic. `within` counts a gate pass, not whether ML compression then
ran.
## How
Both reuse the existing `PrometheusMetrics` singleton and the
established `defaultdict(int)` counter + text-exposition pattern. The
handler records via `self.metrics`; `content_router` records through the
existing `CompressionObserver` hook to avoid an import cycle. Cleared in
`reset_runtime`; exposition blocks are emitted only when non-empty.
## Verification
- `tests/test_prometheus_obs_counters.py` (6 tests): per-reason/outcome
bucketing, empty-string default buckets, exposition format, conditional
absence until recorded, and reset clearing. All green.
- Counter increments and well-formed exposition (HELP/TYPE balanced,
labels escaped) confirmed by direct exercise; gate `within`/`exceeded`
shown mutually exclusive across the eligible-block call sites.
Single commit, rebased on current `main`.
Addresses #1567.
|
||
|
Zhenjia ZHOU
|
f8915067f6
|
feat(evals): register multilingual multi-wiki-qa (zh/ja/ko) dataset (#1530)
## Description The eval framework's `DATASET_REGISTRY` (`headroom/evals/datasets.py`) only had English datasets (squad, hotpotqa, longbench, …), so the LLM-in-the-loop `BeforeAfterRunner` could not be pointed at Chinese/Japanese/Korean. This registers `alexandrainst/multi-wiki-qa` as `multi_wiki_qa` — the only HF-loadable dataset with **uniform zh/ja/ko** extractive QA: SQuAD-style, paper-guaranteed **verbatim-span** answers over full Wikipedia articles. It makes multilingual compression eval first-class in the framework. Pairs with #1527 (which fixes the CJK-broken F1 tokenization + token estimation the framework's metrics use), so registered CJK data feeds into CJK-correct metrics. ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - `headroom/evals/datasets.py`: add `load_multi_wiki_qa(n, lang)` — mirrors `load_longbench` exactly (the `_check_datasets_installed()` guard, `EvalCase` construction, `EvalSuite` return); reads the verified schema `answers["text"][0]` (a verbatim substring of `context`). - Register it in `DATASET_REGISTRY` under a new `rag_multilingual` category (same 4-key shape as every other entry). - `tests/test_evals_multilingual.py`: offline registration/shape tests (the live HF load is exercised in Real Behavior Proof, matching the other loaders). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`/`format`) - [x] Type checking passes (`mypy headroom/evals/datasets.py`) - [x] New tests added for new functionality - [x] Manual testing performed (see Real Behavior Proof) ### Test Output ```text $ .venv/bin/python -m pytest tests/test_evals_multilingual.py 2 passed $ ruff check / ruff format --check headroom/evals/datasets.py # clean $ mypy headroom/evals/datasets.py # clean ``` ## Real Behavior Proof - Environment: macOS (Darwin 25.3.0), Python in a uv venv with the `[evals]` extra (`datasets`), branch `feat/evals-multilingual-dataset` off `main`. - Exact command / steps: called the new loader against the live dataset — `load_multi_wiki_qa(n=3, lang="ja")`. - Observed result: it returned an `EvalSuite` named `multi_wiki_qa_ja` with 3 `EvalCase`s, each with non-empty `id`/`context`/`query`/`ground_truth`; the `ground_truth` is a **verbatim substring** of its `context` (the property the answer-retention eval relies on); contexts are full-article length (~2,796 chars). Sample answer: `'1988年10月'`. ```text suite: multi_wiki_qa_ja cases: 3 case fields ok: True ground_truth verbatim-substring of context: True ctx len: 2796 | answer: '1988年10月' ``` - Not tested: an end-to-end `BeforeAfterRunner` run against a live LLM (that costs API calls and is out of scope for the loader); zh-cn/ko configs (same schema, verified present via the dataset's splits). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation — N/A (internal eval tooling) - [x] My changes generate no new warnings - [x] I have added tests that prove the feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A: `headroom/evals/` is internal dev tooling, not user-facing runtime ## Additional Notes - **No new dependency.** `multi-wiki-qa` loads through the existing `[evals]` `datasets` extra (guarded by `_check_datasets_installed()`); the dataset is fetched at run time and **never vendored** into the repo. - **License:** `alexandrainst/multi-wiki-qa` is CC-BY-NC-SA-4.0 (non-commercial) — consistent with how the repo already references externally-licensed datasets (SQuAD/LongBench) by id without committing their data. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Abhay Singh
|
a5d7e12c90
|
fix(proxy/batch): preserve sibling tool configs on Google batch requests (#2177)
## Description
When Headroom optimizes a Google/Gemini batch request, it silently drops
every tool config that isn't `functionDeclarations`.
In `handle_google_batch_create` the per-item optimizer extracts the
function declarations:
```python
tools = req_content.get("tools")
existing_funcs = None
if tools:
for tool in tools:
if "functionDeclarations" in tool:
existing_funcs = tool["functionDeclarations"]
break
```
and then rebuilds the forwarded request's tools as a single entry:
```python
if existing_funcs is not None:
compressed_req_content["tools"] = [{"functionDeclarations": existing_funcs}]
```
Gemini's `tools` array is a list of heterogeneous entries —
`{"functionDeclarations": [...]}` can sit alongside `{"googleSearch":
{}}` and `{"codeExecution": {}}`. Collapsing the array to one
`functionDeclarations` entry discards those siblings, so a batch request
that combines function calling with Google Search or code execution
reaches Google with those features stripped out. The request still
succeeds, so the loss is silent — the model just never grounds against
Search / never runs code.
The branch fires whenever the item had any `functionDeclarations` (or
CCR injected a retrieval tool), i.e. exactly the requests most likely to
also declare Search/code-execution.
## Fix
Rebuild the tools list from the original, replacing only the
`functionDeclarations` entry with the (possibly CCR-injected) funcs and
appending a new entry when the original had none:
```python
rebuilt_tools = []
replaced = False
for tool in tools or []:
if "functionDeclarations" in tool:
rebuilt_tools.append({**tool, "functionDeclarations": existing_funcs})
replaced = True
else:
rebuilt_tools.append(tool)
if not replaced:
rebuilt_tools.append({"functionDeclarations": existing_funcs})
compressed_req_content["tools"] = rebuilt_tools
```
Sibling entries (`googleSearch`, `codeExecution`, ...) are preserved in
place; the search/no-search behavior of the request is unchanged.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/batch.py`: preserve
non-`functionDeclarations` tool entries when rebuilding the optimized
Gemini batch request's tools array.
- `tests/test_proxy_handlers_batch.py`: new regression test asserting
`googleSearch` / `codeExecution` survive alongside
`functionDeclarations` in the forwarded body (uses the existing
`RealConvHandler` harness with the real converters).
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/proxy/handlers/batch.py tests/test_proxy_handlers_batch.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/handlers/batch.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx
mypy@1.20.2`. A full `pytest` OOM-kills this box (ML stack import), so I
reproduced the array rebuild with a dependency-free script and left the
full pytest to CI.
- Exact command / steps: fed a tools array of
`[{functionDeclarations:[get_weather]}, {googleSearch:{}},
{codeExecution:{}}]` (plus a CCR-injected retrieval function) through
the OLD single-entry rebuild and the NEW preserving rebuild; also the
search-only case where CCR injects the first `functionDeclarations`.
- Observed result: OLD → `[{functionDeclarations:[...]}]` only
(googleSearch and codeExecution gone); NEW → all three entries retained
with the injected retrieval function present in `functionDeclarations`;
the search-only case gains a `functionDeclarations` entry while keeping
`googleSearch`.
- Not tested: a live Gemini batch submission; full local `pytest`
deferred to CI (OOM).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" box is unchecked because the full suite
imports the ML stack, which I can't run here. The new test reuses the
in-file `RealConvHandler` harness (same one the existing
`..._preserves_functioncall_response_order` test uses) so it runs under
the normal CI pytest job; behaviour is additionally verified by the
standalone proof above.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Abhay Singh
|
195ed90ced
|
fix(savings): record pre-compression original as ledger before, not forwarded count (#2176)
## Description
`headroom savings` overstates the proxy reduction percentage because the
durable ledger is written with the wrong `before` value.
In `PrometheusMetrics.record_request` the proxy appends a savings event:
```python
if tokens_saved > 0 and not self._stateless:
savings_ledger.record_savings_event(
tokens_before=input_tokens,
tokens_after=max(input_tokens - tokens_saved, 0),
...
)
```
But `input_tokens` here is the optimized, **post-compression** count
that was actually forwarded, not the original. `emit_request_outcome`
(the single funnel that calls `record_request`) passes
`input_tokens=outcome.optimized_tokens`.
The ledger derives the reported reduction as `saved / before`
(`savings_ledger._Bucket.savings_percent`), with `saved = max(before -
after, 0)`. Passing the forwarded count as `before` (and `before -
saved` as `after`) keeps `saved` correct but understates `before` by
`tokens_saved`, so the percentage is inflated:
- original input 1000 tokens, forwarded 600, saved 400 → true reduction
40%.
- recorded as `before=600, after=200` → `400 / 600` = **66.7%** on the
dashboard.
So `headroom savings` (which aggregates this ledger across restarts and
processes) reports a reduction percent well above what actually happened
for all proxy traffic.
## Fix
Reconstruct the original as forwarded + saved:
```python
tokens_before=input_tokens + tokens_saved, # the pre-compression original
tokens_after=input_tokens, # what we forwarded
```
`saved` (= `before - after` = `tokens_saved`) and the stored `cost_usd`
(derived from `saved`) are unchanged; only the `before`/`after` labels
are corrected, so the reduction percent becomes honest.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/prometheus_metrics.py`: pass
`tokens_before=input_tokens + tokens_saved` and
`tokens_after=input_tokens` to `record_savings_event`, with a comment
explaining that `input_tokens` is the forwarded count.
- `tests/test_savings_ledger_before_forwarded.py`: new regression guard
on the call shape.
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/proxy/prometheus_metrics.py tests/test_savings_ledger_before_forwarded.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/prometheus_metrics.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx
mypy@1.20.2`. A full `pytest` OOM-kills this box (ML stack import), so I
reproduced the math with a dependency-free script modelling the ledger's
own `saved = before - after` and `saved / before * 100`, and left the
full pytest to CI.
- Exact command / steps: fed a request with original=1000,
forwarded=600, saved=400 through the OLD call shape
(`before=input_tokens`, `after=input_tokens-saved`) and the NEW shape
(`before=input_tokens+saved`, `after=input_tokens`).
- Observed result: OLD → `before=600, after=200`, reported 66.7%; NEW →
`before=1000, after=600`, reported 40.0% (the true reduction). `saved`
is 400 in both, so the cost figure is unaffected.
- Not tested: a live proxy end-to-end run; full local `pytest` deferred
to CI (OOM).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" box is unchecked because the full suite
imports the ML stack, which I can't run here. The regression test
asserts on the source of `record_request` (the enclosing module imports
the ML stack, so it executes the assertion against the source text
rather than calling the method); the behavioural verification is the
standalone proof above.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Abhay Singh
|
f723925be7
|
fix(proxy/gemini): forward a non-JSON upstream body with its real status (#2174)
## Description
`handle_gemini_generate_content` turns a non-JSON upstream error
response into a generic 502, hiding the real status and body.
After the upstream call it extracts usage from `response.json()`:
```python
try:
resp_json = response.json()
usage = resp_json.get("usageMetadata", {})
...
cache_read_tokens = usage.get("cachedContentTokenCount", 0)
except (KeyError, TypeError, AttributeError) as e: # <-- missing JSONDecodeError / ValueError
...
```
`response.json()` raises `json.JSONDecodeError` (a `ValueError`
subclass) for a non-JSON body. That isn't in the tuple, so it escapes to
the function's outer `except Exception`, which returns a synthetic 502
and discards the real `response.status_code` / `response.content` (which
the success path forwards verbatim). An overloaded Google/Vertex/Copilot
frontend commonly returns a 503/500/429 with an HTML or empty body, so
the client sees a generic 502 instead of the true status — defeating
retry/backoff and dropping the diagnostic.
The all-non-text early-exit branch in the same handler already handles
this correctly with the full tuple (`except (json.JSONDecodeError,
ValueError, KeyError, TypeError, AttributeError)`) and then forwards the
real status/content.
## Fix
Add `json.JSONDecodeError, ValueError` to the token-extraction `except`,
matching that sibling. On a non-JSON body the extraction is skipped
(token metrics keep their fallbacks) and the handler falls through to
`return Response(content=response.content,
status_code=response.status_code, ...)` — the real status and body.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/gemini.py`: broaden the token-extraction
`except` in `handle_gemini_generate_content` to include
`json.JSONDecodeError, ValueError`.
- `tests/test_gemini_nonjson_status.py`: new test asserting that except
clause catches the JSON/ValueError family (guards against the
regression).
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/proxy/handlers/gemini.py tests/test_gemini_nonjson_status.py
All checks passed!
$ python -m py_compile headroom/proxy/handlers/gemini.py tests/test_gemini_nonjson_status.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. A full
`pytest` OOM-kills this box (ML stack import), so I verified the
exception handling with a dependency-free script that models the
token-extraction try/except plus the handler's verbatim-forward return,
and left the full pytest to CI.
- Exact command / steps: sent a 503 response whose `.json()` raises
`JSONDecodeError` (non-JSON body) through the old tuple and the new
tuple, plus a normal JSON 200 as a control.
- Observed result: old lets `JSONDecodeError` escape (→ the outer
handler's synthetic 502); new catches it and forwards the real 503; the
JSON 200 still extracts tokens under both. The new test asserts the real
handler's except clause includes the JSON/ValueError family.
- Not tested: a live overloaded Gemini upstream; full local `pytest`
deferred to CI (OOM).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" / "type checking" boxes are unchecked
because the full suite imports the ML stack, which I can't run here; the
change adds two exception types matching an existing, tested sibling
branch, verified by the standalone proof and a source-level regression
guard (a full handler-integration harness for Gemini doesn't exist
in-tree, and the existing gemini integration tests hit a live API).
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Abhay Singh
|
8f867e4622
|
fix(install): guard non-dict health config in 'install status' (#2150)
## Description
`headroom install status` crashes with an `AttributeError` when the
probed health endpoint returns a non-dict `config`.
```python
if payload and isinstance(payload, dict):
click.echo(f"Health URL: {manifest.health_url.replace('/readyz', '/health')}")
click.echo(f"Backend: {payload.get('config', {}).get('backend', manifest.backend)}")
```
`payload` is guarded as a dict, but `payload['config']` is not.
`dict.get('config', {})` only substitutes the `{}` default when the key
is **absent** — a present-but-non-dict `config` (`null`, a string, a
list) is returned as-is, and the chained `.get('backend', ...)` then
raises `AttributeError`, crashing the command with a raw traceback.
Reachability: the Headroom proxy normally returns `config` as an object,
so this bites when `install status` probes a port that a different or
older service is occupying (which can emit `config: null` or a
non-object), or a build that emits `config: null`. The correctly-guarded
sibling already exists in the codebase — `wrap.py`'s
`_proxy_health_config` does `config = payload.get("config"); return
config if isinstance(config, dict) else None`.
## Fix
Guard the `config` value with `isinstance(config, dict)` before the
`.get('backend', ...)` lookup, mirroring `_proxy_health_config`. A
non-dict (or missing) `config` falls back to the manifest's backend.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/cli/install.py`: `install status` guards `config` with
`isinstance(config, dict)` before reading `backend`.
- `tests/test_cli/test_install_cli.py`: add
`test_install_status_survives_non_dict_config` (health payload with
`config: null` must not crash; backend falls back to the manifest).
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/cli/install.py tests/test_cli/test_install_cli.py
All checks passed!
$ python -m py_compile headroom/cli/install.py tests/test_cli/test_install_cli.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified the access with a
dependency-free script that replicates the old vs guarded lookup, and
left the full pytest (including the new CLI test) to CI.
- Exact command / steps: ran the old `payload.get('config',
{}).get('backend', ...)` and the new guarded lookup against `config`
values of `null`, a string, a list, a proper object, and a missing key.
- Observed result: the old lookup raises `AttributeError` for every
non-dict `config`; the new lookup falls back to the manifest backend for
those and returns the real backend for a proper object (and the
missing-key case is unchanged). The new CLI test drives `install status`
with `probe_json` returning `{"config": null}` and asserts a clean exit
with the manifest backend.
- Not tested: a live foreign service occupying the port; full local
`pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" and "type checking" boxes are unchecked
because the full suite imports the ML stack, which I can't run in this
environment; the change adds an `isinstance` guard mirroring an existing
sibling, verified by the standalone proof and a new CLI test that reuses
the file's existing `install status` mocking harness.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Abhay Singh
|
0cddac632d
|
fix(telemetry): only advance usage-report baseline after a 200 (#2149)
## Description
The usage reporter permanently drops a reporting window's usage whenever
the send to the cloud fails.
`UsageReporter._report_usage` computes usage as a **delta** against the
last snapshot, POSTs it, and then rebases the baseline:
```python
try:
resp = await client.post(f"{self._cloud_url}/v1/license/usage", json=payload, timeout=10.0)
if resp.status_code == 200:
...
else:
logger.warning("Usage report returned status %d", resp.status_code)
except Exception:
logger.warning("Failed to send usage report", exc_info=True)
# Update snapshot
self._snapshot_metrics() # runs on success, non-200, AND exception
self._last_report_time = now
```
`_snapshot_metrics()` rebases `_last_tokens_saved_by_model` /
`_last_tokens_sent_by_model` / `_last_requests_by_model` to the current
cumulative counters. Because it runs unconditionally after the POST, a
report that fails to send — non-200 or a raised exception — still
advances the baseline. The module is explicitly built to tolerate a
briefly-unreachable cloud (7-day grace, cached license), so this is a
normal, recurring situation.
The consequence: the failed window's requests and tokens are never
re-included. The next report is a delta from the advanced baseline, so
that window is silently and permanently lost from usage-based billing /
quota. Every transient network blip under-counts usage. (The
`total_requests == 0` early-return already gets this right — it advances
only `_last_report_time`, without snapshotting, since there's nothing to
lose.)
## Fix
Advance the baseline (`_snapshot_metrics()` and `_last_report_time`)
only inside the `resp.status_code == 200` branch. On a non-200 or an
exception, both baselines are left intact, so the next report covers the
full period since the last successful send and re-includes the
previously-failed window.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/telemetry/reporter.py`: move `_snapshot_metrics()` +
`_last_report_time = now` into the 200 branch of `_report_usage`.
- `tests/test_usage_reporter_snapshot.py`: new tests — baseline advances
on 200, and stays intact on a non-200 and on an exception (window
preserved).
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/telemetry/reporter.py tests/test_usage_reporter_snapshot.py
All checks passed!
$ python -m py_compile headroom/telemetry/reporter.py tests/test_usage_reporter_snapshot.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified the delta accounting with a
dependency-free script that models two windows across a failed then
successful send, and left the full pytest to CI.
- Exact command / steps: window 1 saves 100 tokens and the send fails;
window 2 saves another 50 (cumulative 150) and the send succeeds. Ran
under the old (unconditional snapshot) and new (snapshot-on-200) logic.
- Observed result: old delivers only 50 tokens total (window 1's 100
dropped when the baseline advanced on the failed send); new delivers the
full 150 (window 2's delta re-includes window 1). The new tests assert
the baseline advances on 200 and is untouched on a non-200 / exception,
driving the real `_report_usage` with a fake proxy + client.
- Not tested: a live cloud round-trip; full local `pytest` deferred to
CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" and "type checking" boxes are unchecked
because the full suite imports the ML stack, which I can't run in this
environment; the change moves two lines into the success branch,
verified by the standalone delta-accounting proof and new tests that
drive the real `_report_usage` via `object.__new__` with a fake proxy
and HTTP client (200, non-200, and exception).
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Abhay Singh
|
fb17156bfa
|
fix(savings): don't bill free models at the $3/M fallback in the ledger (#2147)
## Description
The durable savings ledger records phantom cost-avoided for free
(0-priced) models, billing them at the `$3/M` blended fallback.
`estimate_cost_usd` prices a known model via
`_estimate_compression_savings_usd`, but gates the result on `> 0`:
```python
if model and model != UNKNOWN:
priced = _estimate_compression_savings_usd(model, tokens_saved)
if priced > 0: # <-- the bug
return round(priced, 6)
return round(float(tokens_saved) * float(fallback_rate), 6) # ~$3/M
```
`_estimate_compression_savings_usd` deliberately distinguishes three
cases: a litellm-priced model (`> 0`), a model litellm can't price
(returns the blended fallback itself), and a model that is *legitimately
free* — litellm has an entry with `input_cost_per_token == 0.0`, so it
returns `tokens_saved * 0.0 == 0.0`. Its own comment calls this out:
"`if not ...` treated a real 0.0 as unavailable and billed the $3/M
fallback — phantom savings for a model that costs nothing."
The ledger's `if priced > 0` re-introduces exactly that defect: a free
model's `0.0` is treated as "unpriced" and the code falls through to the
`$3/M` fallback. Every saved token on a free/local/promo model is then
written into the durable JSONL ledger — and surfaced by `headroom
savings` / `aggregate_savings` — as cost-avoided that never existed.
## Fix
Trust `_estimate_compression_savings_usd`'s return verbatim for known
models. It already returns the blended fallback for models litellm can't
price and `0.0` for free ones, so the `> 0` gate is not needed — and is
the source of the double-fallback. The `UNKNOWN`/empty-model path still
uses `fallback_rate` as before.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/savings_ledger.py`: `estimate_cost_usd` returns
`_estimate_compression_savings_usd(...)` unconditionally for known
models instead of gating on `> 0`.
- `tests/test_savings_ledger.py`: add
`test_free_model_is_not_billed_at_fallback` (free model → $0) and
`test_priced_model_uses_litellm_estimate` (priced model → estimate),
both monkeypatching the helper.
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/savings_ledger.py tests/test_savings_ledger.py
All checks passed!
$ python -m py_compile headroom/savings_ledger.py tests/test_savings_ledger.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified the pricing with a
dependency-free script that replicates the gate and the helper's three
cases, and left the full pytest to CI.
- Exact command / steps: priced 1,000,000 saved tokens for a free model,
a priced model, a litellm-unknown named model, and the explicit
`UNKNOWN` sentinel, under the old (`> 0` gate) and new (unconditional)
logic.
- Observed result: the old logic bills the free model `$3.00` (phantom);
the new logic bills `$0.00`. The priced model (`$2.00`), the
litellm-unknown fallback (`$3.00`), and the `UNKNOWN`-path
`fallback_rate` are unchanged. The new tests assert the free-model `$0`
and the priced-model estimate via a monkeypatched helper.
- Not tested: a live litellm lookup for a real free model; full local
`pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" and "type checking" boxes are unchecked
because the full suite imports the ML stack, which I can't run in this
environment; the change removes a `> 0` gate in a pure pricing function,
verified by the standalone proof and the new tests. This is the same
category as the earlier zero-price-model fix, but at a distinct,
still-buggy call site (the durable ledger) — the earlier fix landed
inside `_estimate_compression_savings_usd`.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Ingmar Krusch
|
d125805589
|
fix(proxy/savings): append history point on cache-only savings too (#2194)
## Description `SavingsTracker.record_request()`'s history-append guard only fired when `tokens_saved > 0` (headroom's own lossy compression). In `--mode cache`, `tokens_saved` is near-always 0 by design — the frozen prefix is byte-replayed rather than compressed, to keep the provider's prompt cache warm. That silently dropped every history point on a cache-mode deployment even when `cache_read_tokens`/`cache_savings_usd` were large, making `headroom-monthly`-style tooling read as a total savings collapse. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/savings_tracker.py`: widen the history-append guard in `record_request()` to fire on `tokens_saved > 0` OR `cache_read_tokens > 0`, and carry `cache_read_tokens`/`cache_savings_usd` on the appended history entry so downstream consumers can show them. `_normalize_history_entry` defaults both fields to `0`/`0.0` for legacy entries that predate this change. - `tests/test_proxy_savings_history.py`: regression coverage that a cache-only request (zero `tokens_saved`, nonzero `cache_read_tokens`) still appends a history point, that the new fields round-trip through normalization, and that legacy history entries without the new keys still normalize cleanly. - `CHANGELOG.md`: added a `### Fixed` entry under `Unreleased`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_proxy_savings_history.py tests/test_savings_tracker_zero_price.py tests/test_proxy_project_savings.py -q collected 62 items tests/test_proxy_savings_history.py .................................... [ 58%] ...... [ 67%] tests/test_savings_tracker_zero_price.py .... [ 74%] tests/test_proxy_project_savings.py ................ [100%] ======================== 62 passed, 1 warning in 5.81s ========================= $ uv run ruff check headroom/proxy/savings_tracker.py tests/test_proxy_savings_history.py All checks passed! $ uv run mypy headroom/proxy/savings_tracker.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - **Environment:** personal fork deployed as a real proxy (macOS launchd service, `headroom install apply`) with `--backend bedrock --mode cache`, fronting a live Claude Code session where `cache_read_tokens`/`cache_savings_usd` are the dominant savings mechanism and headroom's own compression (`tokens_saved`) is near-always 0. - **Exact command / steps:** ran a multi-turn Claude Code session against this deployment, then inspected `proxy_savings.json`'s `history` array and the `headroom-monthly` savings-history rollup that reads it. - **Observed result:** before the fix, `history` stayed empty (or stopped growing) across the whole cache-mode session despite the lifetime `cache_read_tokens`/`cache_savings_usd` counters climbing turn over turn, because every request had `tokens_saved == 0` and never passed the append guard. `headroom-monthly` therefore rendered a flat/zero savings trend for a deployment that was, by its own lifetime counters, saving real money. After the fix, the same session appends a history point on every cache-hit turn, `cache_read_tokens`/`cache_savings_usd` populate on each new entry, and the monthly rollup tracks the lifetime counters instead of reading as a collapse. - **Not tested:** this deployment is `--mode cache`-only; the `tokens_saved > 0` branch of the widened guard (plain `--mode token` compression-driven savings) was not independently re-verified live post-fix, only via the existing/updated test suite, since it was already covered by pre-existing behavior. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — backend logic change, no UI surface. ## Additional Notes - No linked issue number: found via independent investigation of a personal deployment, not filed as a `headroomlabs-ai/headroom` issue first. - Also incidentally fixes a pre-existing test isolation gap in `test_savings_tracker_helpers_normalize_inputs_and_paths` (it unset `HEADROOM_SAVINGS_PATH` but not `HEADROOM_WORKSPACE_DIR`, so the default-path assertion could pick up whatever workspace a live deployment on the test machine had exported, instead of the library default). --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: Tejas Chopra <chopratejas@gmail.com> |
||
|
Abhay Singh
|
84f66da36f
|
fix(init/codex): merge into hooks.json instead of overwriting it (#2173)
## Description
`headroom init codex` destroys a user's existing Codex hooks.
`_ensure_codex_hooks` builds a fresh payload containing only Headroom's
two hooks and writes it wholesale:
```python
payload = { "hooks": { "SessionStart": [...headroom...], "PreToolUse": [...headroom...] } }
path.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
```
It never reads the existing file, so any user-managed hooks (and any
other top-level keys) in `~/.codex/hooks.json` are silently replaced
with just Headroom's entries — data loss on every `init codex` run.
The sibling registrars do it correctly: `_ensure_claude_hooks` and
`_ensure_copilot_hooks` both read via `_json_file`, then merge per event
and dedup on the Headroom marker, preserving unrelated user entries. The
codex path was the lone writer that overwrote.
## Fix
Read-merge-write, mirroring `_ensure_claude_hooks`: load the existing
payload, keep each event's entries that don't carry the
`headroom-init-codex` marker, append Headroom's, and write back. User
hooks and other top-level keys survive; Headroom's are deduped
(idempotent re-runs).
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/cli/init.py`: `_ensure_codex_hooks` reads via `_json_file`,
merges per event with marker dedup, and writes via `_write_json` (no
more wholesale overwrite).
- `tests/test_cli/test_init_cli.py`: add
`test_ensure_codex_hooks_preserves_user_hooks` — a user hook and an
unrelated top-level key survive; Headroom's hook is appended once.
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/cli/init.py tests/test_cli/test_init_cli.py
All checks passed!
$ python -m py_compile headroom/cli/init.py tests/test_cli/test_init_cli.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified the merge with a
dependency-free script that replicates the old (overwrite) vs new
(merge) logic, and left the full pytest to CI.
- Exact command / steps: gave a config with a user `SessionStart` hook
(`echo my-own-hook`) and an unrelated top-level key (`notify: true`),
then ran both the old and new logic.
- Observed result: old drops both the user hook and the top-level key;
new keeps both and appends Headroom's hook exactly once. The new test
asserts this against the real `_ensure_codex_hooks`.
- Not tested: a live `headroom init codex` end to end; full local
`pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" and "type checking" boxes are unchecked
because the full suite imports the ML stack, which I can't run in this
environment; the change mirrors the existing `_ensure_claude_hooks`
merge logic, verified by the standalone proof and the new test (which
reuses the file's existing hooks-test harness).
|
||
|
Abhay Singh
|
8da4384bfc
|
fix(init/codex): don't delete per-profile provider settings (#2146)
## Description `headroom init codex` silently deletes a user's per-profile provider settings. `_ensure_codex_provider` owns the root-level `model_provider` / `openai_base_url` keys, and to avoid emitting a duplicate top-level key it strips any prior assignment before re-inserting its block: ```python content = re.sub(r"(?m)^[ \t]*model_provider[ \t]*=.*\r?\n", "", content) content = re.sub(r"(?m)^[ \t]*openai_base_url[ \t]*=.*\r?\n", "", content) ``` Those multiline regexes match the keys at any indentation, **in any TOML table**. Codex supports per-profile overrides: ```toml [profiles.work] model_provider = "azure" [profiles.gpt5] model_provider = "openai" ``` So a user with named Codex profiles who runs `headroom init codex` has every `[profiles.*]` `model_provider` / `openai_base_url` line silently removed. Those profiles then fall through to the injected root `model_provider = "headroom"` default — their routing is quietly changed. That collateral deletion isn't needed to prevent the root-level duplicate the strip exists for (#260); the unwrap-side sibling `_strip_codex_init_block` proves the intent is precise (it only removes the Headroom-owned value). ## Fix Scope the strip to the document root — everything before the first table header. Root-level `model_provider` / `openai_base_url` are still replaced (init owns them), but keys inside `[profiles.*]` (or any other table) are left untouched. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cli/init.py`: `_ensure_codex_provider` splits the config at the first table header and strips `model_provider`/`openai_base_url` only from the root section. - `tests/test_cli/test_init_cli.py`: add `test_ensure_codex_provider_preserves_profile_overrides` — a `[profiles.work]` override survives init while the root key is replaced by `headroom`. - `CHANGELOG.md`: Bug Fixes entry. ## Testing - [ ] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 check headroom/cli/init.py tests/test_cli/test_init_cli.py All checks passed! $ python -m py_compile headroom/cli/init.py tests/test_cli/test_init_cli.py OK ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing `headroom` pulls in the torch/transformers stack and a full `pytest` gets OOM-killed on this box, so I verified the strip with a dependency-free script that replicates the old (whole-file) vs new (root-scoped) regex, and left the full pytest to CI. - Exact command / steps: ran both strippers on a config with a root `model_provider = "openai"` and a `[profiles.work]` block overriding `model_provider`/`openai_base_url`. - Observed result: the old strip deletes the `[profiles.work]` overrides too; the new strip keeps them and still removes the root assignment. The new test asserts the profile override survives and the root becomes `headroom`. - Not tested: a live `headroom init codex` end to end; full local `pytest` deferred to CI (OOM, per above). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes The "unit tests pass locally" and "type checking" boxes are unchecked because the full suite imports the ML stack, which I can't run in this environment; the change scopes an existing regex strip to the document root, verified by the standalone proof and the new test (the two existing `_ensure_codex_provider` tests only exercise root-level and block-placement behavior, both preserved). I kept the fix to root-scoping rather than also matching only the `"headroom"` value, since that preserves the #260 duplicate-key guard without the broad deletion. Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Abhay Singh
|
8a71947023
|
fix(proxy): reject rate_limit_requests_per_minute=0 when limiting is enabled (#2142)
## Description
A `rate_limit_requests_per_minute` of 0 makes the proxy return a 500 on
every rate-limited request instead of failing configuration early.
The token-bucket wait computation divides by the per-minute rate:
```python
def consume_from_bucket(*, available_tokens, requested_tokens, rate_per_minute):
if available_tokens >= requested_tokens:
return True, available_tokens - requested_tokens, 0.0
wait_seconds = (requested_tokens - available_tokens) * (60.0 / rate_per_minute)
return False, available_tokens, wait_seconds
```
With `rate_limit_requests_per_minute == 0`, the bucket initializes to 0
tokens, so the first request reaches the division and raises
`ZeroDivisionError`. The CLI guards `--rpm` with
`click.IntRange(min=1)`, but `HEADROOM_PROXY_CONFIG_JSON` and
programmatic `ProxyConfig(...)` construction bypass that guard.
## Fix
Validate `rate_limit_requests_per_minute >= 1` in
`ProxyConfig.__post_init__` when `rate_limit_enabled`, mirroring the
existing `retry_max_attempts` validation. Bad enabled configs now fail
fast with a clear message. When rate limiting is disabled, `rpm=0`
remains inert and is not rejected.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/models.py`: reject `rate_limit_requests_per_minute <
1` when `rate_limit_enabled`.
- `tests/test_proxy_config_rate_limit.py`: cover zero/negative enabled
values, disabled zero, and a valid enabled value.
- `CHANGELOG.md`: add a bug-fix entry.
- Merged current `main` to pick up the repository-wide memory factory
type annotation fix that was breaking the PR lint job.
## Testing
- [x] Unit tests pass (`pytest` focused locally; broader CI passed on
the pre-merge head and fresh CI is running on the main-merged head)
- [x] Linting passes (`ruff check` focused locally)
- [x] Type checking passes for the prior mypy blocker after merging
current `main`
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
uvx ruff@0.15.17 check headroom/proxy/models.py tests/test_proxy_config_rate_limit.py headroom/memory/factory.py
All checks passed!
uvx ruff@0.15.17 format --check headroom/proxy/models.py tests/test_proxy_config_rate_limit.py headroom/memory/factory.py
3 files already formatted
git diff --check headroomlabs/main...HEAD
# no output
uv run --extra dev python -m pytest tests/test_proxy_config_rate_limit.py -q
4 passed
```
## Real Behavior Proof
- Environment: Windows 11 review worktree, Python 3.13.3.
- Exact command / steps: ran the focused rate-limit config test file and
targeted lint/format checks.
- Observed result: enabled zero and negative rpm raise `ValueError`;
disabled zero is accepted; valid enabled rpm is accepted.
- Not tested: full suite; fresh CI is queued after the main merge.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The validation is intentionally at the config boundary to match the
CLI's `IntRange(min=1)` contract and the existing fail-fast
`retry_max_attempts` check.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
Abhay Singh
|
def2f9a728
|
fix(learn): don't desync verbosity pairing on empty assistant turns (#2123)
## Description
`verbosity._ordered_events` and `_parse_session` disagree about empty
assistant turns, which desyncs the response list and produces spurious
fast-skips.
`_parse_session` only creates a `_Response` when an assistant message
actually said something:
```python
if words > 0 or out_tok > 0:
responses.append(_Response(...))
```
But `_ordered_events` consumes one `responses[ri]` for **every**
assistant line, with no matching filter:
```python
if ltype == "assistant" and ri < len(responses):
out.append((responses[ri].ts, "assistant", responses[ri]))
ri += 1
```
So an assistant turn with no text and no output tokens — for example a
pure `tool_use` turn where `usage` is absent — creates no `_Response` at
parse time, yet still consumes a slot in `_ordered_events`. That slot
actually belongs to a *later* real response, so the two lists drift by
one. A human reply that follows the real answer is then paired with the
next answer's (future) timestamp, `ts - last_resp.ts` goes negative, and
since a negative gap is always below the read-fraction threshold, a
spurious `fast_skip` is recorded. That inflates `fast_skip_rate`, which
feeds `pressure`, which lowers the recommended verbosity level.
The user side of `_ordered_events` already replicates its parse-site
filter (`_human_text(...) is None -> continue`); only the assistant side
was missing the equivalent guard. That asymmetry is the bug.
## Fix
In `_ordered_events`, compute `words`/`out_tok` for the assistant line
the same way `_parse_session` does and only consume a response when
`words > 0 or out_tok > 0`, keeping the two functions in lockstep.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/learn/verbosity.py`: `_ordered_events` applies the `words >
0 or out_tok > 0` guard on the assistant branch before consuming a
response, with a comment explaining the desync.
- `tests/test_verbosity_learn.py`: add `_empty_assistant` helper and
`test_empty_assistant_message_does_not_desync_fast_skip` (an empty
assistant turn before a real answer + a slow reply must not record a
fast skip).
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [x] Unit tests pass (`uv run --extra dev pytest
tests/test_verbosity_learn.py::TestSignalExtraction::test_empty_assistant_message_does_not_desync_fast_skip
-q`)
- [x] Linting passes (`uvx ruff@0.15.17 check
headroom/learn/verbosity.py tests/test_verbosity_learn.py
headroom/memory/factory.py`)
- [x] Type checking passes (`uvx mypy==1.20.2
headroom/memory/factory.py`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/learn/verbosity.py tests/test_verbosity_learn.py
All checks passed!
$ python -m py_compile headroom/learn/verbosity.py tests/test_verbosity_learn.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified the alignment with a
dependency-free script that models the parse-site filter, the old vs new
`_ordered_events` consume, and the resulting human-to-response pairing,
and left the full pytest to CI.
- Exact command / steps: built an event stream `[empty assistant, real
answer #1, fast human reply, real answer #2, reply]`, computed the
response list from the parse filter, then walked the old (unfiltered)
and new (filtered) consume to find the gap between the first human and
the response paired before it.
- Observed result: old consume pairs the reply with answer #2 (a future
timestamp) -> gap `-8` (spurious fast_skip); new consume keeps alignment
and pairs it with answer #1 -> gap `+1`. The new test builds a session
with an empty assistant turn and a genuinely slow reply and asserts
`fast_skips == 0`.
- Not tested: a real Claude Code transcript end to end; full local
`pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
Merged current `main` to pick up the repository-wide mypy cache-key
annotation fix, then verified the focused regression locally. the change
adds the existing parse-site filter to one branch of a pure file-parsing
function, verified by the standalone alignment proof and the new
regression test for CI.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Abhay Singh
|
faed4dcfe7
|
fix(wrap/claude): bind _wrap_settings_path before the try (#2126)
## Description
`headroom wrap claude` crashes with an `UnboundLocalError` from its
cleanup `finally` whenever the proxy fails to start, which both hides
the real error and skips cleanup.
`claude()` initializes its cleanup state before the `try` so the
`finally` can always reference it — `proxy_holder`, `_saved_base_url`,
`_settings_foundry`, `port_holder`, `_settings_vertex` are all bound up
front. But `_wrap_settings_path` was the exception: it was assigned only
inside the `try`, after `_ensure_proxy`:
```python
try:
...
proxy_holder[0], actual_port = _ensure_proxy(port, ...) # can raise
...
_wrap_settings_path = Path.cwd() / ".claude" / "settings.local.json" # assigned here
...
finally:
_restore_claude_wrap_base_url(..., settings_path=_wrap_settings_path) # referenced here
cleanup()
```
`_ensure_proxy` raises when the requested port is unavailable and the
range is exhausted, or when the proxy subprocess fails to start. When it
does, control jumps to the `finally`, which evaluates
`settings_path=_wrap_settings_path` — a local that was never assigned —
and raises `UnboundLocalError`. That replaces the real failure with a
raw traceback, and because the `finally` aborts on that line,
`cleanup()` never runs, so proxy cleanup and wrap-marker clearing are
skipped too.
## Fix
Bind `_wrap_settings_path` before the `try`, next to the other cleanup
holders, so the `finally` can always reference it. The value is
unchanged (the in-`try` assignment is removed since it computed the same
path).
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/cli/wrap.py`: hoist the `_wrap_settings_path` initialization
to before the `try` (alongside `proxy_holder`/`_saved_base_url`/…) and
drop the redundant in-`try` assignment.
- `tests/test_cli/test_wrap_claude_finally_unbound.py`: new test — drive
`wrap claude` with `_ensure_proxy` patched to raise and assert the
`finally` completes (no `UnboundLocalError`, and both restore and
cleanup ran).
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [x] Unit tests pass (`uv run --extra dev pytest
tests/test_cli/test_wrap_claude_finally_unbound.py -q`)
- [x] Linting passes (`uvx ruff@0.15.17 check headroom/cli/wrap.py
tests/test_cli/test_wrap_claude_finally_unbound.py
headroom/memory/factory.py`)
- [x] Type checking passes (`uvx mypy==1.20.2
headroom/memory/factory.py`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/cli/wrap.py tests/test_cli/test_wrap_claude_finally_unbound.py
All checks passed!
$ python -m py_compile headroom/cli/wrap.py tests/test_cli/test_wrap_claude_finally_unbound.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and running the CLI
test locally OOM-kills this box, so I verified the control flow with a
dependency-free script that reproduces the try/finally with the variable
assigned inside vs before the try, and left the full pytest (including
the new CLI test) to CI.
- Exact command / steps: ran the flow with the variable bound inside the
try (old) and before the try (new), each with an early failure that
fires before the in-try assignment.
- Observed result: old raises `UnboundLocalError` from the finally and
skips restore/cleanup; new runs the finally cleanly and lets the real
`RuntimeError` propagate. The new CLI test drives `wrap claude` with
`_ensure_proxy` raising and asserts no `UnboundLocalError` and that
restore and cleanup both ran.
- Not tested: a live proxy port-exhaustion end to end; full local
`pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
Merged current `main` to pick up the repository-wide mypy cache-key
annotation fix, then verified the focused regression locally. the change
hoists one assignment to before the `try` (mirroring the four sibling
holders three lines above), verified by the control-flow proof and a new
CLI test that reuses the same mocking pattern the existing `wrap claude`
vertex tests use.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Abhay Singh
|
d236b27c60
|
fix(wrap/codex): export the detected custom upstream base URL (#2125)
## Description `headroom wrap codex` detects a user's custom upstream gateway but never tells Codex to use it, so the user's gateway key is sent to `api.openai.com`. `_inject_codex_provider_config` handles a Codex user who has an OpenAI-compatible gateway declared in `~/.codex/config.toml`, e.g. ```toml model_provider = "freemodel" [model_providers.freemodel] base_url = "https://api.freemodel.dev" ``` It injects the Headroom provider with `env_http_headers = { ... "X-Headroom-Base-Url" = "HEADROOM_CODEX_UPSTREAM_BASE_URL" }` and **returns the preserved upstream URL** so the caller can export it. Its docstring even says: *"Callers that go on to launch Codex should export this value into `HEADROOM_CODEX_UPSTREAM_BASE_URL`."* But `_prepare_codex_wrap_state` called it as a bare statement and discarded the return, and `_run_codex_wrap` / `_build_codex_launch_env` only ever set `OPENAI_BASE_URL`. A repo-wide grep confirms `HEADROOM_CODEX_UPSTREAM_BASE_URL` (`_UPSTREAM_BASE_URL_ENV_VAR`) is never assigned into any process env — it appears only at its definition and in that docstring. Since Codex only emits the `X-Headroom-Base-Url` header when the env var exists, the header is omitted, the proxy's OpenAI handler falls back to its hardcoded `https://api.openai.com`, and the user's `freemodel.dev` key is sent to OpenAI, which rejects it. This is a regression: the wiring existed in the original `#1614` fix (`_codex_custom_upstream = _inject_codex_provider_config(...)` then `env[_UPSTREAM_BASE_URL_ENV_VAR] = ...`) and was dropped by a later refactor that extracted `_prepare_codex_wrap_state`. ## Fix Restore the wiring: `_prepare_codex_wrap_state` now captures and returns `_inject_codex_provider_config`'s value, and `_run_codex_wrap` exports it into the launch env (`env[_UPSTREAM_BASE_URL_ENV_VAR] = custom_upstream`) when it is non-None and not already set, so a user-provided value still wins. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cli/wrap.py`: `_prepare_codex_wrap_state` returns the detected custom upstream URL; `_run_codex_wrap` exports it into the launch env (and its display list) when set. - `tests/test_cli/test_wrap_codex.py`: add `TestCodexLaunchExportsCustomUpstream` — drives `_run_codex_wrap` with mocked prepare/launch and asserts the launch env carries `HEADROOM_CODEX_UPSTREAM_BASE_URL` when a custom upstream is detected, and does not when there isn't one. - `CHANGELOG.md`: Bug Fixes entry. ## Testing - [x] Unit tests pass (`uv run --extra dev pytest tests/test_cli/test_wrap_codex.py::TestCodexLaunchExportsCustomUpstream -q`) - [x] Linting passes (`uvx ruff@0.15.17 check headroom/cli/wrap.py tests/test_cli/test_wrap_codex.py headroom/memory/factory.py`) - [x] Type checking passes (`uvx mypy==1.20.2 headroom/memory/factory.py`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 check headroom/cli/wrap.py tests/test_cli/test_wrap_codex.py All checks passed! $ python -m py_compile headroom/cli/wrap.py tests/test_cli/test_wrap_codex.py OK ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing `headroom` pulls in the torch/transformers stack and running the CLI test locally OOM-kills this box, so I verified the wiring with a dependency-free script that models prepare -> run -> the proxy's upstream fallback, and left the full pytest (including the new CLI test) to CI. - Exact command / steps: modelled the old flow (inject return discarded) and the new flow (return exported into the launch env), then applied the proxy's rule that a missing `HEADROOM_CODEX_UPSTREAM_BASE_URL` falls back to `api.openai.com`. - Observed result: old effective upstream is `https://api.openai.com` (the gateway key is misrouted); new effective upstream is `https://api.freemodel.dev` (the user's gateway). The new CLI test asserts the launch env carries the var when a custom upstream is present and omits it otherwise. - Not tested: a live Codex process reading the env and emitting the header; full local `pytest` deferred to CI (OOM, per above). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes Merged current `main` to pick up the repository-wide mypy cache-key annotation fix, then verified the focused regression locally. the change threads one return value through two functions and exports it, verified by the wiring proof and a new CLI test that drives `_run_codex_wrap` with the heavy prepare/launch steps mocked so only the env-export logic is exercised. Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: Tejas Chopra <chopratejas@gmail.com> |
||
|
Abhay Singh
|
f8eaaeb26a
|
fix(cache): normalize embeddings before the semantic similarity check (#2122)
## Description
The semantic tier of the dynamic-content detector compares an
unnormalized dot product against a cosine threshold, so it flags almost
everything as dynamic and strips the static content it is supposed to
protect.
`SemanticDetector` pre-computes exemplar embeddings and, per sentence,
scores similarity with `np.dot` and compares to `semantic_threshold`:
```python
self._exemplar_embeddings = self._model.encode(self.DYNAMIC_EXEMPLARS, convert_to_numpy=True)
...
sentence_embeddings = self._model.encode(sentence_texts, convert_to_numpy=True)
similarities = np.dot(sentence_embeddings, self._exemplar_embeddings.T)
...
if max_sim < self.config.semantic_threshold: # semantic_threshold defaults to 0.7
continue
```
`sentence_transformers.encode(..., convert_to_numpy=True)` does **not**
normalize by default. So `np.dot` here is an inner product whose
magnitude scales with the embedding norms (typically ~5-15 for MiniLM),
not a cosine similarity in [0, 1]. Comparing that against
`semantic_threshold=0.7` (documented and configured as a 0-1 similarity)
is a scale mismatch: nearly every sentence clears the threshold, so the
semantic tier classifies almost all text as dynamic, moves it into
`dynamic_content`, and empties `static_content` — busting the very cache
the detector exists to protect.
A standalone repro: an unrelated sentence with a true cosine of ~0.1 to
an exemplar produces a raw dot of ~9.1 (well over 0.7); normalized, it
correctly scores ~0.09 and stays static.
The correct behavior is used by the in-repo siblings:
`prediction/feature_extractor.py` passes `normalize_embeddings=True`,
and `memory/adapters/embedders.py` L2-normalizes before dot-product
similarity. This detector did neither.
## Fix
Pass `normalize_embeddings=True` to both `encode` calls (exemplars in
`__init__` and sentences in `detect`). Both sides of the dot product are
then unit vectors, so `np.dot` is a true cosine similarity in [-1, 1],
comparable to `semantic_threshold`.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/cache/dynamic_detector.py`: add `normalize_embeddings=True`
to the exemplar encode (`__init__`) and the sentence encode (`detect`),
with comments explaining the cosine requirement.
- `tests/test_cache/test_dynamic_detector.py`: add
`TestSemanticDetectorNormalization` — a recording fake model asserts
both encode calls pass `normalize_embeddings=True` (via `object.__new__`
for `detect`, and a monkeypatched registry for `__init__`). No model
download needed.
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [x] Unit tests pass (`uv run --extra dev pytest
tests/test_cache/test_dynamic_detector.py::TestSemanticDetectorNormalization
-q`)
- [x] Linting passes (`uvx ruff@0.15.17 check
headroom/cache/dynamic_detector.py
tests/test_cache/test_dynamic_detector.py headroom/memory/factory.py`)
- [x] Type checking passes (`uvx mypy==1.20.2
headroom/memory/factory.py`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/cache/dynamic_detector.py tests/test_cache/test_dynamic_detector.py
All checks passed!
$ python -m py_compile headroom/cache/dynamic_detector.py tests/test_cache/test_dynamic_detector.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`, numpy.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the scale mismatch
with a dependency-free numpy script (no sentence-transformers), and left
the full pytest to CI.
- Exact command / steps: built a MiniLM-dimension exemplar direction and
a sentence direction with a true cosine of ~0.1 (genuinely not dynamic),
gave them realistic un-normalized magnitudes (~9 and ~11), and computed
the old `np.dot` of the raw vectors versus the new `np.dot` of the
normalized vectors, against the 0.7 threshold.
- Observed result: old raw dot ~9.1 (far above 0.7 -> the unrelated
sentence is wrongly flagged dynamic); new cosine ~0.09 (below 0.7 ->
correctly kept static), and always within [-1, 1]. The new tests assert
both encode calls pass `normalize_embeddings=True`.
- Not tested: a real sentence-transformers model end to end; full local
`pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
Merged current `main` to pick up the repository-wide mypy cache-key
annotation fix, then verified the focused regression locally. the change
adds one keyword argument to two `encode` calls, verified by the numpy
proof and the new fake-model tests for CI. The fix brings this detector
in line with the two sibling call sites that already normalize.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Rod Boev
|
fb683e18d4
|
fix(litellm): forward chat_template_kwargs and other vendor top-level fields to OpenAI-compatible backends via extra_body (#2128) (#2163)
## Description
When Headroom forwards a `/v1/chat/completions` request to an
OpenAI-compatible backend (vLLM) via the LiteLLM backend, the
non-standard-but-OpenAI-compatible top-level field
`chat_template_kwargs` (e.g. `{"chat_template_kwargs":
{"enable_thinking": false}}`, used by vLLM to toggle Qwen3-family
"thinking" mode per request) never reaches the upstream model. A caller
that needs thinking *off* for a specific request has no way to disable
it through Headroom: the reasoning model spends its whole output-token
budget on hidden `<think>...</think>` content and returns
empty/truncated visible content.
Root cause: `LiteLLMBackend.send_openai_message`
(`headroom/backends/litellm.py:1101-1210`) and `stream_openai_message`
(`headroom/backends/litellm.py:1285+`) build the outgoing LiteLLM
`kwargs` from an explicit allowlist of recognized OpenAI params
(`headroom/backends/litellm.py:1129-1141`: `max_tokens`, `temperature`,
`top_p`, `stop`, `tools`, `tool_choice`, `response_format`, `seed`,
`n`). Only `model` and `messages` are copied unconditionally; anything
not in the list — including `chat_template_kwargs` — is dropped before
`acompletion(**kwargs)`. This is exactly the "litellm-backed forwarding
only passes fields it recognizes as standard OpenAI params" the reporter
suspected.
LiteLLM already forwards arbitrary vendor fields to an OpenAI-compatible
backend verbatim through its documented `extra_body` parameter — the
same mechanism vLLM users use directly. This change collects the
top-level body fields Headroom does not consume as standard params and
forwards them under `extra_body`, so `chat_template_kwargs` (and any
other vendor top-level field) reaches vLLM unchanged, on both the
buffered and streaming paths.
Closes #2128.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- In `LiteLLMBackend.send_openai_message` and `stream_openai_message`,
after populating the standard-param allowlist, collect top-level `body`
keys not consumed by Headroom/LiteLLM (everything outside the standard
allowlist plus `model`/`messages`/`stream`/`stream_options` and internal
markers) and forward them to the backend via LiteLLM's `extra_body`.
- `chat_template_kwargs` and other vendor-specific top-level fields now
reach the OpenAI-compatible upstream verbatim.
- Left the standard-param allowlist, region/profile config, API-key
forwarding, and the cache-stats usage block untouched; standard params
stay first-class LiteLLM kwargs (not moved into `extra_body`).
- Scoped to the OpenAI-format methods; the Anthropic-format
`send_message`/`stream_message` and the metadata-only
`OpenAICompatibleProvider` are unchanged.
## Testing
- [x] Unit tests pass (`uv run pytest
tests/test_litellm_openai_passthrough.py`)
- [x] Linting passes (`uv run ruff check .`)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality when applicable
- [x] Manual testing performed
### Test Output
```text
$ uv run pytest tests/test_litellm_openai_passthrough.py -q
.... [100%]
4 passed in 1.88s
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uv run`; `acompletion` mocked
(no live vLLM).
- Exact command / steps: `uv run pytest
tests/test_litellm_openai_passthrough.py -q`, which drives
`send_openai_message` and `stream_openai_message` with a body containing
`chat_template_kwargs: {"enable_thinking": false}` and inspects the
captured `acompletion` call kwargs.
- Observed result: on both the buffered and streaming paths
`acompletion` is now called with `extra_body={"chat_template_kwargs":
{"enable_thinking": false}}`; a standard-only body produces no
`extra_body`, and standard params (`max_tokens`, `temperature`, …)
remain first-class kwargs. Before the change the same body reaches
`acompletion` with `chat_template_kwargs` absent.
- Not tested: live vLLM run confirming Qwen3 thinking mode toggles off
end-to-end.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- This implements reporter option (a): pass unrecognized top-level body
fields through verbatim (via `extra_body`), which needs no new config
surface. Reporter option (b), an explicit allowlist/config on the
provider, is a deliberate non-goal here and can follow if maintainers
prefer it. The Anthropic-format `send_message`/`stream_message`
translation path and the direct-httpx passthrough path (which already
forwards the full body) are out of scope.
- Issue diagnosed by George Stephanis (`@georgestephanis`) with Claude
Code assistance, per the report's AI disclosure.
- `mypy` left unchecked: not part of the focused validation for this
change.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Rod Boev
|
cc072f0821
|
fix(cache-aligner): hash the frozen conversation prefix so Claude Code cache invalidation is detected (#2085) (#2161)
## Description Running Headroom as the API proxy for Claude Code, provider prompt-cache reuse collapsed: uncached input tokens went from ~755 to ~4.5M, cache-creation (write) tokens inflated ~4.4×, and one session burned ~36% of a weekly model cap. Roughly 96%-cached traffic became uncached+rewrite traffic — a net cost multiplier, not a saving. The `CacheAligner` owns the pipeline's "is the cacheable prefix byte-stable across requests?" signal (`stable_prefix_hash` / `prefix_changed` on `CachePrefixMetrics`). But `CacheAligner.apply()` computes that hash over **only `role == "system"` messages** (`headroom/transforms/cache_aligner.py:314-325`). Under Claude Code the system prompt is the stable part; what actually churns between requests is the conversation head — earlier user turns and tool-result blocks — the range Claude Code relies on for provider cache reads. `apply()` already receives the authoritative freeze boundary (`frozen_message_count`, produced by `PrefixCacheTracker`) and uses it to skip volatile-content detection, but the hash ignores it. So `prefix_changed` reports "prefix stable" even while the real cacheable prefix churns: the budget-burning regression is invisible and the byte-stability invariant the issue asks for is neither asserted nor enforced. This change scopes the aligner's stable-prefix hash to the actual frozen cacheable prefix (`messages[:frozen_message_count]` plus system messages), keyed on the authoritative `frozen_message_count`, so `prefix_changed` becomes a true cache-invalidation signal — and adds the replay regression test the issue specifies, locking the invariant "for `messages[0..k]` identical to the previous request, the emitted prefix bytes and hash are identical." `apply()` remains strictly detector-only and byte-equal; no rewrite is introduced. Closes #2085. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Scoped `CacheAligner.apply()`'s `stable_prefix_hash` to the frozen cacheable prefix: the byte content of `result_messages[:frozen_message_count]` (the frozen conversation head, in order) combined with the system messages, keyed on the authoritative `frozen_message_count` kwarg from `PrefixCacheTracker`. - `prefix_changed` now reflects churn in the true provider-cacheable prefix (a changed tool-result block that leaves the system prompt untouched is now detected), so Claude Code cache invalidation is observable via the existing `CachePrefixMetrics` and the `stable_prefix_hash:<hash>` marker. - Preserved first-turn behavior: when `frozen_message_count == 0` the hash falls back to the current system-only scope, so the first request in a session is byte-for-byte unchanged. - Kept `apply()` detector-only (deep copy, never mutates messages) and left the `should_apply` skip gate, volatile-content warning, token counts, and `TransformResult` shape unchanged. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_cache_aligner_prefix_stability.py`) - [x] Linting passes (`uv run ruff check .`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_cache_aligner_prefix_stability.py -q ..... [100%] 5 passed in 0.40s ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, `uv run`; no live provider. - Exact command / steps: `uv run pytest tests/test_cache_aligner_prefix_stability.py -q`, which replays consecutive `apply()` calls in the Claude Code shape (stable system prompt + accumulated tool-result prefix) with `frozen_message_count > 0`. - Observed result: when a frozen tool-result block changes between requests while the system prompt is byte-identical, `prefix_changed` is now `True` and `stable_prefix_hash` differs; when the frozen prefix + system are identical, `prefix_changed` is `False`; when only the live/unfrozen tail changes, `prefix_changed` stays `False`; `apply()` output stays byte-equal to input. Before the change the same frozen-prefix churn reports `prefix_changed=False` because the hash covers only the system prompt. - Not tested: live Anthropic prompt-cache accounting over a full Claude Code session. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - Scope: this slice corrects and locks the byte-stability invariant **at the CacheAligner boundary** — the exact acceptance criterion in the issue (a cache-preservation invariant over identical prefixes plus a replay regression test). It is distinct from, and does not touch, the upstream sources of prefix churn (ContentRouter per-block verdict flaps under `min_ratio` drift, #1619), the `headroom stats` cache-delta surfacing (#960), or parallel-subagent stream misclassification (#1949); those remain separate follow-ups. `PrefixCacheTracker`'s independent forwarded-prefix byte check (`headroom/cache/prefix_tracker.py`) is unchanged. - `mypy` left unchecked: not part of the focused validation for this change. Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: Tejas Chopra <chopratejas@gmail.com> |
||
|
nangsontay
|
5d9bbbeea0
|
fix(codex): rewrite config.toml properly so Codex will route through … (#2102)
## Description `headroom install apply --providers manual --target codex --scope provider` silently failed to route Codex through the proxy whenever `~/.codex/config.toml` already had a `[table]` section (e.g. `[features]`, `[mcp_servers.*]`). `apply_provider_scope` appended the managed `model_provider = "headroom"` block after the last existing table, so TOML scoped the bare key into that table instead of the document root — Codex silently ignored it and kept routing through its default provider. The same code path never overrode a pre-existing top-level `model_provider` assignment either, so a user's `model_provider = "openai"` kept winning even when Headroom's block was appended elsewhere in the file. This mirrors a bug already fixed in the `headroom init` path (`_ensure_codex_provider`, #260) that was never ported to the persistent-install path. Closes: reported via user session (no tracked issue number yet). ## Type of Change - [x] Bug fix (non-breaking change which fixes an issue) ## Changes Made - **`headroom/providers/codex/install.py`**: Added `_insert_block_at_root()`, which walks the document line-by-line and inserts the managed marker block immediately above the first `[table]`/`[[array-of-tables]]` header, falling back to end-of-file append only when no table exists. Mirrors the root-insertion logic already used by `cli/init.py:_ensure_codex_provider`. - Added `_ANY_MODEL_PROVIDER` / `_ANY_OPENAI_BASE_URL` patterns (match any value, not just `"headroom"`) so `apply_provider_scope` strips **any** prior top-level `model_provider` / `openai_base_url` assignment before re-inserting the managed block — the managed keys now override the user's config outright instead of losing to it. - `apply_provider_scope` merge order is now: strip old managed block → strip prior top-level assignments → insert fresh block at document root. ## Testing - [x] **New regression test**: `test_apply_codex_provider_scope_lands_model_provider_at_root` (`tests/test_install/test_providers.py`) — asserts `model_provider = "headroom"` lands before `[features]`, overrides a prior `"openai"` value, and the user's own table content survives. - [x] **Existing tests**: `tests/test_install/test_providers.py` — 42/42 pass (includes prior codex apply/revert/replace/orphan-cleanup coverage). - [x] **Adversarial (ad-hoc, not committed)**: 6-case TOML round-trip proof — parses output with `tomllib` (not substring matching) across: prior provider before a table, no prior provider, empty file, scalars-only (no tables), CRLF line endings, multiple tables. All 6 pass after the fix; first pass caught a false failure from a stale globally pip-installed `headroom` copy shadowing the repo source when tests run outside the project directory — re-verified from inside the repo to confirm the fix itself is correct. - [x] **Lint**: `ruff check` and `ruff format --check` pass on both changed files. ```text $ uv run pytest tests/test_install/test_providers.py -q 42 passed in 0.21s $ uv run --with ruff ruff check headroom/providers/codex/install.py tests/test_install/test_providers.py All checks passed! $ uv run --with ruff ruff format --check headroom/providers/codex/install.py tests/test_install/test_providers.py 2 files already formatted ``` ## Real Behavior Proof - Environment: macOS 26.4.1 (arm64), Python 3.13.14, headroom branch `patch/install-codex` - Exact command / steps: constructed a temp `config.toml` with `[features]\nweb_search = true` (no existing Headroom block), invoked `apply_provider_scope(manifest)` against it with `codex_config_path` patched to the temp file, then parsed the result with `tomllib.loads()`. - Observed result: before the fix, `tomllib.loads(result)["model_provider"]` raised `KeyError` — the key was nested inside `[features]` due to end-of-file append. After the fix, `parsed["model_provider"] == "headroom"` and `parsed["features"]["web_search"] is True` — both the managed key and the user's table are present and correctly scoped. Revert removes `model_provider` and preserves the user's table. - Tested local build and behavior is correct as expected of this patch. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review |
||
|
Abhay Singh
|
5fb449e90b
|
fix(subscription): keep efficiency_pct from exceeding 100% (#2121)
## Description
`HeadroomContribution.efficiency_pct` can report values above 100%
because its numerator and denominator disagree about cache-read tokens.
```python
def total_saved(self) -> int:
return (self.tokens_saved_compression + self.cli_filtering_saved()
+ self.tokens_saved_cache_reads) # includes cache reads
def raw_without_headroom(self) -> int:
return (self.tokens_submitted + self.tokens_saved_compression
+ self.cli_filtering_saved()) # excludes cache reads
def efficiency_pct(self) -> float:
raw = self.raw_without_headroom()
if raw == 0:
return 0.0
return round(self.total_saved() / raw * 100, 1)
```
`tokens_saved_cache_reads` are input tokens that were *forwarded* to the
provider and served from the prefix cache at a discount, so they already
live inside `tokens_submitted` (the "raw input tokens actually
forwarded"). They are added to the numerator via `total_saved()` but
never to the denominator, so with `tokens_submitted=100` and
`tokens_saved_cache_reads=1000` the method returns `1000.0%`, which the
dashboard renders verbatim. An efficiency percentage should never exceed
100%.
## Fix
Use the existing sibling `compression_saved()` (compression + CLI
filtering, which already excludes cache reads) as the numerator. Then
`efficiency_pct = compression_saved / (tokens_submitted +
compression_saved)`, which is bounded by its own denominator and is the
meaningful quantity here: the fraction of the pre-Headroom input that
compression and CLI filtering actually removed. Cache reads are a
provider-side discount on forwarded tokens, not tokens Headroom removed,
so they don't belong in a removal-efficiency ratio. `total_saved()` is
left unchanged for its other callers (`to_dict`, etc.).
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/subscription/models.py`: `efficiency_pct` now uses
`compression_saved()` instead of `total_saved()` as the numerator, with
a comment explaining the cache-read inconsistency.
- `tests/test_subscription_contribution.py`: new tests — cache reads
can't push efficiency over 100%, the ratio equals the
compression-removal fraction, and empty input yields 0.
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [x] Unit tests pass (`uv run --extra dev pytest
tests/test_subscription_contribution.py -q`)
- [x] Linting passes (`uvx ruff@0.15.17 check
headroom/subscription/models.py tests/test_subscription_contribution.py
headroom/memory/factory.py`)
- [x] Type checking passes (`uvx mypy==1.20.2
headroom/memory/factory.py`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/subscription/models.py tests/test_subscription_contribution.py
All checks passed!
$ python -m py_compile headroom/subscription/models.py tests/test_subscription_contribution.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified the ratio with a
dependency-free script that replicates the three methods, and left the
full pytest to CI.
- Exact command / steps: computed `efficiency_pct` under the old
numerator (`total_saved`) and the new numerator (`compression_saved`)
for `tokens_submitted=100, tokens_saved_cache_reads=1000` and for a real
compression case (`submitted=1000, compression=400, cache_reads=300`).
- Observed result: old returns `1000.0%` for the cache-read case
(impossible) and the new returns `0.0%`; for the compression case old
returns `50.0%` (inflated by cache reads) and new returns `28.6%` (= 400
/ 1400), always `<= 100%`. The new tests assert these.
- Not tested: the dashboard render path end to end; full local `pytest`
deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
Merged current `main` to pick up the repository-wide mypy cache-key
annotation fix and current dependency security floors, then verified the
focused regression locally. the change swaps one method call in a pure
dataclass method, verified by the standalone proof and the new tests for
CI.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
Abhay Singh
|
bc24e258b1
|
fix(mcp/claude): don't clobber an unparseable Claude config on register (#1660)
## Description
When the `claude` CLI isn't on PATH (or its `mcp add` fails),
`ClaudeRegistrar`
falls back to `_register_via_file`, which does a full-file
read-modify-write of
`~/.claude/.claude.json`:
```python
config = _read_json(target) # returns {} on JSONDecodeError
servers = config.setdefault("mcpServers", {})
servers[spec.name] = _spec_to_entry(spec)
_write_json(target, config) # overwrites the ENTIRE file
```
`_read_json` returns `{}` for a file that exists but doesn't parse. So
if
`~/.claude/.claude.json` is momentarily corrupt or hand-edited (a
trailing
comma, a crash mid-write), the register path silently rewrites it as
just
`{"mcpServers": {"headroom": {...}}}` — **destroying every other key
Claude Code
keeps there**: `projects`, `oauthAccount`, session history, etc. There's
no
backup. The existing `test_get_server_robust_to_bad_json` only covers
the *read*
path; the destructive *write* path was untested.
Closes: no issue filed — found while auditing the MCP registry
config-write paths.
## Fix
Keep `_read_json` (returning `{}`) for the read-only callers
(`get_server`,
removal), where it's harmless. Add `_read_json_for_write` for the
rewrite path:
it returns `{}` only when the file is **absent or empty** (safe to start
fresh)
and raises `_MalformedConfigError` when the file is present but not a
JSON
object. `_register_via_file` catches it and returns a `FAILED` result
with an
actionable message instead of overwriting.
Result: absent/empty file → registers fresh (unchanged); valid file →
merges,
all other keys preserved (unchanged); present-but-invalid file → refuses
to
touch it and tells the user to fix or remove it.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/mcp_registry/claude.py`: add `_read_json_for_write` +
`_MalformedConfigError`; `_register_via_file` uses it and returns
`FAILED` (without writing) when the target is present-but-unparseable.
`_read_json` is unchanged for read-only callers.
- `tests/test_mcp_registry/test_claude_registrar.py`: regression tests —
register against malformed configs leaves the bytes untouched and
returns `FAILED`; register against a valid config still merges and
preserves unrelated keys.
- `CHANGELOG.md`: Bug Fixes entry under Unreleased.
## Testing
- [x] New tests added for the fixed behavior
- [x] Linting passes (`ruff check`) and formatting is clean (`ruff
format --check`)
- [ ] Full `pytest` deferred to CI (local-OOM reason under Real Behavior
Proof).
```text
$ uv run ruff check headroom/mcp_registry/claude.py tests/test_mcp_registry/test_claude_registrar.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, headroom built from this
branch. Importing `headroom` loads the torch/transformers stack; a full
`pytest` gets OOM-killed on this box, so I verified the write-path logic
with a dependency-free script and left the full pytest to CI.
- Exact command / steps: replicated `_read_json_for_write` and the
`_register_via_file` read-modify-write flow in a standalone script (only
stdlib, no `headroom` import) against real temp files, and exercised:
absent, empty, four malformed variants (`not json`, `{`, `{"projects":
}`, `[]`), and a valid config carrying `projects`/`oauthAccount`.
- Observed result: absent/empty register fresh; every malformed variant
returns FAILED and the original bytes on disk are byte-for-byte
unchanged (no clobber); a valid config merges in `headroom` while
`projects`/`oauthAccount` survive:
```text
OK: absent -> fresh register
OK: empty -> fresh register
OK: malformed -> FAILED, original bytes preserved (no clobber)
OK: valid config -> merged, unrelated keys preserved
MCP CONFIG-WRITE LOGIC VERIFIED
```
- Not tested: driving the real `claude` CLI-absent path end-to-end on a
live `~/.claude/.claude.json` (didn't want to touch a real Claude
install); the file-fallback logic is exercised directly by the
regression tests. Full local `pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- No new dependencies. `headroom/mcp_registry/opencode.py` has the same
read-`{}`-then-clobber shape on its write path (an OpenCode
`opencode.json` with comments/JSONC would be wiped) — I scoped this PR
to the Claude registrar to keep it focused and because OpenCode config
handling is being touched in other open PRs; happy to send a follow-up
for opencode with the same guard if useful.
---------
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
Chester
|
ce141301f1
|
fix(learn): ingest OpenAI Responses HTTP traffic (#2167)
## Description OpenAI Responses HTTP requests currently bypass `TrafficLearner`, so Learn can be enabled and healthy while receiving no preference or tool-result evidence from this transport. This draft adds the first, intentionally narrow part of #2060: HTTP ingestion only. Codex WebSocket per-turn ingestion and transcript baselining remain separate follow-ups. Part of #2060. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Normalize Responses `message`, `function_call`, and tool-output items into the message and tool-result shape already understood by `TrafficLearner`. - Observe the original client payload before memory injection or compression mutates it. - Reuse the existing lazy memory-backend wiring and recent-tool-result limit from the Anthropic path. - Keep ingestion fail-open so learner failures never block proxy traffic. - Add focused normalization and real HTTP-handler regression tests. ## Testing - [x] Focused unit tests pass - [x] Linting passes (`ruff check` on changed files) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual focused test execution performed ### Test Output ```text uvx ruff check headroom/proxy/handlers/openai.py tests/test_openai_responses_traffic_learner.py All checks passed! uvx ruff format --check headroom/proxy/handlers/openai.py tests/test_openai_responses_traffic_learner.py 2 files already formatted Focused source-checkout execution: 2 focused tests passed GitHub CI: All test shards, lint, builds, security checks, and E2E jobs passed ``` ## Real Behavior Proof - Environment: macOS, Python 3.13, in-process FastAPI test client with a fake OpenAI Responses upstream and a recording learner. - Exact command / steps: execute both focused test functions in `tests/test_openai_responses_traffic_learner.py`; the handler test posts a payload containing one user message, one `function_call`, and its failed `function_call_output` to `/v1/responses`. - Observed result: the HTTP response remained successful, the learner received exactly one normalized message batch, and it received the matched failed tool result with parsed arguments. - Not tested: live OpenAI traffic, Codex WebSocket ingestion, or replay/transcript baselining. The complete GitHub CI test matrix passes. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project style guidelines - [x] I have performed a self-review of my code - [x] I have commented the provider-boundary normalization and ingestion behavior - [ ] Documentation changes are not included because this is an internal transport wiring fix - [x] I have added tests that prove the new ingestion path - [x] Focused tests pass locally and the complete GitHub CI test matrix passes - [ ] CHANGELOG update is not included because release notes are generated from conventional commits ## Screenshots (if applicable) Not applicable. ## Additional Notes This PR intentionally excludes Codex WebSocket ingestion, replay/transcript baselining, and scaffolding/noise filters. Keeping those separate avoids coupling transport lifecycle semantics to the basic HTTP parity fix. Maintainer feedback on whether provider-boundary normalization is the preferred ownership layer is welcome. |
||
|
Rod Boev
|
3a39cb99ad
|
install: couple Codex routing to persistent runtime readiness (#2043)
## Description Persistent install currently writes Codex routing before runtime readiness, and it leaves that routing in place while the runtime is stopped or after a failed start. This changes the lifecycle so routing is applied only after the persistent runtime is ready and reverted before stop or removal, which prevents Codex from getting stranded on a dead `127.0.0.1:8787` provider. Closes #2038 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Apply persistent provider mutations only after runtime readiness. - Revert persistent provider mutations before stop and remove. - Clear stale routing before recovery start paths, then reapply after readiness. - Add focused install lifecycle tests for the ordering change. ## Testing - [x] Unit tests pass - [x] Linting passes - [ ] Type checking passes - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text uv run pytest tests/test_cli/test_install_cli.py -q 23 passed, 1 warning in 0.27s uv run ruff check headroom/cli/install.py tests/test_cli/test_install_cli.py All checks passed! uv run ruff format --check headroom/cli/install.py tests/test_cli/test_install_cli.py 2 files already formatted ``` ## Real Behavior Proof - Environment: Windows, Python 3.12.13, no provider (persistent install lifecycle change; no live Codex or proxy call) - Exact command / steps: `uv run pytest tests/test_cli/test_install_cli.py -q`, then `uv run ruff check` and `uv run ruff format --check` on `headroom/cli/install.py` and `tests/test_cli/test_install_cli.py` - Observed result: 23 install-lifecycle tests pass and lint/format checks pass, confirming persistent Codex provider mutations are now applied only after runtime readiness and reverted before stop/remove - Not tested: a live persistent-runtime start/stop cycle with Codex routing ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Rod Boev
|
2678bb1db6
|
fix(update): let Windows self-update replace headroom.exe (#2016)
## Description `headroom update` currently runs pip from the same `headroom.exe` process pip needs to replace. On Windows that leaves the launcher locked and the upgrade fails mid-uninstall. This reroutes Windows pip-based self-update through a short delayed Python helper that replays the original pip argv after the current launcher exits. Other update paths stay synchronous, and the printed upgrade command stays exact for manual recovery. Closes #1941 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Replace the Windows-only `cmd.exe /c` handoff for `pip` and `pip-user` self-update with a short Python helper that replays the original pip argv after the launcher exits. - Keep every pip value, including extras, as argv data through the delayed child and preserve the manual recovery output. - Keep pipx, uv-tool, and non-Windows behavior unchanged. - Add focused regression coverage for extras containing `&`, `|`, and `>`. ## Testing - [x] Unit tests pass - [x] Linting passes - [ ] Type checking passes - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text uv run pytest tests/test_cli_update.py tests/test_update_helpers.py -q 66 passed uv run ruff check headroom/cli/update.py tests/test_cli_update.py tests/test_update_helpers.py All checks passed uv run ruff format --check headroom/cli/update.py tests/test_cli_update.py tests/test_update_helpers.py 3 files already formatted ``` ## Real Behavior Proof - Environment: Windows pip install - Exact command / steps: run `headroom update --extras "foo&calc"`, accept the prompt, and wait for the child pip output - Observed result: Windows pip and pip-user updates now launch the original pip command through a short delayed Python helper argv, so extras stay one argument end to end, while pipx and non-Windows paths remain synchronous in the focused test coverage - Not tested: live Windows run on this host - Scope: Windows pip self-update ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes The printed upgrade command stays unchanged so the manual recovery path remains exact and copy-pasteable. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Rod Boev
|
4364eb8dc4
|
fix(copilot): refresh wrapped subscription tokens (#2156) (#2182)
## Description `headroom wrap copilot --subscription` currently validates a Copilot subscription credential once at launch, exchanges it once, and then pins that short-lived API token into the proxy as an explicit override. When the token expires, long-lived wrapped sessions start returning `transient_auth_error` and then a final HTTP 401 until the entire wrapped session is restarted. This PR keeps the validated launch token for first-request determinism, carries reusable OAuth refresh material into the proxy, and refreshes inside `CopilotTokenProvider` when the seeded token is expired. The explicit `GITHUB_COPILOT_API_TOKEN` override path stays unchanged when no reusable OAuth token exists. The configured `GITHUB_COPILOT_API_URL` also stays pinned across refresh, matching the current wrap contract. Closes #2156. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Extended the Copilot subscription token resolution path to carry reusable OAuth refresh material and expiry metadata instead of discarding it after the wrap-time exchange. - Reworked `CopilotTokenProvider.get_api_token()` so it seeds the wrapper-validated launch token once for the first request, then refreshes through the existing exchange path when that token is expired and reusable OAuth material exists. - Rejected non-finite seeded expiry values such as `inf`, so malformed `GITHUB_COPILOT_API_TOKEN_EXPIRES_AT` inputs cannot pin a stale launch token forever. - Preserved the explicit `GITHUB_COPILOT_API_TOKEN` override path when no reusable OAuth token exists, so non-refreshable overrides keep today's fixed behavior. - Kept explicitly configured `GITHUB_COPILOT_API_URL` values pinned across refresh rather than adopting a refreshed payload's host. - Replaced wrapper-managed seeded `tid_` bearer passthrough with the refresh-aware provider path, so the wrapped CLI no longer bypasses expiry refresh just because it keeps sending the launch token back to the proxy. - Started a dedicated local proxy instance whenever a subscription-seeded session targets a shared or persistent proxy port, so per-session refresh material is not silently dropped on healthy-proxy reuse or cross-wired between concurrent sessions. - Scrubbed inherited Copilot refresh-seed environment variables from both the Copilot child env and the proxy subprocess env before re-injecting the explicit launch-time values. - Added focused auth, wrap, proxy-env, and proxy-reuse regression coverage for expiry refresh, non-finite expiry rejection, session-local proxy isolation, explicit-override preservation, exchange-flag independence, configured API URL pinning, and secret handling. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_copilot_auth.py tests/test_cli/test_wrap_copilot.py tests/test_cli_proxy_env.py tests/test_cli/test_wrap_persistent.py -q`) - [x] Linting passes (`uv run ruff check headroom/copilot_auth.py headroom/cli/wrap.py tests/test_copilot_auth.py tests/test_cli/test_wrap_copilot.py tests/test_cli_proxy_env.py tests/test_cli/test_wrap_persistent.py`) - [x] Formatting passes (`uv run ruff format --check headroom/copilot_auth.py headroom/cli/wrap.py tests/test_copilot_auth.py tests/test_cli/test_wrap_copilot.py tests/test_cli_proxy_env.py tests/test_cli/test_wrap_persistent.py`) - [x] Type checking passes (`uv run mypy headroom/copilot_auth.py`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text 195 passed, 1 skipped in 3.14s ``` ```text All checks passed! ``` ```text 6 files already formatted ``` ```text Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Python 3.12.13, `uv`, no live Copilot credentials. - Exact command / steps: run the focused auth, wrap, proxy-env, and proxy-reuse regression suite after seeding an expired launch token plus reusable OAuth refresh material, then rerun lint and format checks on the touched files. - Observed result: base reproduces the bug because the explicit-token branch never refreshes, accepts non-finite expiry inputs, and shared-proxy reuse can keep the wrong per-session refresh seed alive; head refreshes through the reusable OAuth token, rejects non-finite seeded expiry, replaces the wrapper-managed seeded bearer instead of blindly passing it through, preserves the valid-seed fast path and the fixed override path when no refresh material exists, keeps the configured API URL pinned across refresh, starts a dedicated local proxy when the requested port already belongs to a shared or persistent proxy, and keeps the reusable OAuth token confined to explicit proxy launch env only. The focused suite passed with `195 passed, 1 skipped in 3.14s`. - Not tested: live business-subscription session past the provider's real token-expiry window. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - Scope stays provider-local. The fix remains inside `headroom/copilot_auth.py` and the Copilot wrap handoff in `headroom/cli/wrap.py`; it does not add generic 401 retry logic to provider-neutral proxy layers. - The line that disables token exchange for the Copilot CLI child env is unchanged because it never reached the proxy env and was not the root cause. - Subscription-seeded sessions now get a dedicated local proxy whenever the requested port already belongs to a shared or persistent proxy; existing shared proxies are left alone to avoid disrupting attached wrappers. - Live provider confirmation still needs maintainer or reporter validation because that truth is owned by GitHub's real subscription APIs, not by local stubs. - Headroom's release pipeline generates changelog entries from conventional commits, so `CHANGELOG.md` is intentionally untouched. |
||
|
Rod Boev
|
d2fbd55b8e
|
fix(proxy): protect WebSearch/WebFetch tool results from lossy compression (#2115)
## Description `WebSearch` and `WebFetch` tool results can be large reference payloads whose exact formatting matters. This PR keeps those web-tool outputs verbatim through both the chat/router path and the OpenAI Responses path, including cross-turn dedup, while leaving ordinary compressible tools such as `Bash` unchanged by default. Closes #1810 ## Changes Made - Added `WebSearch`, `WebFetch`, `web_search`, and `web_fetch` to the default excluded tools. - Added a verbatim-only excluded-tool subset for web payloads so those outputs bypass lossy compression, lossless JSON rewriting, and cross-turn dedup folding. - Updated the OpenAI Responses adapter to track protected call IDs for verbatim web outputs. - Added regressions for Anthropic-style tool results, OpenAI Responses tool outputs, cross-turn dedup, and unchanged `Bash` compression behavior. - Merged current `main` and removed unrelated dependency floor changes from the PR diff. ## Testing ```text uv run --extra dev python -m pytest tests/test_websearch_tool_result_protection.py tests/test_content_router_exclude_tools.py tests/test_openai_responses_compression_units.py::test_openai_responses_adapter_keeps_websearch_output_verbatim tests/test_responses_cross_turn_dedup.py::test_protected_websearch_outputs_do_not_fold -q 13 passed uv run --extra dev mypy headroom/transforms/content_router.py headroom/proxy/handlers/openai.py Success: no issues found in 2 source files git diff --check headroomlabs/main...HEAD # no output ``` The local pre-commit hook also passed on the pushed cleanup/type-fix commit. ## Review Readiness - [x] Ready for review - [x] Regression tests added --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Rod Boev
|
09e72125b4
|
fix(proxy): isolate image compression in a subprocess so a native SIGSEGV can't take down the proxy (#2107) (#2162)
## Description On Apple Silicon (arm64) macOS the proxy hard-crashes with SIGSEGV the moment it compresses an image, and because the proxy is the single API endpoint for every routed client (`ANTHROPIC_BASE_URL=http://127.0.0.1:8787`), one image request takes down every agent on the machine at once — they then fail with `ConnectionRefused` and retry into a closed port until the proxy is manually restarted. The faulting stack is inside OpenCV's KleidiCV ARM NEON resize (`kleidicv::neon::kleidicv_resize_generic_stripe_u8`), reached from the SigLIP ONNX image encoder during `ImageCompressor.compress()`. The proxy runs that call on a `ThreadPoolExecutor` (`headroom/proxy/server.py:946`), and both handler call sites (`headroom/proxy/handlers/anthropic.py:1148-1172`, `headroom/proxy/handlers/openai.py:2274-2296`) wrap it in `try/except Exception` intending to fail open. That guard cannot help: a native SIGSEGV is not a Python exception, and a segfault on any worker thread aborts the whole interpreter. Thread isolation is not crash isolation. The defect Headroom owns is that an optional, best-effort, native-heavy transform runs in-process with no crash boundary, so any native fault in it is fatal to the proxy and to every unrelated client it fronts. This change gives image compression a real crash boundary by executing it in a spawned subprocess, so a native crash degrades to "image forwarded uncompressed" instead of killing the proxy. Closes #2107. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `headroom/proxy/image_isolation.py` with `run_image_compression_isolated(messages, provider, *, timeout) -> tuple[list[dict], dict | None]`, which runs an `ImageCompressor.compress` worker inside a lazily-created module-level `ProcessPoolExecutor(max_workers=1)` on a **spawn** multiprocessing context (ONNX sessions are not fork-safe) and carries the compression result (technique, `savings_percent`, token counts) back across the process boundary. The native OpenCV/KleidiCV work now runs in the child address space. - Made the runner fail open for **any** child outcome: `BrokenProcessPool` (the class raised when the child is killed by a signal, i.e. SIGSEGV/SIGABRT), `TimeoutError`, or any other `Exception` all return `(messages, None)` — the original `messages` unchanged, no telemetry — and reset the pool so the next request re-spawns a fresh child. - Routed the two request-path image-compression sites (`handlers/anthropic.py`, `handlers/openai.py`) through the runner, keeping the existing `ImageCompressionDecision` gate and the `image_compression` mutation tag, and emitting the savings `INFO` log line from the runner's returned result on the success path. - Scoped strictly to crash containment: `config.image_optimize` default is unchanged, no dependency pins, no new env switches. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_image_compression_isolation.py tests/test_image_compression_offload.py`) - [x] Linting passes (`uv run ruff check .`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_image_compression_isolation.py tests/test_image_compression_offload.py -q ....... [100%] 7 passed in 1.73s ``` The reproduction test (`test_worker_sigsegv_fails_open_parent_survives`) spawns a real subprocess whose worker dies by signal (`os.abort()`), then asserts `run_image_compression_isolated` returns the original messages and that the parent test process is still alive and continues past the call. ## Real Behavior Proof - Environment: Windows 11, Python 3.12, `uv run`; no live provider. - Exact command / steps: `uv run pytest tests/test_image_compression_isolation.py tests/test_image_compression_offload.py -q`, which drives the runner against a real spawned subprocess that is killed by signal, one that raises, one that times out, and one that returns normally, and also locks the handler wiring to `run_image_compression_isolated(...)`. - Observed result: on a signal-killed child the runner returns the original message list and the parent survives; on a raising or timing-out child it fails open the same way; on a normal child the compressed messages are returned. The handler source-level regression keeps the savings log and mutation-tag path wired through the new isolation helper. Before the change, the handlers called `compressor.compress(...)` through `_run_compression_in_executor(...)`, so a native crash on that worker thread would abort the interpreter. - Not tested: live arm64 macOS run against a real `opencv-python` 5.x KleidiCV fault. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - Non-goals, kept out deliberately to keep this slice shippable now: pinning `opencv-python<5` (a transitive-dependency change that only masks this one fault and does not contain the next native crash), a `HEADROOM_IMAGE_OPTIMIZE=0` env off-switch (distinct config surface; `config.image_optimize=False` already disables the feature), the per-request ONNX model reload, and the negative-savings (`preserve` logged as `-100%`) reporting bug. The last two are independent defects noted in the same report and are better fixed on their own. - `mypy` left unchecked: not part of the focused validation for this change. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
JD Davis
|
fce93bf39a
|
fix(ci/deps): clear audit and release smoke failures (#2190)
## Summary - add a uv constraint floor for `setuptools>=83.0.0` to address `PYSEC-2026-3447` - refresh `uv.lock` so the production audit export resolves with `setuptools 83.0.0` - harden the Release wheel smoke-import gate by retrying Ubuntu-container `apt-get` operations and using `--fix-missing` - keep the generated `requirements-prod.txt` uncommitted; it is produced by the security workflow ## Why This clears the new dependency audit alert that made PRs red: - `setuptools 80.10.2` - `PYSEC-2026-3447` - fixed in `83.0.0` While validating the queue, the same PR class also hit a Release smoke-import failure in the Ubuntu 22.04 ARM container due apt mirror skew: `E: Failed to fetch ... python3-httplib2_0.20.2-2ubuntu0.1_all.deb 404 Not Found` The smoke gate should still fail for broken wheels, but transient apt mirror skew should not make unrelated PRs red. ## Lockfile impact - `setuptools 80.10.2 -> 83.0.0` - `torch 2.12.1 -> 2.13.0`, required for pip resolver compatibility with `setuptools 83.0.0` in the exported audit set - `cuda-toolkit 13.0.2 -> 13.0.3.0`, pulled by the torch lock refresh - uv also refreshed the existing project metadata for the sandbox extra so `uv lock --check` passes ## Validation - `uv lock --check` - `uv export --frozen --no-dev --no-emit-project --no-hashes --extra all --format requirements-txt > requirements-prod.txt` - confirmed generated `requirements-prod.txt` contains `setuptools==83.0.0`, `torch==2.13.0`, `cuda-toolkit==13.0.3.0` - `uvx pip-audit -r requirements-prod.txt` -> No known vulnerabilities found - `python -m pytest tests/test_release_workflows.py -q` -> 32 passed - `uvx ruff@0.15.17 check tests/test_release_workflows.py` -> All checks passed - `git diff --check` |
||
|
Dima Solodukha
|
4e30dde2ac
|
fix(router): compact JSON evades compression via whitespace token counting (#1857)
## Description
`ContentRouter` counts section tokens with `len(content.split())`. On
compact machine-generated JSON — the default output of
`json.dumps(separators=(",", ":"))`, `JSON.stringify`, and boto3 — there
are no spaces, so a large payload counts as ~1 "token". Every section
compression ratio then computes as ~1.0 and the `min_ratio` acceptance
gate silently rejects the compressor's real output: the router logs
`router:noop` while SmartCrusher separately logs `was_modified=true`.
Compression effectively no-ops on the most common agent payload type
(tool results returning JSON), on every provider.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Add `_estimate_tokens(text)` — a size-proportional estimate
(`len(text) // 4`, floored at 1), monotone in content size for any
format.
- Replace the decision-relevant `len(...split())` counts in
`ContentRouter` (section original/compressed token counts feeding the
ratio gates, plus the debug estimates) with `_estimate_tokens(...)`.
- Add `tests/test_content_router_compact_json.py`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_content_router_compact_json.py -q
2 passed, 1 warning
$ ruff check headroom/transforms/content_router.py tests/test_content_router_compact_json.py
All checks passed!
$ mypy headroom/transforms/content_router.py --ignore-missing-imports
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: `ContentRouter` invoked directly on a 150-item
ECS-service JSON tool_result, Python 3.13, estimator tokenizer.
- Exact command / steps: run the same payload two ways — compact
(`json.dumps(..., separators=(",", ":"))`) and the identical data with
spaces (`separators=(", ", ": ")`) — through
`ContentRouter(ContentRouterConfig(skip_user_messages=False))`.
- Observed result: before this change, compact JSON saved 0.0%
(`router:noop`) while the identical data with spaces saved 43.3%
(`router:tool_result:smart_crusher`) — same data, same compressor, only
whitespace differed. After this change, compact JSON compresses
equivalently to the spaced form.
- Not tested: no behavior change expected for content that already
tokenizes with whitespace (prose, code); those counts move from
word-count to chars/4 but the ratio comparison is self-consistent (both
sides use the same estimator).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
Scope kept deliberately narrow: only the counts that feed
compression-acceptance decisions are changed. Non-decision `.split()`
uses elsewhere are left alone. Happy to add a CHANGELOG entry if you'd
like one.
---------
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
gglucass
|
fd9ddaa238
|
fix(proxy): cold-start fast pass — defer only Kompress, not the whole pipeline (#2073)
## Description Since #1850, the freeze path forwards a session's provider-cached prefix byte-identical — so a session is permanently locked to whatever form its cold start put in the provider cache. That fix is correct (it stopped token-mode cache busting measured at +41% cost), but it interacts badly with off-path background compression (#1171): when `HEADROOM_BACKGROUND_COMPRESSION=1` defers a cold-start-large request (frozen=0, ≥50k tokens), the ENTIRE pipeline is deferred and the raw transcript is forwarded, cached, and frozen. The background job's results can never be applied afterward (doing so would rewrite the frozen prefix), so the session forfeits its compression savings for its lifetime. Field data (same day, same session, A/B across a version boundary): ~15k tokens/turn saved when the cold start compressed synchronously vs 0/turn forever when it deferred. Notably, the recurring savings came from `read_lifecycle` stale-read drops completing in ~300ms — deferral throws away sub-second lossless wins to avoid a 30s Kompress pass. Only the Kompress ML stage can blow the request budget (the #1171 cascade). This PR splits the two: - The deferral branch now runs the pipeline synchronously with a new `skip_kompress=True` per-call kwarg — everything except the ML stage — under a bounded budget, and forwards the pruned form. The provider caches (and #1850 freezes) the *compressed* transcript, so the cheap savings persist for the session's lifetime. - The full pipeline (Kompress included) still goes to the background job, unchanged, keyed against the original messages so its content-hash results remain reusable at future cache-miss boundaries. - Fail-open: on fast-pass timeout or error, the request forwards uncompressed exactly as before this change. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/content_router.py`: new per-call `skip_kompress` runtime kwarg (follows the existing `_runtime_force_kompress` pattern). Gates only the Kompress deep-path call site; units routed there take the identical fallback used when the model isn't ready. Wins over `force_kompress`. - `headroom/proxy/helpers.py`: `COLD_START_FAST_PASS_TIMEOUT_SECONDS` (env `HEADROOM_COLD_START_FAST_PASS_TIMEOUT_SECONDS`, default 10s), documented next to `COMPRESSION_TIMEOUT_SECONDS`. - `headroom/proxy/handlers/anthropic.py`: the background-deferral branch runs the fast pass synchronously, stores its result in the session `CompressionCache`, forwards the pruned messages, and tags `deferred:kompress_background` (or `deferred:dropped` when the enqueue was dropped). On failure it constructs the same `_DeferredCompressionResult` as before. The Anthropic handler is the only deferral site (OpenAI/Gemini handlers don't defer). - `tests/test_transforms/test_content_router.py`: `skip_kompress` never invokes the ML stage and wins over `force_kompress` (mirrors the existing `force_kompress` test). - `tests/test_cold_start_fast_pass.py`: handler-level tests — exactly one synchronous `skip_kompress=True` pass, the background job runs the full pipeline, the forwarded body carries the fast-pass form, fast-pass results land in the compression cache; and the fail-open path (executor timeout → original messages forwarded, background job still queued). - `CHANGELOG.md`: Bug Fixes entry. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_cold_start_fast_pass.py -v tests/test_cold_start_fast_pass.py::test_cold_start_runs_fast_pass_and_defers_only_kompress PASSED tests/test_cold_start_fast_pass.py::test_fast_pass_failure_falls_back_to_full_deferral PASSED ============================== 2 passed in 0.28s =============================== $ uv run --frozen --extra dev pytest tests/test_proxy/test_background_compression.py \ tests/test_proxy/test_phase3_byte_identity.py tests/test_anthropic_stage_timings.py \ tests/test_transforms/test_content_router.py tests/test_anthropic_pre_upstream_backpressure.py ======================== 90 passed, 1 warning in 10.47s ======================== $ uv run --frozen --extra dev mypy headroom/proxy/handlers/anthropic.py headroom/proxy/helpers.py headroom/transforms/content_router.py Success: no issues found in 3 source files $ ruff check <changed files> && ruff format --check <changed files> All checks passed! / 5 files already formatted ``` ## Real Behavior Proof - Environment: macOS 15 (darwin 24.6.0), Python 3.10 via `uv run --frozen --extra dev`; field logs from a production desktop deployment (Python 3.12, `HEADROOM_MODE=token`, `HEADROOM_BACKGROUND_COMPRESSION=1`, subscription auth policy). - Exact command / steps: compared per-request PERF log lines for the same Claude Code session served by 0.30.0-lineage (sync cold start) vs 0.31.0-lineage (deferred cold start) on the same day. - Observed result: deferred-cold-start sessions log `tok_saved=0` on every subsequent turn with `Pipeline: freezing first 281/284 messages`; sync-cold-start sessions log `tok_saved=15526-18791` per turn with `read_lifecycle:stale` transforms at `opt_ms≈300`. - Not tested: this patch has not run against a live proxy yet (behavior verified at the handler-test level); `ruff`/`mypy` scoped to changed files. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — proxy pipeline change, no UI. ## Additional Notes - Companion to #2057 (nested tool_result image token counting) and #2058 (new-content-relative savings rate) — all three came out of the same investigation into near-zero reported savings on long 1M-context Claude Code sessions. - Deliberate scope cuts: the OpenAI/Gemini handlers don't have a deferral branch, so nothing to change there; the background job is left keyed to original messages (not the fast-pass output) so its cached results match client-resent bytes at future cache-miss boundaries. - Timeout leak caveat is documented in code: a fast-pass timeout briefly leaks an executor worker, but without the ML stage the pass is bounded by routing + statistical crushers (observed 5-8s worst case on multi-M-token counted transcripts). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Tejas Chopra <chopratejas@gmail.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Abhay Singh
|
fa330f3e2b
|
fix(memory): remove a superseded memory from the search indexes (#2143)
## Description
Superseding a memory leaves the old version live in the search indexes,
so outdated content can still come back from search after supersession.
`HierarchicalMemory.supersede` updates the store and indexes the new
memory, but previously never removed the old entry from the vector/text
indexes. The store sets the old row's `valid_until` and `superseded_by`,
but the indexes keep their own cached metadata copy from first indexing.
Default search filters superseded rows from that cached metadata, so the
old entry could still look live and be returned with stale content.
Concretely: `add("User prefers Python")`, then `supersede(id, "User now
prefers JavaScript frameworks")`, then `search("Python")` could return
the superseded "prefers Python" entry alongside the new one. That
defeats supersession and can recall contradictory facts.
## Fix
After `store.supersede`, remove the old id from the vector and text
indexes, mirroring `delete`. The store keeps the old row for
`get_history`; only the search indexes are corrected. The new memory is
indexed as before.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/memory/core.py`: `supersede` removes the old id from the
vector and text indexes after the store supersession, before indexing
the new memory.
- `tests/test_memory/test_core_operations.py`: adds
`test_superseded_memory_does_not_resurface_in_search`.
- `CHANGELOG.md`: adds a bug-fix entry.
- Merged current `main` to pick up the repository-wide memory factory
type annotation fix that was breaking the PR lint job.
## Testing
- [x] Unit tests pass (`pytest` in CI on the pre-merge head; fresh CI is
running on the main-merged head)
- [x] Linting passes (`ruff check` focused locally; previous CI `ruff
check` and `ruff format` passed before hitting unrelated mypy)
- [x] Type checking passes for the prior mypy blocker after merging
current `main`
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
uvx ruff@0.15.17 check headroom/memory/core.py tests/test_memory/test_core_operations.py headroom/memory/factory.py
All checks passed!
uvx ruff@0.15.17 format --check headroom/memory/core.py tests/test_memory/test_core_operations.py headroom/memory/factory.py
3 files already formatted
git diff --check headroomlabs/main...HEAD
# no output
uv run --extra dev python -m pytest tests/test_memory/test_core_operations.py::TestSupersede::test_superseded_memory_does_not_resurface_in_search -q
# assertion passed; local Windows teardown hit a locked temp SQLite file during fixture cleanup
```
## Real Behavior Proof
- Environment: Windows 11 review worktree, plus GitHub Actions on the
pre-merge head.
- Exact command / steps: ran focused ruff/format/diff checks; ran the
new supersede regression test directly.
- Observed result: focused checks passed; the new test body passed and
confirmed the old memory id does not resurface when searching for old
content while the new memory remains searchable. The local run then
errored during Windows temp SQLite cleanup after the assertion
completed.
- Not tested: full memory suite, because it pulls the ML embedding
stack. CI already passed the broader test matrix on the pre-merge head;
fresh checks are queued after the main merge.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
Removing the old entry from indexes is intentionally aligned with
`delete`; the store row remains available for `get_history`.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Tejas Chopra
|
eca3db62a3
|
fix(savings): coding profile compresses the recent delta (protect_recent 2->0, min_tokens 25->10) (#2145)
## Description
In cache mode, Headroom only compresses the newest delta while keeping
the prefix stable. The coding persona previously set `protect_recent=2`,
which protected that newest delta positionally and could suppress
compression of recent grep/test/build output. This changes the coding
persona to rely on type-specific read protection instead of positional
recent-turn protection.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Changed the coding savings profile from `protect_recent=2` to
`protect_recent=0`.
- Lowered the coding profile's `min_tokens_to_compress` from `25` to
`10` so modest cache-mode deltas are eligible.
- Updated agent savings tests to assert the new coding profile behavior
and runtime pipeline kwargs.
- Included the `_EMBEDDER_CACHE` type annotation correction needed by
current `mypy` after the cache key gained `ollama_base_url`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
GitHub CI on current head:
- lint: pass
- build/build-wheel/build-wheel-windows: pass
- test matrix, test-agno, test-extras, test-dashboard-ui: pass
- docker-native-e2e: pass
```
## Real Behavior Proof
- Environment: GitHub Actions on PR head `
|
