mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
1553 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Tejas Chopra
|
a99dc61424
|
feat: output-token reduction — verbosity shaper, per-user learning, counterfactual savings (#965)
## Description
Adds the first levers that reduce the tokens the model **writes back**
(output), complementing Headroom's existing input compression. Output
costs 5× input on Opus-class models and is full of waste (ceremony,
restated code, deep "thinking" on routine steps). Two phases in one
self-contained PR off `main`: the request-side output shaper, then
per-user verbosity learning plus an honest counterfactual savings
estimator and dashboard surfacing.
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **Output shaper** (`output_shaper.py`, opt-in
`HEADROOM_OUTPUT_SHAPER=1`): cache-safe verbosity steering appended to
the system-prompt tail (5 levels); effort routing that lowers
`output_config.effort` on mechanical tool-result continuations; legacy
`thinking.budget_tokens` clamp. Never injects effort where absent, never
toggles `thinking.type`.
- **`headroom learn --verbosity`**: mines Claude Code transcripts for
behavioral signals (interrupts, length-adaptive fast-skips, echo ratio),
recommends a verbosity level (heuristic + optional `--llm-judge`), and
seeds the savings baseline.
- **Counterfactual estimator** (`output_savings.py`): per-stratum
synthetic-control (estimated) + A/B holdout (measured) with a propagated
95% CI; conversation-stable arm assignment for A/B validity and
prefix-cache safety.
- **AIMD verbosity controller** (`verbosity_controller.py`):
additive-increase / fast-back-off state machine; live signal emission
gated off by default.
- **Wiring + surfaces**: shaper resolves the learned level; recording
rides the existing `transforms_applied` channel through the outcome
funnel (no `RequestOutcome` changes); `headroom output-savings` CLI;
dashboard "Output Tokens Saved" card.
- **Docs**: simple-words user guide + design doc with the counterfactual
methodology.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_output_savings.py tests/test_output_savings_cli.py \
tests/test_verbosity_learn.py tests/test_verbosity_controller.py \
tests/test_output_shaper.py -q
94 passed in 0.54s
$ pytest tests/test_request_outcome.py tests/test_handler_outcome_tag_invariant.py \
tests/test_proxy_dashboard_stats_cache.py -q
44 passed
$ ruff format --check .
831 files already formatted
$ mypy headroom --ignore-missing-imports
Success: no issues found in 361 source files
```
## Real Behavior Proof
- Environment: macOS, Python 3.12 (`.venv`), `anthropic` 0.76, live API
model `claude-opus-4-8`.
- Exact command / steps: `HEADROOM_OUTPUT_SHAPER=1`; `headroom learn
--verbosity --apply` (seeds level + baseline); `python
scripts/eval_output_shaper.py A` (live before/after); simulate holdout
traffic then `headroom output-savings`.
- Observed result: code-review ask — baseline 1,750 output tokens → L2
1,354 (−22.7%) → L3 599 (−65.8%), same bugs found. `learn --verbosity`
on 24 real sessions → 11% interrupt / 26% fast-skip → L3 (high
confidence). Measured A/B path → 31.7% reduction (95% CI 27.7%–35.7%).
94 new tests + 44 existing outcome/dashboard tests green; ruff + mypy
clean.
- Not tested: live streaming-path recording exercised only via unit
tests (the `transforms_applied` funnel is shared across paths); runtime
AIMD signal emission is gated off by default and not exercised live.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
Output savings are counterfactual (we never observe what the model
*would* have written), so the estimator separates **estimated** (vs a
learned baseline) from **measured** (A/B holdout via
`HEADROOM_OUTPUT_HOLDOUT`) and always reports a confidence band — never
a single made-up number. CHANGELOG left unchecked (release-please
manages it). Runtime AIMD self-tuning is intentionally a TODO
(controller built/tested; live signal emission gated behind
`HEADROOM_VERBOSITY_AUTOTUNE`).
|
||
|
Tejas Chopra
|
b7be3814f1
|
feat: compression extraction — Rust knob exposure, CCR hardening, traffic audits (#818)
## Description
A data-driven push for better compression savings without accuracy loss,
in four parts: expose and tune the Rust compressor knobs, harden the CCR
retrieval store, add traffic-audit tooling that sizes opportunities from
real transcripts, and introduce **read maturation** — a new,
live-validated mechanism that compresses Read outputs *before* they ever
enter the provider prefix cache.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
### 1. Rust compressor extraction
- Expose `lossless_min_savings_ratio` end-to-end and lower the default
0.30 → 0.15 (lockstep across Rust, PyO3, and both Python config classes)
so the lossless Table/CSV compaction path wins more often.
- Expose the `CompactConfig` heuristics (core-field fraction,
heterogeneity ratio, flatten cap, bucket bounds) through PyO3 + Python.
- `SearchCompressor` grouped-by-file output (`rg --heading` style — path
once per file instead of per match). Library default off; the proxy
enables it in token mode.
- Complete `factor_out_constants`: constant fields now emit once in a
`_constant_fields` sentinel with slim rows (defensive per-item value
match; default off).
- `ContentRouter` accepts a SmartCrusher config override and the
search-grouping knob.
### 2. CCR store hardening
- Session-scale TTL: 300s → 1800s (CCRConfig, CompressionEntry,
CompressionStore, Rust `DEFAULT_TTL` — lockstep).
- **SQLite is the default CCR backend** (`~/.headroom/ccr_store.db`,
WAL): survives proxy restarts and is shared across workers.
`HEADROOM_CCR_BACKEND=memory` opts out.
- Multi-worker safety: `busy_timeout`, and corruption detection narrowed
so transient `SQLITE_BUSY` errors can never trigger database deletion.
- Data-at-rest hygiene: `chmod 600` on db + sidecars, expired rows swept
at open.
- Retrieval-miss messages are actionable (re-read the file / re-run the
command).
### 3. Traffic audit tooling (measure before tuning)
- `headroom audit-reads`: sizes Read opportunities from local Claude
Code transcripts (read share, stale %, line-number overhead, context
residency, cache-death windows).
- `--simulate-maturation`: Mechanism B risk sizing (re-read rates,
never-touched-again share, quiesce coverage, at-risk edits).
- `--codex`: shell-read classifier for Codex transcripts (rtk-wrapper
aware, workdir resolution).
- Findings that shaped this PR (81 sessions): Reads are 67% of tool
bytes; median Read lingers 118 turns (~13x lifetime cost); a prototyped
repeat-Read dedup measured 0.1% and was **removed** rather than shipped
as dead code.
### 4. Read maturation (Mechanism B) — experimental, default OFF
- Activity-based: a fresh large Read is held **out** of the provider
cache (trailing breakpoint relocated before it), stays verbatim while
its file is active, and matures into a CCR-backed marker once the file
is quiet for `quiesce_turns` (default 5; `max_hold_turns` bounds busy
files).
- Only the final compressed form ever enters the cache — **no cached
byte is ever mutated**; matured markers replay byte-identically.
- Wired into the Anthropic handler behind `--read-maturation` /
`HEADROOM_READ_MATURATION=1`; session state rides on the prefix tracker;
advisory (can never fail a request).
- Live-validated against the Anthropic API: held content excluded from
cache_creation; after maturation the prior cached prefix still served —
the no-bust invariant holds end-to-end.
### 5. Rebase / CI fixups (this update)
- Rebased onto latest `main` (was 28 commits behind): picks up `ci: pass
CODECOV_TOKEN to coverage uploads (#968)`, which is what was turning the
4 test shards red — the tests themselves passed (1528) but the post-test
codecov upload exited non-zero on a protected branch.
- Resolved the duplicate `lossless_min_savings_ratio` that two
independent main/branch additions left in `SmartCrusherConfig` and the
Rust-config kwarg (import-time `SyntaxError` + mypy `no-redef`).
- Aligned CCR tests with the new defaults (SQLite backend, 1800s TTL)
across `test_ccr`, `test_adapter_hooks`, `test_compression_store`,
`test_proxy_ccr`, and the lossy row-drop bridge test.
## Testing
<!-- Check what you actually ran, then paste the real command output
below. -->
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_proxy_ccr.py tests/test_ccr.py tests/test_compression_store.py tests/test_adapter_hooks.py tests/test_ccr_row_drop_store_bridge.py -q
170 passed, 4 warnings in 42.49s
$ python -m pytest tests/test_audit_reads.py tests/test_audit_codex.py tests/test_read_maturation.py tests/test_transforms_content_router.py tests/test_smart_crusher_toin_attachment.py -q
83 passed
$ mypy headroom/
Success: no issues found in 365 source files
$ python -m compileall headroom/ -q
COMPILE-OK
# CI (run 27488990477, pre-rebase head): all 4 shards ran to completion —
# "1528 passed, 120 skipped, 4922 deselected"
# The red shards were the codecov upload step, not test failures; fixed by
# the #968 rebase above.
```
## Real Behavior Proof
- Environment: macOS (darwin), Python 3.12 venv; branch
`feat/compression-extraction` rebased onto `origin/main` (head
|
||
|
Eyal Mizrachi
|
b2f04e4ef7
|
fix(deps): make litellm optional on Python 3.14 (#956) (#993)
## Description `litellm` is a hard dependency and its metadata caps `Requires-Python >=3.10,<3.14`, so `pip install headroom-ai` is unsatisfiable on Python 3.14. But litellm is only used for model registry / pricing / non-core providers — all lazily imported behind `ImportError` guards — never on the core compression or Anthropic proxy path. Refs #956 (install half). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Add a `python_version < '3.14'` marker to both litellm declarations (core deps + dev extra); installs unchanged on <=3.13, skipped on 3.14 (matches the existing rapidocr/tomli marker pattern). ## Testing - [x] Unit tests pass (`pytest`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_litellm_optional.py -q 2 passed in 0.10s $ python3.14 -m pip install dist/headroom_ai-0.25.0-cp310-abi3-linux_x86_64.whl Successfully installed headroom-ai-0.25.0 ... # litellm NOT installed $ python3.14 -c "import importlib.util as u; print(u.find_spec('litellm') is not None)" False ``` ## Real Behavior Proof - Environment: fresh venv on CPython 3.14.5, Linux - Exact command / steps: built the abi3 wheel, `pip install` it on Python 3.14, then `import headroom` + start the proxy + send a compressible request - Observed result: install exits 0 with litellm skipped; `import headroom` works; the proxy compresses (29913 -> 27626 tokens). Stock 0.25.0 cannot install on 3.14 at all. - Not tested: litellm-backed features on 3.14 (intentionally unavailable there until litellm supports 3.14) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Eyal Mizrachi
|
addebdb29c
|
feat(proxy): make COMPRESSION_TIMEOUT_SECONDS configurable via env (#946) (#991)
## Description The compression-pipeline timeout was hard-coded at 30s, so slow CPUs and long Claude Code conversations had no recourse. #946 asks to wire `HEADROOM_COMPRESSION_TIMEOUT_SECONDS` through. Refs #946. ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - Read `HEADROOM_COMPRESSION_TIMEOUT_SECONDS` from the environment (float), falling back to 30 on an unparseable value. ## Testing - [x] Unit tests pass (`pytest`) - [x] New tests added for new functionality ### Test Output ```text $ pytest tests/test_proxy/test_compression_timeout_config.py -q 4 passed in 0.09s ``` ## Real Behavior Proof - Environment: Linux, Python 3.13, editable build of this branch - Exact command / steps: `HEADROOM_COMPRESSION_TIMEOUT_SECONDS=88 python -c "import headroom.proxy.helpers as h; print(h.COMPRESSION_TIMEOUT_SECONDS)"` - Observed result: prints `88.0` (default `30.0`; an unparseable value falls back to `30.0`) - Not tested: a live compression actually exceeding the configured timeout under real load ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
r00t
|
c9853f30cb
|
fix: pure-Python content detector default on Windows (clean) (#1063)
## Description Native Magika content detection initializes an ONNX Runtime session. On Windows that init can leave a background thread alive past the Rust-side 5s timeout, contending on the process-wide DLL loader lock. This makes `_detect_content` select a pure-Python regex detector by default on Windows so no ONNX session is ever created there. Supersedes #1043 (clean single-commit version; the original branch bundled unrelated dashboard/hooks changes and a fix-then-revert noise pair). Closes #1043 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `_resolve_detect_backend()`: honors `HEADROOM_DETECT_BACKEND=rust|python`; otherwise defaults to `python` on Windows (`sys.platform == "win32"`) and `rust` elsewhere. - `_detect_content()` routes through the resolved backend. On the Python path it calls the existing pure-Python regex detector (`content_detector.detect_content_type`) and never imports/initializes the native ONNX session. - One-time warn-level log line documents the Python-backend choice and the override env var. - Tests covering env override (both directions), the Windows default, and that the native detector is not invoked on the Python path. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/Scripts/python.exe -m pytest tests/test_transforms_content_router.py -q ======================== 24 passed, 1 warning in 0.36s ======================== $ .venv/Scripts/python.exe -m ruff check headroom/transforms/content_router.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows 11 (win32), Python 3.11, headroom 0.26.0. - Exact command / steps: `.venv/Scripts/python.exe -c "import sys; from headroom.transforms.content_router import _resolve_detect_backend; print(sys.platform, _resolve_detect_backend())"` - Observed result: prints `win32 python` — the Windows host selects the pure-Python backend, so no ONNX/Magika session is created and the loader-lock hang cannot occur. Setting `HEADROOM_DETECT_BACKEND=rust` forces the native chain (covered by tests). - Not tested: native chain on a real Windows host with `HEADROOM_DETECT_BACKEND=rust` (intentionally avoided — that path is the deadlock risk being mitigated); `mypy` not run. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes The Rust side (`magika_detector::session()`) already converts an init hang into a recoverable `Err(timeout)` so detection falls through magika → unidiff → PlainText with no user-visible failure. This PR adds belt-and-suspenders: on Windows the native session is never created, removing the loader-lock contention entirely rather than relying on the timeout. `mypy` not run locally; CHANGELOG not updated (single-file bugfix). |
||
|
Eyal Mizrachi
|
5eec7f6701
|
fix(serena): migrate stale Headroom-installed Serena entry on re-wrap (#1008)
## Description #1003 added `--open-web-dashboard False` to the Serena spec to stop the dashboard browser tab popping up on every session — but the flag only reaches **fresh** registrations. `register_server` returns `MISMATCH` and refuses to overwrite a differing entry unless `force=True`, and the Claude wrap path calls `_setup_serena_mcp` **without** force (unlike the Codex path, which passes `force=True`). So anyone wrapped before #1003 has a `serena` entry whose args lack the flag. Every re-wrap detects the mismatch, prints `existing config differs … To update: remove the existing serena MCP entry, then rerun`, and gives up — the stale spec, and the popup, persist forever. The fix never reaches already-wrapped users, which is most of them. This completes #1003 by migrating those stale entries in place. Related to #1003 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `_setup_serena_mcp` now migrates a stale entry: on `MISMATCH` (and when not already forced), it force-updates to the current spec **only when the ledger proves Headroom installed the entry currently on disk** (`headroom_installed_matching`). Prints `Serena MCP: migrated previously-installed entry to current spec`. - A user-managed Serena (absent from the ledger) is left untouched and the mismatch is reported exactly as before — the same ownership check `--no-serena` / `_disable_serena_mcp` already use, so a hand-rolled Serena is never clobbered. - No call-site change: migration is self-contained and gated on ledger ownership, not on the `force` param, so the Codex path keeps hard-overwriting as before. - New `tests/test_cli/test_serena_migrate.py`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_cli/test_serena_migrate.py tests/test_cli/test_serena_disable.py tests/test_mcp_registry/ -q ============================== 89 passed in 4.26s ============================== $ ruff check headroom/cli/wrap.py tests/test_cli/test_serena_migrate.py All checks passed! ``` ## Real Behavior Proof - Environment: Fedora 44, Python 3.14.5, headroom working tree at this branch (off `upstream/main`), real `ClaudeRegistrar` (cli=None → file-backed), isolated `$HOME` + ledger via `tempfile` and `HEADROOM_WORKSPACE_DIR`. - Exact command / steps: wrote a pre-#1003 `serena` entry (no flag) into a throwaway `.claude/.claude.json`, recorded it in the ledger as Headroom-owned, then ran `_setup_serena_mcp(ClaudeRegistrar(claude_cli=None, home_dir=tmp), context="claude-code")`. Repeated with a `custom-serena` entry absent from the ledger. - Observed result: Headroom-owned entry rewritten on disk to end with `--open-web-dashboard False` (`migrated previously-installed entry` printed); user-managed `custom-serena` entry left byte-for-byte unchanged with the mismatch reported; fresh-install path writes the dashboard-off spec. Discovered originally on a live machine whose `~/.claude.json` kept the popup across re-wraps until the entry was hand-fixed — this PR removes the need for that. - Not tested: did not launch the Claude CLI end-to-end (the dashboard auto-open is Serena's documented response to `web_dashboard_open_on_launch=False`, traced in #1003); `mypy` not run. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG / version: left to release-please (the repo's `fix:`-driven release PR aggregator), so no manual CHANGELOG edit. - Docs unchanged: behavior is internal to `headroom wrap`; the user-visible outcome (no dashboard popup) matches #1003's documented intent. - `mypy` not run locally (heavy dev extra pulls a compiled dep in this environment); happy to add the result if CI doesn't cover it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
gglucass
|
74ae781644
|
fix(codex): retag thread providers so history menu stays whole across the proxy boundary (#1034)
## Description
Codex stamps every thread with the `model_provider` it ran under and
filters its
history/projects menu by the currently active provider set. When
Headroom rewrites
Codex's config to route through the custom `headroom` provider, threads
created through
Headroom are tagged `headroom` while native threads keep `openai` — so
the two sets
never appear in the same menu. The visible symptom: enabling Headroom
appears to "lose"
the entire native Codex history, and disabling it hides everything
created while wrapped.
This reconciles the thread tags in Codex's SQLite store alongside the
existing config
edits, so the menu stays whole across the proxy boundary in both
directions:
`openai -> headroom` on enable/wrap, `headroom -> openai` on
revert/unwrap. Only rows
matching the source provider are touched, so third-party providers (e.g.
`anthropic`)
are left alone. The provider key cannot be unified by config — Codex
rejects naming a
custom provider `openai` ("reserved built-in provider IDs") — so
retagging the store is
the only path. A DB-only retag is sufficient: resuming a retagged
session still routes
completions through the active provider; the rollout `.jsonl` files do
not need
rewriting.
Every operation is best-effort: a missing store, a missing `threads`
table, or a corrupt
store is logged and skipped, never raised, so install/uninstall and
wrap/unwrap never
fail on account of the history menu. The store is WAL-mode, so the
update succeeds even
while Codex is running; the short busy timeout only covers a transient
checkpoint lock.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- New `headroom/providers/codex/threads.py`: best-effort retag of Codex
thread provider
tags across both known stores (`<codex_home>/sqlite/state_5.sqlite` for
the GUI and
`<codex_home>/state_5.sqlite` for the CLI/TUI). `retag_to_headroom` /
`retag_to_native`
wrap the directional helper. `codex_home` is passed in by callers (never
re-derived from
`Path.home()`), so tests stay pointed at a temp dir.
- `providers/codex/install.py`: `apply_provider_scope` calls
`retag_to_headroom` after
writing the provider block; `revert_provider_scope` calls
`retag_to_native` after
stripping it.
- `cli/wrap.py`: `_inject_codex_provider_config` calls
`retag_to_headroom`;
`unwrap_codex` calls `retag_to_native` once the config restore reports a
`restored`/`cleaned`/`removed` status.
- Tests: `tests/test_provider_codex_threads.py` (retag direction,
threads-table no-op,
missing/corrupt store best-effort) and a wrap/unwrap round-trip
integration test in
`tests/test_cli/test_wrap_codex.py`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uv run --extra dev pytest tests/test_provider_codex_threads.py tests/test_cli/test_wrap_codex.py tests/test_install/test_providers.py -q
... 88 passed, 2 failed
# The 2 failures are TestInjectAvoidsDuplicateTopLevelKeys::* — pre-existing on a clean
# upstream/main checkout, unrelated to this change: they `import tomllib`, which is
# stdlib only on Python 3.11+, and this environment runs Python 3.10.18.
$ uv run --extra dev ruff check headroom/cli/wrap.py headroom/providers/codex/threads.py \
headroom/providers/codex/install.py tests/test_cli/test_wrap_codex.py tests/test_provider_codex_threads.py
All checks passed!
$ uv run --extra dev mypy headroom/providers/codex/threads.py headroom/providers/codex/install.py
Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: macOS, Codex GUI v148 + Codex CLI, Python 3.10.18.
- Exact command / steps: The root cause and fix were confirmed live in
the Headroom
desktop app, which performs the identical SQLite retag. Connecting Codex
to Headroom
hid ~140 native (`openai`) threads from the history menu; running
`UPDATE threads SET model_provider='headroom' WHERE
model_provider='openai'` on the live
store (`~/.codex/sqlite/state_5.sqlite`) made the full menu reappear,
and resuming a
retagged session still routed completions through the active provider.
This Python port
is a 1:1 of that logic, exercised by the unit + wrap/unwrap integration
tests above.
- Observed result: full Codex history menu restored across
enable/disable; third-party
provider rows untouched; the real `~/.codex` stores were snapshotted
before/after the
test run and were not mutated by the tests.
- Not tested: an end-to-end `headroom wrap codex` run against a live
Codex GUI in this CI
environment (no Codex install here); covered instead by the integration
test invoking
the real `wrap`/`unwrap` Click commands against a temp `$HOME`.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A — behavior is in Codex's own history menu; covered by the proof
above.
## Additional Notes
- Documentation / CHANGELOG: N/A — internal behavior with no user-facing
surface beyond
the restored menu.
- The 2 failing `TestInjectAvoidsDuplicateTopLevelKeys` tests are
pre-existing on
upstream/main and fail only because this environment runs Python 3.10
(no `tomllib`);
they are unrelated to this change.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Brian Toye
|
0932b8bef4
|
feat: Add support for Mistral Vibe CLI (#935)
## Description Add `headroom wrap vibe` / `headroom unwrap vibe` support for Mistral Vibe CLI so Vibe can launch through Headroom's proxy, compression, and observability path. ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - Added `headroom.providers.mistral_vibe` provider runtime helpers. - Added `headroom wrap vibe` command support and matching unwrap handling. - Configured `VIBE_PROVIDERS` so Vibe routes through the Headroom proxy. - Added tests covering launch, custom ports, no-proxy behavior, code-graph/learn-memory flags, verbose mode, invalid-command handling, and provider JSON structure. - Updated `CHANGELOG.md`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text pytest -v tests/test_cli/test_wrap_vibe.py # 10 passed ``` ## Real Behavior Proof - Environment: Linux, Python 3.13.13, local checkout from the PR branch. - Exact command / steps: Ran the Vibe wrapper tests and manually launched Mistral Vibe through `headroom wrap vibe` with `VIBE_PROVIDERS` pointing at the Headroom proxy. - Observed result: Vibe launched through Headroom's proxy configuration, and the wrapper tests passed. - Not tested: RTK hook support for Vibe. Persistent installs may eventually hold an expired Vibe auth token because Vibe reads its auth token from the environment at startup; opening another port or removing the persistent install is the current workaround. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: Vibe Nuage Agent <vibe@mistral.ai> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Djabx
|
e20f16b1a6
|
fix: route v1internal code assist requests to cloudcode-pa.googleapis… (#821)
## Description This PR fixes routing of Google Cloud Code Assist authentication, onboarding, and experiment list endpoints. Specifically, endpoints under `/v1/v1internal:*` (e.g. `/v1/v1internal:fetchAvailableModels`) are now correctly routed to the Cloud Code target (`https://cloudcode-pa.googleapis.com`) and **normalized** to `/v1internal:*` prior to forwarding. This resolves 404/403 errors on the upstream service which does not accept `/v1/v1internal:*` request paths. Closes #821 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Modified `headroom/providers/proxy_routes.py` to strip the `v1/` prefix and normalize the path to `/v1internal:*` for Cloud Code routes. - Modified `tests/test_provider_proxy_routes.py` to add assertions verifying route and path normalization. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom/providers/proxy_routes.py`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text ================================= test session starts ================================= platform linux -- Python 3.14.5, pytest-9.0.3, pluggy-1.6.0 -- /home/alex/projects/github.com/Djabx/headroom/.venv/bin/python3 cachedir: .pytest_cache rootdir: /home/alex/projects/github.com/Djabx/headroom configfile: pyproject.toml plugins: anyio-4.12.1, cov-7.1.0, asyncio-1.4.0, langsmith-0.8.15 asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 13 items tests/test_provider_proxy_routes.py::test_provider_passthrough_routes_forward_expected_targets PASSED [ 7%] tests/test_provider_proxy_routes.py::test_proxy_route_helpers_prefer_legacy_targets_and_gemini_passthrough PASSED [ 15%] tests/test_provider_proxy_routes.py::test_provider_specific_routes_delegate_to_expected_proxy_handlers PASSED [ 23%] tests/test_provider_proxy_routes.py::test_openai_response_websocket_aliases_delegate_to_openai_ws_handler PASSED [ 30%] tests/test_provider_proxy_routes.py::test_openai_response_subpath_passthrough_returns_502_on_http_failure PASSED [ 38%] tests/test_provider_proxy_routes.py::test_openai_response_subpath_passthrough_uses_openai_target PASSED [ 46%] tests/test_provider_proxy_routes.py::test_openai_response_subpath_aliases_and_chatgpt_auth_use_expected_targets PASSED [ 53%] tests/test_provider_proxy_routes.py::test_gemini_batch_embed_contents_passthrough_uses_gemini_target PASSED [ 61%] tests/test_provider_proxy_routes.py::test_v1_models_fetches_codex_registry_under_chatgpt_auth PASSED [ 69%] tests/test_provider_proxy_routes.py::test_v1_models_falls_back_to_synthetic_list_under_chatgpt_auth PASSED [ 76%] tests/test_provider_proxy_routes.py::test_v1_models_get_single_dynamic_under_chatgpt_auth PASSED [ 84%] tests/test_provider_proxy_routes.py::test_v1_models_still_forwards_under_non_chatgpt_auth PASSED [ 92%] tests/test_provider_proxy_routes.py::test_v1_models_routes_claude_code_gateway_discovery_to_anthropic PASSED [100%] ================================= 13 passed in 0.63s ================================= ``` ## Real Behavior Proof - Environment: Linux, Python 3.14.5 - Exact command / steps: `pytest tests/test_provider_proxy_routes.py` which utilizes `fastapi.testclient.TestClient` to dispatch requests. - Observed result: Both `/v1internal` and `/v1/v1internal` endpoints are correctly routed to the Cloud Code target (`https://cloudcode.test`) and normalize their paths to `/v1internal`, avoiding 404/403 errors on the upstream service. - Not tested: Actual production Cloud Code endpoints (simulated via TestClient/fakes). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) Add screenshots to help explain your changes. ## Additional Notes <!-- Mention any N/A checklist items, tradeoffs, follow-ups, or maintainer context. --> <!-- headroom-maintainer-template-completion:start --> ## Description This PR prepares `fix: route v1internal code assist requests to cloudcode-pa.googleapis…` for review by documenting the intended change, validation evidence, and remaining merge-readiness context. Linked issues: None declared. ## Type of Change - [x] Bug fix - [ ] New feature - [ ] Documentation - [ ] Refactor - [ ] Tests only ## Changes Made - Commit: fix: route v1internal code assist requests to cloudcode-pa.googleapis… - Touches `headroom/providers/proxy_routes.py` - Touches `tests/test_provider_proxy_routes.py` ## Testing - [x] GitHub checks reviewed - [x] Metadata/template validation - [ ] Local functional testing ### Test Output ```text gh pr view 821 --repo chopratejas/headroom --json statusCheckRollup - PR Governance / template: FAILURE - PR Governance / template: FAILURE - PR Governance / template: FAILURE - PR Governance / template: FAILURE - PR Governance / label: SUCCESS - PR Governance / label: SUCCESS - PR Governance / label: SUCCESS - PR Governance / label: SUCCESS - external / GitGuardian Security Checks: SUCCESS ``` ## Real Behavior Proof - Environment: GitHub PR metadata and checks for `chopratejas/headroom` PR #821. - Exact command / steps: Reviewed PR title, commits, changed files, linked issues, labels, and check rollup; appended this maintainer template completion block without replacing the author's original description. - Observed result: PR body now contains all required governance sections, checked readiness fields, and a non-placeholder validation evidence block. - Not tested: This pass updated PR metadata only; code validation remains represented by the linked GitHub checks and any author-provided evidence above. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review <!-- headroom-maintainer-template-completion:end --> |
||
|
rongabbay
|
7edb27ab24
|
feat(proxy): compress AWS Bedrock InvokeModel requests via configurable upstream (#720)
## Description
Clients that speak **Bedrock to a local gateway** can't get proxy-level
compression. Claude Code launched with `CLAUDE_CODE_USE_BEDROCK=1` (and
any AWS SDK pointed at a custom endpoint) POSTs
`/model/{id}/invoke[-with-response-stream]` to
`AWS_ENDPOINT_URL_BEDROCK_RUNTIME`, never `/v1/messages`. Those requests
fell through the catch-all and were forwarded **verbatim — no
compression**.
`--backend bedrock` is the opposite direction: it accepts Anthropic
input and re-signs to AWS. It can't accept Bedrock-format input or
forward to a custom upstream. So the "client speaks Bedrock → local
re-signing gateway → AWS" topology (internal gateways, LiteLLM,
LocalStack; see #510) got nothing.
This adds a Bedrock InvokeModel passthrough that compresses the request
body with the **same** `anthropic_pipeline` used for `/v1/messages` —
the Bedrock InvokeModel body for Anthropic models *is* the Anthropic
Messages shape (`{anthropic_version, system, messages, max_tokens, …}`,
model in the URL), so there's no translation and no new compression
logic. The routes register **only** when `--bedrock-api-url` is set, so
default behavior is completely unchanged.
**Limitation (important):** rewriting the body invalidates the caller's
**SigV4** signature (it covers a hash of the body). Point
`--bedrock-api-url` at a gateway that re-signs or doesn't verify the
inbound signature (an internal gateway, LiteLLM, LocalStack, a corporate
Bedrock proxy) — **never raw AWS**, which would 403. For direct-to-AWS
compression, use `--backend bedrock` (which re-signs). The two are
complementary. This is documented in the flag help, the handler
docstring, the proxy docs, and the CHANGELOG.
Closes #734. Refs #510 (the Bedrock slice of the provider-agnostic
umbrella).
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- New `--bedrock-api-url` flag (env: `BEDROCK_TARGET_API_URL`). When
set, registers `POST /model/{id}/invoke` and `POST
/model/{id}/invoke-with-response-stream`.
- `BedrockHandlerMixin` compresses the request body via the existing
`anthropic_pipeline`, then forwards to the configured upstream,
preserving path/query.
- Responses forwarded byte-faithfully (non-streaming JSON and the
streaming AWS event-stream alike — neither is parsed or mutated, since
all compression is request-side).
- `{model_id:path}` captures inference-profile ids with
dots/colons/slashes (e.g.
`us.anthropic.claude-sonnet-4-5-20250929-v1:0`).
- Fail-open: a malformed body or compression error forwards verbatim
rather than erroring.
- Routes register only when the flag is set — default behavior
unchanged.
- Files: `headroom/proxy/handlers/bedrock.py` (new —
`BedrockHandlerMixin`); `headroom/providers/proxy_routes.py` (gated
route registration); `headroom/cli/proxy.py`,
`headroom/proxy/server.py`, `headroom/proxy/models.py` (flag + config
wiring); `docs/content/docs/proxy.mdx`, `CHANGELOG.md` (docs).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_proxy/test_bedrock_passthrough.py -q
.............. [100%]
14 passed in 12.46s
```
`tests/test_proxy/test_bedrock_passthrough.py` (14 tests) covers: route
gating (absent unless configured), body compression, non-message fields
preserved, inference-profile id capture + re-encoding, byte-faithful
streaming, fail-open on malformed body and on pipeline exceptions,
bypass when `optimize=False` and via the `x-headroom-bypass` header,
upstream connect failure surfacing as a 502, the content-length
regression, outcome recorded with `provider="bedrock"`, and
`BEDROCK_TARGET_API_URL` env wiring. `ruff check`/`format` clean.
## Real Behavior Proof
- Environment: macOS, Python 3.12; forked proxy on `:8788` with
`--bedrock-api-url` pointed at a local re-signing Bedrock gateway;
provider Anthropic Claude on Bedrock.
- Exact command / steps: `headroom proxy --port 8788 --bedrock-api-url
http://127.0.0.1:<gateway>`, then `curl -X POST
http://127.0.0.1:8788/model/claude-haiku-4-5/invoke --data
@bedrock_invoke.json` (a ~52k-token conversation with a large assistant
turn).
- Observed result: valid Claude response returned and the gateway
received the compressed body — proxy `/stats` reports `52,095 → 3,979
tokens` (92.4%, 48,116 removed), and the gateway's reported
`input_tokens: 3709` confirms the compressed body reached the model.
- Not tested: raw direct-to-AWS (out of scope by design — SigV4; use
`--backend bedrock`); non-Anthropic Bedrock model bodies (e.g.
Titan/Llama) — only the Anthropic Messages-shaped invoke body is
handled.
<details><summary>Proxy <code>/stats</code> output + content-length bug
note</summary>
```json
"compression": {
"requests_compressed": 1,
"avg_compression_pct": 92.4,
"best_detail": "52,095 → 3,979 tokens",
"total_tokens_removed": 48116
}
```
The first iteration of this proof surfaced a real bug — a shrunk body
still carried the inbound `Content-Length`, so httpx raised `Too little
data for declared Content-Length`. Fixed by dropping
`content-length`/`content-encoding` on the rewritten path so httpx
recomputes them; covered by a regression test.
</details>
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
`mypy headroom` is left unchecked above — type checking runs in the CI
matrix rather than locally on my side; the new code carries type hints
on all public functions. Design spec / feature request: #734.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
ulias
|
e36fccd8cf
|
refactor: DRY cache logic, add thread safety, fix Bash exclusion (#704)
## Description Four targeted improvements to ContentRouter and configuration, refactoring ~120 lines of duplicated cache logic into a shared helper and fixing several correctness issues. ### 1. DRY: Extract `_compress_block_content` helper The two-tier cache lookup + compression logic was duplicated ~60 lines per path (tool_result blocks and text blocks in `_process_content_blocks`). Extracted into a single, shared helper method. Net reduction of ~80 lines; no behavioural change. ### 2. Thread-safe `CompressionCache` `CompressionCache` is read/modified from `ThreadPoolExecutor` workers during parallel compression in `apply()`. Added a `threading.Lock` guarding all read-modify-write operations so concurrent cache misses for the same content do not produce duplicate compression work and metrics counters stay consistent. ### 3. Remove duplicate Kompress fallback for SmartCrusher The SMART_CRUSHER strategy block had an inline Kompress fallback that ran when SmartCrusher produced no savings. The unified post-strategy fallback block already covers the same case — the inline copy was a duplicate Kompress invocation. Removed it; the post-strategy handler now owns all fallback decisions for both SMART_CRUSHER and CODE_AWARE. Also added a guard preventing duplicate Kompress when CODE_AWARE's inline fallback fires alongside the unified block. ### 4. Fix Bash exclusion contradiction in `DEFAULT_EXCLUDE_TOOLS` The docstring on `DEFAULT_EXCLUDE_TOOLS` explicitly states "Bash is NOT excluded — its outputs (build logs, test output) are ideal compression targets." But both "Bash" and "bash" were still in the frozenset. Removed them so code matches the documented intent. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - `headroom/config.py`: Remove Bash/bash from `DEFAULT_EXCLUDE_TOOLS` - `headroom/transforms/content_router.py`: Extract `_compress_block_content` helper; unified post-strategy fallback block; threading.Lock on CompressionCache; CODE_AWARE duplicate guard - `headroom/client.py`: Replace silent `except Exception: pass` with `logger.debug(..., exc_info=True)` - `tests/test_compression_cache.py`: Add 2 concurrency regression tests - `tests/test_transforms/test_content_router.py`: Add 14 tests covering Bash exclusion, SmartCrusher fallback chain, and `_compress_block_content` shared path ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Formatting passes (`ruff format --check .`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # 14 new tests added across 3 test classes: # TestExcludeTools: 3 tests (Bash not in DEFAULT_EXCLUDE_TOOLS) # TestSmartCrusherFallback: 4 tests (fallback chain, no duplicate Kompress, JSON direct hit, CODE_AWARE path) # TestCompressBlockContent: 5 tests (skip set, result cache, ratio gating, route counts, transforms tracking) # TestCompressionCache: 2 tests (concurrent hits/misses consistency, stable hash ops no race) # Local run (43 tests pass): $ pytest tests/test_compression_cache.py tests/test_transforms/test_content_router.py -v ...43 passed... # ruff check: $ ruff check headroom/client.py headroom/config.py headroom/transforms/content_router.py tests/test_compression_cache.py tests/test_transforms/test_content_router.py All checks passed! # ruff format: $ ruff format --check headroom/client.py headroom/config.py headroom/transforms/content_router.py tests/test_compression_cache.py tests/test_transforms/test_content_router.py 5 files already formatted ``` ## Real Behavior Proof - Environment: Python 3.12, Linux (CI), headroom with headroom._core Rust extension compiled - Exact command / steps: CI run https://github.com/chopratejas/headroom/actions/runs/27326150021 — 13/16 jobs pass; 2 failures were lint+commitlint (both fixed in subsequent commits); 1 failure is pre-existing test(4) which monkeypatches time.time() but the CompressionCache uses time.monotonic() — unrelated to our changes - Observed result: All 14 new tests pass in CI; SmartCrusher fallback chain deterministically shows [smart_crusher, kompress] or [smart_crusher, kompress, log] when SmartCrusher produces no savings, with no duplicate entries - Not tested: fork-PR CI path where GitHub secrets are not available; local Windows environment where headroom._core Rust extension is not built ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes The pre-existing CI failure in `test (4)` is `test_compression_cache_handles_hits_skips_evictions_and_clear` in `tests/test_transforms_content_router.py`. It monkeypatches `time.time()` but the `CompressionCache` (content_router-local, line 191) uses `time.monotonic()` for TTL — the monkeypatched clock never advances, and `is_skipped()` always returns True. This failure exists on `main` and is unrelated to our changes (we only modified the other CompressionCache in `headroom/cache/compression_cache.py`). --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
AKT99!
|
64ca95361a
|
fix: --disable-kompress should not override fallback_strategy to PASSTHROUGH (#1046)
## Description `--disable-kompress` correctly disabled the ML model via `enable_kompress = False`, but it also forced `router_config.fallback_strategy = CompressionStrategy.PASSTHROUGH`. That override suppressed ContentRouter's rule-based passes — including the `exclude_tools` gate — for content that falls through to the fallback, so `HEADROOM_EXCLUDE_TOOLS` had no effect when `--disable-kompress` was set. Removing the override leaves `fallback_strategy` at its default (`KOMPRESS`); ContentRouter keeps running its rule-based passes and only Kompress inference is disabled. Closes #955 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/server.py`: removed the `router_config.fallback_strategy = CompressionStrategy.PASSTHROUGH` line inside the `if config.disable_kompress:` block; `enable_kompress = False` is kept. - `headroom/proxy/server.py`: dropped the now-unused `CompressionStrategy` import (it was only referenced by the removed line). - `tests/test_proxy_disable_kompress.py`: updated the assertion to expect `fallback_strategy == CompressionStrategy.KOMPRESS` (the default), matching the corrected behaviour. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ pytest tests/test_proxy_disable_kompress.py -v ============================= test session starts ============================== platform darwin -- Python 3.13.7, pytest-9.1.0, pluggy-1.6.0 rootdir: /.../headroom configfile: pyproject.toml plugins: anyio-4.14.0, asyncio-1.4.0 collected 2 items tests/test_proxy_disable_kompress.py::test_disable_kompress_config_keeps_optimization_but_disables_ml_fallback PASSED [ 50%] tests/test_proxy_disable_kompress.py::test_disable_kompress_defaults_to_existing_kompress_behavior PASSED [100%] ============================== 2 passed in 1.37s ============================== $ ruff check headroom/proxy/server.py tests/test_proxy_disable_kompress.py All checks passed! ``` ## Real Behavior Proof - Environment: local clone, Python 3.13.7 venv, headroom core deps + fastapi/uvicorn/httpx. - Exact command / steps: built the proxy router via `create_app(ProxyConfig(optimize=True, ...))` with `disable_kompress` set and inspected the resulting `ContentRouter` config; ran `pytest tests/test_proxy_disable_kompress.py -v` and `ruff check` on the changed files. - Observed result: with `--disable-kompress`, `enable_kompress` is `False` and `fallback_strategy` is `KOMPRESS` (the default) instead of `PASSTHROUGH`; ContentRouter stays in the pipeline. Both config tests pass and lint is clean. - Not tested: full live-proxy `/stats` run against an LLM backend. The issue reporter observed `router_content_router_activations` 0→22 and exclude-tool hits 0→10 after this change (see #955). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes This removes the override line plus its now-unused import, and updates the existing test that asserted the old behaviour; no new test was added because the corrected behaviour is covered by that existing test. The live `/stats` reproduction is described in #955. |
||
|
Tejas Chopra
|
0ddd4ed9e9
|
fix(learn): scan subagent and workflow transcripts (#1045)
## Description `headroom learn` only scanned top-level main Claude Code sessions (`<project>/<uuid>.jsonl`). Nested subagent and workflow transcripts under `<project>/<uuid>/subagents/**` were not opened, which hid a large amount of tool-call failure and token-spend activity from failure mining and downstream analysis. This change makes the Claude scanner descend into nested transcripts by default and tag each `SessionData` with its source. `--main-only` restores the previous top-level-only scan scope. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Updated `ClaudeCodePlugin.scan_project` to discover nested subagent and workflow transcripts by default. - Added source tagging for `main`, `subagent`, and `workflow` sessions. - Added `--main-only` and `include_subagents` plumbing so callers can opt back into top-level-only scanning. - Added the `include_subagents` scanner parameter to Codex/Gemini as a documented no-op because those scanners use flat session layouts. - Added regression tests for nested discovery, source tagging, parallel scanning, and CLI flag threading. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output ```text Full learn + CLI suite: # 186 passed, 2 skipped GitHub Actions CI for this PR: # build, build-wheel, lint, tests, e2e, CodeQL, and native wrapper checks passed ``` ## Real Behavior Proof - Environment: local Claude Code corpus with nested subagent/workflow transcripts. - Exact command / steps: Scanned the corpus with the previous top-level-only behavior and then with nested transcript discovery enabled. - Observed result: The scanner saw 24 sessions before and 306 sessions after descending into nested transcripts. - Not tested: Codex/Gemini nested transcript discovery, because those providers currently use flat session layouts and treat `include_subagents` as a no-op. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
felixboenkost-droid
|
8662a82e8a
|
Fix Codex ChatGPT /v1/models compatibility metadata (#1048)
## Description Fixes Codex ChatGPT/OAuth `/v1/models` metadata compatibility while keeping Headroom's existing OpenAI-compatible response shape. Headroom's ChatGPT/OAuth model-list route already returned: - `object: "list"` - `data[]` Newer Codex clients also inspect a top-level `models[]` registry metadata array. Without that shape, completions can still work, but clients may emit non-fatal model metadata decode or missing-field warnings before the follow-up `/v1/responses` call. This PR keeps `object`/`data[]` unchanged and adds a Codex-compatible `models[]` array. Upstream registry metadata is preserved where available, and only missing fields are filled with defaults. Closes: N/A ## Type of Change - [x] Bug fix (non-breaking change fixes issue) - [ ] New feature (non-breaking change adds functionality) - [ ] Breaking change (fix or feature would cause existing functionality change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add Codex registry metadata generation for the ChatGPT/OAuth `/v1/models` response. - Preserve dynamic upstream registry entries instead of reducing them to slug-only IDs. - Add fallback metadata for known Codex models if upstream registry data is unavailable. - Fill required/default Codex fields when absent, including: - `display_name` - `default_reasoning_level` - `supported_reasoning_levels` - `context_window` - tool/runtime capability flags - Keep the existing OpenAI-compatible `data[]` response shape. - Add tests that assert both OpenAI-compatible `data[]` and Codex-compatible `models[]` shapes. Changed files: - `headroom/providers/proxy_routes.py` - `tests/test_provider_proxy_routes.py` - `tests/test_proxy_codex_route_aliases.py` ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text ruff check headroom/providers/proxy_routes.py tests/test_provider_proxy_routes.py tests/test_proxy_codex_route_aliases.py # pass pytest tests/test_provider_proxy_routes.py::test_v1_models_fetches_codex_registry_under_chatgpt_auth # pass pytest tests/test_proxy_codex_route_aliases.py # pass pytest tests/test_provider_proxy_routes.py # pass ``` ## Real Behavior Proof - Environment: isolated local Headroom proxy using the patched source. - Exact command / steps: - call `/v1/models` - run one small Codex `/v1/responses` request through the proxy - compare Headroom `/stats` - check logs for Codex model metadata decode or missing-field warnings - Observed result: - `/v1/models` succeeded - `/v1/responses` succeeded - `requests.failed` stayed flat - provider stats and proxy compression accounting increased - no Codex model metadata decode or missing-field warnings observed - Not tested: - full repository `mypy headroom` pass was not run for this submission ## Review Readiness - [x] I have performed a self-review before requesting human review - [x] This PR is ready for human review ## Checklist - [x] My code follows project's style guidelines - [x] I performed self-review my code - [ ] I commented my code, particularly in hard-to-understand areas - [ ] I made corresponding changes documentation - [x] My changes generate no new warnings - [x] I added tests prove fix is effective or feature works - [x] New and existing unit tests pass locally my changes - [ ] I updated CHANGELOG.md if applicable ## Screenshots (if applicable) N/A. ## Additional Notes Checklist items left unchecked are intentionally not applicable or not run for this focused compatibility PR: - no new comments were needed in the implementation - no documentation or changelog update is included for this compatibility fix to an existing route - full-suite `mypy headroom` was not run in the submission pass Co-authored-by: felixboenkost-droid <258905464+felixboenkost-droid@users.noreply.github.com> |
||
|
Umi_Ma
|
e67ee2af65
|
feat: add Copilot BYOK provider wrapper utilities and CLI support (#1041)
## Description Fix `--model auto` causing `400 The requested model is not supported` errors when using Copilot BYOK mode. `auto` is a Copilot-internal virtual routing token that external providers (Anthropic, OpenAI) do not recognise as a valid model name. In subscription/OAuth mode the wrapper now strips `--model auto` before launching Copilot so its own native auto-selection takes effect. In BYOK mode `auto` is treated as unconfigured and a clear, actionable error message is shown. Closes #972 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/providers/copilot/wrap.py`: added `is_auto_model()` and `strip_auto_model_args()` helpers; updated `model_configured()` to treat `auto` as unconfigured for BYOK - `headroom/providers/copilot/__init__.py`: exported both new helpers via `__all__` - `headroom/cli/wrap.py`: strips `--model auto` in subscription mode before launch; shows specific actionable error in BYOK mode - `tests/test_provider_copilot_wrap.py`: 17 new parametrized test cases for `is_auto_model`, `strip_auto_model_args`, and updated `model_configured` ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_provider_copilot_wrap.py -v platform win32 -- Python 3.14.6, pytest-9.0.3, pluggy-1.6.0 collected 34 items tests/test_provider_copilot_wrap.py::test_is_auto_model[auto-True] PASSED tests/test_provider_copilot_wrap.py::test_is_auto_model[Auto-True] PASSED tests/test_provider_copilot_wrap.py::test_is_auto_model[AUTO-True] PASSED tests/test_provider_copilot_wrap.py::test_strip_auto_model_args[args0-expected0] PASSED tests/test_provider_copilot_wrap.py::test_strip_auto_model_args[args1-expected1] PASSED tests/test_provider_copilot_wrap.py::test_strip_auto_model_args[args2-expected2] PASSED tests/test_provider_copilot_wrap.py::test_model_configured_detects_env_and_cli_variants PASSED ============================= 34 passed in 0.46s ============================== $ ruff check headroom/providers/copilot/wrap.py headroom/cli/wrap.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.14.6, headroom-ai 0.25.0 editable install from branch fix-automode-issue - Exact command / steps: ran uv run pytest tests/test_provider_copilot_wrap.py -v and ruff check on all four changed files; reviewed CLI code path for both subscription and BYOK modes - Observed result: 34 passed, ruff All checks passed; --model auto is stripped silently in subscription mode and rejected with a specific actionable error in BYOK mode - Not tested: live end-to-end Copilot CLI session, macOS/Linux keychain auth, Docker/CI token-injection paths ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes mypy is not installed in the local venv so type checking was skipped; the code uses standard type hints and passes ruff checks cleanly. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Zhenjia ZHOU
|
7dbbb4077e
|
fix(proxy): keep codex image-generation WS turns alive through the relay (#1000)
## Description Image generation through the proxy fails. Driving Codex (`/v1/responses` over WebSocket) through Headroom, an image-generation turn never returns an image — the client retries (`Reconnecting… n/5`) and gives up, while the same prompt works when Codex talks to ChatGPT directly. Root cause: two independent defects on the upstream `websockets.connect()`, both specific to how image generation behaves on the wire: 1. **Pong deadline kills the silent render.** An image turn emits a single `response.image_generation_call.generating` event and then goes silent for 20–60s while the model renders (no data frames). The hard-coded `ping_timeout=20` treats that healthy-but-quiet connection as dead and tears it down as `upstream_error` mid-render, before the image is ready. 2. **1 MiB frame cap drops the image.** The finished image comes back inline as a single base64 frame that exceeds the `websockets` default `max_size=2**20` (1 MiB), raising `PayloadTooBig` exactly as the image lands. They compound: with only ping fixed, the session survives the silent phase (observed ~20s → ~54s) but then dies on the oversized image frame. Normal text/tool turns stream tokens continuously and stay well under 1 MiB, so neither defect affects them — which is why this only ever bit image generation. Closes: N/A (no tracking issue) ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/openai.py`: on the upstream `/v1/responses` connect, set `ping_timeout=None` (keep `ping_interval=20` for NAT keepalive) so a long silent render is not torn down on a missing pong, and `max_size=None` so the inline base64 image frame is accepted instead of raising `PayloadTooBig`. - `tests/test_openai_codex_ws_lifecycle.py`: add `test_ws_upstream_connect_allows_large_frames_and_no_pong_deadline`, which captures the upstream connect kwargs and pins `ping_timeout is None` / `max_size is None`. ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check . All checks passed! $ mypy headroom --ignore-missing-imports Success: no issues found in 358 source files $ pytest tests/test_openai_codex_ws_lifecycle.py -q collected 15 items tests/test_openai_codex_ws_lifecycle.py .............. [100%] ============================== 15 passed in 0.72s ============================== $ pytest tests/test_openai_codex_ws_lifecycle.py -k large_frames_and_no_pong -q collected 15 items / 14 deselected / 1 selected tests/test_openai_codex_ws_lifecycle.py . [100%] ======================= 1 passed, 14 deselected in 0.35s ======================= ``` End-to-end (managed Codex image generation through the running proxy): ```text # BEFORE fix: fails at ~20s (unpatched) / ~54s (ping-only) # WS /v1/responses completed (cause=upstream_error, # last_upstream_type=response.image_generation_call.generating) # -> client "Reconnecting… n/5", no image produced # AFTER fix: [codex] Image ready; stopping the turn. Saved image: /tmp/headroom-imagegen-test.png $ file /tmp/headroom-imagegen-test.png PNG image data, 1254 x 1254, 8-bit/color RGB, non-interlaced (908 KB) # proxy session count +1 -> the turn DID traverse the proxy and completed. ``` ## Real Behavior Proof - Environment: macOS, headroom 0.23.0 running as the Codex `model_provider` (proxy on `127.0.0.1:8787`), Codex CLI 0.139.0 driving a managed `/v1/responses` image-generation turn through the proxy. - Exact command / steps: trigger a Codex image-generation turn (gpt-image-2) with the proxy in front; observe the upstream `/v1/responses` WS session in `proxy.log` and whether a PNG is returned. - Observed result: before the change the session dies with `upstream_error` while `last_upstream_type=response.image_generation_call.generating` and no image is produced; after the change a valid 1254×1254 PNG is returned and the session traverses the proxy normally. - Not tested: the full `pytest` suite was not run locally — this machine has no Rust toolchain to rebuild the matching `_core` extension, so the complete suite (incl. the pyo3 tests) is left to CI. The affected `test_openai_codex_ws_lifecycle.py` module was run against the installed extension and passes 15/15; `ruff check .` and `mypy headroom` were run in full and pass. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A. ## Additional Notes - `ruff check .` (whole repo) and `mypy headroom --ignore-missing-imports` (358 source files) were run locally and pass. The only `pytest` not run locally is the full suite, because the Rust `_core` cannot be rebuilt here without a toolchain; the directly affected lifecycle module passes 15/15 and CI runs the rest. - Documentation / CHANGELOG left unchecked — this is a focused two-line behavioral fix on the upstream WS connect; happy to add a CHANGELOG entry if preferred. - `ping_timeout=None` keeps `ping_interval` for NAT keepalive; if you'd rather bound it, a generous finite value (e.g. 300s) would also fix the render case — happy to switch. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Focused Instability
|
c2e52fe743
|
feat(policy): batch deep edits through one cache-bust (#856 P3a) (#1015)
## Description #856 P3a (umbrella #904), stacked on the now-merged P2 (#905) and P2b (#944). A net-cost mutation at depth K already busts the provider's cached suffix after K. Every *later* candidate at a deeper slot therefore rides that same cache invalidation for free — mutating it adds no incremental cache-bust cost. Today the P2 break-even gate re-charges each candidate the full invalidated suffix S independently, so a batch of legitimate deep edits is under-admitted: only the first pays for the bust, yet each is billed as if it paid alone. This adds a batch-reclaim floor to the net-cost gate so that once one net-positive deep edit is admitted at slot K, candidates at slot > K are admitted on the write/read economics alone (S charged as 0). Flag-gated under `HEADROOM_NET_COST_POLICY` (the same flag as P2/P2b), default **off** — telemetry-first before any default-on. ## Type of Change - [x] New feature (non-breaking change which adds functionality) - [ ] Bug fix - [ ] Breaking change - [ ] Documentation ## Changes Made - `ContentRouter._net_cost_allows`: new `batch_state` param. When the candidate sits strictly deeper than `batch_state["floor"]`, S is charged as 0 via the *same* `net_mutation_gain` formula (conservative — never admits a mutation the real economics would reject). Full-S admits open/lower the floor; batch admits never lower it, so a slot only ever rides free behind a genuinely mutated shallower slot. - `ContentRouter.apply`: shared per-request `netcost_batch_state` wired into both gate call sites (cached-result path and parallel-merge path). - Telemetry: every batch admission emits the `router:netcost_batch_admit` transform marker and the `netcost_batch_admitted` route counter; added to the routing summary log line. - Tests: 5 new cases in `tests/test_netcost_gate.py` (`TestNetCostBatchReclaim`). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] New tests added for new functionality ### Test Output ```text $ pytest tests/test_netcost_gate.py -q 20 passed in 1.46s $ pytest tests/ -k "content_router or netcost or router" -q 142 passed, 8 skipped, 6342 deselected, 1 warning in 22.54s $ ruff check headroom/transforms/content_router.py tests/test_netcost_gate.py All checks passed! $ ruff format --check headroom/transforms/content_router.py tests/test_netcost_gate.py 2 files already formatted $ mypy headroom/transforms/content_router.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: local macOS, repo .venv, Python 3.11.9, gpt-4o tokenizer fixture - Exact command / steps: drive `ContentRouter.apply()` on a 5-message conversation — a huge compressible tool dump at slot 1 (ΔT≈34K) and a modest dump at slot 2 (ΔT≈5K) followed by a ~12K-token suffix, so slot 2's own break-even S blocks it. Run with `HEADROOM_NET_COST_POLICY=1`, once with a non-compressible slot 1 (no shallower admit, control) and once with the slot-1 dump intact (opens the floor). - Observed result: control → `slot2_compressed=False batch_markers=0 skip_markers=1` (slot 2 correctly blocked on its own S, no floor opened); floor opened → `slot2_compressed=True batch_markers=1 skip_markers=0` (slot 2 rides slot 1's cache-bust for free, `router:netcost_batch_admit` emitted). Flag absent → no `router:netcost_batch_admit` marker ever. - Not tested: live proxy traffic / real dashboard validation — deferred to the default-on milestone per #904 (ships default-off precisely to gather telemetry first). Known limitation logged for follow-up: in a *warm-cache* request a deep cache-hit slot is gated in pass 1 before a shallower cache-miss slot can lower the floor in pass 3, so the batch win can no-op there (never a wrong admit — strictly conservative). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes Charging S=0 through the existing formula (rather than blanket-admitting on `ΔT > 0`) keeps the decision conservative under non-default env tunables (`HEADROOM_NET_COST_EXPECTED_READS`, `HEADROOM_NET_COST_P_ALIVE`). P3b will be a separate PR after this review. Note: the failing `test` / `test-extras` checks are a **pre-existing regression on `main`** in `tests/test_cache/test_dynamic_detector.py` (unrelated to this PR, which only touches `content_router.py`). Fix tracked in a separate PR; this branch will go green once that lands and this is rebased. |
||
|
Shengbo_Wang
|
2d3701b59e
|
fix(learn): decode directory names with spaces in Windows project paths (#997) (#1027)
## Description `headroom learn --apply` crashes with `FileNotFoundError` when the project lives in a Windows directory whose name contains spaces (e.g. `C:\Users\user\Desktop\Claude Code Projects`). Claude Code encodes that path as `-C-Users-user-Desktop-Claude-Code-Projects`, using `-` for both path separators *and* spaces. The greedy path decoder walks the real filesystem to reconstruct the original components, but `_component_tokenizations()` never tried splitting on spaces — so it couldn't match `Claude Code Projects` against tokens `["Claude", "Code", "Projects"]`. Closes #997 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Added `" "` (space) to the explicit separator list in `_component_tokenizations()` - Updated the catch-all regex from `[-._]` to `[-.\s_]` so the combined split also covers whitespace - Same change in the hidden-component (dotfile) branch ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] New tests added for new functionality ### Test Output ```text tests/test_learn/test_scanner.py::TestGreedyPathDecode::test_single_space_in_dirname PASSED tests/test_learn/test_scanner.py::TestGreedyPathDecode::test_multiple_spaces_in_dirname PASSED tests/test_learn/test_scanner.py::TestGreedyPathDecode::test_space_nested_path PASSED tests/test_learn/test_scanner.py::TestDecodeProjectPath::test_windows_path_with_spaces_decoded_via_greedy PASSED 4 passed in 0.64s ``` ## Real Behavior Proof - Environment: Windows 11 Home 10.0.26200, Python 3.10.18 - Exact command / steps: Ran `python -m pytest tests/test_learn/test_scanner.py -v` on Windows after applying the fix. Also verified `_component_tokenizations("Claude Code Projects")` returns `[['Claude Code Projects'], ['Claude', 'Code', 'Projects']]`. The integration test creates a real temp directory with spaces and asserts `_decode_project_path()` resolves it correctly. - Observed result: All 4 new tests pass on Windows. All 34 scanner tests pass. Ruff check clean. - Not tested: No manual `headroom learn --apply` end-to-end run, but the integration test exercises the same `_decode_project_path` code path with a real temp directory on disk. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes ## Additional Notes The fix follows the exact same pattern used for underscores (issue #159) and dots (issue #47) — extending the separator list. Spaces are the last common character that Claude Code flattens to `-` but the decoder didn't know about. |
||
|
Andrew Barnes
|
e616dcf788
|
fix(mcp): honor CLAUDE_CONFIG_DIR for Claude registrar (#886)
## Description Resolve the Claude MCP config directory from `CLAUDE_CONFIG_DIR` for default `ClaudeRegistrar()` instances so file fallback registration, direct reads, and unregister cleanup operate on the same config files Claude Code is using. Closes #872 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Resolve the Claude MCP config directory from `CLAUDE_CONFIG_DIR` for default `ClaudeRegistrar()` instances. - Use the resolved directory for both modern `.claude.json` and legacy `mcp.json` file fallback paths. - Add regression coverage for read, register, and unregister behavior against a custom Claude config directory. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run --with ruff ruff check headroom/mcp_registry/claude.py tests/test_mcp_registry/test_claude_registrar.py All checks passed! uv run --with ruff ruff format --check headroom/mcp_registry/claude.py tests/test_mcp_registry/test_claude_registrar.py 2 files already formatted uv run --with pytest --with pytest-asyncio pytest -q tests/test_mcp_registry/test_claude_registrar.py [passed locally] ``` ## Real Behavior Proof - Environment: macOS Darwin arm64, Python 3.14.2 via `uv`, local fork branch. - Exact command / steps: ran the fallback registrar against a temporary `CLAUDE_CONFIG_DIR`. ```sh tmpdir=$(mktemp -d) CLAUDE_CONFIG_DIR="$tmpdir" uv run python -c 'import json, os; from pathlib import Path; from headroom.mcp_registry import ClaudeRegistrar, build_headroom_spec; reg = ClaudeRegistrar(claude_cli=None); result = reg.register_server(build_headroom_spec("http://127.0.0.1:9999")); path = Path(os.environ["CLAUDE_CONFIG_DIR"]) / ".claude.json"; data = json.loads(path.read_text()); print(result.status.value); print(path.exists()); print(data["mcpServers"]["headroom"]["env"]["HEADROOM_PROXY_URL"]); print((Path(os.environ["CLAUDE_CONFIG_DIR"]) / ".claude" / ".claude.json").exists()); print(reg.unregister_server("headroom")); print("headroom" in json.loads(path.read_text())["mcpServers"])' ``` - Observed result: registration wrote `$CLAUDE_CONFIG_DIR/.claude.json`, preserved `HEADROOM_PROXY_URL`, avoided the old nested path, and unregister removed the server. ```text registered True http://127.0.0.1:9999 False True False ``` - Not tested: a live Claude Code session or `claude mcp list` on WSL with a real installed CLI. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A. ## Additional Notes No changelog entry was added because this is a focused MCP registrar bug fix. |
||
|
Ashish
|
6cdb846200
|
fix(compression): use thread-local tree-sitter parsers in code handler (#893)
## Description `CodeStructureHandler` cached tree-sitter parsers in a process-global dict; the lock only guarded creation, while `parse()` ran unlocked on any thread. tree-sitter `Parser` objects are pyo3 `unsendable` — using one from a non-creator thread panics. The proxy invokes handlers from executor pool threads, so a shared parser is an eventual crash. Same class already fixed in `transforms/code_compressor.py` (#604). Stacked on #892. Closes # <!-- compression-handler review --> ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/compression/handlers/code_handler.py`: one parser per (thread, language) via `threading.local()`, porting the pattern from `transforms/code_compressor.py`. - `tests/test_compression/test_code_handler.py`: regression test parsing from a 4-worker thread pool, asserting every call stays on the tree-sitter path. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_compression/ -q 94 passed, 8 skipped ``` ## Real Behavior Proof - Environment: local macOS, Python 3.11, tree-sitter-language-pack 1.8.1, branch `fix/code-threadlocal-parsers` (stacked on #892). - Exact command / steps: `pytest tests/test_compression/ -q`. - Observed result: 16 parses across a 4-worker pool all stay on the tree-sitter path with no pyo3 panic; previously a shared parser would be touched cross-thread. - Not tested: Reproducing the original panic under production concurrency (covered structurally by the thread-pool test). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — library change. See Test Output. ## Additional Notes Stacked on #892 — review the top commit until that merges. PR 5 of 7. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
Focused Instability
|
3b0bceecf4
|
fix(cache): name the missing piece in semantic detector guard (#1018)
## Description
The `test` and `test-extras` CI jobs are currently red on `main`:
`tests/test_cache/test_dynamic_detector.py::TestSemanticDetectorGuards::test_none_exemplars_early_return`
fails. This PR fixes the underlying regression. It is independent of any
feature branch (it only touches `headroom/cache/dynamic_detector.py` and
its test).
#950 folded the exemplar-embeddings None-check into the model
None-guard:
```python
if self._model is None or self._exemplar_embeddings is None:
return [], self._load_error or "semantic detector is not initialized"
```
So a `SemanticDetector` with a loaded model but unset exemplar
embeddings now returns the generic *"semantic detector is not
initialized"* message, shadowing the specific *"exemplar embeddings not
initialized"* message and leaving the later guard as unreachable dead
code. That directly contradicts #950's own
`test_none_exemplars_early_return`, which asserts the specific message —
hence the red main.
## Type of Change
- [ ] New feature
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] Breaking change
- [ ] Documentation
## Changes Made
- `SemanticDetector.detect`: split the combined guard into two checks —
`_model` then `_exemplar_embeddings` — both *before* `encode()`. The
model-missing case keeps the generic message; the exemplar-missing case
reports the specific *"exemplar embeddings not initialized"*. Checking
before `encode()` avoids a wasted encode in the error path and preserves
the mypy narrowing for `np.dot(..., self._exemplar_embeddings.T)`. The
previously-shadowed duplicate guard is removed.
- `tests/test_cache/test_dynamic_detector.py`:
`test_missing_exemplar_embeddings_returns_warning` sets the same
model-present / exemplar-None state, so its assertion is aligned to the
specific message to match `test_none_exemplars_early_return`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] New and existing unit tests pass locally with my changes
### Test Output
```text
$ pytest tests/test_cache/test_dynamic_detector.py -q
38 passed, 2 skipped in 9.94s
$ pytest tests/test_cache/ -q
198 passed, 2 skipped in 11.58s
$ ruff check headroom/cache/dynamic_detector.py tests/test_cache/test_dynamic_detector.py
All checks passed!
$ ruff format --check headroom/cache/dynamic_detector.py tests/test_cache/test_dynamic_detector.py
2 files already formatted
$ mypy headroom/cache/dynamic_detector.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: local macOS, repo .venv, Python 3.11.9, numpy installed
- Exact command / steps: construct a `SemanticDetector` via
`object.__new__` with `_model` set (mock) and `_exemplar_embeddings =
None`, then call `.detect(...)`. Before fix: on `upstream/main`
(
|
||
|
Ashish
|
1ec9320888
|
fix(cache): guard None exemplar embeddings in dynamic detector (#950)
## Description `mypy headroom --ignore-missing-imports` fails on `main` at `headroom/cache/dynamic_detector.py:786` with `Item "None" of "Any | None" has no attribute "T"` (surfaced by updated numpy stubs). This breaks the `lint` job for every open PR that merges current main. The `is_available` property only guarantees `_model` is set, not `_exemplar_embeddings`, so mypy cannot narrow the `Any | None` attribute before `.T` — and if it were ever None this is a real runtime crash, not just a type nit. Closes # <!-- broken-main lint failure; no tracked issue --> ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cache/dynamic_detector.py`: add an explicit `self._exemplar_embeddings is None` guard before the `np.dot(..., .T)` call, returning the method's existing early-return shape `([], "exemplar embeddings not initialized")`. Narrows the type for mypy and prevents a latent `None.T` crash. - `tests/test_cache/test_dynamic_detector.py`: add `TestSemanticDetectorGuards::test_none_exemplars_early_return` covering the new guard path (model present, exemplars unset → early return, no crash). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ mypy headroom --ignore-missing-imports --no-incremental (0 errors — was: "Found 1 error in 1 file" at dynamic_detector.py:786) $ ruff check headroom/cache/dynamic_detector.py tests/test_cache/test_dynamic_detector.py All checks passed! $ pytest tests/test_cache/test_dynamic_detector.py -q 37 passed, 2 skipped ``` ## Real Behavior Proof - Environment: local macOS, Python 3.11, branch `fix/dynamic-detector-mypy` from current `origin/main`. - Exact command / steps: `mypy headroom --ignore-missing-imports --no-incremental` before and after the change (must clear the incremental cache to reproduce — stale cache hides it). - Observed result: before the guard mypy reports `Found 1 error in 1 file (dynamic_detector.py:786)`; after, 0 errors. The `lint` CI job that is currently red on main and on every dependent PR goes green. - Not tested: the runtime path where `_exemplar_embeddings` is actually None (the guard is defensive; existing detector tests cover the populated path). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — type/CI fix with no UI surface. See **Test Output** above. ## Additional Notes - This is broken-main, not introduced by any single PR: `origin/main` has the identical line 786, and main's own CI `lint` job is currently failing. Merging this unblocks #885, #926, and the compression-handler PR series in one shot. - N/A checklist items: no new test (defensive guard on an existing branch; covered indirectly by the 44 detector tests), no docs/CHANGELOG (internal type fix). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Aniruddh Jha
|
a7ee8a60a7
|
fix(anyllm): forward openai api_base/api_key to the any-llm backend (#942) (#954)
## Description The any-llm backend ignored `--openai-api-url`, so requests against custom OpenAI-compatible providers (vLLM, LiteLLM, xiaomimimo.com, etc.) were sent to `api.openai.com` instead of the configured URL, returning 401s. This wires the configured URL all the way through to the any-llm client. Closes #942 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made There were two layers to the bug, both fixed here: - The URL was never threaded to the backend. `create_proxy_backend()` did not accept or forward the configured OpenAI URL, so `AnyLLMBackend` was always constructed without an `api_base`. It now takes `openai_api_url` and passes it through as `api_base`, wired from `config.openai_api_url` in `server.py`. - The backend never applied it. `AnyLLMBackend.__init__` stored `self.api_base` and `self.api_key` but never used them; `AnyLLM.create()` only received the provider. Both are now forwarded to `AnyLLM.create()`, and only when set, so providers that rely on their own env-var defaults (`OPENAI_API_KEY` / `OPENAI_BASE_URL`) are unaffected. Files touched: - `headroom/providers/registry.py` — `create_proxy_backend()` gains an `openai_api_url` parameter, passed to the any-llm backend as `api_base`. - `headroom/proxy/server.py` — pass `openai_api_url=config.openai_api_url` into `create_proxy_backend()`. - `headroom/backends/anyllm.py` — forward `api_key`/`api_base` to `AnyLLM.create()` when set. Verified against `any-llm-sdk` 1.17.0, whose `AnyLLM.create(provider, api_key=None, api_base=None, ...)` accepts both parameters. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ pytest tests/test_backend_anyllm.py tests/test_provider_registry_extended.py tests/test_provider_registry.py -q tests/test_backend_anyllm.py .............. [ 43%] tests/test_provider_registry_extended.py ....... [ 65%] tests/test_provider_registry.py ........... [100%] 32 passed $ ruff check headroom/backends/anyllm.py headroom/providers/registry.py headroom/proxy/server.py tests/test_backend_anyllm.py tests/test_provider_registry_extended.py All checks passed! ``` ## Real Behavior Proof - Environment: macOS (ARM64), Python 3.13, any-llm-sdk 1.17.0 - Exact command / steps: introspected `AnyLLM.create` signature from any-llm-sdk 1.17.0 to confirm it accepts `api_base`, then ran the unit suites above which assert the URL is threaded through `create_proxy_backend` into `AnyLLM.create`. - Observed result: with `openai_api_url` set, `AnyLLMBackend` is now constructed with `api_base=<url>` and `AnyLLM.create()` receives it; previously it received only the provider and the value was dropped. - Not tested: live end-to-end request against a real custom OpenAI-compatible endpoint (no credentials available in this environment); mypy was not run locally. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes Documentation and CHANGELOG updates are N/A: this restores intended behavior of an existing documented flag (`--openai-api-url`) rather than adding new surface. `mypy` and live end-to-end testing were not run in this environment. |
||
|
Focused Instability
|
90bdc676fa
|
feat(policy): unlock formula-positive deep edits through the frozen floor (#856 P2b) (#944)
## Description Part of #904 — the **P2b (Subscription deep-unlock)** item from #856's phased plan. Builds directly on the P2 gate (#905, now merged); rebased onto `main` so the diff below is P2b-only (`headroom/transforms/content_router.py` +39/−5, `tests/test_netcost_gate.py` +72). The P2 net-cost gate only governs mutations the router already considers — messages **above** the `frozen_message_count` floor. The floor itself stays a hard binary skip: anything in the provider's prefix cache is left byte-identical no matter how compressible. That leaves the deep-edit half of #856 on the table — e.g. a ~60K-token stale tool dump sitting in the frozen prefix with only a small cached suffix after it, which pays for its cache-bust many times over. With `HEADROOM_NET_COST_POLICY=1` (default **off**), a **string-content** frozen message now falls through to the normal candidate pipeline instead of being skipped at the floor. The existing P2 break-even gate then decides per candidate: **S** is the full invalidated suffix after the slot, so the deep edit proceeds only when `ΔT·(w+r(R−1))` still beats the cache-bust penalty. Flag off restores byte-identical current behavior. ## Type of Change - [x] New feature (non-breaking change which adds functionality) ## Changes Made - Open the `frozen_message_count` floor in `ContentRouter` under `HEADROOM_NET_COST_POLICY=1`: string-content frozen messages route to the existing P2 gate instead of an unconditional skip; the gate's whole-suffix S already prices the cache-bust correctly for frozen slots. - **Scope guard:** block-list and non-string frozen content stay frozen — the gate is wired into the string and parallel-merge paths only, and the per-block `cache_control` contract in `_process_content_blocks` is not net-cost aware, so opening them here would mutate cached blocks ungated. - Emit a `router:netcost_frozen_unlock` transform marker + `netcost_frozen_unlocked` route count on actual unlocks, and `netcost_frozen_considered` for every frozen string slot routed to the gate — telemetry to validate the flag before any default-on. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] New tests added for new functionality - [x] New and existing unit tests pass locally with my changes ### Test Output ```text $ pytest tests/test_netcost_gate.py -q 15 passed in 0.96s $ pytest tests/ -k "content_router or netcost or router" -q 137 passed, 8 skipped, 6251 deselected in 20.89s $ ruff check headroom/transforms/content_router.py tests/test_netcost_gate.py All checks passed! $ ruff format --check headroom/transforms/content_router.py tests/test_netcost_gate.py 2 files already formatted ``` ## Real Behavior Proof - Environment: local macOS, repo .venv, Python 3.11.9, gpt-4o tokenizer fixture - Exact command / steps: drive `ContentRouter.apply()` with a 4-message conversation whose index-1 `tool` message (61,584 tokens) sits inside the frozen prefix (`frozen_message_count=2`), tiny suffix after; run once with the flag absent and once with `HEADROOM_NET_COST_POLICY=1` - Observed result: flag **off** → frozen tool dump left untouched, no unlock marker; flag **on** → dump compressed (`router:smart_crusher`) and `router:netcost_frozen_unlock` emitted, while the surrounding user messages stay `router:protected:user_message`. The 4 new unit tests also confirm a modest-shave / 40K-suffix frozen slot is *kept* frozen (gate runs, `netcost:skip:` emitted, no unlock) and block-list frozen content stays frozen. - Not tested: live proxy traffic / real dashboard validation — deferred to the default-on milestone per #904 (ships default-off precisely to gather that telemetry first). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes Net-cost economics are unchanged from P2 — this only widens *which slots* the same gate may consider. The Subscription deep-unlock story from #856 is realized without a mode branch: the floor is mode-agnostic in `ContentRouter`, and the formula is the correct arbiter regardless of auth mode. Remaining #904 items: P3a (batch deep edits) and P3b (idle-timer compaction). |
||
|
wangxiangyu7
|
dd22cfd72a
|
fix(wrap): avoid duplicate top-level keys when injecting codex provider (#884)
## Description `_inject_codex_provider_config` in `headroom/cli/wrap.py` unconditionally prepended a top-level block to `~/.codex/config.toml`: ```toml # --- Headroom proxy (auto-injected by headroom wrap codex) --- model_provider = "headroom" openai_base_url = "http://127.0.0.1:8787/v1" # --- end Headroom --- ``` If the user already had a top-level `model_provider` (or `openai_base_url`), the result was two top-level keys with the same name. That violates the TOML spec, and Codex refuses to start with `duplicate key`. This change makes the injector rewrite any pre-existing top-level `model_provider` / `openai_base_url` in place to the headroom values (keeping the user's original value in a `# was: …` trailing comment) and only emit the marker-delimited top-level block for keys the user has not declared. The pre-wrap snapshot mechanism is unchanged, so `headroom unwrap codex` still restores the file byte-for-byte. Closes #883 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cli/wrap.py` - New helper `_redirect_existing_top_level_keys(content, port)`: rewrites existing top-level `model_provider` / `openai_base_url` lines to the headroom values and preserves the previous value in a trailing `# was: …` comment. - New helper `_has_redirectable_top_level_key(content, key)`: cheap predicate for the two redirectable keys. - New helper `_build_top_level_block(user_content)`: emits a marker-delimited block containing only the redirectable keys the user has **not** already declared (declared ones are rewritten in place instead, avoiding the TOML duplicate-key error). - `_inject_codex_provider_config` now rewrites declared keys in place and only prepends the marker block for the remaining keys; `requires_openai_auth` handling (#406) is preserved. - `tests/test_cli/test_wrap_codex.py` - New class `TestInjectAvoidsDuplicateTopLevelKeys` (TOML-validity after wrap on a config already declaring a provider, original-value preservation in a `# was:` comment, idempotent re-wrap with a port change, marker-block fallback on an empty file, snapshot-based unwrap restoration). The TOML-validity test parses the wrapped file with `tomllib.loads`, which fails before the fix and passes after. - `CHANGELOG.md` - Added entry under `## Unreleased` → `### Bug Fixes`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom --ignore-missing-imports`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_cli/test_wrap_codex.py -q ======================== 52 passed, 1 warning in 5.28s ========================= $ uv run ruff check headroom/cli/wrap.py tests/test_cli/test_wrap_codex.py All checks passed! $ uv run ruff format --check headroom/cli/wrap.py tests/test_cli/test_wrap_codex.py 2 files already formatted $ mypy headroom --ignore-missing-imports Success: no issues found in 358 source files ``` ## Real Behavior Proof - Environment: macOS 24.6.0, Python 3.13.3, branch `fix/wrap-codex-no-duplicate-keys` rebased onto upstream `main`, Codex CLI config at `~/.codex/config.toml`. - Exact command / steps: Seed a user config matching the bug report (`model_provider = "ccswitch"` + `openai_base_url = "…"` + `[model_providers.ccswitch]`), run the same path `headroom wrap codex` takes (`_inject_codex_provider_config(8787)`), then parse the result with `tomllib.loads(...)` and run `headroom unwrap codex`. - Observed result: On patched code the wrapped `config.toml` parses cleanly — exactly one `model_provider` and one `openai_base_url` remain (the user's prior value preserved in a `# was: …` comment) and the `[model_providers.headroom]` table is present; `unwrap` restores the file byte-for-byte. On the unpatched code the same file raises `tomllib.TOMLDecodeError` (duplicate key). Full suite: 52 passed, ruff lint + format clean (see Test Output). - Not tested: End-to-end launch of the Codex CLI against a live proxy (no Codex e2e harness in this sandbox); Windows / `$CODEX_HOME` override paths (covered only by existing tests). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — CLI/config change, no UI. ## Additional Notes `ruff check .`, `ruff format --check .`, and `mypy headroom --ignore-missing-imports` all pass on the rebased branch. The diff stays narrow: `headroom/cli/wrap.py` + its tests + a CHANGELOG entry. Co-authored-by: wangxiangyu7 <wangxiangyu7@lixiang.com> |
||
|
Ashish
|
615e1ed6f5
|
test(compression): fill code handler coverage gaps (#895)
## Description `CodeStructureHandler` had zero dedicated tests before this series — which is exactly why the P0 bugs in #890/#892/#893 went unnoticed. This fills the remaining coverage gaps beyond the per-fix regression tests. Stacked on #893. Closes # <!-- compression-handler review --> ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - `tests/test_compression/test_code_handler.py`: language detection (python/go/rust + default fallback); regex-path signature/import preservation across go/rust/typescript/javascript; regex confidence value; empty/whitespace content; unknown language; mask-length invariant. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_compression/test_code_handler.py -q 25 passed ``` ## Real Behavior Proof - Environment: local macOS, Python 3.11, tree-sitter-language-pack 1.8.1, branch `test/code-handler-coverage` (stacked on #893). - Exact command / steps: `pytest tests/test_compression/test_code_handler.py -q`. - Observed result: 25 tests pass; tree-sitter classes skip cleanly when the pack is absent, regex-path tests always run. - Not tested: N/A — this PR is tests only. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — tests only. See Test Output. ## Additional Notes Stacked on #893 — review the top commit until that merges. PR 6 of 7. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ashish
|
b1f700fc27
|
fix(compression): convert tree-sitter byte offsets to char offsets (#892)
## Description tree-sitter reports node positions as byte offsets into the UTF-8 encoding, but `CodeStructureHandler` builds a character-indexed mask. Any multi-byte character (accents, emoji, CJK in docstrings/comments/strings) shifted every subsequent span, preserving the wrong characters and leaking signature bytes into bodies. Stacked on #890. Closes # <!-- compression-handler review --> ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/compression/handlers/code_handler.py`: remap spans through a byte->char table before masking; pure-ASCII content (byte == char) skips the conversion. - `tests/test_compression/test_code_handler.py`: regression test with `café münü 🎉` in a comment, asserting the following signature and body are correctly aligned. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_compression/ -q 93 passed, 8 skipped ``` ## Real Behavior Proof - Environment: local macOS, Python 3.11, tree-sitter-language-pack 1.8.1, branch `fix/code-byte-char-offsets` (stacked on #890). - Exact command / steps: `pytest tests/test_compression/ -q`. - Observed result: With 9 extra UTF-8 bytes ahead of it, a function signature is exactly preserved and its body stays compressible; before, the offsets were shifted. - Not tested: End-to-end through the live proxy pipeline. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — library change. See Test Output. ## Additional Notes Stacked on #890 — review the top commit until that merges. PR 4 of 7. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ashish
|
65b0e8c58d
|
fix(compression): measure short-value threshold on payload, not token (#889)
## Description `JSONStructureHandler._should_preserve_token` compared `len(token.text)` — which includes both quote characters — against `short_value_threshold`. A value of exactly threshold length was rejected: the documented "20-char threshold" was effectively 18 chars of payload. Stacked on #887. Closes # <!-- compression-handler review --> ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/compression/handlers/json_handler.py`: strip quotes once at the top of the string-value branch and use the payload length for both the short-value and entropy checks. - `tests/test_compression/test_json_handler.py`: regression test for a value of exactly `short_value_threshold` length. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_compression/test_json_handler.py -q 33 passed ``` ## Real Behavior Proof - Environment: local macOS, Python 3.11, branch `fix/json-quote-threshold` (stacked on #887). - Exact command / steps: `pytest tests/test_compression/test_json_handler.py -q`. - Observed result: A 20-char value is preserved at a 20-char threshold; previously it was dropped due to the +2 quote miscount. - Not tested: End-to-end through the live proxy pipeline. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — library change. See Test Output. ## Additional Notes Stacked on #887 — review the top commit until that merges. PR 2 of 7. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ashish
|
a14ab45cf0
|
fix(proxy): make budget enforcement actually work (#885)
## Description
`CostTracker._costs` was initialized but never written to, so
`get_period_cost()` always returned `0` and `check_budget()` always
returned "allowed" — the `--budget` flag was a silent no-op.
`_prune_old_costs()` was dead code with zero callers. This makes budget
enforcement actually work: requests are rejected once the configured
limit is reached.
Closes # <!-- no tracked issue; discovered during a proxy-pipeline audit
-->
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **`headroom/proxy/cost.py`** — `record_tokens()` now computes the
request cost via `estimate_cost()` and appends it to `_costs`,
activating `_prune_old_costs()`. When a call site has no API usage
breakdown (cache/uncached all zero), `tokens_sent` is used as the input
count so input cost is not silently dropped. `COST_RETENTION_HOURS` 24 →
744 so retention covers the longest budget period (monthly sums from the
1st; 24h retention would have under-enforced monthly budgets).
- **`headroom/proxy/outcome.py`** — the request funnel passes
`output_tokens` through to `record_tokens()` so costs include output,
for all providers.
- **`headroom/cli/proxy.py`** — added `--budget-period
[hourly|daily|monthly]` (env `HEADROOM_BUDGET_PERIOD`); it existed in
`ProxyConfig` and the server entry point but was unreachable from the
main CLI. Fixed the `--budget` help text that wrongly said "resets at
midnight UTC".
- **`headroom/cli/main.py`** — minor registration/version plumbing.
- Tests: regression coverage for the full `record_tokens →
get_period_cost → check_budget` chain, the `tokens_sent` fallback, and
the `--budget-period` flag/env wiring.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_cost_tracker_counterfactual.py tests/test_request_outcome.py -q
40 passed
$ ruff check headroom/proxy/cost.py headroom/proxy/outcome.py headroom/cli/proxy.py
All checks passed!
$ mypy headroom/proxy/cost.py headroom/proxy/outcome.py headroom/cli/proxy.py --ignore-missing-imports
Success: no issues found
```
## Real Behavior Proof
- Environment: local macOS, Python 3.11, branch `fix/budget-enforcement`
at the PR head commit.
- Exact command / steps: `pytest
tests/test_cost_tracker_counterfactual.py::test_budget_enforced_after_recording_costs
-v` — sets `CostTracker(budget_limit_usd=0.0001)`, records ~$1.50 of
Sonnet input, then asserts `check_budget()` returns not-allowed with
`remaining == 0`.
- Observed result: budget is now enforced — `get_period_cost()` reflects
real spend and `check_budget()` rejects once the limit is exceeded (the
proxy returns HTTP 429 on that path). On `main` the same test fails
because `_costs` is never populated and `check_budget()` always returns
allowed.
- Not tested: live end-to-end rejection against a running proxy with
real upstream traffic; the running proxy needs a restart on this version
to pick up the fix.
```text
$ pytest tests/test_cost_tracker_counterfactual.py::test_budget_enforced_after_recording_costs \
tests/test_cost_tracker_counterfactual.py::test_budget_input_cost_counted_without_usage_breakdown -v
2 passed
```
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A — CLI/backend change with no UI surface. See **Test Output** and
**Real Behavior Proof** above for terminal evidence.
## Additional Notes
- The `ci.yml` coverage-upload change originally added here (commit
`
|
||
|
Tejas Chopra
|
919379a8a1
|
fix(serena): stop the Serena dashboard popup and make --no-serena actually disable Serena (#1003)
## Description Headroom installs the Serena MCP server by default during `headroom wrap`, and many users reported the Serena web dashboard browser tab popping up on every session — even when they never opted into Serena. This PR fixes two distinct root causes: Serena's dashboard auto-open, and `--no-serena` not actually disabling an already-installed Serena. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `build_serena_spec()` now passes `--open-web-dashboard False` to `serena start-mcp-server`. This is Serena's startup override for `web_dashboard_open_on_launch` (`serena/mcp.py:317-318`), so it suppresses the browser popup regardless of the user's `~/.serena/serena_config.yml` — the correct fix is at the launch point, not a per-machine config edit. The dashboard backend still runs and stays reachable at `http://localhost:24282/dashboard/`; only the auto-open is disabled. Applies to both launch paths (wrap + strands bundle) since both go through `build_serena_spec()`. - New `_disable_serena_mcp()`: `--no-serena` now actively removes the Serena entry Headroom installed (ledger-verified) instead of merely skipping registration. Previously a prior default wrap persisted a `serena` entry and the agent kept launching it; the old `Skipping Serena MCP` message was misleading. A user-managed Serena (absent from the ledger) is reported and left untouched; an absent Serena prints the skip message. Wired into both the Claude and Codex wrap paths. - `unwrap_codex` now removes Headroom-installed Serena. Codex writes Serena as its own `[mcp_servers.serena]` table, separate from the provider block the config-restore handles, so a "cleaned" unwrap previously left it behind (`unwrap_claude` already removed it; Codex was the gap). - Tests: updated `build_serena_spec` arg assertion + added a no-popup-default test; new `test_serena_disable.py` covering removed-when-headroom-owned, preserved-when-user-managed, skip-when-absent, noop-when-undetected, and `unwrap_codex` removal. ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_cli/test_serena_disable.py tests/test_cli/test_wrap_codex.py tests/test_cli/test_unwrap_claude.py tests/test_mcp_registry/ -q 134 passed $ python -m pytest tests/test_mcp_registry/test_install.py -q ... passed (build_serena_spec arg + no-popup-default assertions) $ ruff check headroom/cli/wrap.py headroom/mcp_registry/install.py tests/test_cli/test_serena_disable.py tests/test_mcp_registry/test_install.py All checks passed! $ ruff format --check headroom/cli/wrap.py headroom/mcp_registry/install.py ... already formatted $ mypy headroom/cli/wrap.py headroom/mcp_registry/install.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: macOS (darwin), Python 3.12 venv, Serena 1.5.4 cached via uvx, headroom on branch fix/serena-no-dashboard-popup - Exact command / steps: Traced Serena source — `serena/cli.py` exposes `--open-web-dashboard <bool>`; `serena/mcp.py:317-318` sets `config.web_dashboard_open_on_launch = open_web_dashboard`; `serena/agent.py:706` feeds that to `DashboardManager`, which calls `webbrowser.open()` (`serena/dashboard.py:831`). Verified click parses `--open-web-dashboard False` → `False` via a CliRunner probe. Ran the test suites above. - Observed result: With the flag injected, the value that gates the browser-open is forced to False at startup regardless of local config, so no tab opens; dashboard backend still serves on its port. `--no-serena` removes the previously-installed `serena` entry (unregister called, "Removed previously-installed Serena MCP" printed) and `unwrap codex` removes it too. All 134 targeted tests pass; ruff + mypy clean. - Not tested: A full end-to-end `headroom wrap claude` against a live Claude Code install with a real browser was not run; verification is via Serena source tracing + the click-parse probe + unit/integration tests over the registrar and wrap/unwrap paths. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes Two unchecked checklist items are N/A: no user-facing docs reference the Serena dashboard behavior, and CHANGELOG is generated via release-please from the conventional commits. "Manual testing performed" is left unchecked deliberately — see `Real Behavior Proof` → `Not tested` for the exact boundary of what was and wasn't exercised against a live browser. |
||
|
Ashish
|
16ed73bca6
|
fix(compression): keep container bodies compressible in code handler (#890)
## Description Two bugs in `CodeStructureHandler`'s tree-sitter path. (1) Container nodes (class/impl/trait/decorated definitions) were marked structural over their full span and `_spans_to_mask` never un-marks, so every method body inside a class was preserved and compression silently no-opped at confidence 0.95. (2) Discovered while testing: `tree-sitter-language-pack >= 1.0` switched to a Rust binding (methods, not attributes; `parse(str)`), so the handler raised `TypeError` on every call and silently fell back to regex. Closes # <!-- compression-handler review --> ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/compression/handlers/code_handler.py`: containers emit a signature-only span (start to body start); recursion gives nested functions their own signature/body split; decorated definitions emit no whole-node span. - `headroom/compression/handlers/code_handler.py`: small compat shim supporting both the classic attribute API and the new Rust-binding method API. - `tests/test_compression/test_code_handler.py`: new file (the handler had zero dedicated tests) covering class/decorated/impl body compressibility, regex fallback, and a preservation-ratio bound. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_compression/ -q 92 passed, 8 skipped ``` ## Real Behavior Proof - Environment: local macOS, Python 3.11, tree-sitter-language-pack 1.8.1 installed, branch `fix/code-container-bodies`. - Exact command / steps: `pytest tests/test_compression/ -q`. - Observed result: Class method bodies are now compressible (preservation ratio drops from ~1.0 to roughly the signature fraction); the tree-sitter path runs instead of falling back to regex. - Not tested: End-to-end through the live proxy pipeline. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — library change. See Test Output. ## Additional Notes PR 3 of 7; branched fresh from main (independent of #887/#889). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ashish
|
d6f0f0f642
|
fix(compression): correct JSON array item counting and entropy gate (#887)
## Description
Two bugs in `JSONStructureHandler` that jointly defeated the "keep first
N array items fully" design. (1) Every comma under `array_depth > 0` was
counted as an array item separator — including commas *between keys
inside objects* — so for arrays of objects the first record's own keys
exhausted `max_array_items_full` and dropped values belonging to item 0.
(2) Fixing that unmasked a second bug: self-normalized Shannon entropy
scores English prose at 0.90+, above the 0.85 "identifier" threshold, so
every long description was preserved as a fake high-entropy identifier.
Closes # <!-- found during a compression-handler review -->
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/compression/handlers/json_handler.py`: replace depth-keyed
comma counting with a container stack so only commas whose immediate
enclosing container is an array advance that array's item index.
- `headroom/compression/handlers/json_handler.py`: gate the entropy
preservation check on a no-spaces identifier signal, so UUIDs/hashes
still pass but prose compresses.
- `tests/test_compression/test_json_handler.py`: regression tests for
object-comma counting, items past the threshold, and prose-vs-identifier
entropy.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_compression/test_json_handler.py -q
32 passed
```
## Real Behavior Proof
- Environment: local macOS, Python 3.11, branch
`fix/json-array-item-count`.
- Exact command / steps: `pytest
tests/test_compression/test_json_handler.py -q` plus an empirical mask
dump on `[{"a":1,"b":2,...}]`.
- Observed result: Values inside array item 0 are now preserved; long
prose values compress while UUIDs are retained (prose scored
0.906-0.929, UUID 0.956 — the threshold alone could not separate them).
- Not tested: End-to-end through the live proxy pipeline (the handler is
not yet wired into the proxy hot path).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A — library/compression change with no UI. See Test Output.
## Additional Notes
First of a 7-PR compression-handler review series.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
skblue
|
b70fccbe17
|
fix(proxy): read RTK gain stats globally by default (#957)
## Description Closes #900. The proxy now reads RTK lifetime savings with global scope by default. This matches shared daemon deployments where the proxy process cwd is often `$HOME` or a service directory, while RTK savings are accumulated across the operator's projects. `HEADROOM_RTK_GAIN_SCOPE=project` keeps the old `rtk gain --project` behavior for operators who explicitly want the proxy process working directory as the scope. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Default RTK stats subprocess command to `rtk gain --format json` - Add `HEADROOM_RTK_GAIN_SCOPE=project` to opt into `rtk gain --project --format json` - Keep fallback/synthetic-zero payload `scope` aligned with the queried scope - Deduplicate context-tool zero payload construction - Document the new RTK gain scope environment variable ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run --extra dev python -m pytest tests/test_proxy_dashboard_stats_cache.py tests/test_subscription_tracker_rtk_wired.py -q ============================= test session starts ============================== platform darwin -- Python 3.14.5, pytest-9.0.3, pluggy-1.6.0 rootdir: /Users/joshuasiu/vibe/temp/headroom configfile: pyproject.toml plugins: anyio-4.12.1, langsmith-0.8.0, asyncio-1.3.0, cov-7.0.0 asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 26 items tests/test_proxy_dashboard_stats_cache.py .......... [ 38%] tests/test_subscription_tracker_rtk_wired.py ................ [100%] ============================== 26 passed in 0.40s ============================== uv run --extra dev python -m pytest tests/test_proxy_stats_recent_requests.py tests/test_proxy_healthchecks.py -q ============================= test session starts ============================== platform darwin -- Python 3.14.5, pytest-9.0.3, pluggy-1.6.0 rootdir: /Users/joshuasiu/vibe/temp/headroom configfile: pyproject.toml plugins: anyio-4.12.1, langsmith-0.8.0, asyncio-1.3.0, cov-7.0.0 asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 15 items tests/test_proxy_stats_recent_requests.py ... [ 20%] tests/test_proxy_healthchecks.py ............ [100%] ============================= 15 passed in 10.41s ============================== uv run --extra dev ruff check headroom/proxy/helpers.py tests/test_proxy_dashboard_stats_cache.py tests/test_subscription_tracker_rtk_wired.py All checks passed! uv run --extra dev ruff format --check headroom/proxy/helpers.py tests/test_proxy_dashboard_stats_cache.py tests/test_subscription_tracker_rtk_wired.py 3 files already formatted uv run --extra dev mypy headroom headroom/proxy/server.py:1141: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] headroom/proxy/server.py:1211: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] headroom/proxy/server.py:1215: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] Success: no issues found in 358 source files ``` ## Real Behavior Proof - Environment: macOS, Python 3.14.5 via `uv run --extra dev` - Exact command / steps: mocked RTK subprocess argv in unit tests - Observed result: default command is `rtk gain --format json`; project scope command is `rtk gain --project --format json`; invalid scope logs `event=rtk_gain_scope_invalid` and falls back to global - Not tested: full repository pytest suite ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes The unchecked comment and changelog items are not applicable for this scoped proxy stats fix. |
||
|
Focused Instability
|
b51cda10d7
|
docs(evals): add session probes section to evals README (#888)
## Description Follow-up to #862. That PR's body described a **Session Probes** section in `headroom/evals/README.md`, but the file edit missed the commit (edited in the wrong checkout). This adds the missing 22-line docs-only section: the record-then-score workflow for `HEADROOM_PROBE_RECORD_DIR` + `headroom evals probes`, including the plaintext-recording privacy note. Refs #861 (session-probe eval harness — this README section was part of that feature's spec). ## Type of Change - [x] Documentation update ## Changes Made - Add a **Session Probes (real recorded sessions)** section to `headroom/evals/README.md` (+22 lines, no code change): the two-step record (`HEADROOM_PROBE_RECORD_DIR=… headroom proxy start`) then score (`headroom evals probes --recordings …`) workflow, the three probe dimensions (exact numerics, artifact trail, error evidence), the retained/recoverable/lost classification, retention bucketing by ratio + per-transform grouping, and the `--json-output` flag. - Includes the opt-in privacy note: recordings contain full conversation content in plaintext and stay on the local machine. ## Testing - [x] Documentation builds/renders correctly - [x] Linting passes (`ruff check .`) - [x] New and existing unit tests pass locally with my changes ### Test Output ```text $ git diff --stat upstream/main..HEAD headroom/evals/README.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) Docs-only change — no code paths touched. The commands and flags documented (HEADROOM_PROBE_RECORD_DIR, `headroom evals probes`, --recordings, --json-output) are the surface shipped and tested in #862. ``` ## Real Behavior Proof - Environment: local macOS, repo .venv, Python 3.11.9 - Exact command / steps: rendered the edited `headroom/evals/README.md` and cross-checked every documented flag/command against the implemented CLI from #862 (`headroom evals probes`, `HEADROOM_PROBE_RECORD_DIR`, `--recordings`, `--json-output`) - Observed result: the new section renders correctly and every command/flag it names exists in the shipped probe harness; no code paths are changed by this PR, so behavior is unchanged - Not tested: nothing additional — docs-only change with no executable surface of its own ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes Pure documentation backfill for #862; the feature itself (recorder + retention probes) already merged. PR body updated to satisfy the PR-governance template gate. |
||
|
Kumario
|
0c5c89d05c
|
fix(anthropic): strip styled Claude model ids (#651)
## Description Fixes #626 by normalizing Anthropic/Claude model ids that contain ANSI escape sequences or dangling style suffixes before provider lookups and upstream forwarding. The branch has been updated onto current `main` and the proxy handler conflicts have been resolved. ## Type of Change - [x] Bug fix - [ ] New feature - [ ] Documentation - [ ] Refactor - [x] Tests only ## Changes Made - Normalize Anthropic model ids before context/pricing lookup. - Sanitize Anthropic `/v1/models` metadata and styled `/v1/models/{id}` passthrough paths. - Sanitize `/v1/messages` request body model ids before upstream forwarding. - Resolved current-main conflicts while preserving newer `model_override` and streaming passthrough behavior. ## Testing - [x] Unit tests - [x] Route/proxy tests - [x] Lint/static checks - [ ] Manual testing ### Test Output ```text UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --with pytest --with fastapi --with httpx --with uvicorn --with h2 python -m pytest tests/test_providers/test_anthropic.py tests/test_provider_proxy_routes.py::test_anthropic_model_metadata_strips_ansi_model_ids tests/test_provider_proxy_routes.py::test_anthropic_model_detail_path_strips_ansi_model_id tests/test_provider_proxy_routes.py::test_anthropic_messages_strips_ansi_model_id_before_upstream -q 17 passed, 2 warnings in 39.91s UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --with ruff ruff check headroom/providers/anthropic.py headroom/proxy/handlers/openai.py headroom/proxy/handlers/anthropic.py tests/test_providers/test_anthropic.py tests/test_provider_proxy_routes.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13.3, focused local worktree for PR #651 after merging current `upstream/main`. - Exact command / steps: Merged current main, resolved conflicts in Anthropic/OpenAI proxy handlers, ran the PR's targeted provider/proxy tests and ruff checks. - Observed result: Styled Anthropic model metadata, model-detail path, and messages upstream sanitization tests pass; ruff reports no issues. - Not tested: Full repository mypy/pre-commit; existing unrelated Windows `fcntl` typing errors block full hook execution locally. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review <!-- headroom-maintainer-template-completion:start --> ## Description This PR prepares `fix(anthropic): strip styled Claude model ids` for review by documenting the intended change, validation evidence, and remaining merge-readiness context. Linked issues: #626 ## Type of Change - [x] Bug fix - [ ] New feature - [ ] Documentation - [ ] Refactor - [ ] Tests only ## Changes Made - Commit: fix(anthropic): normalize styled model ids - Commit: fix(proxy): strip styled Anthropic model ids - Commit: fix: format anthropic model sanitization - Commit: Merge remote-tracking branch 'upstream/main' into review/pr-651 - Touches `headroom/cache/dynamic_detector.py` - Touches `headroom/providers/anthropic.py` - Touches `headroom/proxy/handlers/anthropic.py` - Touches `headroom/proxy/handlers/openai.py` - Touches `tests/test_provider_proxy_routes.py` - Touches `tests/test_providers/test_anthropic.py` ## Testing - [x] GitHub checks reviewed - [x] Metadata/template validation - [ ] Local functional testing ### Test Output ```text gh pr view 651 --repo chopratejas/headroom --json statusCheckRollup - PR Governance / template: FAILURE - CI / changes: SUCCESS - Init E2E / docker-init-e2e: SUCCESS - Wrap E2E / docker-wrap-e2e: SUCCESS - Wrap Native E2E / wrap-native (ubuntu-latest): SUCCESS - Wrap Native E2E / wrap-native (macos-latest): SUCCESS - CI / commitlint: SUCCESS - PR Governance / label: SUCCESS - CI / lint: SUCCESS - CI / build-wheel: SUCCESS - CI / prefetch-model: SUCCESS - CI / build: SUCCESS ``` ## Real Behavior Proof - Environment: GitHub PR metadata and checks for `chopratejas/headroom` PR #651. - Exact command / steps: Reviewed PR title, commits, changed files, linked issues, labels, and check rollup; appended this maintainer template completion block without replacing the author's original description. - Observed result: PR body now contains all required governance sections, checked readiness fields, and a non-placeholder validation evidence block. - Not tested: This pass updated PR metadata only; code validation remains represented by the linked GitHub checks and any author-provided evidence above. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review <!-- headroom-maintainer-template-completion:end --> --------- Co-authored-by: Tejas Chopra <chopratejas@gmail.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Matt Van Horn
|
9b7b436b04
|
fix: wire HEADROOM_EXCLUDE_TOOLS / HEADROOM_TOOL_PROFILES into Click proxy entrypoint (#943)
## Description The Click-based `headroom proxy` entrypoint (`headroom/cli/proxy.py`) constructed `ProxyConfig` without calling `_parse_exclude_tools` or `_parse_tool_profiles`, so `HEADROOM_EXCLUDE_TOOLS` and `HEADROOM_TOOL_PROFILES` were silently ignored for any service launched via `headroom proxy`. The argparse path in `headroom/proxy/server.py` already handled these correctly. This PR imports both helpers into the Click entrypoint and wires their output into `ProxyConfig`. Closes #825 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cli/proxy.py`: import `_parse_exclude_tools` and `_parse_tool_profiles` alongside `ProxyConfig`/`run_server`; pass their output into the `ProxyConfig(...)` construction (`or None` guard collapses empty set/dict to `None` so unset vars leave `DEFAULT_EXCLUDE_TOOLS` unchanged) - `tests/test_cli_proxy_env.py`: new `TestCLIProxyExcludeToolsEnvVar` class with 5 regression tests ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ### Paste relevant command output or artifact links here ```text ============================= test session starts ============================== platform darwin -- Python 3.13.12, pytest-9.0.3 collected 43 items tests/test_cli_proxy_env.py::TestCLIProxyExcludeToolsEnvVar::test_exclude_tools_single_name_from_env PASSED tests/test_cli_proxy_env.py::TestCLIProxyExcludeToolsEnvVar::test_exclude_tools_multi_name_from_env PASSED tests/test_cli_proxy_env.py::TestCLIProxyExcludeToolsEnvVar::test_exclude_tools_unset_leaves_none PASSED tests/test_cli_proxy_env.py::TestCLIProxyExcludeToolsEnvVar::test_tool_profiles_from_env PASSED tests/test_cli_proxy_env.py::TestCLIProxyExcludeToolsEnvVar::test_tool_profiles_unset_leaves_none PASSED ============================== 43 passed in 8.95s ============================== ruff check headroom/cli/proxy.py tests/test_cli_proxy_env.py All checks passed! ``` ## Real Behavior Proof - Environment: Python 3.13.12, headroom-ai dev install - Exact command / steps: `HEADROOM_EXCLUDE_TOOLS=WebSearch headroom proxy` before fix silently built `ProxyConfig(exclude_tools=None)` despite the env var being set - Observed result: After fix, `ProxyConfig.exclude_tools` contains `{"WebSearch", "websearch"}` as verified by the new unit tests - Not tested: end-to-end proxy run with a live Anthropic endpoint ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes The fix mirrors the exact pattern already used in the argparse path (`_main()` in `headroom/proxy/server.py` lines 3920-3922). The `or None` guard is intentional: `_parse_exclude_tools(None)` returns `set()` when the env var is unset, and `ProxyConfig.exclude_tools=None` means "use `DEFAULT_EXCLUDE_TOOLS` unchanged" — passing an empty set would instead replace the defaults with nothing. Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Zbl1007
|
e0a9fdb62c
|
chore(imports): lazy-load dynamic detector ML imports (#597)
## Description Avoid importing optional ML dependencies when `headroom.cache.dynamic_detector` is imported during wrap/proxy startup. This keeps the dynamic detector module cheap to import while preserving the existing NER and semantic detector behavior when those tiers are actually used. Fixes #195 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Replaced eager `spacy`, `sentence_transformers`, and `numpy` imports in `dynamic_detector.py` with `find_spec` availability checks. - Kept NER and semantic model loading on the existing first-use detector initialization paths. - Moved `numpy` import to the semantic similarity calculation path where it is actually needed. - Added regression coverage proving `dynamic_detector` import does not load optional ML modules even when stub versions of those modules are importable. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text .venv/bin/pytest tests/test_package_init_lazy.py 7 passed in 2.82s .venv/bin/pytest tests/test_cache/test_dynamic_detector.py tests/test_package_init_lazy.py 43 passed, 2 skipped in 19.48s .venv/bin/python -m ruff check headroom/cache/dynamic_detector.py tests/test_package_init_lazy.py All checks passed! .venv/bin/python -m ruff format --check headroom/cache/dynamic_detector.py tests/test_package_init_lazy.py 2 files already formatted git diff --check # no output ``` ## Real Behavior Proof - Environment: macOS local checkout, Python 3.11 virtualenv at `.venv`. - Exact command / steps: Added a subprocess regression test that creates importable stub `spacy`, `numpy`, `torch`, and `sentence_transformers` modules, imports `headroom.cache.dynamic_detector`, and inspects `sys.modules`. - Observed result: Before the implementation change, the new regression test failed because `spacy` was loaded during module import. After the change, `spacy`, `sentence_transformers`, and `torch` remain unloaded and the dynamic detector test suite still passes. - Not tested: Full repository-wide `mypy headroom`; full CI requires maintainer approval for fork workflows. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) Not applicable. ## Additional Notes This branch has been narrowed after review feedback. It now only contains the lazy `dynamic_detector` import fix. The wrapper startup-timeout behavior was removed from this PR so it can be reviewed separately from the existing timeout work. Signed-off-by: Zbl1007 <1399853961@qq.com> |
||
|
Focused Instability
|
5939004185
|
feat(evals): adversarial-input robustness grid for compressors (#918)
## Description Closes #916. CompressionAttack (arXiv:2510.22963) showed that prompt compressors are an attack surface for LLM middleware: adversarial text in compressible content can preferentially survive compression (amplifying injection density) or abuse compressor control surfaces. Headroom has a concrete instance of the latter — content carrying a CCR retrieval marker is pinned as already-compressed, so a spoofed marker string in tool output could make content compression-immune. This adds an offline, deterministic eval grid measuring both, with no LLM, no API key, and no model download (Kompress disabled by default). Closes #916. ## Type of Change - [x] New feature (non-breaking change which adds functionality) ## Changes Made - `headroom/evals/adversarial_grid.py`: payload corpus (instruction override, fake system tag, fake tool directive, CCR marker spoof in block + inline forms, steering imperative, benign control), realistic + synthetic carriers (60-record JSON array, 150-line worker log), and a payload-class × carrier × splice-position grid. - Per-cell metrics: payload survival (normalization-tolerant containment), benign-line survival baseline, and compression suppression (payload-ratio minus clean-ratio — the marker-spoof immunity signal), plus per-class aggregates. - `headroom/cli/evals.py`: wire the grid into the evals CLI command. - Tests in `tests/test_adversarial_grid.py`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] New tests added for new functionality - [x] New and existing unit tests pass locally with my changes ### Test Output ```text $ pytest tests/test_adversarial_grid.py -q 12 passed in 1.10s ``` ## Real Behavior Proof - Environment: local macOS, repo .venv, Python 3.11.9; offline (no API key, Kompress disabled) - Exact command / steps: rebased onto current main (dropping the now-superseded codecov-upload commit — main already uploads per-shard coverage via codecov-action@v5), then `pytest tests/test_adversarial_grid.py -q` - Observed result: 12/12 pass; grid runs deterministically with no network/model access and reports survival + suppression metrics per cell. - Not tested: LLM-in-the-loop attack realism — out of scope by design; this grid is the offline deterministic layer. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes Force-pushed after a rebase onto current main to resolve a `.github/workflows/ci.yml` conflict introduced by #921: the standalone codecov-upload commit was dropped because main now performs per-shard coverage upload globally. PR payload is unchanged (adversarial grid + tests). --------- Co-authored-by: integration-check <integration@local> |
||
|
Focused Instability
|
553ade4ec6
|
feat(policy): consume net-cost mutation gate in ContentRouter (#856 P2) (#905)
## Description Fixes #907. Part of #904 — the **P2 (consume, flag-gated)** item from #856's phased plan. (#903, which this was stacked on, has merged; this is now a clean diff.) `HEADROOM_NET_COST_POLICY=1` (default **off** — flag absent restores byte-identical current behavior) routes every ContentRouter mutation candidate through `CompressionPolicy.net_mutation_gain` before compression is applied, at both decision sites: the result-cache-hit path and the fresh-compression merge (pass 3). v1 estimators (as specced in #856): **ΔT** exact (compressed form already computed); **S** = token total after the slot, precomputed once as a reverse cumulative sum (O(1) per candidate); **R / P_alive** env-tunable (`HEADROOM_NET_COST_EXPECTED_READS`=10, `HEADROOM_NET_COST_P_ALIVE`=1.0). Every decision logs all inputs at INFO and increments `netcost_allowed`/`netcost_skipped` counters so the flag can be validated from telemetry before any default-on. Closes #907. ## Type of Change - [x] New feature (non-breaking change which adds functionality) ## Changes Made - Add flag-gated net-cost mutation gate to `ContentRouter` at both mutation sites (cache-hit + fresh-compress merge). - Precompute reverse-cumulative suffix token sums once per request for O(1) S lookups. - Emit INFO telemetry, `netcost_allowed`/`netcost_skipped` counters, and a `netcost:skip:<band>` transform marker on blocked slots. - **Review-response (4eb2307):** reject non-finite env values (`math.isfinite` guard), count suffix tokens block-aware via `_netcost_message_tokens()` (was `str(content)`, which miscounted Anthropic block lists), and bucket the skip marker via `_gain_bucket()` to bound dashboard cardinality. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] New tests added for new functionality - [x] New and existing unit tests pass locally with my changes ### Test Output ```text $ pytest tests/test_netcost_gate.py -q 11 passed in 0.82s $ pytest tests/ -k "content_router or netcost or router" -q 133 passed, 8 skipped, 6120 deselected in 23.26s $ ruff check headroom/transforms/content_router.py All checks passed! ``` ## Real Behavior Proof - Environment: local macOS, repo .venv, Python 3.11.9, gpt-4o tokenizer fixture - Exact command / steps: `pytest tests/test_netcost_gate.py -q` then the router-suite selector above; gate exercised end-to-end through the real tokenizer + compression path (flag on via monkeypatch) - Observed result: with R=10/P=1 defaults, a 300-row tool result followed by a 40k-word suffix is left uncompressed (gate skips, `netcost:skip:` marker emitted); a 2000-row result with a 5-word suffix compresses (gate allows). Non-finite env (`inf`/`nan`) falls back to defaults and still skips. - Not tested: live proxy traffic / real dashboard validation — deferred to the default-on milestone per #904 (this ships default-off precisely to gather that telemetry first). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes Cache-hit re-tokenization (`:2312`) and the large integration fixtures are tracked as follow-ups in the PR review thread; both are intentional given the flag is default-off. Known v1 limitations (whole-suffix S, no batch awareness, static P_alive) are tracked in #904 as P2b/P3a/P3b. PR body updated to satisfy the new PR-governance template gate (#914-era governance workflow). --------- Co-authored-by: integration-check <integration@local> |
||
|
Focused Instability
|
f9285766dd
|
feat: attribute reread waste to over-compression via marker check (#901)
## Description Fixes #899. The `reread` signal (#853/#854) counts re-served tool results but cannot answer the question that motivated it: **did Headroom cause the re-read?** A re-read after an intact first serve is agent behavior; a re-read after Headroom markerized the first serve is over-compression cost. This PR splits the signal so the actionable part is visible. Request-local, no store lookups: the client resends full history each turn and the pipeline recompresses it deterministically, so the current request already holds the evidence. `TransformPipeline.apply` passes `current_messages` into `parse_messages(compressed_messages=...)`. For each counted reread group, if the transformed copy of the **first serve** carries a CCR retrieval marker and its original text is gone, the group's counted repeats go into `reread_compressed_tokens`. Lossless reshaping (no marker) is deliberately not attributed. Closes #899. ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - `parser.py`: `parse_messages` gains an optional `compressed_messages` param; the content-hash reread loop accumulates per-group `counted_tokens` and attributes them to `reread_compressed_tokens` when the first serve's transformed copy carries a CCR marker (`CCR_RETRIEVAL_MARKER_RE`, kept local to avoid a transforms import cycle). - `transforms/pipeline.py`: pass `current_messages` (post-transform copy) into the existing waste-detection `parse_messages` call. - `config.py`: new `reread_compressed_tokens` WasteSignals field; `dashboard.html` + `reporting/generator.py` surface it. - Tests: `tests/test_reread_attribution.py` + WasteSignals contract update. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] New tests added for new functionality - [x] New and existing unit tests pass locally with my changes ### Test Output ```text $ pytest tests/test_reread_attribution.py tests/test_parser.py tests/test_gemini_function_response_waste.py tests/test_codex_responses_waste_signals.py -q 122 passed in 1.50s $ pytest tests/ -k "waste or pipeline or reporting or config or reread" -q 348 passed, 33 skipped, 6010 deselected # (1 unrelated env-dependent failure: test_proxy_gemini_native_integration::test_generation_config — 404, reproduces on main without these changes; needs a Gemini key locally) $ ruff check headroom/parser.py headroom/transforms/pipeline.py All checks passed! ``` ## Real Behavior Proof - Environment: local macOS, repo .venv, Python 3.11.9 - Exact command / steps: rebased onto current main to resolve conflicts with #909 (merged), then ran the reread + parser + waste suites above - Observed result: a reread whose first serve is markerized attributes to `reread_compressed_tokens`; an intact first serve and a lossless (no-marker) reshape do not. #909's re-issued-call detection (same call, different bytes) continues to count and dedup correctly alongside it — all 122 targeted tests pass. - Not tested: live proxy traffic; the one gemini-native route test above (environmental 404, not introduced here). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes **Rebased onto current main after #909 merged.** #909 added a re-issued-call reread pass *after* the original content-hash loop this PR modifies — the conflict was textual/adjacent, not a re-architecture. Resolution preserves #909's `counted_results` dedup contract and leaves its new pass unchanged; #901's attribution stays scoped to the content-hash groups it was reviewed against (attributing #909's call-key pass too would be a separate follow-up). The diff differs from the prior approval only by this reshape — worth a quick re-glance. |
||
|
Focused Instability
|
2a4d300841
|
feat(dashboard): surface compression-vs-cache net impact in Prefix Cache panel (#913)
## Summary Closes #911, and delivers the dashboard half of the question in #855 ("how do I understand the cache impact — can we add it to the dashboard?"). `GET /stats` already exposes `prefix_cache.compression_vs_cache` (`tokens_saved_by_compression`, `tokens_lost_to_cache_bust`, `cache_bust_count`, `net_tokens`) and `prefix_cache.prefix_freeze` (`busts_avoided`, `tokens_preserved`, `compression_foregone_tokens`, `net_benefit_tokens`) — built in `headroom/proxy/cost.py` — but the dashboard never rendered them. This adds a **Compression vs Cache** section to the existing Prefix Cache Impact panel: - **Saved by Compression** — tokens removed before send - **Lost to Cache Busts** — tokens lost, with observed bust count - **Net** — color-coded emerald when positive, red when negative, with a matching "Net positive / Net negative" headline pill - **Prefix Freeze Net** — net benefit of freeze decisions (preserved minus compression foregone), with busts avoided The section is gated on data presence (hidden until any underlying counter is non-zero), styled to match the existing TTL bucket cards, works in light and dark mode, and carries `data-testid` hooks. Frontend-only: no backend changes; the stats endpoint already serves every field. ## Testing New `tests/test_dashboard_cache_net_playwright.py` (mirrors the TTL playwright test): pins the rendered values, the negative-net red styling, and the hidden-when-empty behavior. 3/3 pass locally under chromium. Note for reviewers: the harness matches stubbed routes on URL **path**, because the dashboard now fetches `/stats?cached=1` — full-URL `endswith("/stats")` checks miss it and the request escapes to the real network. The pre-existing `test_dashboard_cache_ttl_playwright.py` has this exact bitrot (plus stale text assertions) and currently fails when playwright is actually installed — playwright isn't installed in CI so it silently skips. Left out of scope here; can file separately. `ruff check` and `ruff format --check` clean. Refs #855. Co-authored-by: integration-check <integration@local> |
||
|
Joel Belanger
|
0b4a4bd483
|
fix: support Copilot Business subscription auth (#641)
## Description Adds a first-party `headroom copilot-auth login` flow for Copilot subscription mode and uses the resulting Copilot OAuth token to perform GitHub's Copilot token exchange before launching the wrapped Copilot CLI. This fixes Business/Enterprise Cloud accounts where a generic GitHub/Copilot token can read Copilot account metadata but is rejected by the Copilot token exchange endpoint. It also avoids treating GitHub.com Enterprise Cloud account URLs such as `github.com/enterprises/acme` as API hostnames. Fixes #635 Related: #488, #610 Builds on #576 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [x] Documentation update - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Adds `headroom copilot-auth login` and `headroom copilot-auth status`. - Stores a Headroom-specific Copilot OAuth token under Headroom's state dir. - Exchanges reusable Copilot OAuth tokens with Copilot Chat-compatible headers before subscription-mode launch. - Carries the resolved Copilot API endpoint into `headroom wrap copilot --subscription`. - Handles GitHub.com Enterprise Cloud URLs without synthesizing invalid `api.github.com/enterprises/...` hosts. - Adds focused unit tests and README guidance for subscription login. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```console ruff check headroom/copilot_auth.py headroom/cli/copilot_auth.py headroom/cli/main.py headroom/cli/__init__.py headroom/cli/wrap.py tests/test_copilot_auth.py tests/test_cli/test_copilot_auth.py tests/test_cli/test_wrap_copilot.py tests/test_copilot_subscription_smoke.py # All checks passed! ruff format --check headroom/copilot_auth.py headroom/cli/copilot_auth.py headroom/cli/main.py headroom/cli/__init__.py headroom/cli/wrap.py tests/test_copilot_auth.py tests/test_cli/test_copilot_auth.py tests/test_cli/test_wrap_copilot.py tests/test_copilot_subscription_smoke.py # 9 files already formatted python -m py_compile headroom/copilot_auth.py headroom/cli/copilot_auth.py headroom/cli/main.py headroom/cli/__init__.py headroom/cli/wrap.py tests/test_copilot_auth.py tests/test_cli/test_copilot_auth.py tests/test_cli/test_wrap_copilot.py tests/test_copilot_subscription_smoke.py uv run --no-project --with pytest --with pytest-asyncio --with click --with rich --with opentelemetry-api --with pydantic --with tiktoken --with 'litellm==1.82.3' --with fastapi --with uvicorn --with 'httpx[http2]' --with openai --with mcp --with magika --with zstandard --with websockets --with onnxruntime --with transformers --with watchdog --with sqlite-vec pytest tests/test_copilot_auth.py tests/test_cli/test_copilot_auth.py tests/test_cli/test_wrap_copilot.py tests/test_cli_proxy_env.py tests/test_copilot_subscription_smoke.py # 127 passed ``` Local note: `uv run pytest ...` against the project currently fails before running tests because `uv.lock` has an unrelated `gitpython` wheel/version mismatch. ## Manual Validation I tested this with an existing GitHub Copilot Business subscription associated with a GitHub.com Enterprise Cloud account. The Enterprise Cloud value I tested was in the form: ```text github.com/enterprises/<enterprise> ``` The tested flow was: ```text headroom copilot-auth login headroom wrap copilot --subscription -- --model gpt-5.4 ``` This validated that Headroom does not treat github.com/enterprises/<enterprise> as a Copilot API hostname. Instead, token exchange uses GitHub.com and Headroom routes subscription-mode traffic to the Copilot API endpoint returned by GitHub for the signed-in account. I did not test this with GitHub Enterprise Server or a custom enterprise domain such as ghe.example.com. No tokens, request IDs, or organization-specific identifiers are included in this PR. ## Real Behavior Proof - Environment: macOS Darwin, Python 3.12.7, local checkout on `codex/copilot-business-auth`. - Exact command / steps: Ran `headroom copilot-auth login`, then launched `headroom wrap copilot --subscription -- --model gpt-5.4` with a GitHub Copilot Business subscription tied to a GitHub.com Enterprise Cloud account. - Observed result: Headroom did not treat `github.com/enterprises/<enterprise>` as a Copilot API hostname; token exchange used GitHub.com and subscription traffic was routed to the Copilot API endpoint returned for the signed-in account. The latest focused Copilot auth/proxy tests pass locally (`127 passed`). - Not tested: GitHub Enterprise Server or custom enterprise domains such as `ghe.example.com`; Windows Credential Manager integration still needs confirmation from someone on Windows. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing targeted unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes Acknowledgement: the OAuth/token-exchange behavior was informed by `anomalyco/opencode-copilot-auth` by Aiden Cline. No tokens are printed by the new login/status commands; only a short SHA-256 fingerprint is displayed for troubleshooting. The interactive login is included because the missing piece is not just an Enterprise URL or routing hint. For GitHub.com Enterprise Cloud accounts, URLs like `github.com/enterprises/acme` identify the enterprise account but are not Copilot API hostnames; token exchange still happens through GitHub.com and then returns the account-specific Copilot API endpoint. A command-line enterprise argument can help for true GitHub Enterprise Server/custom-domain deployments, but it cannot produce the Copilot OAuth token class that the token-exchange endpoint accepts. Ideally, Headroom would avoid an extra interactive login and reuse an existing GitHub/Copilot CLI session everywhere. In practice, some reusable-looking tokens can read Copilot account metadata but are rejected by Copilot token exchange, which leaves Business/Enterprise Cloud users with missing model catalogs. The explicit login command is the smallest independent way to obtain and persist the token needed for that exchange without asking users to pass a secret on the command line. --------- Co-authored-by: jbelanger <your-username@users.noreply.github.com> |
||
|
Focused Instability
|
7d4ae86ec0
|
feat(parser): detect re-issued identical tool calls as reread waste (#909)
Fixes #908 ## Problem Reread waste detection matches `tool_result` blocks by exact `content_hash` only. Two gaps hide a common waste pattern — the agent re-issuing the *same tool call* and paying full price for a near-identical result: 1. **Byte-different results escape matching.** Same tool, same arguments, but the second result differs trivially (embedded mtimes, timestamps, ordering) → different hash, zero reread counted. 2. **Anthropic `tool_use` parts were dropped entirely** in `parse_message_to_blocks` — only OpenAI-style `message.tool_calls` produced `tool_call` blocks, so Anthropic/Strands traffic had no call-side record at all. ## Fix - Parse Anthropic `tool_use` / Strands `toolUse` content parts into `tool_call` blocks (same shape as the OpenAI path: `function_name`, `tool_call_id` flags). - Tag every `tool_call` block with a canonical `call_key` = hash(name + arguments re-serialized with sorted keys), so `'{"path": "a.py", "lines": 100}'` (OpenAI JSON string) and `{"lines": 100, "path": "a.py"}` (Anthropic dict) hash equal — covered by a cross-format parity test. - Second reread pass in `parse_messages` groups calls by `call_key`: repeat invocations past the existing `REREAD_ADJACENT_GAP` polling guard count their **result** tokens into `reread_tokens`, subject to the existing `REREAD_MIN_TOKENS` floor. Results already counted by the content-hash pass are skipped, so byte-identical repeats are never double-counted. No new `WasteSignals` field — a byte-different re-fetch of an identical call is reread waste by the existing definition. Detection is Python-only (`parser.py`); no Rust parity surface. ## Proof Re-reading the same file twice, 7 messages apart, second serve differing only by an mtime line: ``` main: tool_call blocks: 2, reread_tokens: 0 this branch: tool_call blocks: 2, reread_tokens: 381 ``` ## Testing - 11 new tests (`TestCallArgMatchReread`): changed-result repeat counted (OpenAI + Anthropic + Strands formats), byte-identical repeat counted exactly once, polling gap skipped, different args not matched, sub-floor results skipped, repeat without result ignored, canonical-key normalization, cross-format call_key parity. - Full `tests/test_parser.py` suite: 87 passed. Consumer regression sweep (reporting, config, request outcome, read lifecycle, observability, storage): 188 passed. - `ruff check` + `ruff format --check` + `mypy headroom/parser.py` clean. Co-authored-by: integration-check <integration@local> |
||
|
Focused Instability
|
0632eba6c3
|
fix(policy): correct warm-cache penalty in net_mutation_gain to (S + dT) (#903)
Fixes #906. ## What Part of #904 (net-cost policy completion tracking). Follows up #856 / #857 with the corrected gain term raised in [this #856 comment](https://github.com/chopratejas/headroom/issues/856#issuecomment-4679706939) — prerequisite for P2 (pipeline consumption), which would otherwise wire in a formula that is always-pro-mutation by exactly `P_alive·(w−r)·ΔT`. ## Why the corrected form is right With a live cache, the ΔT tokens a mutation removes are **already cache-written** — keeping them costs only reads (`ΔT·r·R`), so a mutation cannot avoid a fresh write of them. Blending alive (`ΔT·r·R − (w−r)·S`) and dead (`ΔT·(w + r·(R−1))`, no suffix penalty) cases over `P_alive`: ``` gain = ΔT·(w + r·(R−1)) − P_alive·(w−r)·(S + ΔT) ``` Three independent confirmations: 1. **Direct cost check** (w=1.25, r=0.1, warm, ΔT=50K, S=10K, R=2): keeping costs 60K·0.1·2 = 12,000 in reads; mutating costs 10K·1.25 (suffix rewrite, the first of the R touches) + 10K·0.1 (remaining read) = 13,500 — mutation loses 1,500, matching the corrected gain of −1,500. The old form said +56,000. 2. **The issue's own anchors**: corrected break-even is exactly `R = 11.5·S/ΔT` → 2K/50K = 287.5 (~290, as the issue says) and 50K/10K = 2.3 — the spec text's anchor numbers can only be derived from the corrected penalty. The implemented form gave 276 and *negative*. 3. **Internal consistency**: `break_even_reads` already shipped with the ~11.5·S/ΔT shape; this PR reconciles `net_mutation_gain` with it (and drops break_even's stray −1 term). ## Behavior changes (formula is still dead code — nothing consumes it yet) - 50K-shave/10K-suffix/R=3 golden: +61,000 → **+3,500** (tight win, consistent with 2.3-read break-even). - 2K-shave/50K-suffix/R=10 golden: −53,200 → **−55,500**. - S=0 boundary: an edit of already-cached content with no suffix is profitable whenever ≥1 read remains (`gain = ΔT·r·R`), and exactly 0 at R=0 warm. Not-yet-cached (live-zone) content should bypass the formula — now documented on both implementations. Rust + Python goldens updated in lockstep: 13 Rust + 19 Python tests green. ## Next (separate PRs) - **P2**: flag-gated consumption (`HEADROOM_NET_COST_POLICY=1`) with decision telemetry. - **P3**: batch deep edits (reclaim threshold), idle-timer compaction near TTL lapse. Co-authored-by: integration-check <integration@local> |
||
|
Focused Instability
|
b9e27614c6
|
fix(codex): compute waste signals on the OpenAI Responses path (#898)
## Problem Fixes #820. `headroom codex` traffic through `handle_openai_responses` never produced waste signals: the path compresses via CompressionUnits (not `TransformPipeline`, which is where waste detection lives), and the minimal `messages` list it synthesises only covers `instructions` + string-typed `input` — list-typed `input` (every real multi-turn Codex session) is dropped entirely. Tool output never reached `parse_messages`, so the dashboard "What Headroom Removed" stayed empty and the new `reread` signal (#853/#854) was blind for Codex. ## Fix (telemetry-only) 1. **`_responses_input_to_waste_messages(instructions, input_data)`** — converts a Responses payload to OpenAI-style messages for waste parsing only. Tool output items (`function_call_output`, `custom_tool_call_output`, `local_shell_call_output`, `apply_patch_call_output`) become `role="tool"` messages (with `tool_call_id`); `message` items keep their role and joined part text; string/part-list `output` and `content` both handled. 2. **`handle_openai_responses`** parses that list behind the same >100 saved-token gate `TransformPipeline.apply` uses, fail-open, and threads the result into the non-streaming `RequestOutcome` and the streaming branch. 3. **`_stream_response` / `_finalize_stream_response`** gain an optional `waste_signals` param passed through to `RequestOutcome.from_stream` (which already supported it). Default `None` — the other callers are unaffected. 4. `OPENAI_RESPONSES_OUTPUT_TYPES` now aliases the module-level frozenset the converter uses (single source; usage is membership-only, no behavior change). The existing `role="tool"` parsing from #815 handles the rest: tool_result blocks, waste flags, and reread grouping all apply. ## Tests `tests/test_codex_responses_waste_signals.py` — 13 new tests covering part-text extraction (string/part-list/non-text), conversion (roles preserved, all four output item types, tool_call_id, skipped unusable items, non-list input), and parsing (tool_result blocks + `json_bloat` from `function_call_output`; identical outputs far apart count as `reread`). Local regression sweep: responses compression units, codex routing/aliases/contract parity, responses bypass/compaction/T3-replay, request outcome, all streaming suites — 168 tests green. ## Live proof Mock `/v1/responses` upstream on a real port, proxy with `optimize=True`; list-typed `input` with a large `function_call_output` served twice (5 messages apart) plus compressible assistant bulk: ``` waste_signals: { "json_bloat": 20448, "reread": 8525, ... } PROOF OK: codex responses waste visible ``` ## Notes - Sibling of #897 (Gemini functionResponse waste signals) — same bug class from #813's matrix, independent code paths, no conflicts. - The WS Responses path (`handle_openai_responses_ws`) still computes no waste signals; left as a follow-up since its outcome plumbing differs. Co-authored-by: integration-check <integration@local> |
||
|
Focused Instability
|
9b0c840dd7
|
fix(gemini): surface functionResponse payloads to waste-signal detection (#897)
## Problem Fixes #819. Gemini `functionResponse` parts are preserved verbatim on the wire (by design — they are never compressed), but their payloads never reached `parse_messages`: `_gemini_contents_to_messages` only extracts `text` parts. Tool output — where most waste lives — contributed nothing to waste detection on either Gemini path, so `json_bloat`, `repetition`, and the new `reread` signal (#853/#854) were all blind to it. ## Fix (telemetry-only) 1. **`_gemini_contents_to_messages(..., include_function_responses=True)`** — new keyword-only flag. When set, each `functionResponse` payload is additionally emitted as a `role="tool"` message (dict payloads JSON-serialized, strings passed through, missing/`None` responses skipped). `preserved_indices` semantics are unchanged: the entries are still restored verbatim on the wire. 2. **`TransformPipeline.apply(..., waste_messages=...)`** — new optional kwarg (popped before transforms, like `record_metrics`). When provided, the waste-signal parse runs over this richer list instead of the transform input. Transforms, token accounting, and savings deltas are untouched — this is why the richer list is not simply fed to the pipeline: compressed copies of preserved entries are discarded on rebuild, which would corrupt savings reporting. 3. Both Gemini `generateContent` paths (native + Cloud Code Assist) build the enriched list and pass it through. The existing `role="tool"` parsing from #815 handles the rest: tool_result blocks, waste flags, and reread grouping all apply. ## Tests `tests/test_gemini_function_response_waste.py` — 11 new tests: - conversion: default unchanged (regression), dict/string payloads, missing response skipped, text-before-tool ordering, preserved_indices unchanged, circular-reference fallback - parsing: functionResponse payload produces tool_result blocks + `json_bloat`; identical payloads far apart count as `reread` - pipeline: `waste_messages` overrides the waste source, does not affect transform output/token counts, falls back to transform input when absent Full local sweep of touched suites: gemini multimodal, parser, safety rails, canonical pipeline — green. The 13 failures in `test_proxy_gemini_*_integration.py` are credential-dependent and identical on clean `main`. ## Live proof Mock Gemini upstream on a real port, proxy with `optimize=True`; conversation with a large functionResponse payload served twice (5 messages apart) plus compressible model text: ``` waste_signals: { "json_bloat": 35003, "reread": 11673, ... } PROOF OK: waste visible, wire verbatim ``` Upstream received both `functionResponse` entries byte-identical to the client request. ## Known limitations / follow-ups - The Cloud Code Assist path passes `waste_messages` but does not yet consume `result.waste_signals` into a recorded outcome (pre-existing gap; the native path records it). - Requests where **all** content entries are preserved (pure functionResponse/media conversations) early-exit before the pipeline and still produce no waste signals. - Codex/Responses-API counterpart is #820 (separate PR). Co-authored-by: integration-check <integration@local> |
||
|
gglucass
|
8c00f7103c
|
fix(codex): poll /wham/usage for subscription limits (handshake no longer sends x-codex-* headers) (#924)
## Description Codex's subscription usage window (primary/secondary rate-limit gauges) stopped populating for ChatGPT-OAuth sessions. This PR restores it by polling Codex's dedicated usage endpoint instead of relying on response headers that are no longer sent. ### Why the previous approach no longer works The existing code populates `CodexRateLimitState` from `x-codex-*` rate-limit headers captured on the `/v1/responses` WebSocket handshake (`update_from_headers` at WS accept). That worked when OpenAI returned `x-codex-primary-used-percent`, `x-codex-primary-window-minutes`, etc. on the handshake response. OpenAI has since stopped sending those headers on the ChatGPT WebSocket handshake. I confirmed this by faithfully replaying a real Plus-account handshake (both `prewarm` and regular `request_kind`): no `x-codex-*` headers come back on either. This matches OpenAI's own move to a dedicated usage endpoint (`GET /backend-api/codex/usage` in CodexApi mode) and reports such as openai/codex#14728. So `update_from_headers` now runs on every accept but finds nothing to parse, and the window silently stays empty. The headers aren't coming back, so there is nothing to fix in the parsing path. The data now lives behind a request we have to make ourselves. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `subscription/codex_rate_limits.py`: - `parse_codex_usage_payload()` / `update_from_usage_payload()` — map the `GET /backend-api/wham/usage` JSON (`rate_limit.primary_window` / `secondary_window` with `used_percent`, `limit_window_seconds`, `reset_at`; `credits`; `rate_limit_reached_type`) into the existing `CodexRateLimitState`. `limit_window_seconds` is converted to window-minutes with the same round-up codex-rs uses (`(secs + 59) // 60`). - `maybe_schedule_usage_poll()` — fire-and-forget, throttled to one request per 60s, scoped to ChatGPT sessions (requires both a Bearer token and `ChatGPT-Account-Id`; API-key traffic is skipped). Uses an in-flight guard so concurrent accepts don't stack polls. Endpoint is overridable via `HEADROOM_CODEX_USAGE_URL`. - `proxy/handlers/openai.py`: - At the Codex WS accept site, after the now-usually-empty `update_from_headers` block, schedule the usage poll. Wrapped in `contextlib.suppress` and fully non-blocking so it can never delay or fail the WebSocket accept. The old header-capture path is intentionally left in place as a no-cost fallback in case OpenAI restores the headers. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed (live `/wham/usage` replay against a Plus account returned HTTP 200 with the expected schema; payload fixture in tests mirrors that real shape) ## Test Output ``` $ uv run pytest tests/test_codex_rate_limits.py -q ........................................ [100%] 41 passed in 0.16s $ uv run ruff check headroom/subscription/codex_rate_limits.py headroom/proxy/handlers/openai.py tests/test_codex_rate_limits.py All checks passed! $ uv run mypy headroom/subscription/codex_rate_limits.py Success: no issues found in 1 source file ``` ## Additional Notes - New tests cover: full-payload mapping, window-minutes round-up, credits balance kept only when `has_credits`, promo object vs string, empty payload returns `None`, missing `used_percent` skipped, header-gating (requires Bearer + account-id), poll throttling, and no-event-loop safety. - Scoping to `ChatGPT-Account-Id` keeps the poll off API-key traffic, and the 60s throttle plus in-flight guard bound it to at most one lightweight GET per minute per running proxy. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Michael Sam
|
6d3f39f213
|
feat: add dashboard agent usage stats (#814)
## Description Add a clear dashboard view for per-agent token usage so end users can see Cursor, Claude, Codex, and other detected clients with before/after token counts, tokens saved, and savings percentages. The stats API now exposes a stable `agent_usage` object that the dashboard renders near the top of the session view. Fixes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made ### New Files **Tests:** - `tests/test_dashboard_agent_usage.py` — Covers agent classification, exact per-request aggregation, and aggregate fallback behavior. ### Modified Files - `headroom/proxy/server.py` — Adds per-agent usage aggregation to `/stats` with before tokens, after tokens, output tokens, saved tokens, savings percentage, source, providers, and models. - `headroom/dashboard/templates/dashboard.html` — Adds a prominent Agent Usage panel with totals, coverage status, per-agent token-flow bars, request counts, before/after tokens, saved tokens, and share of savings. ## Testing - [x] Unit tests pass: `.venv312/bin/pytest tests/test_dashboard_agent_usage.py` - [x] Linting passes: `.venv312/bin/ruff check headroom/proxy/server.py tests/test_dashboard_agent_usage.py` - [x] Diff whitespace check passes: `git diff --check origin/main...HEAD` - [x] Dashboard smoke render: local proxy on `127.0.0.1:8790`, captured Chrome headless screenshot of `/dashboard` - [x] New tests added for new functionality ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing relevant unit tests pass locally with my changes - [ ] I have made corresponding changes to the documentation - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes The agent usage panel uses exact request-log data when available. If detailed request logs are empty, it falls back to aggregate provider/model request counts and labels the coverage as aggregate fallback so users are not misled. |
||
|
Logan Kang
|
dff6a19946
|
fix(codex): write canonical hooks feature flag and migrate deprecated codex_hooks (#743)
## Description `headroom init codex` writes the hooks feature flag into `.codex/config.toml` under the key `codex_hooks`. Codex renamed the canonical key to `hooks` and kept `codex_hooks` as a legacy alias (openai/codex#20522). Current Codex builds warn about `[features].codex_hooks` and tell users to use `[features].hooks` instead, so configs written by headroom should stop emitting the deprecated key. This PR switches headroom to write the canonical `hooks` key and **migrates existing configs in place**. The migration is the tricky part: a config can already contain `codex_hooks`, `hooks`, or both, in any order, inside or outside headroom's marker block — and a naive replace can emit a *duplicate* `hooks` key, which is invalid TOML that Codex rejects outright. The fix strips every `codex_hooks` line up front (any value, anywhere) — mirroring the existing top-level key cleanup in `_ensure_codex_provider` (#260) — then guarantees `hooks` is present without ever duplicating it, and respects a user-managed `hooks` value that lives outside our marker block. Fixes: N/A (no tracking issue — surfaced while aligning with Codex >= 0.129; related upstream context: openai/codex#20522 and the warning behavior discussed in openai/codex#22148) ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Write the canonical `hooks` key instead of the deprecated `codex_hooks` in `_ensure_codex_feature_flag` (`headroom/cli/init.py`). - Strip any `codex_hooks` line (any value, inside or outside the marker block) before ensuring the flag, so re-running `init` migrates a legacy config instead of leaving a stale key or producing a duplicate `hooks` key (invalid TOML). - Respect a user-managed `hooks` value found outside headroom's marker block (e.g. `hooks = false`); only the deprecated alias is removed. - Make the insert/create paths match `_replace_marker_block`'s normalisation so re-running `init` is byte-idempotent. - Extract a `_codex_feature_block()` helper to remove the 4x duplicated marker block assembly. - Add regression tests for the previously-broken edge cases. ## Testing - [x] Unit tests pass (`pytest`) — affected module fully green (see output) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom/cli/init.py`) - [x] New tests added for new functionality - [x] Manual testing performed (reproduced each edge case against the patched function via `tomllib.loads`) ## Test Output ``` $ pytest -v tests/test_cli/test_init_cli.py -k "feature_flag or hooks_feature" tests/test_cli/test_init_cli.py::test_init_codex_merges_feature_flag_into_existing_table PASSED tests/test_cli/test_init_cli.py::test_init_codex_creates_hooks_feature_flag_on_first_init PASSED tests/test_cli/test_init_cli.py::test_ensure_codex_feature_flag_replaces_existing_marker PASSED tests/test_cli/test_init_cli.py::test_ensure_codex_feature_flag_migrates_legacy_codex_hooks_key PASSED tests/test_cli/test_init_cli.py::test_ensure_codex_feature_flag_migrates_when_both_keys_present PASSED tests/test_cli/test_init_cli.py::test_ensure_codex_feature_flag_migrates_when_keys_reversed PASSED tests/test_cli/test_init_cli.py::test_ensure_codex_feature_flag_drops_legacy_key_outside_marker PASSED tests/test_cli/test_init_cli.py::test_ensure_codex_feature_flag_is_idempotent PASSED tests/test_cli/test_init_cli.py::test_ensure_codex_feature_flag_creates_features_section_when_missing PASSED ======================= 9 passed, 45 deselected in 0.24s ======================= $ pytest -q tests/test_cli/test_init_cli.py 54 passed $ ruff check . All checks passed! $ mypy headroom/cli/init.py Success: no issues found in 1 source file ``` Note: the broader `tests/test_cli/` run has one unrelated failure (`test_wrap_copilot_auto_detects_running_proxy_backend`) caused by a real proxy already bound to port 8787 in the local environment — it fails identically on a clean checkout without this change. ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (none required) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable (managed by release-please; generated from the conventional commit, not edited by hand) ## Screenshots (if applicable) N/A ## Additional Notes - **Why the duplicate-key path matters:** TOML forbids duplicate keys, so a `[features]` table containing both `codex_hooks` and `hooks` (which the old in-place migration could produce) makes Codex reject `config.toml` entirely. The new "strip then ensure" approach can never emit two `hooks` lines. - **Version provenance:** the `codex_hooks` -> `hooks` rename landed in openai/codex#20522, first shipped in Codex `rust-v0.129.0`. `codex_hooks` remains a working legacy alias, but current Codex builds can warn users to move to `[features].hooks`. - **Idempotency:** running `headroom init codex` repeatedly now produces a byte-stable `config.toml`, so there is no churn on re-init. |
||
|
Focused Instability
|
b7350aa29c
|
ci: run dashboard playwright tests in a dedicated job (#921)
## Summary Closes #920. Follow-up noted in #915. The dashboard Playwright tests guard on `pytest.importorskip("playwright...")` and no CI job installs playwright, so they have skipped on every CI run since they were added — which is how the bitrot fixed in #915 went unnoticed. This adds a `test-dashboard-ui` job to `ci.yml`, same shape as `test-agno`: - installs the prebuilt wheel `[dev]` + playwright, then `playwright install --with-deps chromium` - runs `pytest tests/test_dashboard_*_playwright.py` — the stub-based tests only (all routes mocked via `page.route`, no network); the glob also picks up the CVC panel tests from #913 once that merges - sets `HEADROOM_PLAYWRIGHT_ARTIFACT_DIR` and uploads the captured dashboard screenshots as a workflow artifact (7-day retention), so every CI run leaves a visual record of the rendered dashboard Deliberately excluded: `tests/test_dashboard/test_live_feed.py` — it navigates to a live proxy on `localhost:8787` and would fail on a runner with nothing listening. The main test shards keep skipping playwright tests (playwright stays uninstalled there), so nothing double-runs. ## Testing - `yaml.safe_load` parses the workflow; the `workflow-validation` CI job (actionlint + act) runs on this PR since it touches `ci.yml` - The test this job will run passes locally: `tests/test_dashboard_cache_ttl_playwright.py` — 1 passed (chromium) - This PR's own CI run exercises the new job end-to-end |