mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
1553 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Devanshi Vyas
|
05bd56bcb6
|
fix(wrap): track shared proxy clients with markers (#877)
## Description Replace argv-based proxy client detection with per-port wrap client markers so cleanup and ephemeral restarts do not tear down a shared proxy while another wrapped session is still attached. Also prune stale markers, guard against PID reuse when process identity is available, and add coverage for the marker-based lifecycle behavior. Fixes #804 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Testing Describe the tests you ran to verify your changes: - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
gglucass
|
2269e40bde
|
feat(proxy): log compressed messages alongside original request (#261)
## Description Expose the post-compression message list that was actually sent upstream as a new `compressed_messages` field on `RequestLog`, paired with the existing (now consistently pre-compression) `request_messages`. Consumers of `/transformations/feed` — dashboards and any downstream observability — can now diff the two sides of a compression to see exactly what the pipeline stripped, replaced, or kept. Turns an abstract "saved N tokens" into a legible before/after. Gated by the same `log_full_messages` flag as `request_messages` so the two sides stay in sync; it's pointless to store one without the other. Also fixes a latent correctness bug: today's `request_messages` field is inconsistent across the four `RequestLog` construction sites — sometimes it's the pre-compression snapshot, sometimes it's the mutated `body["messages"]` (which is the compressed list, because the proxy mutates `body` in place before the log call). After this change, `request_messages` always means pre-compression and `compressed_messages` always means what went upstream. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) Note on "breaking": strictly speaking this is a semantic correction of an inconsistently-populated field, not a schema break. The field name `request_messages` is unchanged and the JSON shape is unchanged; what changes is that the field now consistently holds the pre-compression list. Consumers that treated it as "whatever messages we have" continue to work. Consumers that depended on the accidental post-compression value (if any existed) would shift to `compressed_messages`. ## Changes Made - **`headroom/proxy/models.py`**: `RequestLog` gains `compressed_messages: list[dict] | None = None`. Doc comment explains it's paired with `request_messages` and gated by the same `log_full_messages` flag. - **`headroom/proxy/handlers/anthropic.py`** (2 sites — Bedrock non-streaming and main non-streaming): `request_messages` now consistently sources from `original_messages` (the pre-compression snapshot at line 724), `compressed_messages` sources from `body["messages"]` (the compressed list after in-place mutation at line 1189). Both gated symmetrically. - **`headroom/proxy/handlers/streaming.py`** (2 sites — main streaming in `_finalize_stream_response`, Bedrock streaming in `_stream_response_bedrock`): same treatment. `_stream_response_bedrock` gains a new `original_messages: list[dict] | None = None` parameter so it has access to the pre-compression snapshot; the sole caller in `anthropic.py` now threads it through. - **`headroom/proxy/server.py`**: `/transformations/feed` adds `compressed_messages` to the JSON payload alongside the existing `request_messages` / `response_content`. *Split into a separate preceding commit is a one-time EOL normalization to LF — the file blob in history carries CRLF but `.gitattributes` declares `*.py text eol=lf`, so any contributor editing `server.py` triggers the same whole-file renormalization. Separating the two commits keeps this feature commit's diff at a single line.* - **`headroom/proxy/request_logger.py`**: `compressed_messages` is stripped from the JSONL file log and from `get_recent()` alongside the existing `request_messages` / `response_content` stripping. `get_memory_stats()` also counts it toward the deque's byte budget. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .` and `ruff format --check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed (via the Headroom Desktop client that consumes `/transformations/feed` — confirmed both fields arrive and render) Test coverage added/extended: - `tests/test_proxy/test_request_logger.py` (new file): round-trip unit tests for `RequestLogger`. Confirms `get_recent` strips both sides (pre + post), `get_recent_with_messages` exposes both, and the JSONL file log drops both when `log_full_messages=False`. - `tests/test_proxy/test_transformations_feed.py`: extended to assert `compressed_messages` appears in the endpoint payload alongside `request_messages` / `response_content`. - `tests/test_proxy_streaming_request_logger.py`: existing include/omit tests updated to assert both sides populate when the flag is on and both are `None` when it's off. ## Test Output ``` $ uv run ruff check headroom tests All checks passed! $ uv run ruff format --check headroom tests 614 files already formatted $ uv run pytest tests/test_proxy/test_request_logger.py tests/test_proxy_streaming_request_logger.py tests/test_proxy/test_transformations_feed.py -v ... tests/test_proxy/test_request_logger.py::test_get_recent_strips_compressed_messages_alongside_request_and_response PASSED tests/test_proxy/test_request_logger.py::test_get_recent_with_messages_returns_compressed_messages PASSED tests/test_proxy/test_request_logger.py::test_jsonl_file_strips_both_sides_when_log_full_messages_disabled PASSED tests/test_proxy_streaming_request_logger.py::test_finalize_stream_response_logs_original_and_compressed_messages PASSED tests/test_proxy_streaming_request_logger.py::test_finalize_stream_response_omits_messages_when_log_full_messages_disabled PASSED tests/test_proxy/test_transformations_feed.py::test_transformations_feed_returns_messages PASSED ... ============================== 11 passed in 5.36s ============================== ``` ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas (the two-sided gating at each log site, the `_stream_response_bedrock` parameter addition, and the `get_memory_stats` accounting) - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes ### Non-Anthropic backends `handlers/openai.py` and `handlers/gemini.py` do not currently emit `RequestLog` entries at all — only Anthropic and the shared streaming paths do. This PR therefore only populates `compressed_messages` on Anthropic traffic (which is what `/transformations/feed` shows today). Wiring OpenAI and Gemini into `RequestLogger` end-to-end is a separate, larger gap worth its own PR. ### `server.py` EOL normalization The feature change in `server.py` is a single line. To keep the diff readable, the preceding commit is a whitespace-only `chore(proxy): normalize server.py to LF per .gitattributes` — the file blob was stored with CRLF terminators but `.gitattributes` declares `*.py text eol=lf`. Any contributor touching `server.py` triggers this renormalization; isolating it here keeps the feature commit reviewable. Happy to rebase / drop / reshape as preferred. ### Downstream desktop compatibility The Headroom Desktop client I work on now consumes `compressed_messages` and renders the pre/post pair side-by-side on the "Recent large compression" card. The desktop was updated to handle both shapes: proxies without the field render the legacy single "Request" block; proxies with the field render "Request (original, N tokens)" + "Request (compressed, M tokens)" where N/M come from `input_tokens_original` / `input_tokens_optimized`. No changes needed downstream if this PR lands. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Chris Yau
|
b4395993ae
|
fix(init): suppress hook recovery output (#760)
## Summary - silence best-effort profile recovery while `headroom init hook ensure` runs from installed hooks - suppress both Python-level stdout/stderr and child process file-descriptor output so SessionStart hooks do not emit invalid JSON - add a regression test for noisy supervisor recovery failures ## Verification - `python3 -m py_compile headroom/cli/init.py` - live local hook probe: `headroom init hook ensure --profile default --marker headroom-init-codex` exits 0 with empty output - targeted pytest was not runnable locally because `uv.lock` currently fails to parse due to an inconsistent GitPython wheel version entry |
||
|
Michael Sam
|
d2cdab268d
|
feat(proxy): add agent-90 savings profile (#830)
## Summary - add an `agent-90` savings profile with cross-agent proxy env exports - wire the profile into proxy/router runtime kwargs, including force-Kompress routing and a smaller read-protection window - expose effective savings-profile config in `/stats` and add focused regression coverage ## Type of change - [x] feat (non-breaking change which adds functionality) - [ ] fix (non-breaking change which fixes an issue) - [ ] docs - [ ] test/CI-only - [ ] refactor-only ## Testing - [x] `python3 -m py_compile headroom/agent_savings.py headroom/cli/agent_savings.py headroom/cli/main.py headroom/cli/proxy.py headroom/proxy/models.py headroom/proxy/server.py headroom/transforms/content_router.py tests/test_agent_savings.py tests/test_proxy_healthchecks.py tests/test_cli/test_wrap_persistent.py tests/test_transforms/test_content_router.py` - [x] `git diff --check` - [x] manual smoke: `agent-savings --profile agent-90 --format json` returns `HEADROOM_TARGET_RATIO=0.10` - [x] manual smoke: `proxy_pipeline_kwargs(ProxyConfig(savings_profile="agent-90"))` enables `force_kompress`, system/user compression, and `read_protection_window=2` - [x] manual smoke: Anthropic-style `tool_result` routes through Kompress with `target_ratio=0.10` - [ ] `pytest` suite not run: pytest is not installed in the available local Python environments ## Notes This keeps agent-90 as an opt-in profile. Existing defaults remain unchanged unless `HEADROOM_SAVINGS_PROFILE=agent-90` or `ProxyConfig(savings_profile="agent-90")` is set. |
||
|
jimu
|
058bcedab8
|
feat(plugins): Hermes agent headroom_retrieve plugin (#824)
## Summary Implements the Hermes-side retrieval plugin proposed in #796 (as invited — thanks for the quick response!). When Hermes routes traffic through `headroom proxy`, compressed markers are a one-way street: Hermes registers its own tools, so it never gets the `headroom_retrieve` CCR tool that Claude Code receives via MCP injection. In practice the model either re-runs the original command or — observed in the wild — treats `ccr:abc123` as a file path and tries to `cat` it. This plugin uses Hermes's user-plugin system (`~/.hermes/plugins/`) to register a native `headroom_retrieve` tool that calls the proxy's `POST /v1/retrieve` endpoint. ## What's included - `plugins/hermes/headroom_retrieve/` — `plugin.yaml` + `__init__.py` (single-file, httpx, ~100 lines) - `plugins/hermes/README.md` — install steps and proxy-side recommendations ## Design notes - **Both marker formats covered**: Kompress emits `[N items compressed ... hash=KEY]`, SmartCrusher's opaque-blob walker emits `<<ccr:HASH[,KIND,SIZE]>>`. The tool description teaches both and explicitly says markers are NOT file paths; the handler normalizes whole-marker input (`<<ccr:abc,base64,4.5KB>>` → `abc`). - **Re-compression loop guard**: retrieved originals travel back through the proxy on the next request and get re-compressed into a fresh marker, looping forever. README documents `HEADROOM_EXCLUDE_TOOLS=read_file,headroom_retrieve` as the fix (Hermes tool names don't match `DEFAULT_EXCLUDE_TOOLS`, which targets Claude Code's `Read`/`Grep`/...). - **Actionable failure modes**: 404 (TTL expired / proxy restarted) and connection-refused both return guidance to re-run the original command rather than retry. ## Relationship to existing PRs Complementary to #707 / #556 (`headroom wrap hermes`, proxy-side): those launch/route Hermes through the proxy; this gives the agent the retrieval capability once it's routed. Notably #707 disables CCR tool injection in Hermes mode precisely because Hermes must register its own tool — this plugin is that registration. ## Testing Running in production on macOS (headroom 0.23.0, pipx) and Linux (0.22.4, systemd) for a day. Verified: fresh-marker retrieval roundtrip, whole-marker hash normalization (6 input shapes), expired-hash 404 messaging, proxy-down messaging, and end-to-end via live Hermes sessions (fresh ≥500B `read_file` returns original with the documented exclude config). Closes #796 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: akb4q <zhunyunjiang@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Devanshi Vyas
|
249af6cc7b
|
fix(ccr): use shared compression store (#875)
## Description Use shared get_compression_store() singleton in MCP _get_local_store so headroom_retrieve sees proxy-compressed content. Fixes #860 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Testing Describe the tests you ran to verify your changes: - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality |
||
|
Federico Rao
|
f9384ef4b7
|
fix: suppress LiteLLM provider banner before import (#874)
## Summary - set `LITELLM_SUPPRESS_DEBUG_INFO` before importing `litellm` in the LiteLLM provider - keep the existing post-import suppression flags as a fallback - add a regression test that verifies the env flag exists before `litellm` import Fixes #613 ## Tests - `uv run pytest tests/test_startup_log_noise.py::TestLiteLLMLogSuppression -q` - `uv run ruff check headroom/providers/litellm.py tests/test_startup_log_noise.py` - `python3 -m py_compile headroom/providers/litellm.py tests/test_startup_log_noise.py` |
||
|
Focused Instability
|
bfcb07d78e
|
fix(ccr): key Rust search/diff/log markers with explicit_hash (#852)
Fixes #816 ## What The three `_persist_to_python_ccr` shims (`search_compressor.py`, `diff_compressor.py`, `log_compressor.py`) called `store.store(original, compressed)` with the default key — `SHA-256(original)[:24]` since PR #395 — while the Rust side embeds `MD5(original)[:24]` in the emitted `Retrieve more: hash=...` marker. Marker key and storage key never matched, so **every retrieval of a Rust search/diff/log marker returned "Entry not found or expired"** (inside any TTL — the symptom class reported in #714). Fix is exactly what #816 proposed: pass the marker's key via `explicit_hash=cache_key` at all three call sites, the same contract SmartCrusher has used since PR #395. No store changes needed — `store()` already validates and honors `explicit_hash`. Also corrected the stale comment in `search_compressor.py` that still claimed "both use MD5(original)[:24]". ## Tests `tests/test_ccr_rust_marker_hash_bridge.py` (companion to `test_ccr_row_drop_store_bridge.py`, which pinned the same bug class for SmartCrusher in #389): for each shim, the store entry must be retrievable under the Rust marker key AND absent under the SHA-256 default key. Verified red→green: all 3 tests fail on main with the exact issue symptom ("store has no entry under the Rust marker key ...; the marker dangles") and pass with the fix. `test_ccr_row_drop_store_bridge.py` still green. `ruff check` + `ruff format --check` clean. --------- Co-authored-by: Ash Rhodes <ashley.rhodes@king.com> |
||
|
Focused Instability
|
5f1d88ad27
|
feat: detect re-served tool results as over-compression waste signal (#854)
Closes #853 ## What Adds a `reread` waste signal: identical `tool_result` content appearing at more than one message position means the agent re-fetched something already in context — the dominant failure signature of over-compression (Manus context-engineering; JetBrains "Complexity Trap", arXiv:2508.21433). Per-request savings can't see this cost; this signal makes it visible. - `WasteSignals.reread_tokens` — new field, in `total()`, exported as `"reread"` in `to_dict()`. - `parse_messages()` groups `tool_result` blocks by their **existing** `content_hash` and counts every repeat beyond the first serve. No new hashing or tokenization; one O(blocks) dict pass. - `REREAD_MIN_TOKENS = 50` guard: short outputs ("ok", empty diffs) legitimately repeat and are skipped. Duplicates within a single message (same `source_index`) are not counted. - Works across all formats the parser already normalizes to `tool_result` blocks: OpenAI `role=tool`, Anthropic `tool_result`, Strands/Bedrock `toolResult` (#813/#815). - Flows through existing generic plumbing with zero handler changes: pipeline → `RequestOutcome.waste_signals` → Prometheus `headroom_waste_signal_tokens_total{signal="reread"}` → dashboard "Waste Detected" panel. Dashboard gains label/color entries for the new key. ## Tests 7 new tests in `tests/test_parser.py::TestRereadDetection` (red before, green after): OpenAI + Anthropic format detection, repeat-counting semantics (first serve free), single-occurrence, short-duplicate guard, same-message guard, `total()`/`to_dict()` participation. Updated 2 exact-shape assertions in `tests/test_config.py`. Local runs: `tests/test_parser.py` (72 passed), `tests/test_config.py` + outcome/reporting/observability/storage/proxy-hooks suites (190 passed), `tests/test_canonical_pipeline.py` + `tests/test_proxy_pipeline_lifecycle.py` (11 passed). `ruff check` + `ruff format --check` clean. ## Real behavior proof **Setup:** macOS (Darwin 25.5), Python 3.11.9, this branch, real proxy server (`python -m headroom.proxy.server --port 18970 --anthropic-api-url http://127.0.0.1:18971`) with a local mock Anthropic upstream returning a canned `/v1/messages` response (no real key needed). **Steps:** POSTed an Anthropic-format conversation to the live proxy: agent fetches a 14 KB JSON log array via `get_logs` tool, then fetches the identical content again under a different `tool_use_id` (the re-read). **Observed result** — `curl http://127.0.0.1:18970/metrics` after the request: ``` # HELP headroom_waste_signal_tokens_total Tokens attributed to detected waste signals # TYPE headroom_waste_signal_tokens_total counter headroom_waste_signal_tokens_total{signal="json_bloat"} 9858 headroom_waste_signal_tokens_total{signal="reread"} 4935 ``` `reread` = 4935 tokens, exactly the second serve of the ~4.9k-token tool result (json_bloat counts both occurrences ≈ 2×). `/stats` shows the same: `"waste_signals": {"json_bloat": 9858, "reread": 4935, ...}` — which is what the dashboard panel renders. Also verified the negative path live: a conversation whose tool results contain non-compressible plain code text produced no waste-signal entries (the pipeline only attributes waste when compression actually engaged, unchanged behavior). **Not tested:** Gemini `functionResponse` path (parser doesn't produce `tool_result` blocks for it — pre-existing gap tracked in #819); dashboard rendering only verified via the `/stats` payload the panel binds to, not a browser screenshot. ## Out of scope (per #853) Tool-call argument matching, compression-marker attribution, tokens-per-task metric, cache hit-rate panel. --------- Co-authored-by: Ash Rhodes <ashley.rhodes@king.com> |
||
|
Focused Instability
|
d5f58026e2
|
feat: net-cost cache mutation formula on CompressionPolicy (#856 P1) (#857)
Closes #856 **P1 of the #856 phased plan** — pure functions, zero behavior change. (Closing keyword links the issue; if P2 hasn't started when this merges, reopen #856 or it remains the design record for the P2/P3 follow-up PRs.) ## What Adds the break-even decision rule for deep (pre-cache-marker) edits to `CompressionPolicy`: ``` gain = ΔT · (w + r·(R−1)) − P_alive · (w − r) · S ``` - `net_mutation_gain()`, `should_mutate_deep()` (gain > 0), `break_even_reads()` (R = ((w−r)/r)·(S/ΔT−1) ≈ 11.5·S/ΔT) on the Rust struct (source of truth) and the Python hand-mirror, following the existing F2.1/F2.2 parity pattern. - `CACHE_WRITE_MULTIPLIER = 1.25` / `CACHE_READ_MULTIPLIER = 0.1` public constants (Anthropic 5-minute tier). - Inputs clamped (`expected_reads ≥ 0`, `p_alive ∈ [0,1]`); methods take `&self`/`self` so a follow-up can add per-mode margins. - The formula derives the existing Subscription live-zone policy as its S=0 special case rather than contradicting it. **No callers yet.** P2 (consuming this in `TransformPipeline` behind `HEADROOM_NET_COST_POLICY`, replacing the binary `live_zone_only` gate, with decision telemetry) is specified in #856 and awaits maintainer direction — this PR just lands the audited arithmetic both dispatchers will share. ## Tests Golden-value parity: 6 new Rust unit tests and 7 new Python tests assert the **identical scenario numbers** (loss −53 200 for a 2K shave under a 50K warm suffix at R=10; win +61 000 for a 50K shave under a 10K suffix at R=3; S=0 always profitable; P_alive=0 always profitable — the idle-timer window; clamping; break-even 276 reads for the 2K/50K anchor). A drift on either side trips the pair loudly, same contract as the existing field-map parity test. - `cargo test -p headroom-core --lib compression_policy`: 12 passed (6 existing + 6 new) - `pytest tests/test_compression_policy.py`: 17 passed (10 existing + 7 new) - `cargo fmt --check`, `cargo clippy -p headroom-core` clean; `ruff check` + `ruff format --check` clean ## Real behavior proof Not applicable in the runtime sense — this PR intentionally adds **no runtime behavior** (pure functions, no call sites). The arithmetic is validated against the research anchors above in both languages' test suites; live decision telemetry arrives with P2 where the formula first gates real traffic. ## Out of scope P2 (flag-gated pipeline consumption + telemetry), P3 (deep-edit batching, idle-timer compaction near TTL lapse), retiring the deprecated `volatile_token_threshold`/`max_lossy_ratio` fields — all tracked in #856. --------- Co-authored-by: Ash Rhodes <ashley.rhodes@king.com> |
||
|
Focused Instability
|
06b2625b17
|
feat: gated Markdown-KV compaction formatter (serialization-aware output) (#859)
Closes #858. ## What Adds an opt-in **Markdown-KV** renderer to the lossless-first compaction stage, plus the plumbing to pick a compaction formatter by name. Default behavior is unchanged (`csv-schema`). Format-comprehension benchmarks show models retrieve values from Markdown-KV substantially more reliably than from CSV (~60.7% vs ~44.3%) — token-cheapest is not the same as most comprehensible. This makes the trade-off selectable per workload. ## How - **`MarkdownKvFormatter`** (`compaction/formatter.rs`): keeps the `[N]{cols}` declaration line, renders each row as a Markdown list item with `key: value` lines. - Missing cells omitted entirely (the KV advantage over positional CSV). - Strings ambiguous on a line (newlines, leading/trailing whitespace, empty) render JSON-quoted; everything else raw — commas and quotes need no escaping. - Nested cells inline compact JSON; opaque cells keep the fixed `<<ccr:HASH,KIND,SIZE>>` marker contract shared by all formatters. - **`CompactionStage::from_format_name`** maps `"csv-schema" | "json" | "markdown-kv"` to presets. - **Core**: `SmartCrusher::with_compaction_format(config, name)` — standard OSS composition with the named formatter. - **PyO3 bridge**: `SmartCrusher.with_compaction_format(config, format_name)` staticmethod; `ValueError` on unknown names (loud, no silent fallback). - **Python**: `SmartCrusher(compaction_format=...)` kwarg, falling back to the `HEADROOM_COMPACTION_FORMAT` env var, default `"csv-schema"`. ## Safety - **Default-off**: the default constructor path still calls the Rust `new()` constructor, so byte-parity coverage stays on the exact production codepath. A test asserts default output is byte-identical to an explicit `csv-schema` opt-in. - The existing `lossless_min_savings_ratio` gate (0.30) still applies. Markdown-KV repeats field names per row, so it clears the gate less often than CSV and falls through to the lossy path — we never inline a "lossless" rendering that isn't actually smaller. - CCR marker format unchanged across formatters; downstream retrieval pattern-matching keeps working. - No user/assistant content dropped — the formatter is a pure rendering of the same Compaction IR. ## Tests - Rust: 10 new unit tests in `compaction/formatter.rs` (table/buckets rendering, missing-cell omission, string quoting, CCR markers, drop summary, byte-size sanity vs raw JSON). `cargo test -p headroom-core`: 894 passed. Clippy + fmt clean. - Python: `tests/test_compaction_markdown_kv.py` (10 tests) — bridge rendering end-to-end, name→preset parity with the default constructor, kwarg/env knob precedence, loud failure on unknown names, default-output-unchanged guarantee. Existing smart_crusher suite: 38 passed. --------- Co-authored-by: Ash Rhodes <ashley.rhodes@king.com> |
||
|
Focused Instability
|
c2106cbdab
|
feat: probe-based retention scoring of recorded compression events (#862)
Closes #861 ## What First piece of compression quality measurement on **real proxied sessions** (vs the existing public-benchmark evals): an opt-in recorder captures (original, compressed) message pairs at each compression event, and a deterministic offline prober scores what survived. **Recorder** (`headroom/proxy/probe_recorder.py`) - `CompressionEventRecorder` implements the existing `PipelineExtension` protocol, subscribed to `INPUT_COMPRESSED`. Registered ONLY when `HEADROOM_PROBE_RECORD_DIR` is set — off means not even constructed, zero request-path overhead. - One JSONL line per compression event that changed tokens: `{ts, request_id, provider, model, tokens_before, tokens_after, transforms_applied, original_messages, compressed_messages}`. One file per PID (no interleaving), directory mode 0700. - Fail-open everywhere: construction failure logs a warning and disables recording; runtime exceptions are already swallowed by `PipelineExtensionManager.emit`. - Enabling handler change: the two `INPUT_COMPRESSED` emit sites (anthropic + openai) add a read-only `original_messages` reference to event metadata. No copies, no behavior change for other consumers. **Probes** (`headroom/evals/session_probes.py` + `headroom evals probes` CLI) - Probe targets extracted from ORIGINAL tool-result content across three dimensions: **exact numerics** (number + key context, incl. JSON-quoted keys), **artifact trail** (paths, URLs, hex hashes, UUIDs), **error evidence** (lines matching the existing `is_error_content` heuristic). - Each target classified as **retained** (verbatim, or surviving a legitimate format conversion — punctuation-normalized match; numerics require key AND value to survive; error lines tolerate dropped JSON key prefixes), **recoverable** (absent but a CCR retrieval marker is present), or **lost** (gone with no retrieval path). - Report: aggregate retention per dimension, bucketed by compression ratio (the quality-per-ratio curve), and grouped per transform. `--json-output` for machine-readable results. - Fully offline: no LLM, no API key. The recording format is designed to feed an LLM-judge pass later (out of scope per #861). ## Tests 33 new tests (red before, green after): `tests/test_probe_recorder.py` (11 — event filtering, JSONL shape, env activation, fail-open on unusable path, 0700 dir mode) and `tests/test_session_probes.py` (22 — extraction per dimension incl. JSON-quoted numerics, retained/recoverable/lost classification, format-change survival for numerics and error lines, ratio bucketing, transform dedup, malformed-line skipping, report rendering/serialization). Both proxy lifecycle tests additionally assert the INPUT_COMPRESSED `original_messages` metadata contract end-to-end through the real anthropic/openai handlers, so a refactor cannot silently disable the recorder. Local runs: new tests + `tests/test_proxy_pipeline_lifecycle.py` + `tests/test_canonical_pipeline.py` + `tests/test_pipeline.py` — 45 passed. `ruff check` + `uvx ruff format --check` clean. ### Self-review hardening (second commit) - Hex artifact regex now requires at least one `a-f`, so bare decimal runs (timestamps, counters) no longer inflate the artifact dimension. - Inflation events (ratio > 1, the #847 territory) get an explicit `1.00+ (inflated)` ratio bucket instead of silently dropping out of the bucketed view. - `run_probes` streams recording files line by line instead of slurping them. - Documented honestly: marker recoverability is event-scoped (comparative metric, not absolute); recorder writes synchronously on the request path (diagnostic sessions, not always-on). - `headroom/evals/README.md` gained a Session Probes usage section. ## Real behavior proof **Setup:** macOS (Darwin 25.5), Python 3.11.9, this branch, real proxy (`python -m headroom.proxy.server --port 18994 --anthropic-api-url http://127.0.0.1:18995`) with a local mock Anthropic upstream (no key), `HEADROOM_PROBE_RECORD_DIR=/tmp/headroom-probe-proof/recordings`. **Steps:** POSTed three Anthropic-format conversations whose tool results carry large JSON arrays (220-row uniform logs, 3000-row logs, 800 heterogeneous events), each containing known numerics, paths, trace hashes, and one error line. **Observed** — recorder wrote `compression-events-<pid>.jsonl` (one line per event); `headroom evals probes --recordings ...`: ``` Probed 3 compression events Aggregate retention: numerics 97.7% retained, 2.3% recoverable, 0.0% lost (527 targets) artifacts 100.0% retained, 0.0% recoverable, 0.0% lost (6555 targets) errors 100.0% retained, 0.0% recoverable, 0.0% lost (3 targets) By compression ratio (tokens_after / tokens_before): ratio 0.50-0.75: numerics 100.0% retained (CSV compaction — lossless, correctly recognized) ratio 0.75-1.00: numerics 95.7% retained, 4.3% recoverable (SmartCrusher sampling — dropped values carried a CCR marker) ``` All three classifications exercised: verbatim/format-change retention on the CSV-compacted events, **recoverable** on the heterogeneous event where SmartCrusher sampled rows out behind a `Retrieve more: hash=` marker, and the injected error lines retained in every event (the error-protection gate held). The `lost` path is covered by unit tests. The first iteration of this proof exposed two real bugs — naive verbatim matching misreported lossless JSON→CSV compaction as 100% lost, and duplicated transform markers double-counted tallies — both fixed with regression tests. **Not tested live:** Gemini path (no `INPUT_COMPRESSED` emit parity — pre-existing, same gap as #819); LLM-judge scoring (out of scope per #861). ## Security Recordings contain full conversation content in plaintext: opt-in env var only, local disk only, dir mode 0700, documented in CLI help. ## Out of scope (per #861) LLM-judge dimensions (decisions/intent, next steps), ACON-style counterfactual replay, automatic rule revision. --------- Co-authored-by: Ash Rhodes <ashley.rhodes@king.com> |
||
|
Logan Kang
|
c71592d421
|
feat(memory): add opt-in Apple-GPU (MPS) embedding runtime (#766)
## Description
On Apple-Silicon Macs — especially fanless models like the MacBook Air
(M5) — running the proxy with memory context injection can pin the CPU
while embedding. The embedding work runs an uncapped session on the CPU,
saturating multiple cores, which starves the proxy's asyncio loop and
leads to request timeouts.
This PR adds an **opt-in** runtime that offloads the memory embedder to
the Apple GPU (MPS). Setting `HEADROOM_EMBEDDER_RUNTIME=pytorch_mps`
routes embedding through the torch `sentence-transformers` backend on
MPS instead of the default ONNX CPU embedder, moving the work off the
CPU and keeping the proxy responsive.
The default behavior is unchanged — the feature is strictly opt-in,
env-var only, and falls through to the existing default embedder
selection (with a warning) whenever MPS or the torch dependencies are
unavailable.
Fixes: N/A — no tracking issue (surfaced while running codex auto-review
through
the proxy on a fanless MacBook Air (M5)).
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **Runtime selection** (`headroom/proxy/memory_handler.py`): read
`HEADROOM_EMBEDDER_RUNTIME`; when set to `pytorch_mps` **and** MPS is
actually available, route the memory embedder to the torch
`sentence-transformers` backend (Apple GPU).
- If MPS is unavailable or torch/sentence-transformers is not installed,
log a warning and fall through to the existing default embedder
selection (ONNX when available, else the pre-existing local
sentence-transformers fallback) — no crash. The default (env var unset)
is unchanged. Env-var only
- **MPS serialization** (`headroom/memory/adapters/embedders.py`):
`LocalEmbedder` now funnels every `encode()` through a dedicated
single-worker `ThreadPoolExecutor` when the resolved device is MPS.
torch-MPS is not thread-safe, and the existing `run_in_executor(None,
...)` dispatch would otherwise let concurrent proxy requests call MPS
from multiple threads. CPU/CUDA keep the shared default executor
(behavior unchanged). `close()` also drops the cached model so re-use
after close re-initializes cleanly.
- **Packaging** (`pyproject.toml`): new `pytorch-mps` extra (`torch` +
`sentence-transformers`), **platform-gated to macOS** (`; sys_platform
== 'darwin'`) since MPS is Apple-Silicon-only. Deliberately left out of
`[all]` (its deps already arrive via `[ml]`/`[memory]`).
- **Tests** (`tests/test_memory/test_embedder_mps_serialization.py`):
regression coverage for the serialized executor, concurrency safety (no
SIGABRT), CPU-path default behavior, and close/re-use re-initialization.
- **Docs**: `wiki/{configuration,memory,macos-deployment}.md`,
`docs/content/docs/{configuration,installation,memory}.mdx`,
`README.md`, `CHANGELOG.md`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed (CPU-offload + concurrency profiling on
Apple Silicon)
## Test Output
```
$ pytest -v tests/test_memory/test_embedder_mps_serialization.py
collected 4 items
tests/test_memory/test_embedder_mps_serialization.py::test_cpu_uses_shared_executor PASSED [ 25%]
tests/test_memory/test_embedder_mps_serialization.py::test_mps_creates_single_worker_executor PASSED [ 50%]
tests/test_memory/test_embedder_mps_serialization.py::test_mps_concurrent_embeds_do_not_crash PASSED [ 75%]
tests/test_memory/test_embedder_mps_serialization.py::test_mps_reembed_after_close_recreates_executor PASSED [100%]
============================== 4 passed in 6.92s ===============================
$ ruff check headroom/memory/adapters/embedders.py headroom/proxy/memory_handler.py tests/test_memory/test_embedder_mps_serialization.py
All checks passed!
$ mypy headroom/memory/adapters/embedders.py headroom/proxy/memory_handler.py
Success: no issues found in 2 source files
$ pytest -q tests/test_memory/ tests/test_memory_handler_concurrent_init.py tests/test_memory_handler_native_ops.py
553 passed, 1 skipped in 13.51s
```
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
**Why MPS (and not CoreML or a thread cap):** measured on an
Apple-Silicon Mac, the default uncapped CPU embedding session saturates
the cores; the same model on MPS runs at a fraction of the CPU (≈8x
lower sustained CPU utilization in profiling) while producing
**byte-identical embeddings** (cosine distance ≈ 0 across runtimes), so
relevance/ranking is unchanged. A CoreML execution-provider path was
evaluated and rejected: the default optimized ONNX model uses fused ops
that fall back to CPU under CoreML (no offload), and a full-precision
re-export was impractical (very low throughput + multi-GB memory). MPS
via `sentence-transformers` was the only practical GPU offload.
**Why serialization is mandatory:** torch-MPS is not thread-safe —
concurrent encode calls from a multi-worker executor abort with
`-[IOGPUMetalCommandBuffer validate]: failed assertion 'commit an
already committed command buffer'` (reproduced deterministically; a
single-worker executor resolves it).
Under concurrent load the serialized single-GPU-stream throughput meets
or exceeds the parallel CPU path while using a fraction of the cores.
**Scope / boundary:** this targets the Python **memory** embedder, which
is live on the proxy request path (memory context injection). The
Rust-backed SmartCrusher compression path is unaffected and remains
non-configurable from Python by design.
**Safety:** default behavior is unchanged (ONNX, no torch). The feature
is opt-in, env-var only, macOS-gated at the packaging layer, and
degrades gracefully (warn + the existing default embedder selection)
when MPS or the dependencies are unavailable.
|
||
|
@aaronjmars
|
b4b50253f1
|
fix(security): block DNS-rebinding on /debug/* and /stats/reset via Host-header allowlist (#605)
## Summary `require_loopback` in `headroom/proxy/loopback_guard.py` guards `/debug/tasks`, `/debug/ws-sessions`, `/debug/warmup`, and `/stats/reset` by checking `request.client.host` only. A malicious website can use DNS rebinding to make a victim's browser send requests to `127.0.0.1` while the page origin (and inbound `Host:` header) still reads `attacker.com`. The IP check passes — the browser IS on loopback — and `app.add_middleware(CORSMiddleware, allow_origins=["*"], ...)` (`headroom/proxy/server.py:1678`) lets the attacker's JS read the response. This PR adds the canonical second gate: the `Host:` header must also name loopback (`127.0.0.1[:port]`, `[::1][:port]`, or `localhost[:port]`). ## Impact While the proxy binds to `127.0.0.1` by default (`headroom/proxy/server.py:2956`), any browser the user opens can be used by an unrelated tab to reach the proxy via DNS rebinding. Concretely: - `GET /debug/tasks` → leaks asyncio task names, coroutine qualnames, and (with `?stack=true`) stack depths — reveals which coroutines are mid-flight and which codepaths are warm. Useful for fingerprinting + targeting follow-on attacks. - `GET /debug/ws-sessions` → leaks live WebSocket session metadata including `upstream_url` per session. - `GET /debug/warmup` → leaks warmup registry shape (lower risk). - `POST /stats/reset` → resets in-memory proxy stats (state mutation; minor DoS of dashboards/observability). The fix path matches the OWASP DNS-rebinding mitigation and Starlette's `TrustedHostMiddleware` posture. ## Location - `headroom/proxy/loopback_guard.py:73` — guarded the IP check only, no `Host:` validation - `headroom/proxy/server.py:1819-1845`, `:2312` — endpoints reachable via DNS rebinding under the old gate ## Fix `require_loopback` now runs two gates: (1) the existing `request.client.host` loopback IP check, and (2) a new `Host:` header allowlist via `is_loopback_host_header(...)`. The header helper accepts `127.0.0.1[:port]`, `[::1][:port]`, `localhost[:port]`, and IPv6-mapped IPv4 (`::ffff:127.0.0.1`) — strips brackets and ports, then delegates to the existing `is_loopback_host` logic. Cross-origin browser fetches always carry the attacker's hostname in `Host:`, so rebinding requests now 404 alongside any other external attempt. Same 404 (not 403) semantics, so debug endpoints remain invisible to external scanners. The new helper is in `__all__` for explicit re-use. The existing manual-`Request`-stub unit test path (no `.headers` attribute) is preserved by an `if headers is None: return` fallback so older callers that pass a bare stub still work. Test fixtures in `tests/test_proxy_debug_endpoints.py` were updated to pin `base_url="http://127.0.0.1"` so the loopback-`Host:` invariant holds in the green-path tests, plus a new `app_and_rebinding_client` fixture and `test_debug_endpoints_block_dns_rebinding` exercising the gate at the HTTP level. Six new unit tests cover `is_loopback_host_header` (canonical accept, external reject, malformed reject, and rebinding signature). Net: +219 lines, -4 lines, in two files. **Compatibility note for operators:** a deployment that fronts the proxy behind a reverse proxy with a non-loopback Host (e.g. `headroom.local` mapped to `127.0.0.1`) and still wants `/debug/*` exposed would need to either point that reverse proxy at the upstream API endpoints only, or extend `is_loopback_host_header` with an env-var allowlist in a follow-up. The default `HEADROOM_HOST=127.0.0.1` path is unaffected. ## Detected by Aeon (manual review — no scanner rule for this; pattern matches the threat-model-claims + local-HTTP-server axes from [`skills/vuln-scanner`](https://github.com/aaronjmars/aeon-aaron/blob/main/skills/vuln-scanner/SKILL.md), priors that have surfaced 10+ similar finds across 5 languages over the last 6 weeks). - Severity: medium - CWE-350 (Reliance on Reverse DNS Resolution for a Security-Critical Action) - CWE-352 (Cross-Site Request Forgery) — adjacent - Class match: DNS-rebinding-via-loopback-bind (recurring axis in the tracker) ## Verification - `python3 -m py_compile headroom/proxy/loopback_guard.py` — parses. - The new unit tests in `tests/test_proxy_debug_endpoints.py` cover: canonical loopback Host headers (accept), external Host headers (reject), malformed/empty (reject), an HTTP-level DNS-rebinding simulation (`app_and_rebinding_client` fixture with `base_url="http://attacker.com"` + loopback client tuple), and the unchanged IP-only stub path. - Other test files that use `require_loopback` (`test_proxy_openai_responses_integration.py`, `test_proxy_openai_responses_bypass.py`, `test_proxy_dashboard_stats_cache.py`) all override it via `app.dependency_overrides[require_loopback] = lambda: None`, so they are not affected by the new gate. - I was unable to run `pytest` locally in this sandbox; please confirm the new fixtures slot in cleanly when the CI matrix runs. --- Filed by [Aeon](https://github.com/aaronjmars/aeon-aaron). --------- Co-authored-by: aeonframework <noreply@anthropic.com> |
||
|
Abhinav Kaurav
|
6b0c09ffd5
|
fix: harden Copilot API auth token handling (#557)
## Description Improve Copilot API authentication behavior by correctly handling incoming bearer tokens and ensuring required Copilot headers are present. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added token classification logic to distinguish short-lived Copilot API tokens (`tid_`) from GitHub OAuth tokens. - Updated auth flow to pass through valid existing Copilot API bearer tokens and replace unsuitable bearer tokens. - Added default `Copilot-Integration-Id` and `editor-version` headers when missing. - Improved Windows credential lookup to consider both GitHub CLI (`gh:`) and Copilot CLI credential target prefixes. - Added regression tests for pass-through, replacement, header injection, and token prefix classification. ## Testing Describe the tests you ran to verify your changes: - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ## Test Output ```text pytest -q tests/test_copilot_auth.py 29 passed in 0.40s ``` ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes This change is intentionally scoped to auth behavior and tests in: - `headroom/copilot_auth.py` - `tests/test_copilot_auth.py` --------- Co-authored-by: Abhinav Kaurav <abhinav.kaurav@e2open.com> |
||
|
Focused Instability
|
c0cadccff9
|
feat: compression safety rails — error-output protection, pipeline circuit breaker, library inflation guard (#851)
Closes #847 ## What Three safety rails, each of which only ever makes compression LESS aggressive — zero behavior change for content that compresses normally: 1. **Error-output protection** (`ContentRouter`) — failed tool calls pass through verbatim on both the OpenAI `role=tool` string path and the Anthropic `tool_result` block path. Triggered by the explicit `is_error: true` flag or the existing Rust error-indicator detector (`headroom._core.content_has_error_indicators`, previously only used for TOIN signatures). Capped by `error_protection_max_chars` (8000, ~2K tokens) so big error-laden CI logs still reach `LogCompressor`, which preserves error lines — the two features stay complementary. `protect_error_outputs=False` disables. 2. **Pipeline circuit breaker** (`TransformPipeline`) — after 3 consecutive transform failures, `apply()` passes messages through untouched for a 60s cooldown instead of re-running (and re-failing) transforms on every request. Env-tunable: `HEADROOM_PIPELINE_BREAKER_THRESHOLD` (0 disables), `HEADROOM_PIPELINE_BREAKER_COOLDOWN_S`. Passthrough results tagged `pipeline:circuit_open`; a clean run closes the breaker. Thread-safe (lock-guarded counters, `time.monotonic`). 3. **Library inflation guard** (`compress()`) — all four proxy handlers already revert when "optimization" inflates tokens; the public library path returned inflated messages as-is. Now mirrors the proxy guard and tags `inflation_guard:reverted`. ## Why Production agent research backs each rail: keeping error outputs verbatim measurably improves agent recovery (Manus context-engineering; JetBrains "Complexity Trap", arXiv:2508.21433); Claude Code added its consecutive-compaction-failure cap after telemetry showed failure loops; the inflation guard closes a library/proxy asymmetry. All three follow CONTRIBUTING's "Safety first: never drop user/assistant content, prefer false negatives." ## Tests `tests/test_compression_safety_rails.py` — 10 tests: - error protection: string path, `is_error` flag (with neutral text proving the flag alone triggers), indicator scan, size-cap fall-through, config-disable - circuit breaker: opens after threshold + passthrough, success resets count, cooldown expiry closes, env-disable - inflation guard: inflated result reverts to originals Regression: `test_transforms_content_router`, `test_pipeline`, `test_compress_api`, `test_compress_failure`, `test_canonical_pipeline`, `test_proxy_pipeline_lifecycle`, `test_observability_*`, `test_compression_policy` — 69 passed. `ruff check` + `ruff format --check` clean. --------- Co-authored-by: Ash Rhodes <ashley.rhodes@king.com> |
||
|
gglucass
|
841663da16
|
fix(proxy): make Kompress eager preload cache-only so a cold cache can't block startup (#783)
## Description `ContentRouter.eager_load_compressors()` runs a network `hf_hub_download` of the Kompress ONNX model on the **blocking startup/lifespan path**, before the proxy binds its port. On a cold cache this is unsafe: - the download can hang long enough to blow the supervisor's bind timeout, or - a native crash in the download/ML stack (an **uncatchable `Fatal Python error: Aborted` / SIGABRT**) kills the interpreter before it ever `listen()`s. Either way the supervisor sees "proxy never opened its port" and gives up. We observed this in the field from the desktop app (process aborted during `eager_load_compressors -> _load_kompress_onnx -> hf_hub_download` of `onnx/kompress-int8.onnx`, while the only Python thread was parked in the HuggingFace download file-lock; the abort came from a native thread, so `try/except` at the call site cannot catch it). The eager preload is a latency optimization and must never be able to block — or kill — startup. This change makes startup preload **cache-only**: if the model isn't already cached, we defer the download to first use (off the startup path) and bind the port normally. Warm starts are unchanged. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `onnx_runtime.hf_hub_download_local_first(...)`: added `allow_network` (default `True`). When `False`, a cache miss re-raises the local-lookup error instead of falling back to a network download. - `kompress_compressor`: added `allow_download` (default `True`) threaded through `preload()` -> `_load_kompress()` -> `_load_kompress_onnx()` / `_load_kompress_pytorch()` and the ModernBERT tokenizer load. Added `KompressModelNotCached`, raised when a cache-only load misses. Auto-mode no longer falls back to a PyTorch network download on a cache-only miss — it propagates so the caller can defer. - `content_router.eager_load_compressors()`: calls `preload(allow_download=False)`. On `KompressModelNotCached` it logs and reports the component as `"deferred"` (a status `warmup.merge_transform_status` already handles gracefully) instead of letting a cold download run on the startup path. Default (first-request) loading behavior and warm-start preload are unchanged. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed New tests in `tests/test_kompress_preload_deferral.py` cover: cache-only `hf_hub_download_local_first` never hits the network; default still falls back; cache-only ONNX load raises `KompressModelNotCached`; auto-mode does **not** trigger a PyTorch download on a cache-only miss; and `eager_load_compressors` reports `deferred` (cold) / `enabled` (warm). Existing `_load_kompress` dispatch tests updated for the new keyword-only param. > Note on environment: I do not have a clean reproduction of the native SIGABRT itself (it depends on a specific machine's HF download/ML native stack), so the "Manual testing performed" box is left unchecked. The tests target the structural fix — that startup preload can no longer perform a network download — which is the precondition for the crash. ## Test Output ``` $ uv run pytest -v tests/test_kompress_preload_deferral.py tests/test_kompress_preload_deferral.py::test_local_first_no_network_when_disallowed PASSED tests/test_kompress_preload_deferral.py::test_local_first_falls_back_to_network_by_default PASSED tests/test_kompress_preload_deferral.py::test_load_kompress_onnx_cache_miss_raises_not_cached PASSED tests/test_kompress_preload_deferral.py::test_load_kompress_auto_does_not_pytorch_download_on_cache_miss PASSED tests/test_kompress_preload_deferral.py::test_eager_load_defers_when_model_not_cached PASSED tests/test_kompress_preload_deferral.py::test_eager_load_enabled_when_model_cached PASSED 6 passed in 4.82s $ uv run pytest tests/test_transforms/test_kompress_compressor.py tests/test_transforms_content_router.py tests/test_onnx_runtime.py tests/test_proxy_warmup.py 63 passed $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/onnx_runtime.py headroom/transforms/kompress_compressor.py headroom/transforms/content_router.py Success: no issues found ``` ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable (auto-generated from conventional commits) ## Additional Notes This contains the cold-start case. A native crash in onnxruntime *session init* (as opposed to the download) on first request would still be a separate issue; it is not what was observed here (the abort was during the HF download), and isolating it would be a larger, separate change. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
gglucass
|
9bff5752bb
|
fix(learn): claude-cli streams output with idle timeout (#373)
## Description `headroom learn` with the claude-cli backend used `subprocess.run` with a hard 120s wall-clock cap and no liveness signal. A successful long analysis and a hung connection looked identical — exit 0 with "0 recommendations" was the only user-visible signal when the LLM call timed out, which silently hides genuine learnings. This PR makes the CLI backend timeout-aware, with progress detection for claude-cli and configurable wall-clock caps for every backend. Fixes #(issue number) ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Streaming claude-cli with idle timeout**: invoke `claude -p --output-format stream-json --verbose` and run a watchdog loop that drains stdout/stderr via reader threads. Each stream-json event resets an idle deadline. Kill the process if no output for `HEADROOM_LEARN_CLI_IDLE_TIMEOUT_SECS` (default 60s) or if total elapsed exceeds `HEADROOM_LEARN_CLI_TIMEOUT_SECS` (default 300s, was 120s). The final `type:"result"` event carries the assistant response, which is then parsed as JSON. Reader threads (rather than `select`) are used so the watchdog works on Windows where `select` does not support pipe handles. - **Bumped default `_CLI_TIMEOUT` from 120s to 300s** as the hard cap for all CLI backends. The previous 120s was too tight for large digests on slower networks. - **Env-var overrides** via new helper `_resolve_timeout_secs(env_var, default)`: - `HEADROOM_LEARN_CLI_TIMEOUT_SECS` — hard wall-clock cap (all CLI backends) - `HEADROOM_LEARN_CLI_IDLE_TIMEOUT_SECS` — idle cap (streaming claude-cli only) - Non-positive or non-integer values log a warning and fall back to defaults, so a typo can't disable the timeout. - **gemini-cli and codex-cli** keep `subprocess.run(timeout=hard_cap)` since they do not emit progress events. They benefit from the bumped default and the env-var override. - **CHANGELOG.md** updated under `[Unreleased]` → `### Fixed`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed (existing repro: 16k-call digest that previously timed out at 120s) New test coverage in `tests/test_learn/test_analyzer.py`: - `test_claude_cli_streams_and_parses_result_event` — happy path, fake Popen yields system/assistant/result events - `test_claude_cli_parses_fenced_result` — markdown fences in the result event still parse - `test_claude_cli_idle_timeout_kills_hang` — `HEADROOM_LEARN_CLI_IDLE_TIMEOUT_SECS=1` + a hanging stdout iterator triggers the idle watchdog - `test_claude_cli_hard_cap_kills_continuous_chatter` — continuous events with a low hard cap fire the wall-clock kill (proves idle reset alone can't keep a runaway alive) - `test_claude_cli_missing_result_event_raises` — graceful failure when no `result` event is emitted - `test_claude_cli_nonzero_exit_raises` / `test_claude_cli_unparseable_result_raises_with_context` / `test_claude_cli_not_installed_raises` — error paths - Parallel codex-cli error coverage (timeout-honors-env-override included) so the wall-clock path is exercised - `TestResolveTimeoutSecs` — unset / empty / non-integer / non-positive / valid override ## Test Output ``` $ uv run pytest tests/test_learn/test_analyzer.py ============================== 67 passed in 2.14s ============================== $ uv run ruff check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py All checks passed! $ uv run ruff format --check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py 2 files already formatted $ uv run mypy headroom/learn/analyzer.py Success: no issues found in 1 source file ``` ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - The contract assumed for claude-cli stream-json output is: each line is a JSON object with a `type` field; the final event has `type:"result"` with a string `result` field carrying the assistant text. This matches the documented Anthropic CLI behavior. If the contract changes upstream, `_call_claude_cli_streaming` raises a clear "did not emit a final \`result\` event" error rather than silently succeeding. - Backwards-compatible for users without env-var configuration: behavior just becomes "longer hard cap, plus idle watchdog for claude-cli", neither of which can falsely succeed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
gglucass
|
8f374263d3
|
feat(transforms): attribute read_lifecycle + smart_crush tags (#249)
## What Enrich the `transforms_applied` tags emitted by `ReadLifecycleManager` and `SmartCrusher` so each tag carries the specific target it acted on, instead of being an opaque counter: - `read_lifecycle:<state>` -> `read_lifecycle:<state>:<file_path>` - `smart_crush:<n>` -> `smart_crush:<n>:<tool1,tool2,...>` (tool names resolved from the assistant's `tool_calls` / `tool_use` metadata; falls back to `smart_crush:<n>` when no name resolves) Downstream UIs can then show *what* a compression acted on (which file was a stale read, which tools had their output crushed), not just that it happened. ## Note on the rebase The original revision targeted `ToolCrusher` / `tool_crush:<n>`. That transform has since been retired and replaced by the Rust-backed `SmartCrusher` (which emits `smart_crush:<n>`), so the tool-name attribution moved to `smart_crusher.py`. The `read_lifecycle` half is unchanged. ## Response-header compatibility `x-headroom-transforms` is built as `",".join(transforms_applied)`. A tag containing a comma (tool-name lists; file paths) would make that header ambiguous to split back into tags. To keep the header backward compatible, `header_safe_transforms` (`headroom/proxy/cost.py`) collapses the enriched tags back to their legacy counter shape **for the header only** -- the full enriched detail still flows through the structured `transforms_applied` list (dashboards, request logs, activity feed). Applied at all three header sites (openai / anthropic / gemini handlers). Paths containing `:` survive in `transforms_applied` because consumers bound their split to 3 parts. ## Tests - `tests/test_transforms/test_read_lifecycle.py` -- OpenAI + Anthropic tag shape, colon-in-path preservation - `tests/test_transforms/test_smart_crusher_attribution.py` -- OpenAI + Anthropic tool-name resolution, dedup, no-name fallback, id/name-missing skips - `tests/test_proxy/test_header_safe_transforms.py` -- header normalization keeps the joined header unambiguous (incl. comma-in-path) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
yehsuf
|
5dfb446da1
|
fix(health): readyz verifies upstream connectivity, not just process liveness (#744)
Closes #740 ## What `/readyz` and `/health` previously reported healthy even when the upstream API was completely unreachable (e.g. SSL certificate errors, wrong URL, network failure). The proxy would accept traffic and return 502 on every `/v1/messages` request. ## Changes - Added `_check_upstream()` async function that probes the configured upstream base URL with a HEAD request (5s timeout, result cached 30s) to verify TLS + TCP reachability without triggering an inference call - `/readyz` now calls `_check_upstream()` before building its response; returns HTTP 503 if the upstream is unreachable - `/health` exposes an `upstream` sub-check entry with `enabled`, `ready`, `status`, and `error` fields - `HEADROOM_SKIP_UPSTREAM_CHECK=1` opts out (for air-gapped or test environments) - Existing tests updated to set `HEADROOM_SKIP_UPSTREAM_CHECK=1` so unit tests don't make live network calls - Three new tests covering: opt-out via env var, 503 on upstream failure, `/health` includes upstream check ## Behaviour | Endpoint | Before | After | |---|---|---| | `/livez` | process alive | unchanged | | `/readyz` | process alive | process alive AND upstream reachable | | `/health` | no upstream info | includes `checks.upstream` with status + error | --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Focused Instability
|
34dafe69d9
|
feat(dashboard): per-model savings breakdown and expected-vs-actual cost on historical charts (#807)
Fixes #806 ## Type of Change - [x] New feature ## Changes Made **Tracker (`headroom/proxy/savings_tracker.py`)** - History checkpoints now persist the `model` alongside the existing `provider` (both `record_compression_savings` and `record_request` already receive it — it was dropped at write time). - `_normalize_model` mirrors `_normalize_provider`: legacy checkpoints without a model collapse into `"unknown"` instead of disappearing from the breakdown. No schema version bump — fields are additive. - Daily/weekly/monthly/hourly rollup buckets gain a `by_model` breakdown with the same delta fields as `by_provider` (`tokens_saved`, `compression_savings_usd_delta`, `total_input_tokens_delta`, `total_input_cost_usd_delta`). The expected no-Headroom cost per bucket/model is derivable as `total_input_cost_usd_delta + compression_savings_usd_delta`, so no pricing logic is duplicated client-side. **Dashboard (`headroom/dashboard/templates/dashboard.html`)** - The Historical Savings Trend chart gains a **Tokens / Cost** mode toggle next to the granularity toggle. - **Tokens** mode: existing aggregate area chart plus cumulative per-model savings lines for the top 5 models, with a color legend. - **Cost** mode: solid cyan line = actual input cost (with Headroom), dashed amber line = expected input cost without Headroom (actual + compression savings) — e.g. "claude-sonnet-4-6 without Headroom: $X, with Headroom: $Y" per time bucket. - Model names are **clickable** (legend entries and table rows) to isolate a single model on the chart; clicking again restores all models. The filtered model rescales to its own axis. - New **Per-Model Breakdown** table: per model, tokens saved, cost with Headroom, expected cost without Headroom, and dollars saved for the selected granularity. - The raw Checkpoints view keeps the aggregate line only: per-model lines are derived from rollup buckets and would not share its x-axis. ## Testing - 2 new tests: `test_savings_tracker_rollup_attributes_savings_per_model` (per-model attribution, deltas sum back to bucket totals, expected-cost derivation) and `test_legacy_checkpoints_without_model_collapse_into_unknown` (backward compat with pre-existing savings files). - 3 existing exact-shape assertions extended with the new `model` field; dashboard markers test extended for the new UI. ``` $ pytest tests/test_proxy_savings_history.py tests/test_dashboard tests/test_proxy_dashboard_stats_cache.py -q 24 passed, 1 skipped, 1 warning in 12.53s $ ruff check headroom/proxy/savings_tracker.py tests/test_proxy_savings_history.py All checks passed! $ mypy headroom/proxy/savings_tracker.py Success: no issues found in 1 source file ``` ## Real behavior proof Ran `headroom proxy --port 8799` against a seeded savings file (3 models, 30 checkpoints over ~3 weeks): `/stats-history` now serves per-model attribution in every rollup bucket: ```json "weekly": [{ "by_model": { "claude-sonnet-4-6": {"tokens_saved": 6900, "compression_savings_usd_delta": 6.9, "total_input_tokens_delta": 16800, "total_input_cost_usd_delta": 42.0}, "claude-opus-4-8": {"tokens_saved": 4500, "compression_savings_usd_delta": 4.5, ...}, "gpt-4o": {"tokens_saved": 4700, "compression_savings_usd_delta": 4.7, ...} }, ... }] ``` Browser-verified with Chrome DevTools against the live dashboard (no Alpine/JS console errors in any state): - Tokens mode renders 3 per-model lines + legend; Cost mode renders actual-vs-expected pair with correct legend. - Per-Model Breakdown table shows with/without-Headroom dollars per model (e.g. gpt-4o: $202.50 with vs $238.00 without). - Clicking a model (legend or table row) isolates its line, dims other legend entries, highlights the row; clicking again restores the full view. - Checkpoints granularity correctly hides per-model lines; legacy files without model fields collapse into an `unknown` row. ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] CHANGELOG.md — skipped; it is generated by release-please --------- Co-authored-by: Ash Rhodes <ashley.rhodes@king.com> |
||
|
Hc
|
2533f7703e
|
fix(ccr): make retrieval TTL configurable (#715)
## Description Make the CCR retrieval store TTL configurable so long-running agent jobs can keep `headroom_retrieve` markers resolvable beyond the previous hard-coded 300-second window. Fixes #714 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `HEADROOM_CCR_TTL_SECONDS` for the global CCR `CompressionStore` default TTL, with validation and fallback to 300 seconds. - Expose the effective CCR TTL as `store.default_ttl_seconds` from `/v1/retrieve/stats`. - Distinguish missing vs expired CCR retrieval failures in `/v1/retrieve`, `/v1/retrieve/{hash}`, tool-call handling, and CCR response handling. - Update CCR docs, README wording, and CHANGELOG for the user-facing TTL behavior. - Add regression tests for env-configured TTL, invalid env fallback, explicit TTL precedence, stats exposure, and expired retrieval detail. ## Reproduction Before this change, the proxy-global CCR store always used the default 300-second TTL when callers used `get_compression_store()` without an explicit `default_ttl`. A long-running agent job could receive a `<<ccr:...>>` marker and fail later with a generic not-found/expired response after the fixed 5-minute retention window. The new tests cover this by setting `HEADROOM_CCR_TTL_SECONDS`, creating CCR entries through the same global store used by the proxy, and asserting that stored entries and `/v1/retrieve/stats` use the configured retention. ## Real behavior proof Setup tested: - macOS 15.7.2 - Python 3.13.9 via `uv run --frozen --extra dev --extra proxy` - Local Headroom proxy subprocess: `python -m headroom.proxy.server` - Proxy config: `HEADROOM_TELEMETRY=off`, `HEADROOM_CACHE_ENABLED=false`, `HEADROOM_RATE_LIMIT_ENABLED=false` - Provider/model: no live upstream provider needed; exercised local `/v1/compress` -> `/v1/retrieve` CCR HTTP flow with model `gpt-4o` Exact steps run after the patch: 1. Start a real Headroom proxy process with `HEADROOM_CCR_TTL_SECONDS=7200`. 2. POST a 200-item tool-result payload to `/v1/compress`. 3. Extract the emitted `<<ccr:...>>` marker hash from the compressed messages. 4. GET `/v1/retrieve/stats`. 5. POST `/v1/retrieve` with the marker hash. 6. Repeat with `HEADROOM_CCR_TTL_SECONDS=1`, wait 1.5 seconds, and retrieve the same way to verify explicit expiration reporting. Observed result: ```json { "long_ttl": { "ccr_hash": "b473e632aa47", "retrieve_status": 200, "retrieved_content_has_result_199": true, "stats_default_ttl_seconds": 7200, "stats_entry_count": 1, "ttl_seconds": 7200 }, "short_ttl_expired": { "ccr_hash": "b473e632aa47", "retrieve_detail": "Entry expired (CCR TTL: 1 seconds; age: 2 seconds)", "retrieve_status": 404, "stats_default_ttl_seconds": 1, "stats_entry_count": 1, "ttl_seconds": 1 } } ``` What I did not test: - A live OpenRouter/OpenAI/Anthropic upstream request. - A durable/non-memory CCR backend. - A literal 5+ minute wall-clock wait; the process E2E used `7200` to prove configured retention and `1` second to prove expiration behavior quickly. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ## Test Output ```bash UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy pytest tests/test_compression_store.py tests/test_proxy_ccr.py tests/test_ccr_response_handler.py tests/test_ccr_response_handler_extra.py # 152 passed, 2 warnings in 12.87s UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy pytest tests/test_adapter_hooks.py tests/test_toin_feedback.py tests/test_toin_fixes.py # 55 passed, 13 skipped, 1 warning in 0.53s UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy ruff check . # All checks passed! UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy ruff format --check . # 776 files already formatted UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy mypy headroom # Success: no issues found in 346 source files ``` Existing warnings observed in the targeted tests were unrelated to this change: - AnthropicProvider tiktoken approximation warning in proxy CCR tests. - Deprecated `ToolIntelligenceNetwork.get_recommendation()` warning in existing TOIN assertions. ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) Not applicable. ## Additional Notes No new dependencies. Default behavior remains 300 seconds unless `HEADROOM_CCR_TTL_SECONDS` is set. |
||
|
JD Davis
|
b723874d12
|
ci: limit commitlint to pull requests (#843)
## Summary - limit the CI commitlint job to pull request events - prevent squash-merge commit subjects on `main` from failing post-merge CI - keep commitlint as a pre-merge PR gate ## Context - fixes the main-branch CI failure from https://github.com/chopratejas/headroom/actions/runs/27320913096/job/80711562040 ## Validation - `diff --check` - `actionlint .github/workflows/ci.yml .github/workflows/release.yml .github/workflows/release-please.yml .github/workflows/docker.yml` - `act workflow_dispatch -W .github/workflows/release.yml -e .github/act/dry-run.json -n` - `act release -W .github/workflows/release.yml -e .github/act/release-published.json -n` - `act push -W .github/workflows/release-please.yml -e .github/act/push-feat.json -n` - `act workflow_dispatch -W .github/workflows/docker.yml -e .github/act/docker-version.json -n` |
||
|
kiyo-e
|
6d30054f82
|
Add option to disable Kompress fallback (#514)
## Summary - add HEADROOM_DISABLE_KOMPRESS / --disable-kompress to disable only Kompress ML fallback - keep the proxy optimization pipeline enabled so structural compressors can still run - expose the setting in proxy health output and direct env config path ## Tests - uv run --with pytest --with fastapi --with click --with httpx --with uvicorn pytest tests/test_cli_proxy_env.py tests/test_proxy_disable_kompress.py - uv run --with ruff ruff check headroom/cli/proxy.py headroom/proxy/models.py headroom/proxy/server.py tests/test_cli_proxy_env.py tests/test_proxy_disable_kompress.py - git diff --check Reviewed by local agent before PR; no blocking findings. |
||
|
Yui(ゆい)
|
4b1c449c73
|
[codex] fix(proxy): parse CRLF SSE event terminators (#649)
## Summary - support CRLF (`\r\n\r\n`) SSE event terminators in the byte-buffer parser - parse completed SSE events with `splitlines()` so LF and CRLF line endings are handled consistently - add a regression test for CRLF-terminated SSE events ## Why SSE streams may be emitted with CRLF line endings by HTTP stacks. The existing byte-buffer parser only looked for `\n\n`, so a complete CRLF-terminated event could remain buffered and never reach usage/event parsing. ## Validation - `.venv/bin/pytest tests/test_sse_utf8_split.py tests/test_streaming_usage_parser.py -q` - `.venv/bin/ruff check headroom/proxy/helpers.py tests/test_sse_utf8_split.py` ## Risk Low. The change is isolated to complete-event boundary detection and keeps the existing invalid UTF-8 behavior loud for complete events. |
||
|
marko1olo
|
0d458e5d45
|
test: add missing type hints to FakeProvider in test_utils (#631) | ||
|
Ashish
|
30078f8465
|
fix(ccr): skip CCR when model calls headroom_retrieve alongside user tools (#839)
## Summary - When the LLM calls `headroom_retrieve` **and** a non-CCR tool (e.g. `read_file`) in the same turn, the previous code attempted a continuation with only the CCR result - Anthropic requires every `tool_use` block to have a matching `tool_result` — the continuation was rejected with 400, a round-trip was wasted, and the original response (with unresolved `headroom_retrieve`) was returned anyway - Fix: if `other_calls` is non-empty alongside `ccr_calls`, log a warning and return the original response immediately — no continuation attempted ## Root cause `_parse_ccr_tool_calls` correctly separates CCR and non-CCR calls, but `handle_response` never checked `other_calls` before building the continuation. `_create_tool_result_message` only adds results for CCR calls, leaving the non-CCR `tool_use` blocks without matching `tool_result` entries. ## Files changed - `headroom/ccr/response_handler.py` — guard at top of `while` loop in `handle_response` - `tests/test_ccr_response_handler.py` — regression test: asserts `api_call_count == 0` and original response returned unchanged when model uses mixed tools ## Test plan - [x] `pytest tests/test_ccr_response_handler.py::TestCCRResponseHandling::test_handle_response_mixed_tools_skips_ccr` — passes - [x] `pytest tests/test_ccr_response_handler.py tests/test_ccr_response_handler_extra.py tests/test_ccr_tool_injection.py tests/test_ccr_tool_always_on.py` — 85 passed - [x] Pre-commit hooks (ruff, mypy) — clean 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Ashish
|
8db5efc6f9
|
fix(anthropic): CCR exception must re-raise, not silently swallow (#838)
## Summary
- When `ccr_response_handler.handle_response` throws on the Anthropic
path, the old code logged a `WARNING` and continued — silently returning
the raw `headroom_retrieve` tool-call block to the client (compressed
content never retrieved, client sees an unknown internal tool)
- The OpenAI handler already does `logger.error + raise` (commit
`
|
||
|
Ashish
|
0ffe2b6ea4
|
fix: correct preserved-entry index mapping in Gemini content round-trip (#836)
## Summary - `_gemini_contents_to_messages` excludes entries with no text parts (pure `functionCall` / `functionResponse` / image-only) from `messages[]`, but their original `contents[]` indices are stored in `preserved_indices` - After compression, `optimized_contents` has a shorter, different index space — the old restoration loop used raw `orig_idx` to overwrite `optimized_contents[orig_idx]`, silently corrupting text entries at colliding positions and silently dropping preserved entries when `orig_idx >= len(optimized_contents)` - Affects all three Gemini handlers (`generateContent`, `cloudCodeAssist`, `countTokens`) — any agentic session with function calls where compression fires **Concrete failure case:** ``` contents = [user:text, model:functionCall, user:functionResponse, model:text] messages = [user:text, model:text] # only 2 — FC/FR have no text optimized_contents = [user:text, model:text] # positions 0 and 1 old loop: orig_idx=1 → optimized_contents[1] = functionCall ← overwrites model text! orig_idx=2 → 2 < 2 is False → functionResponse silently dropped ``` ## Fix Added `_rebuild_gemini_contents()` helper that walks `original_contents` in order, placing preserved entries at their exact relative positions and consuming optimized text entries sequentially via an iterator. Replaced all three broken loops. ## Test plan - [ ] `TestRebuildGeminiContents::test_text_only_unchanged` — text-only round-trip is identity - [ ] `TestRebuildGeminiContents::test_function_call_sequence_preserved` — functionCall + functionResponse survive at correct positions - [ ] `TestRebuildGeminiContents::test_function_call_at_start` — preserved entry at idx=0 no longer overwrites optimized_contents[0] - [ ] `TestRebuildGeminiContents::test_hybrid_entry_uses_original` — entry with both text and functionCall retains functionCall All 58 tests in `test_google_multimodal.py` pass. Rust CI + mypy + ruff clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Boni Gopalan
|
693d9d20e2
|
feat(proxy): add CLI opt-outs for CCR injection (compression-only mode) (#823)
## What & why Streaming / non-MCP clients can't resolve the injected `headroom_retrieve` (CCR) tool, so CCR injection turns into unresolvable tool calls that error and inflate turn count. Today there's no proxy CLI flag to run **compression-only** — `ccr_inject_tool`, `ccr_inject_marker`, and `ccr_proactive_expansion` are hardcoded `True` defaults — so a faithful compression-only eval requires patching the image. This adds three opt-in `--no-*` flags (with env vars), **all defaulting to current behavior (CCR fully on)**: | flag | env var | effect | |---|---|---| | `--no-ccr-inject-tool` | `HEADROOM_NO_CCR_INJECT_TOOL` | don't inject the retrieve tool | | `--no-ccr-marker` | `HEADROOM_NO_CCR_MARKER` | don't add retrieval markers to compressed content | | `--no-ccr-proactive-expansion` | `HEADROOM_NO_CCR_PROACTIVE_EXPANSION` | disable proactive expansion | `ccr_inject_tool` and `ccr_proactive_expansion` already existed on `ProxyConfig`. `ccr_inject_marker` is added to `ProxyConfig` and threaded into `ContentRouterConfig` in `server.py` (previously it was only ever the router's own default). Per CONTRIBUTING I raised this in #645 first; you accepted the patch offer there. ## Changes to existing behavior None unless a flag is passed. With no flags, all three toggles stay `True` (test `test_ccr_defaults_on`). ## Test plan - `tests/test_cli_proxy_env.py::TestCLICompressionOnlyFlags` — defaults-on, `--no-ccr-inject-tool` in isolation, all three combined, and the `HEADROOM_NO_CCR_MARKER` env path. - `pytest tests/test_cli_proxy_env.py` → 26 passed; `tests/test_proxy_modes.py tests/test_proxy_pipeline_lifecycle.py tests/test_cli_proxy_env.py` → 34 passed. - `ruff check` + `ruff format --check` clean on all changed files. ## Real behavior proof - **Setup:** Linux, Python 3.13.5, `python -m venv .venv && .venv/bin/pip install -e ".[dev]"` at this branch; OpenAI-compatible upstream. - **Ran:** - `headroom proxy --help` → all three flags appear with help text. - Instantiated the live proxy: ```python from headroom.proxy.server import ProxyConfig, HeadroomProxy from headroom.transforms.content_router import ContentRouter cfg = ProxyConfig(host="127.0.0.1", port=1, ccr_inject_tool=False, ccr_inject_marker=False, ccr_proactive_expansion=False) p = HeadroomProxy(cfg) router = [t for t in p.anthropic_pipeline.transforms if isinstance(t, ContentRouter)][0] print(router.config.ccr_inject_marker) # -> False ``` - **Observed:** `router.config.ccr_inject_marker == False`; `cfg.ccr_inject_tool == False`; `cfg.ccr_proactive_expansion == False`. With no flags, all three are `True`. - **Not tested here:** a full live agentic run on this branch. The motivating field evidence (a compression-only run with zero `headroom_retrieve` calls, compression intact) was collected on the v0.23.0 image with these same three defaults flipped — this PR replaces that image patch with first-class flags. Refs #645. |
||
|
Focused Instability
|
929698af10
|
fix(parser): detect waste signals in Anthropic tool_result content blocks (#815)
## Description
The dashboard's "What Headroom Removed" panel (waste signals) stays
permanently empty for Anthropic-format traffic.
`parse_message_to_blocks()` only extracted text from content parts with
`type == "text"`, so the `tool_result` blocks that carry the bulk of
agentic conversations (Claude Code, and aider/cursor/copilot in
anthropic mode) were invisible to `detect_waste_signals()`. The pipeline
then reported `waste_signals=None` and `/stats` returned
`"waste_signals": {}` forever.
This PR emits a dedicated `tool_result` Block per Anthropic
`tool_result` content part — handling both string-form content and the
nested text-block-list form — with waste detection and a `tool_call_id`
pairing flag. OpenAI chat-completions behavior is unchanged (parity test
included).
Fixes #813
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/parser.py`: new `_extract_tool_result_text()` helper;
`parse_message_to_blocks()` collects `tool_result` content parts and
emits a `Block(kind="tool_result")` per part with waste signals and
`tool_call_id` flags
- `tests/test_parser.py`: 7 new tests — nested text-list form, string
form, mixed text+tool_result, empty content, non-text inner blocks,
`parse_messages` aggregation, and waste parity with the OpenAI `role:
"tool"` format
## Testing
- [x] Unit tests pass (`pytest tests/test_parser.py` — 60 passed)
- [x] Linting passes (`ruff check`, `ruff format --check`)
- [x] New tests added for new functionality
- [x] Manual testing performed (real pipeline run below)
Also ran `tests/test_pipeline.py`, `tests/test_canonical_pipeline.py`,
`tests/test_proxy_pipeline_lifecycle.py`: 3 failures there are
pre-existing on a clean `upstream/main` checkout (verified via `git
stash`) and unrelated to this change.
## Test Output
```
$ pytest tests/test_parser.py -q
60 passed in 0.14s
$ ruff check headroom/parser.py tests/test_parser.py
All checks passed!
```
## Real behavior proof
Real `TransformPipeline` (CacheAligner + ContentRouter, same
construction as the proxy server) over an Anthropic-format conversation
with four large JSON `tool_result` blocks:
```
# before this fix
before=37265 after=16013 saved=21252
transforms: ['router:protected:user_message', 'router:tool_result:smart_crusher']
waste_signals: None <- SmartCrusher removed 21k tokens, dashboard shows nothing
# after this fix (identical input)
before=37265 after=16013 saved=21252
transforms: ['router:protected:user_message', 'router:tool_result:smart_crusher']
waste_signals: {'json_bloat': 37140, 'html_noise': 0, 'base64': 0, 'whitespace': 0, 'dynamic_date': 0, 'repetition': 0}
```
Parser-level parity (same JSON payload, both wire formats):
```
anthropic tool_result waste total: 0 -> 1745 after fix
openai role:"tool" waste total: 1745 (unchanged)
```
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] CHANGELOG.md — not edited manually; release-please generates
entries from the conventional `fix:` commit
## Additional Notes
Scoped to the Anthropic `tool_result` parsing bug per issue #813. Two
related-but-separate gaps noted there: `handle_openai_responses` (codex)
never computes waste signals at all, and Gemini `functionResponse` parts
are preserved verbatim — both deserve their own issues/PRs.
---------
Co-authored-by: Ash Rhodes <ashley.rhodes@king.com>
|
||
|
Focused Instability
|
914a60a2b0
|
feat(proxy): per-project savings breakdown on the dashboard (claude, codex, aider, copilot, cursor) (#803)
## Summary Adds a **per-project savings breakdown** to the proxy dashboard, covering **all wrap-supported agents: Claude Code, Codex, aider, Copilot, and Cursor**. Spec and rationale in #802 (feature-request issue — happy to adjust scope per maintainer feedback). How it works — two attribution channels, by client capability: **Header channel** (clients that can send custom headers): - `headroom wrap claude` appends `X-Headroom-Project: <basename(cwd)>` to `ANTHROPIC_CUSTOM_HEADERS` (a user-supplied x-headroom-project header always wins; other user headers preserved). - `headroom wrap codex` extends the injected `[model_providers.headroom]` block with `env_http_headers = { "X-Headroom-Project" = "HEADROOM_PROJECT" }` and sets `HEADROOM_PROJECT` per launch — Codex only sends the header when the env var is set, so the static config stays inert outside wrap. **Base-URL prefix channel** (clients that cannot send custom headers): - The proxy accepts `/p/<url-encoded-name>/...`; middleware strips the prefix before routing (before Starlette caches the URL) and binds the project. The explicit header wins over the prefix. - `headroom wrap aider` points `OPENAI_API_BASE` / `ANTHROPIC_BASE_URL` at the prefixed URL. - `headroom wrap copilot` points `COPILOT_PROVIDER_BASE_URL` at the prefixed URL (BYOK anthropic/openai provider types and the GitHub-subscription path). - `headroom wrap cursor` prints the prefixed Override Base URL in its setup instructions, with a note explaining the attribution. **Shared plumbing:** - New `headroom/proxy/project_context.py`: header classification, `/p/` prefix split/strip, URL-prefix builder, and a contextvar bound per request (HTTP middleware + WS accept for the Codex responses bridge); the outcome funnel resolves it (explicit `RequestOutcome.project` wins), stamps `RequestLog.tags["project"]`, and forwards it through `PrometheusMetrics.record_request` into the `SavingsTracker`. - `SavingsTracker`: persisted state gains a `projects` map (requests, tokens saved, savings USD, input tokens/cost, last activity). Schema v2 → v3 with transparent forward migration (v2 files load cleanly, `projects` starts empty). Names sanitized (printable-only, 128-char cap); map capped at 50 projects, evicting the smallest bucket. - `/stats` exposes `savings.per_project` (and `projects`/`projects_limit` inside `persistent_savings`); `/stats-history` exposes `projects`. - Dashboard gains a "Per-Project Savings" table mirroring the per-model table (Alpine `x-text` only — names are user-supplied, no HTML injection). **Behavior changes:** none for unattributed traffic — no header and no prefix means no bucket, aggregate totals exactly as before (regression-tested: legacy `/stats` and `/stats-history` shapes are pinned by tests). ## Real behavior proof **Setup tested on:** macOS (Darwin 25.5.0), Python 3.11.9, `pip install -e ".[dev]"`, proxy on spare ports with isolated `HEADROOM_SAVINGS_PATH`; real Claude Code CLI (subscription auth) and real Codex CLI (ChatGPT auth) as clients. **Header channel — exact steps:** ``` HEADROOM_SAVINGS_PATH=/tmp/headroom-proof-savings.json .venv/bin/python -m headroom.cli proxy --port 9123 # background cd /tmp/proof-alpha && headroom wrap claude --port 9123 --no-rtk --no-mcp --no-serena -- -p "Reply with exactly: OK" cd /tmp/proof-beta && headroom wrap claude --port 9123 --no-rtk --no-mcp --no-serena -- -p "Reply with exactly: OK" cd /tmp/proof-beta && headroom wrap codex --port 9123 --no-rtk --no-mcp --no-serena -- exec --skip-git-repo-check "Reply with exactly: OK" ``` **Observed** (copied live `/stats` output; all runs replied `OK` through the proxy): ```json { "proof-beta": { "requests": 3, "tokens_saved": 140, "compression_savings_usd": 0.0007, "total_input_tokens": 38581, "total_input_cost_usd": 0.360433, "last_activity_at": "2026-06-10T09:19:17Z", "savings_percent": 0.36 }, "proof-alpha": { "requests": 1, "tokens_saved": 0, "compression_savings_usd": 0.0, "total_input_tokens": 9050, "total_input_cost_usd": 0.32245, "last_activity_at": "2026-06-10T09:18:09Z", "savings_percent": 0.0 } } ``` `proof-beta` aggregates one Claude Code turn + one Codex `exec` run (Codex attribution flows through `env_http_headers`). **Prefix channel — exact steps** (the same mechanism the aider/copilot/cursor wraps emit, driven by a real client): ``` .venv/bin/python -m headroom.cli proxy --port 9124 # background, isolated savings path ANTHROPIC_BASE_URL="http://127.0.0.1:9124/p/aider-style-project" claude -p "Reply with exactly: OK" ``` **Observed:** ```json { "aider-style-project": { "requests": 1, "tokens_saved": 0, "compression_savings_usd": 0.0, "total_input_tokens": 8571, "total_input_cost_usd": 1.018575, "last_activity_at": "2026-06-10T10:10:13Z", "savings_percent": 0.0 } } ``` `/stats-history` returned `schema_version: 3` with the same `projects` keys; `GET /dashboard` HTML contains the new "Per-Project Savings" table; persisted state survived a tracker reload; a hand-written v2 savings file loaded cleanly with an empty `projects` map. **What I did NOT test live:** the actual aider/Copilot/Cursor binaries end-to-end (their wraps emit exactly the prefixed URLs exercised above — unit tests pin the emitted env/URLs); Codex subscription-mode routing via the built-in `openai` provider (no provider headers there — such traffic simply stays unattributed); multi-worker uvicorn; Windows. ## Tests - `tests/test_proxy_project_savings.py` (17 tests): sanitization, header classification, `/p/` prefix split + URL-builder round-trip, tracker aggregation/persistence/migration/cardinality-cap/state-sanitization, funnel→`/stats` end-to-end, middleware binding for header + prefix + precedence, plus regression tests pinning legacy `/stats`/`/stats-history` shape and unattributed-traffic totals. - Wrap/provider tests extended: `test_cli/test_wrap_helpers.py`, `test_cli/test_wrap_codex.py`, `test_provider_aider.py`, `test_provider_cursor.py`, `test_provider_copilot_wrap.py` (prefixed env/URLs, user-override wins, no duplicate header, TOML block contents, block strip, setup-line note). - Full `pytest` suite run locally; `ruff check` + `ruff format` clean on all touched files. - `CHANGELOG.md` updated. ## Dependencies None added or bumped. Closes #802 (pending maintainer 👍 per CONTRIBUTING — raised there first with the full spec). --------- Co-authored-by: Ash Rhodes <ashley.rhodes@king.com> |
||
|
Shengbo_Wang
|
6ea6e31f09
|
fix(init): normalize Windows hook paths to forward slashes (#788)
## Description
On Windows, `_command_string()` preserves backslash paths from
`shutil.which()` (e.g. `C:\Users\...\headroom.exe`). Claude Code
executes hooks via Git Bash, which interprets backslashes as escape
characters, corrupting the path and failing with "command not found".
This PR normalizes backslash separators to forward slashes before
passing parts to `subprocess.list2cmdline()`. Forward slashes work in
bash, PowerShell, and cmd.exe on Windows.
Fixes #724
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/cli/init.py`: Normalize backslash path separators to forward
slashes in `_command_string()` on Windows, before calling
`subprocess.list2cmdline()`
- `tests/test_cli/test_init_cli.py`: Add
`test_command_string_normalizes_backslashes_on_windows` verifying no
backslashes remain in the output and the forward-slash path is preserved
## Real behavior proof
**Setup:** Windows 11 (build 26200), Python 3.10.18, headroom repo at
commit
|
||
|
Kumario
|
84ac332d14
|
fix(copilot): use responses API for subscription reasoning models (#647)
Fixes #644 ## Summary - default `headroom wrap copilot --subscription` to the responses wire API when the selected Copilot model is GPT-5/o1/o3-family - normalize `--subscription` to the OpenAI-compatible provider mode before validating `--wire-api responses` - add provider and CLI regressions for model-derived defaults and explicit `--wire-api responses` ## Tests - `UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev python -m pytest tests/test_provider_copilot_wrap.py tests/test_cli/test_wrap_copilot.py -q` - `UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev python -m ruff check headroom/providers/copilot/wrap.py headroom/providers/copilot/__init__.py headroom/cli/wrap.py tests/test_provider_copilot_wrap.py tests/test_cli/test_wrap_copilot.py` - `UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev python -m compileall -q headroom/providers/copilot/wrap.py headroom/providers/copilot/__init__.py headroom/cli/wrap.py tests/test_provider_copilot_wrap.py tests/test_cli/test_wrap_copilot.py` --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Patrick A
|
028efabb4e
|
feat(cli): comprehensive help text, validation, and exception handling improvements (#640)
## Summary This PR improves the `headroom proxy` CLI command across three dimensions: help text completeness, input validation, and exception handling. ### Help text and env var wiring Several options lacked `envvar=` declarations even though they are documented as env-configurable in their `help=` strings. This caused inconsistent behaviour when operators set these variables in container environments: - `--log-file` now reads `HEADROOM_LOG_FILE` - `--log-messages` now reads `HEADROOM_LOG_MESSAGES` - `--memory-db-path` now reads `HEADROOM_MEMORY_DB_PATH` - `--memory-project-root` now reads `HEADROOM_MEMORY_PROJECT_ROOT` - `--no-memory-tools` now reads `HEADROOM_NO_MEMORY_TOOLS` - `--no-memory-context` now reads `HEADROOM_NO_MEMORY_CONTEXT` - `--memory-top-k` now reads `HEADROOM_MEMORY_TOP_K` - `--retry-max-attempts` now reads `HEADROOM_RETRY_MAX_ATTEMPTS` - `--connect-timeout-seconds` now reads `HEADROOM_CONNECT_TIMEOUT_SECONDS` - `--backend` now reads `HEADROOM_BACKEND` - `--anyllm-provider` now reads `HEADROOM_ANYLLM_PROVIDER` - `--region` now reads `HEADROOM_REGION` Help text improvements: `--log-file` describes the JSONL fields, `--log-messages` adds a privacy warning, `--budget` describes the reset behaviour and rejection semantics. ### Input validation Options that already document a valid range now enforce it at the Click layer so invalid values get a clear error rather than a downstream `ValueError`: | Option | Range | |--------|-------| | `--subscription-poll-interval` | 1-3600 | | `--retry-max-attempts` | 1-10 | | `--connect-timeout-seconds` | 1-300 | | `--memory-top-k` | 1-100 | | `--budget` | >= 0.0 | ### Exception handling - `--learn` + `--no-learn` conflict now prints a yellow warning to stderr rather than silently resolving. - Missing proxy dependencies: ImportError path uses `click.secho(err=True)` with red colour and correct package name (`headroom-ai[proxy]`). - KeyboardInterrupt: exits 130 (SIGINT convention) instead of 0. ### Tests Added `tests/test_cli_proxy_improvements.py` with 44 new tests. All existing CLI tests continue to pass. --- ## Files changed - `headroom/cli/proxy.py` — env var wiring, range validation, help text, exception handling - `tests/test_cli_proxy_improvements.py` (new) — 44 tests - `CHANGELOG.md` — changelog entry > **Note:** `uv.lock` was removed from this PR per reviewer feedback. The lockfile is not tracked in this branch. --------- Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
Matt Van Horn
|
163677b405
|
fix: make headroom wrap readiness probe timeout configurable for slow ML imports (#581)
## Summary Makes `headroom wrap` wait long enough for slow proxy startups instead of failing at a fixed readiness window, with an ML-aware default and an env-var override. ## Why `headroom wrap` failed when the proxy took longer than a fixed startup window to bind its port. Issue #195 reports that on ML-heavy setups the proxy imports large libraries (torch, sentence_transformers, spacy) at startup and routinely exceeds the hardcoded window, so `wrap` aborts on a working proxy and the failure message gives no way to extend the wait. ## Description `headroom wrap` now lets slow proxy startups finish instead of failing at a fixed window. The readiness probe in `headroom/cli/wrap.py` reads a `HEADROOM_WRAP_PROXY_TIMEOUT` environment variable, and when no value is set it picks the default automatically: 90 seconds when an ML stack (torch, sentence_transformers, spacy) is detected via `importlib.util.find_spec` without importing it, otherwise 45 seconds. The failure message now names the active timeout and the env var to raise it. Fixes #195 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Resolve the wrap proxy readiness window from `HEADROOM_WRAP_PROXY_TIMEOUT` in `_start_proxy`, falling back to an ML-aware default. - Detect optional ML extras with `importlib.util.find_spec` so the check itself does not pay the cold-import cost the issue describes. - Include the configured timeout and the env var name in the `RuntimeError` raised when the proxy genuinely never binds the port. ## Testing Describe the tests you ran to verify your changes: - [x] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed New cases in `tests/test_cli_proxy_env.py` cover the default window, an extended window via the env var, an invalid value raising a clear error, and the failure message naming the configured timeout. Covered by the new tests in this PR; full suite runs in CI. ## Test Output ``` # Paste relevant test output here pytest -v tests/test_cli_proxy_env.py ``` The new `tests/test_cli_proxy_env.py` cases pass locally; the full suite runs in CI. ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable (N/A: no CHANGELOG.md is maintained in this repo) ## Screenshots (if applicable) N/A. This is a CLI startup-timeout fix with no visual surface. ## Additional Notes The default is conservative: 90s only when an ML stack is detected via `importlib.util.find_spec` (no import cost), otherwise 45s. `HEADROOM_WRAP_PROXY_TIMEOUT` overrides both, and the failure message now names the active timeout and the env var to raise it. Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> |
||
|
Hc
|
9252d852c5
|
fix(init): guard persistent task startup (#616)
## Description Prevent `headroom init` hooks from spawning duplicate persistent-task runners while a proxy is still starting. Fixes #615 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Problem `_ensure_profile_running()` checked readiness for only one second and then launched `start_detached_agent()` whenever the proxy was not ready yet. When Claude/Codex hooks fired close together, each hook could race through that path and spawn another detached persistent-task runner. ## Changes Made - Add a profile-local, nonblocking runtime start lock around init hook startup. - Re-check readiness after acquiring the lock so late-arriving hooks do not start a duplicate runner. - If a runtime is already alive, wait up to 15 seconds for readiness before stopping and restarting it. - Add regression tests for lock contention, slow startup, and cross-process lock behavior. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ## Test Output ``` UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy pytest tests/test_cli/test_init_cli.py tests/test_cli/test_install_cli.py tests/test_cli/test_wrap_persistent.py tests/test_install/test_runtime.py # 89 passed in 0.61s UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy ruff check . # All checks passed! UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy ruff format --check . # 775 files already formatted UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy mypy headroom --ignore-missing-imports # Success: no issues found in 346 source files ``` Manual sandbox check: ``` # before this change: 3 ensure calls spawned 3 detached starts # after this change: 3 ensure calls spawned 1 detached start while the runtime was still starting ``` ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes Docs and CHANGELOG were left unchanged because this is a small runtime bug fix with no user-facing CLI/API change. |
||
|
mbachaud
|
6367d0b722
|
feat(kompress): warn on unrecognized HEADROOM_KOMPRESS_BACKEND + document backend selection (#204)
## Summary
This PR was originally \"HEADROOM_KOMPRESS_BACKEND env + GPU/MPS
auto-detect\" (for #202). While it sat, main independently shipped the
backend-selection env var in
|
||
|
Patrick A
|
2ad300aff8
|
fix(transforms): use thread-local tree-sitter parsers to prevent pyo3 Unsendable panic (#604)
## Problem
pyo3 marks `_native::Parser` as `#[pyclass(unsendable)]`, which causes a
hard thread-assertion panic when a parser created on one thread is
accessed from another:
```
thread '<unnamed>' panicked at pyo3-0.28.3/src/impl_/pyclass.rs:1055:9:
assertion `left == right` failed: _native::Parser is unsendable, but sent to another thread
left: ThreadId(2)
right: ThreadId(1)
```
The prior implementation stored parsers in a module-level `dict[str,
Any]` (`_tree_sitter_languages`). When `_run_compression_in_executor`
dispatched compression work to a `ThreadPoolExecutor`, pool workers
grabbed parsers from that shared dict that were originally created on
the main asyncio thread and panicked.
This produces a 500 on every request where code compression is attempted
via a pool thread.
## Fix
Replace the global dict with `threading.local()` so each thread creates
and owns its own parser instances. No cross-thread parser access is
possible.
```python
# before
_tree_sitter_languages: dict[str, Any] = {} # shared — crosses threads
# after
_tree_sitter_local = threading.local() # per-thread — isolated
```
`is_tree_sitter_loaded()` and `unload_tree_sitter()` updated to operate
on the current thread's local cache (semantics unchanged for
single-threaded callers).
## Tests
9 regression tests added in
`tests/test_transforms/test_tree_sitter_thread_safety.py`:
- Thread isolation: two threads get distinct parser instances
- Within-thread reuse: same thread gets the same cached instance
- Thread pool: parsers usable from `ThreadPoolExecutor` workers without
panic
- Concurrent workers: each distinct pool thread owns a unique parser
- `is_tree_sitter_loaded` / `unload_tree_sitter` lifecycle
Also adds a `filterwarnings` entry for
`PytestUnraisableExceptionWarning`: pyo3 emits this when short-lived
test threads drop parsers at teardown; it does not occur in production
where pool threads are long-lived.
## Relation to #564
PR #564 proposes the same `threading.local()` approach but was blocked
on missing tests (`CHANGES_REQUESTED`). This PR includes the full test
suite.
|
||
|
Gonzalo Zanelli
|
96abf38b09
|
fix(codex): respect CODEX_HOME for wrap config (#731)
> ⚠️ This PR was opened using Codex (gpt-5.5 on `xhigh`) Fixes #730. ## Summary - centralize Codex config path resolution in `headroom wrap codex` so it honors `CODEX_HOME` when set - make the Codex MCP registrar use `$CODEX_HOME/config.toml` instead of always writing to `~/.codex/config.toml` - route optional memory MCP and global Codex `AGENTS.md` injection through the same Codex home helper - make `headroom unwrap codex` print a warning, but still succeed, when `CODEX_HOME` is unset and the default Codex config has no Headroom markers - add regression coverage for provider injection, prepare-only wrapping, MCP registration under a custom Codex home, and the ambiguous unwrap warning - update `CHANGELOG.md` under `Unreleased > Bug Fixes` ## Real behavior proof Setup tested on: - Linux `7.0.10-2-cachyos` - Python 3.14.3 via `uv` - local fork branch `fix/codex-home` - custom Codex home created outside `~/.codex` Exact command run after the patch: ```bash tmp_home=$(mktemp -d) mkdir -p "$tmp_home/codex_custom" HOME="$tmp_home" USERPROFILE="$tmp_home" CODEX_HOME="$tmp_home/codex_custom" \ UV_SKIP_WHEEL_FILENAME_CHECK=1 PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 \ uv run --with fastapi --with uvicorn --with httpx --with websockets \ headroom wrap codex --no-context-tool --no-serena --prepare-only --port 8787 find "$tmp_home" -maxdepth 3 -type f -print | sort sed -n '1,140p' "$tmp_home/codex_custom/config.toml" test -e "$tmp_home/.codex/config.toml" && echo yes || echo no HOME="$tmp_home" USERPROFILE="$tmp_home" \ UV_SKIP_WHEEL_FILENAME_CHECK=1 PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 \ uv run --with fastapi --with uvicorn --with httpx --with websockets \ headroom unwrap codex --no-stop-proxy ``` After-fix evidence + observed result: Interactive check: - A full `CODEX_HOME="$HOME/.codex_p" headroom wrap codex --no-serena` launch was tested locally after the patch and worked as expected. - The prepare-only proof below shows the same config path behavior without requiring an interactive Codex session in CI/reviewer environments. ```text MCP retrieve tool: registered (restart OpenAI Codex CLI if it was already running) Codex config: injected Headroom provider (WS + HTTP) into /tmp/tmp.UPUvloGNYE/codex_custom/config.toml --- files --- /tmp/tmp.UPUvloGNYE/codex_custom/config.toml /tmp/tmp.UPUvloGNYE/codex_custom/config.toml.headroom-backup --- custom config --- # --- Headroom proxy (auto-injected by headroom wrap codex) --- model_provider = "headroom" openai_base_url = "http://127.0.0.1:8787/v1" # --- end Headroom --- # --- Headroom MCP server --- [mcp_servers.headroom] command = "headroom" args = ["mcp", "serve"] # --- end Headroom MCP server --- # --- Headroom proxy (auto-injected by headroom wrap codex) --- [model_providers.headroom] name = "OpenAI via Headroom proxy" base_url = "http://127.0.0.1:8787/v1" supports_websockets = true # --- end Headroom --- --- default config exists? --- no Warning: found no Headroom wrap markers in the default Codex config. If you wrapped Codex with CODEX_HOME, rerun unwrap with the same environment variable, e.g. CODEX_HOME=/path/to/codex-home headroom unwrap codex. Nothing to undo: .../.codex/config.toml has no Headroom wrap markers. ``` What I did not test: - Windows/macOS path behavior - full repository test suite, because this local Python 3.14 environment hits optional dependency/build constraints outside this patch ## Testing ```bash UV_SKIP_WHEEL_FILENAME_CHECK=1 PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 uv run --with pytest --with fastapi --with uvicorn --with httpx --with websockets pytest tests/test_mcp_registry/test_codex_registrar.py tests/test_cli/test_wrap_codex.py -q UV_SKIP_WHEEL_FILENAME_CHECK=1 PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 uv run --with ruff ruff check headroom/cli/wrap.py headroom/mcp_registry/codex.py tests/test_cli/test_wrap_codex.py tests/test_mcp_registry/test_codex_registrar.py UV_SKIP_WHEEL_FILENAME_CHECK=1 PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 uv run --with ruff ruff format --check headroom/cli/wrap.py headroom/mcp_registry/codex.py tests/test_cli/test_wrap_codex.py tests/test_mcp_registry/test_codex_registrar.py ``` Results: ```text 63 passed, 1 warning in 1.48s All checks passed! 4 files already formatted ``` Notes: - Python 3.14 required `PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1` for the editable build. - `uv` required `UV_SKIP_WHEEL_FILENAME_CHECK=1` because the existing `uv.lock` has a GitPython wheel filename/version mismatch. |
||
|
Ashish
|
53a08c63bf
|
feat(evals): add zero-cost tool schema compaction integrity eval (#817)
## Summary - Adds `evaluate_tool_schema_compaction()` and `generate_tool_schema_cases()` to `CompressionOnlyRunner` - Four built-in cases cover the property-name vs annotation-key distinction: `title`, `deprecated`, `readOnly`, and all four at once - Each case asserts: byte count shrinks (annotations stripped), all `must_preserve` property names survive in `properties`, no `required` entry points to a stripped key, root-level schema annotations (`$schema`, `title`) are dropped - Wires the new eval into `.github/workflows/eval.yml` alongside the existing CCR round-trip smoke step — runs on every PR touching `headroom/transforms/**`, `headroom/evals/**`, or `headroom/compress.py`, at zero API cost ## Motivation PR #785 fixed a bug where the compaction pass stripped property *names* that happened to match DROP_KEYS (e.g. a field literally called `title`). This eval encodes the invariant that fix established so future changes to the compaction logic can't silently regress it. ## Test plan - [ ] `pytest tests/test_evals_metrics.py::test_tool_schema_compaction_integrity` — all 4 cases pass, `total_tokens_saved > 0` - [ ] CI smoke step "Run tool schema compaction integrity eval (zero cost)" passes with no API key required ## Real behavior proof ``` $ pytest tests/test_evals_metrics.py::test_tool_schema_compaction_integrity -v PASSED [100%] 1 passed in 0.53s ``` Zero API calls, zero cost. Runs in under 1 second. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Andrew Rich
|
93c69372e6
|
fix(proxy): lazy-import server to avoid fastapi crash (#442)
## Summary - Lazy-import `create_app`/`run_server` in `headroom/proxy/__init__.py` via PEP 562 `__getattr__` to prevent CLI crash when `fastapi` is not installed (i.e., installed without `[proxy]` extras) - Fix `.pre-commit-config.yaml` to use `python3` instead of `python` (unavailable on macOS Homebrew) - Add graceful `ImportError` skip in `scripts/sync-plugin-versions.py` for environments without dev dependencies Fixes #441 ## Test plan - [x] `headroom --help` works without `[proxy]` extras installed - [x] `headroom proxy --help` works with `[proxy]` extras installed - [x] `headroom proxy --port 18787` starts and serves traffic - [x] Lazy imports resolve correctly: `from headroom.proxy import create_app, run_server` - [x] `AttributeError` raised for invalid attributes on `headroom.proxy` - [x] Pre-commit hooks pass (ruff, ruff-format, mypy, sync-plugin-versions) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Code Bot <claude-code@smartwatermelon.github> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
JD Davis
|
3c77e52ce4
|
feat: add Vertex AI proxy routing (#793)
## Description Adds first-class GCP Vertex AI proxy routing for publisher REST endpoints so Vertex requests are forwarded to a configurable regional Vertex host instead of falling through to the generic OpenAI/Anthropic/Gemini passthrough selection. Fixes #792 ## Type of Change - [x] New feature (non-breaking change that adds functionality) - [x] Documentation update ## Changes Made - Added a `vertex` provider target with `VERTEX_TARGET_API_URL` and `--vertex-api-url` support. - Registered explicit Vertex publisher routes for Google `generateContent`, `streamGenerateContent`, `countTokens` and Anthropic publisher `rawPredict`, `streamRawPredict` passthrough. - Added startup banner/routing output for Vertex AI. - Added focused tests for provider target resolution, CLI/env config, banner output, and route delegation. - Added `wiki/vertex.md` with usage examples and Google Cloud source links. ## Sources - Vertex AI Gemini inference reference: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference - Google Cloud REST authentication: https://docs.cloud.google.com/docs/authentication/rest - Google Application Default Credentials: https://docs.cloud.google.com/docs/authentication/application-default-credentials ## Testing - [x] Linting passes (`python -m ruff check .`) - [x] New tests added for new functionality - [x] Focused unit tests pass - [ ] Full unit suite completed locally - [ ] Rust tests completed locally - [ ] Type checking passes locally ## Test Output ```text $ python -m ruff check . All checks passed! $ python -m pytest tests/test_provider_registry.py tests/test_provider_proxy_routes.py tests/test_cli_proxy_env.py tests/test_banner_upstream_targets.py -q 57 passed, 1 warning in 13.75s ``` Local limitations: - `python -m pytest tests scripts/tests -q` timed out after 1 hour on this Windows machine before completing. - `cargo test -p headroom-proxy --test integration_vertex_raw_predict` could not run because `cargo` is not installed on PATH in this environment. - The commit hook's `mypy` step fails locally on an existing Windows `fcntl` typing issue in `headroom/subscription/tracker.py`; `ruff`, `ruff-format`, and plugin-version hooks passed, and the commit was made with only `mypy` skipped. ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have made corresponding changes to the documentation - [x] I have added tests that prove the feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable |
||
|
Ashish
|
ae2122fda8
|
fix: schema compaction must not drop property names that match DROP_KEYS (#785)
Fixes #759 ## Summary `_compact_openai_tool_schema_value()` strips every key matching `_OPENAI_TOOL_SCHEMA_DROP_KEYS` (which includes `title`, `readOnly`, `deprecated`, `writeOnly`, etc.) regardless of where in the schema tree it appears. This is wrong when those same strings are used as **property names** inside a `properties` object — they're valid business fields, not annotation metadata. The result is an invalid strict schema sent upstream: ``` "required key 'title' not in properties" ``` **Root cause (single function, two lines):** ```python # before — drops "title" everywhere, even as a property name for key, child in value.items(): if key in _OPENAI_TOOL_SCHEMA_DROP_KEYS: continue compacted[key] = _compact_openai_tool_schema_value(child) ``` **Fix — add `_parent_key` context, skip drop only when not inside `properties`:** ```python def _compact_openai_tool_schema_value(value, _parent_key=None): ... for key, child in value.items(): if _parent_key != "properties" and key in _OPENAI_TOOL_SCHEMA_DROP_KEYS: continue compacted[key] = _compact_openai_tool_schema_value(child, key) ``` Schema-level annotations (e.g. `title: "ReadFileParameters"` at schema root) are **still stripped**. Only property names whose string value happens to match a drop-key are preserved. ## Test plan - [x] Added `test_openai_tool_schema_compaction_preserves_property_named_title` in `tests/test_openai_responses_context_compaction.py` — reproduces the exact OMP `eval` tool schema from the issue report - [x] All 9 existing compaction tests still pass (including `test_openai_tool_schema_compaction_preserves_invocation_shape` which verifies schema-level `title` is still stripped) ``` tests/test_openai_responses_context_compaction.py::test_openai_tool_schema_compaction_preserves_invocation_shape PASSED tests/test_openai_responses_context_compaction.py::test_openai_tool_schema_compaction_preserves_property_named_title PASSED tests/test_openai_responses_context_compaction.py::test_openai_tool_schema_compaction_is_deterministic PASSED 9 passed ``` ## Real behavior proof - **OS**: macOS darwin arm64, Python 3.11.0 - **Tested**: ran the new and existing compaction tests locally against the patched handler - **Not tested**: live OMP / Venice.ai / Codex endpoint (no API key for those); the fix is a pure schema-transform function with no network side effects 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
gglucass
|
0ce68dedd7
|
fix(proxy): restore Codex usage headers on WS and streaming SSE transports (#577) (#794)
## Description Codex's subscription/rate-limit window (the `x-codex-*` headers) was being **stripped on every transport Codex actually uses**, so session/weekly usage never reached the Codex CLI's own `/status` display, Headroom `/stats`/dashboard, or any consumer that sniffs the client-facing handshake. This PR restores it on **both** the WebSocket and streaming-SSE paths — the two halves of #577 — in one place. Fixes #577 **Supersedes #582 and #590.** This PR incorporates #582's SSE fix (carried verbatim with a `Co-authored-by` trailer) and additionally forwards the window onto the client `101` on the WS path, which #582/#590's capture-only WS code cannot do. Both can be closed as superseded once this merges — GitHub closing keywords only auto-close issues (hence `Fixes #577` above), not PRs, so #582/#590 need a manual close. ### WebSocket (`gpt-5.4+`) OpenAI delivers `x-codex-*` **only** on the upstream WS handshake response, never in data frames. `handle_openai_responses_ws` accepted the client WS *before* it connected upstream and never read `upstream.response.headers`, so the window was dropped. This reorders the handler to **connect upstream first**, extract the `x-codex-*` subset, then **accept the client WS with those headers attached** to the `101`, and refresh the Python state for `/stats` parity. ### Streaming SSE (incorporated from #582, @m16khb) Codex CLI almost always streams. `streaming.py` neither captured `x-codex-*` into `CodexRateLimitState` nor forwarded it — the forwarded-header filter matched only the substring `"ratelimit"`, which `x-codex-*` does not contain. This calls `update_from_headers()` **before** the `>=400` early-return (so a streaming 429/5xx still refreshes the window, matching the non-streaming handlers) and widens the forward filter to pass `x-codex-*`. > Credit: the SSE fix is @m16khb's work from #582, carried here verbatim with a > `Co-authored-by` trailer so the maintainer gets a single PR covering both > transports. This supersedes #582/#590's **WS** capture (which only writes > `/stats`); the connect-before-accept reorder additionally forwards the window to > the client `101`, which capture-only cannot do. #590's optional snapshot > persistence is intentionally left out (separable; hot-path sync write; doesn't > help the `101`-sniff consumers). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `openai.py`: add `_extract_codex_handshake_headers()` (strictly `x-codex-*`, via `raw_items()` to avoid `MultipleValuesError`; never `set-cookie`/`authorization`). - `openai.py`: reorder `handle_openai_responses_ws` — connect-only retry loop runs before `accept()`; `accept(headers=...)` carries the forwarded window; first client frame read afterward. HTTP fallback preserved; it now also refreshes `/stats` from the HTTP response headers. - `streaming.py`: capture `x-codex-*` on all statuses + widen the forwarded-header filter (from #582). ### Diff-size note The bulk of the `openai.py` line count is **whitespace-only relocation**: the relay block dedents one level out of the old per-attempt `async with`. Logical change is ~290 lines. **Review with `?w=1`.** In API-key mode the handshake carries no `x-codex-*`, so the accept-header list is empty and the path behaves exactly as before — the fix only activates for ChatGPT-subscription auth. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed - WS: `test_ws_connect_happens_before_accept`, `test_ws_forwards_codex_headers_to_client_accept` (only `x-codex-*` forwarded; `set-cookie`/`authorization` excluded; `/stats` refreshed), `test_ws_connect_failure_falls_back_to_http`, `test_ws_first_frame_timeout_after_connect_closes_upstream`. - Fallback: `test_fallback_refreshes_codex_rate_limit_state`. - SSE: `test_codex_rate_limit_headers_captured_and_forwarded_in_streaming`, `test_codex_rate_limit_captured_on_streaming_429` (from #582). - Wire-level e2e: `tests/e2e_ws_codex_usage_headers.py` boots the real proxy + fake upstream + real `websockets` client and reads the client `101` — closes the gap the unit tests stub (that uvicorn/starlette actually write `accept(headers=...)`). ## Test Output ``` $ uv run pytest tests/test_proxy_streaming_ratelimit_headers.py \ tests/test_ws_http_fallback.py \ tests/test_openai_codex_ws_lifecycle.py \ tests/test_openai_codex_ws_timings.py \ tests/test_codex_rate_limits.py -q 63 passed in 0.83s $ .venv/bin/python tests/e2e_ws_codex_usage_headers.py [codex-hdr-e2e] client 101 headers: x-codex-primary-used-percent: 42 x-codex-primary-window-minutes: 300 x-codex-secondary-used-percent: 7 x-codex-secondary-window-minutes: 10080 [codex-hdr-e2e] /stats reflects codex window (primary-used=42) === CODEX-HDR E2E ALL GREEN === $ uv run ruff check . && uv run ruff format --check <touched files> All checks passed! ``` ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - **Why connect-before-accept (not capture-only).** Once `accept()` sends the `101`, headers can no longer be added; the `x-codex-*` window only exists after we connect upstream. Capturing into Python state (as #582/#590's WS code does) fixes `/stats` but not the Codex CLI's native display or any `101`-sniffing consumer — those need the headers *on the client handshake*, which requires the reorder. - **Security.** Forwarding is filtered strictly to `x-codex-*`; `set-cookie`, `authorization`, and all other upstream headers are never forwarded to the client (asserted by both the unit test and the e2e). 🤖 Generated with [Claude Code](https://claude.com/claude-code) ## Contract Schemas Per maintainer request: a JSON Schema (draft 2020-12) artifact enshrining the OpenAI interaction expectations this changeset relies on, so drift is detectable later. Committed following the repo's parity convention: - schema: `tests/parity/fixtures/codex_openai_contracts/codex-openai-interaction.schema.json` - test: `tests/test_codex_openai_contract_parity.py` binds the schema to the **live code** in both directions, so drift fails CI rather than living only in this description - every declared `x-codex-*` header must be consumed by `parse_codex_rate_limits`, and `_extract_codex_handshake_headers` must forward exactly the declared subset and never `set-cookie`/`authorization`. No new dependency (does not pull in `jsonschema`). It covers, as `$defs`: - `WSUpstreamHandshakeResponse` / `StreamingUpstreamResponseHeaders` - the upstream `x-codex-*` header family (full superset, with per-header wire pattern + the parsed semantic type) the WS and SSE captures read. Source of truth: `parse_codex_rate_limits`. - `ClientForwardedHandshakeHeaders` - the WS-101 **allow/deny** contract: only `x-codex-*` may be forwarded; `set-cookie`/`authorization` are explicitly forbidden (`propertyNames` + `not`). - `ClientForwardedStreamingHeaders` - the wider SSE forward set (`*ratelimit*` OR `x-codex*`). - `WSClientRequestFrame` / `WSRelayEvent` / `HTTPFallbackRequestBody` - the WS frame envelopes and the unwrapped HTTP-fallback POST body. - `CodexRateLimitStatsOutput` - the headroom `/stats` shape the parity tests assert. Validated with `jsonschema` (Draft202012 `check_schema` passes; positive instances from the e2e validate; negative instances - a leaked `set-cookie`, a fallback body still carrying a top-level `type` - are correctly rejected). <details> <summary><code>codex-openai-interaction.schema.json</code> (draft 2020-12)</summary> ```json { "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://github.com/chopratejas/headroom/contracts/codex-openai-interaction.schema.json", "title": "Codex <-> OpenAI interaction contracts (PR #794)", "description": "Enshrines the OpenAI interaction expectations this changeset depends on, so drift is detectable. Header values are transported as strings on the wire; the `x-headroom-parsed-type` annotation on each records the semantic type the parser (headroom/subscription/codex_rate_limits.py) coerces them to. Sources: codex_rate_limits.parse_codex_rate_limits (header family + gating), openai._extract_codex_handshake_headers (WS-101 forward filter), streaming.py (SSE forward filter).", "$defs": { "OpenAICodexWindowHeaders": { "title": "x-codex-*-{primary,secondary} window headers", "description": "A rolling rate-limit/subscription window. A window is materialized iff its `*-used-percent` header is present and numeric; `*-window-minutes` and `*-reset-at` are optional. `primary` and `secondary` are independent and either may be absent.", "type": "object", "properties": { "x-codex-primary-used-percent": { "type": "string", "pattern": "^\\d+(?:\\.\\d+)?$", "x-headroom-parsed-type": "float (0-100, NaN-guarded)", "description": "Percent of the primary window consumed. Gates creation of the primary window." }, "x-codex-primary-window-minutes": { "type": "string", "pattern": "^\\d+$", "x-headroom-parsed-type": "int", "description": "Primary window size in minutes." }, "x-codex-primary-reset-at": { "type": "string", "pattern": "^\\d+$", "x-headroom-parsed-type": "int (Unix epoch seconds)", "description": "Absolute reset time of the primary window." }, "x-codex-secondary-used-percent": { "type": "string", "pattern": "^\\d+(?:\\.\\d+)?$", "x-headroom-parsed-type": "float (0-100, NaN-guarded)", "description": "Percent of the secondary window consumed. Gates creation of the secondary window." }, "x-codex-secondary-window-minutes": { "type": "string", "pattern": "^\\d+$", "x-headroom-parsed-type": "int" }, "x-codex-secondary-reset-at": { "type": "string", "pattern": "^\\d+$", "x-headroom-parsed-type": "int (Unix epoch seconds)" } }, "additionalProperties": true }, "OpenAICodexCreditsHeaders": { "title": "x-codex-credits-* headers", "description": "OpenAI credits balance. A credits snapshot is materialized iff `x-codex-credits-has-credits` is present; `unlimited` defaults to false; `balance` is optional.", "type": "object", "properties": { "x-codex-credits-has-credits": { "type": "string", "pattern": "^(?:[Tt][Rr][Uu][Ee]|[Ff][Aa][Ll][Ss][Ee]|[01])$", "x-headroom-parsed-type": "bool (true|false|1|0, case-insensitive)", "description": "Gates creation of the credits snapshot." }, "x-codex-credits-unlimited": { "type": "string", "pattern": "^(?:[Tt][Rr][Uu][Ee]|[Ff][Aa][Ll][Ss][Ee]|[01])$", "x-headroom-parsed-type": "bool (defaults false when absent/unparseable)" }, "x-codex-credits-balance": { "type": "string", "x-headroom-parsed-type": "str (empty -> null)", "description": "Free-form server string, e.g. \"$5.00\"." } }, "additionalProperties": true }, "OpenAICodexMetaHeaders": { "title": "x-codex meta headers", "type": "object", "properties": { "x-codex-limit-name": { "type": "string", "x-headroom-parsed-type": "str (empty -> null)", "description": "Active limit/model label, e.g. \"gpt-5.2-codex-sonic\"." }, "x-codex-promo-message": { "type": "string", "x-headroom-parsed-type": "str (empty -> null)", "description": "Server announcement. Also gates snapshot creation when present." } }, "additionalProperties": true }, "OpenAICodexRateLimitHeaders": { "title": "Full x-codex-* header family OpenAI may emit", "description": "Superset of every x-codex-* header headroom reads. parse_codex_rate_limits returns a snapshot iff at least one of: a primary window, a secondary window, a credits snapshot, or a non-empty promo message is present; otherwise null (treated as a non-Codex response). All members are individually optional.", "type": "object", "allOf": [ { "$ref": "#/$defs/OpenAICodexWindowHeaders" }, { "$ref": "#/$defs/OpenAICodexCreditsHeaders" }, { "$ref": "#/$defs/OpenAICodexMetaHeaders" } ], "additionalProperties": true }, "WSUpstreamHandshakeResponse": { "title": "OpenAI WS handshake (101) response headers consumed by the WS fix", "description": "On the Codex WebSocket transport the x-codex-* window is delivered ONLY on the upstream handshake response (never in data frames). handle_openai_responses_ws reads upstream.response.headers here. This is the contract the connect-before-accept reorder depends on: if OpenAI ever moves these headers off the handshake (e.g. into a frame), the WS half of the fix goes stale.", "$ref": "#/$defs/OpenAICodexRateLimitHeaders" }, "StreamingUpstreamResponseHeaders": { "title": "OpenAI streaming/HTTP response headers consumed by the SSE fix", "description": "On the streaming SSE/HTTP transport the same x-codex-* headers ride the HTTP response. streaming.py captures them on ALL statuses (including >=400) via update_from_headers, and forwards a wider set to the client (see ClientForwardedStreamingHeaders).", "$ref": "#/$defs/OpenAICodexRateLimitHeaders" }, "ClientForwardedHandshakeHeaders": { "title": "Headers forwarded onto the CLIENT-facing WS 101 (allow/deny contract)", "description": "_extract_codex_handshake_headers forwards ONLY headers whose (lowercased) name starts with `x-codex-`. Every other upstream handshake header - notably set-cookie and authorization - MUST NOT appear on the client 101. Enforced by propertyNames below and asserted by the unit tests + tests/e2e_ws_codex_usage_headers.py.", "type": "object", "propertyNames": { "pattern": "^[Xx]-[Cc][Oo][Dd][Ee][Xx]-" }, "not": { "anyOf": [ { "required": ["set-cookie"] }, { "required": ["Set-Cookie"] }, { "required": ["authorization"] }, { "required": ["Authorization"] } ] }, "additionalProperties": { "type": "string" } }, "ClientForwardedStreamingHeaders": { "title": "Headers forwarded to the client on the streaming SSE path", "description": "streaming.py forwards a header iff `\"ratelimit\" in name.lower()` OR `name.lower().startswith(\"x-codex\")`. This is a SUPERSET of the WS allow-list: it additionally passes generic *ratelimit* headers (e.g. the Anthropic streaming path) which do not contain the x-codex prefix.", "type": "object", "propertyNames": { "pattern": "(?:[Rr][Aa][Tt][Ee][Ll][Ii][Mm][Ii][Tt])|^[Xx]-[Cc][Oo][Dd][Ee][Xx]" }, "additionalProperties": { "type": "string" } }, "WSClientRequestFrame": { "title": "Client -> proxy WS data frame (Responses API over WS)", "description": "Codex sends the request as a response.create envelope. The HTTP fallback unwraps `.response` for the POST body, forces stream=true, and strips any top-level `type`. A flattened variant (no envelope, fields at top level) is also tolerated by the fallback.", "type": "object", "properties": { "type": { "const": "response.create" }, "response": { "type": "object", "properties": { "model": { "type": "string", "description": "e.g. gpt-5.4" }, "input": { "description": "String prompt or Responses-API structured input array.", "type": ["string", "array"] }, "stream": { "type": "boolean" } }, "required": ["model"], "additionalProperties": true } }, "required": ["type", "response"], "additionalProperties": true }, "WSRelayEvent": { "title": "proxy -> client WS data frame (relayed Responses API event)", "description": "SSE `data:` payloads relayed verbatim as WS text frames. `[DONE]` sentinels are dropped (not relayed). Every relayed event is a JSON object carrying a `type`. response.completed additionally carries usage under `response.usage`. anyOf (not oneOf): an error event also satisfies the looser lifecycle shape, which is fine.", "anyOf": [ { "title": "lifecycle event", "type": "object", "properties": { "type": { "type": "string", "examples": [ "response.created", "response.output_item.added", "response.completed" ] }, "response": { "type": "object", "additionalProperties": true } }, "required": ["type"], "additionalProperties": true }, { "title": "error event", "type": "object", "properties": { "type": { "const": "error" }, "error": { "type": "object", "properties": { "message": { "type": "string" } }, "required": ["message"], "additionalProperties": true } }, "required": ["type", "error"], "additionalProperties": true } ] }, "HTTPFallbackRequestBody": { "title": "proxy -> OpenAI HTTP POST body on WS->HTTP fallback", "description": "Derived from WSClientRequestFrame: the inner `.response` object, with `stream` forced to true and any top-level `type` removed.", "type": "object", "properties": { "model": { "type": "string" }, "stream": { "const": true }, "input": { "type": ["string", "array"] } }, "required": ["model", "stream"], "not": { "required": ["type"] }, "additionalProperties": true }, "CodexRateLimitStatsOutput": { "title": "headroom /stats output for the codex tracker (CodexRateLimitSnapshot.to_dict)", "description": "Internal (headroom-emitted) shape produced from the headers above; the WS and SSE update_from_headers parity tests assert this is refreshed. Included so drift in our own surface is also caught.", "type": "object", "properties": { "limit_id": { "const": "codex" }, "limit_name": { "type": ["string", "null"] }, "primary": { "$ref": "#/$defs/CodexWindowDict" }, "secondary": { "$ref": "#/$defs/CodexWindowDict" }, "credits": { "oneOf": [ { "type": "null" }, { "type": "object", "properties": { "has_credits": { "type": "boolean" }, "unlimited": { "type": "boolean" }, "balance": { "type": ["string", "null"] } }, "required": ["has_credits", "unlimited", "balance"], "additionalProperties": false } ] }, "promo_message": { "type": ["string", "null"] }, "captured_at": { "type": "number", "description": "Unix epoch seconds (float)." } }, "required": ["limit_id", "limit_name", "primary", "secondary", "credits", "promo_message", "captured_at"], "additionalProperties": false }, "CodexWindowDict": { "oneOf": [ { "type": "null" }, { "type": "object", "properties": { "used_percent": { "type": "number" }, "window_minutes": { "type": ["integer", "null"] }, "window_label": { "type": "string", "description": "e.g. \"5h\", \"7d\"-style label; \"unknown\" when window_minutes is null." }, "resets_at": { "type": ["integer", "null"], "description": "Unix epoch seconds." }, "seconds_until_reset": { "type": ["integer", "null"] } }, "required": ["used_percent", "window_minutes", "window_label", "resets_at", "seconds_until_reset"], "additionalProperties": false } ] } } } ``` </details> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: m16khb <m16khb@gmail.com> |
||
|
gglucass
|
0b8b8d92de
|
feat(proxy): attribute savings history rollups per provider (#791)
## Description Adds per-provider attribution to the durable savings history rollups so consumers can show how savings and spend are distributed across providers within a given time period. Today the per-provider numbers on `/dashboard` come from `cache_by_provider`/`requests_by_provider` in `prometheus_metrics.py`, which are cumulative-since-start counters with no timestamp. The time-series that powers the savings history (`/stats-history` -> `SavingsTracker.history_response`) had no provider dimension at all, so a per-time-period provider breakdown was impossible. This change threads the provider into the history buckets. Fixes #(none) ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `prometheus_metrics.py`: forward the `provider` already known at the `record_request` call site into `SavingsTracker.record_request`. - `savings_tracker.py`: add an optional `provider` arg to `record_request` and `record_compression_savings`, persist it on each history checkpoint, and preserve it through `_normalize_history_entry`. - `savings_tracker.py`: in `_build_rollup`, attribute each checkpoint's delta to its provider, emitting a `by_provider` map per bucket (`tokens_saved`, `compression_savings_usd_delta`, `total_input_tokens_delta`, `total_input_cost_usd_delta`). Each checkpoint is produced by a single request, so its delta is wholly owned by one provider. Providers only appear in a bucket where they moved a counter; legacy checkpoints with no provider collapse into `"unknown"`. - Additive and schema-compatible: `schema_version` is unchanged, old persisted state loads unchanged (provider defaults to `"unknown"`), and existing rollup fields are untouched. CSV export is unaffected (it filters to its fixed columns). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed Added `test_savings_tracker_rollup_attributes_savings_per_provider`, covering: two providers sharing one hour bucket, a bucket with a single provider, per-provider deltas summing back to the bucket total, and a no-provider checkpoint collapsing to `"unknown"`. Updated three existing exact-equality history-shape assertions to include the new `provider` field. ## Test Output ``` $ uv run pytest tests/test_proxy_savings_history.py -q 14 passed, 2 warnings in 7.75s $ uv run ruff check headroom/proxy/savings_tracker.py headroom/proxy/prometheus_metrics.py tests/test_proxy_savings_history.py All checks passed! $ uv run ruff format --check ... 3 files already formatted $ uv run mypy headroom/proxy/savings_tracker.py headroom/proxy/prometheus_metrics.py Success: no issues found in 2 source files ``` ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes The new `by_provider` field is purely additive; existing consumers that read the flat bucket totals are unaffected. The keys are providers (`anthropic`, `openai`, ...), not per-client/connector identities -- the proxy buckets requests by provider at record time, so finer per-client attribution would be a separate change. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Ashish
|
574bbae2cb
|
fix: don't inject empty tools:[] when client omitted the tools field (#772)
Fixes #728 ## Summary - `apply_session_sticky_ccr_tool` and `apply_session_sticky_memory_tools` always return a list — returning `[]` when `existing_tools=None` and nothing was injected - The old handler guard `if tools is not None:` evaluated `True` for `[]`, causing `body["tools"] = []` to be sent upstream on every request - vLLM-based providers (Venice.ai, etc.) strictly reject empty `tools` arrays with a 400 error **Fix:** Change the guard in both the OpenAI and Anthropic handlers from: ```python if tools is not None: body["tools"] = tools ``` to: ```python if tools or _original_tools is not None: body["tools"] = tools ``` The `_original_tools` variable is already defined in both handlers (`_original_tools = body.get("tools")`). This condition correctly handles all four cases: | Scenario | `tools` | `_original_tools` | Result | |---|---|---|---| | No client tools, no injection | `[]` | `None` | `False` → don't inject ✅ | | No client tools, CCR injected | `[ccr_tool]` | `None` | `True` → inject ✅ | | Client sent `tools: []` | `[]` | `[]` | `True` → preserve ✅ | | Client sent tools | `[A, ...]` | `[A, ...]` | `True` → preserve ✅ | ## Test plan - [x] New test file `tests/test_issue_728_empty_tools_injection.py` with 7 tests covering the guard condition and helper behavior - [x] All 51 existing CCR/golden-bytes tests still pass - [x] Zero changes to helper function return types or signatures ## Real behavior proof Tested against the helpers directly: ``` tests/test_issue_728_empty_tools_injection.py::TestHandlerGuardCondition::test_no_tools_no_injection_does_not_inject PASSED tests/test_issue_728_empty_tools_injection.py::TestHandlerGuardCondition::test_client_sent_empty_tools_is_preserved PASSED tests/test_issue_728_empty_tools_injection.py::TestHandlerGuardCondition::test_ccr_injection_sets_body_tools PASSED tests/test_issue_728_empty_tools_injection.py::TestHandlerGuardCondition::test_client_tools_always_set PASSED tests/test_issue_728_empty_tools_injection.py::TestCCRHelperNoToolsNoCompression::test_returns_empty_list_and_false_when_no_session_ccr PASSED tests/test_issue_728_empty_tools_injection.py::TestCCRHelperNoToolsNoCompression::test_returns_tool_list_when_compression_occurred PASSED tests/test_issue_728_empty_tools_injection.py::TestCCRHelperNoToolsNoCompression::test_no_double_injection_when_client_pre_registered_ccr_tool PASSED 7 passed in 1.81s ``` **What I did not test:** end-to-end against a live Venice.ai endpoint (no API key available), or passthrough mode with a real vLLM backend. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Tejas Chopra <chopratejas@gmail.com> |
||
|
JD Davis
|
11ab5f83a1
|
feat: add differential network capture harness (#761)
## Summary - add a containerized differential network capture harness for Claude Code direct vs Claude Code routed through Headroom - capture both Headroom client-side traffic and Headroom upstream traffic with sanitized mitmproxy JSONL output - add `headroom capture network-diff` to compare captures and produce Markdown/JSON reports, including Anthropic tool-count/tool-byte deltas for deferred-tool investigations - add an on-demand GitHub Actions workflow for the harness; it only runs via `workflow_dispatch`, with live Claude Code/Anthropic capture gated on `ANTHROPIC_API_KEY` - document the workflow and ignore generated capture artifacts ## Validation - `C:\git\headroom\.venv\Scripts\python.exe -m pytest tests/test_network_diff_capture.py` - `ruff check headroom/capture headroom/cli/capture.py tests/test_network_diff_capture.py` - `ruff format --check headroom/capture headroom/cli/capture.py tests/test_network_diff_capture.py` - `C:\git\headroom\.venv\Scripts\python.exe -m mypy headroom/capture/network_diff.py headroom/cli/capture.py` - `docker compose -f docker/differential-network-capture/docker-compose.yml --profile run config` - `docker compose -f docker/differential-network-capture/docker-compose.yml --profile run build claude-direct` - `docker run --rm -e CLAUDE_COMMAND="claude --version" headroom-network-diff-claude-direct:latest` - parsed `.github/workflows/network-diff-capture.yml` with PyYAML and confirmed manual-only trigger Live Claude API capture was not run locally because `ANTHROPIC_API_KEY` is not set in this environment. The workflow can run it manually in GitHub Actions when that secret is present; otherwise it emits a visible skip warning and uploads a skipped artifact. ## Notes - Full pre-commit mypy still fails on unrelated Windows `fcntl` attributes in `headroom/subscription/tracker.py`; the feature commit skipped only that hook after narrow mypy passed for the new modules. - `tests/test_release_workflows.py` has two Windows-local failures because it shells out to a missing Unix/Rust command; unrelated workflow checks in that file passed before those failures. - Motivated by https://github.com/chopratejas/headroom/issues/746#issuecomment-4651276818 / Issue #746. |
||
|
yehsuf
|
e50fbb3e0d
|
fix(ssl): upstream httpx client inherits SSL_CERT_FILE, REQUESTS_CA_BUNDLE, NODE_EXTRA_CA_CERTS (#745)
Closes #741 ## What Headroom is commonly deployed alongside Claude Code, which sets `NODE_EXTRA_CA_CERTS` to a custom CA bundle for corporate or internal CAs. Node.js inherits this automatically; Python's `httpx` does not. Every upstream request silently failed with `SSL: CERTIFICATE_VERIFY_FAILED`, causing 502s and retry loops in the client. ## Changes - New `headroom/proxy/ssl_context.py` with `build_ssl_context()` helper that checks `SSL_CERT_FILE` → `REQUESTS_CA_BUNDLE` → `NODE_EXTRA_CA_CERTS` (first match wins) and builds an `ssl.SSLContext` with the custom CA bundle loaded - `HeadroomProxy.start()` calls `build_ssl_context()` and passes the result as `verify=` to `httpx.AsyncClient`; falls back to `verify=True` (default httpx behaviour) when no env var is set - Logs which env var and path was used at `INFO` level; warns on set-but-missing paths - 10 unit tests covering: no env var → `None`, each var returns `SSLContext`, priority order, nonexistent paths skipped ## Priority order 1. `SSL_CERT_FILE` — standard POSIX/Python ssl override 2. `REQUESTS_CA_BUNDLE` — standard requests/httpx convention 3. `NODE_EXTRA_CA_CERTS` — Node.js / Claude Code convention |
||
|
JD Davis
|
ec7d0065cc
|
Merge pull request #558 from devdeepsarkar/refactor-model-resolution
refactor: extract litellm model resolution to shared utility |