mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
1576 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Ashish
|
d6f0f0f642
|
fix(compression): correct JSON array item counting and entropy gate (#887)
## Description
Two bugs in `JSONStructureHandler` that jointly defeated the "keep first
N array items fully" design. (1) Every comma under `array_depth > 0` was
counted as an array item separator — including commas *between keys
inside objects* — so for arrays of objects the first record's own keys
exhausted `max_array_items_full` and dropped values belonging to item 0.
(2) Fixing that unmasked a second bug: self-normalized Shannon entropy
scores English prose at 0.90+, above the 0.85 "identifier" threshold, so
every long description was preserved as a fake high-entropy identifier.
Closes # <!-- found during a compression-handler review -->
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/compression/handlers/json_handler.py`: replace depth-keyed
comma counting with a container stack so only commas whose immediate
enclosing container is an array advance that array's item index.
- `headroom/compression/handlers/json_handler.py`: gate the entropy
preservation check on a no-spaces identifier signal, so UUIDs/hashes
still pass but prose compresses.
- `tests/test_compression/test_json_handler.py`: regression tests for
object-comma counting, items past the threshold, and prose-vs-identifier
entropy.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_compression/test_json_handler.py -q
32 passed
```
## Real Behavior Proof
- Environment: local macOS, Python 3.11, branch
`fix/json-array-item-count`.
- Exact command / steps: `pytest
tests/test_compression/test_json_handler.py -q` plus an empirical mask
dump on `[{"a":1,"b":2,...}]`.
- Observed result: Values inside array item 0 are now preserved; long
prose values compress while UUIDs are retained (prose scored
0.906-0.929, UUID 0.956 — the threshold alone could not separate them).
- Not tested: End-to-end through the live proxy pipeline (the handler is
not yet wired into the proxy hot path).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A — library/compression change with no UI. See Test Output.
## Additional Notes
First of a 7-PR compression-handler review series.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
skblue
|
b70fccbe17
|
fix(proxy): read RTK gain stats globally by default (#957)
## Description Closes #900. The proxy now reads RTK lifetime savings with global scope by default. This matches shared daemon deployments where the proxy process cwd is often `$HOME` or a service directory, while RTK savings are accumulated across the operator's projects. `HEADROOM_RTK_GAIN_SCOPE=project` keeps the old `rtk gain --project` behavior for operators who explicitly want the proxy process working directory as the scope. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Default RTK stats subprocess command to `rtk gain --format json` - Add `HEADROOM_RTK_GAIN_SCOPE=project` to opt into `rtk gain --project --format json` - Keep fallback/synthetic-zero payload `scope` aligned with the queried scope - Deduplicate context-tool zero payload construction - Document the new RTK gain scope environment variable ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run --extra dev python -m pytest tests/test_proxy_dashboard_stats_cache.py tests/test_subscription_tracker_rtk_wired.py -q ============================= test session starts ============================== platform darwin -- Python 3.14.5, pytest-9.0.3, pluggy-1.6.0 rootdir: /Users/joshuasiu/vibe/temp/headroom configfile: pyproject.toml plugins: anyio-4.12.1, langsmith-0.8.0, asyncio-1.3.0, cov-7.0.0 asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 26 items tests/test_proxy_dashboard_stats_cache.py .......... [ 38%] tests/test_subscription_tracker_rtk_wired.py ................ [100%] ============================== 26 passed in 0.40s ============================== uv run --extra dev python -m pytest tests/test_proxy_stats_recent_requests.py tests/test_proxy_healthchecks.py -q ============================= test session starts ============================== platform darwin -- Python 3.14.5, pytest-9.0.3, pluggy-1.6.0 rootdir: /Users/joshuasiu/vibe/temp/headroom configfile: pyproject.toml plugins: anyio-4.12.1, langsmith-0.8.0, asyncio-1.3.0, cov-7.0.0 asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 15 items tests/test_proxy_stats_recent_requests.py ... [ 20%] tests/test_proxy_healthchecks.py ............ [100%] ============================= 15 passed in 10.41s ============================== uv run --extra dev ruff check headroom/proxy/helpers.py tests/test_proxy_dashboard_stats_cache.py tests/test_subscription_tracker_rtk_wired.py All checks passed! uv run --extra dev ruff format --check headroom/proxy/helpers.py tests/test_proxy_dashboard_stats_cache.py tests/test_subscription_tracker_rtk_wired.py 3 files already formatted uv run --extra dev mypy headroom headroom/proxy/server.py:1141: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] headroom/proxy/server.py:1211: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] headroom/proxy/server.py:1215: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] Success: no issues found in 358 source files ``` ## Real Behavior Proof - Environment: macOS, Python 3.14.5 via `uv run --extra dev` - Exact command / steps: mocked RTK subprocess argv in unit tests - Observed result: default command is `rtk gain --format json`; project scope command is `rtk gain --project --format json`; invalid scope logs `event=rtk_gain_scope_invalid` and falls back to global - Not tested: full repository pytest suite ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes The unchecked comment and changelog items are not applicable for this scoped proxy stats fix. |
||
|
Focused Instability
|
b51cda10d7
|
docs(evals): add session probes section to evals README (#888)
## Description Follow-up to #862. That PR's body described a **Session Probes** section in `headroom/evals/README.md`, but the file edit missed the commit (edited in the wrong checkout). This adds the missing 22-line docs-only section: the record-then-score workflow for `HEADROOM_PROBE_RECORD_DIR` + `headroom evals probes`, including the plaintext-recording privacy note. Refs #861 (session-probe eval harness — this README section was part of that feature's spec). ## Type of Change - [x] Documentation update ## Changes Made - Add a **Session Probes (real recorded sessions)** section to `headroom/evals/README.md` (+22 lines, no code change): the two-step record (`HEADROOM_PROBE_RECORD_DIR=… headroom proxy start`) then score (`headroom evals probes --recordings …`) workflow, the three probe dimensions (exact numerics, artifact trail, error evidence), the retained/recoverable/lost classification, retention bucketing by ratio + per-transform grouping, and the `--json-output` flag. - Includes the opt-in privacy note: recordings contain full conversation content in plaintext and stay on the local machine. ## Testing - [x] Documentation builds/renders correctly - [x] Linting passes (`ruff check .`) - [x] New and existing unit tests pass locally with my changes ### Test Output ```text $ git diff --stat upstream/main..HEAD headroom/evals/README.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) Docs-only change — no code paths touched. The commands and flags documented (HEADROOM_PROBE_RECORD_DIR, `headroom evals probes`, --recordings, --json-output) are the surface shipped and tested in #862. ``` ## Real Behavior Proof - Environment: local macOS, repo .venv, Python 3.11.9 - Exact command / steps: rendered the edited `headroom/evals/README.md` and cross-checked every documented flag/command against the implemented CLI from #862 (`headroom evals probes`, `HEADROOM_PROBE_RECORD_DIR`, `--recordings`, `--json-output`) - Observed result: the new section renders correctly and every command/flag it names exists in the shipped probe harness; no code paths are changed by this PR, so behavior is unchanged - Not tested: nothing additional — docs-only change with no executable surface of its own ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes Pure documentation backfill for #862; the feature itself (recorder + retention probes) already merged. PR body updated to satisfy the PR-governance template gate. |
||
|
Praneet Singh
|
14281abc26
|
Normalize headroom_stats MCP input schema (#780)
## Description
Normalize the `headroom_stats` MCP tool input schema to match other
no-argument MCP tools in the repository.
The `headroom_stats` tool previously advertised the following schema:
```json
{
"type": "object",
"properties": {}
}
```
This change updates it to:
```json
{
"type": "object",
"properties": {},
"required": []
}
```
This makes the schema consistent with other MCP tool definitions in the
codebase that do not require input parameters.
Related to #736. While this change does not conclusively fix the Copilot
ACP tool discovery issue, it removes an inconsistency in the advertised
MCP tool schema and may improve compatibility with MCP clients that
perform stricter schema validation.
## Type of Change
* [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
* Added `"required": []` to the `headroom_stats` MCP tool schema.
* Normalized `headroom_stats` to match other no-argument MCP tool
definitions in the repository.
* Improved MCP schema consistency across tool registrations.
## Testing
Describe the tests you ran to verify your changes:
* [ ] Unit tests pass (`pytest`)
* [ ] Linting passes (`ruff check .`)
* [ ] Type checking passes (`mypy headroom`)
* [ ] New tests added for new functionality
* [x] Manual testing performed
## Test Output
Not run. This change is limited to MCP tool schema metadata and does not
modify runtime tool behavior.
## Checklist
* [x] My code follows the project's style guidelines
* [x] I have performed a self-review of my code
* [ ] I have commented my code, particularly in hard-to-understand areas
* [ ] I have made corresponding changes to the documentation
* [x] My changes generate no new warnings
* [ ] I have added tests that prove my fix is effective or that my
feature works
* [ ] New and existing unit tests pass locally with my changes
* [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
The repository already uses `"required": []` for other MCP tool schemas
with no required parameters (for example in the memory MCP
implementation). This change aligns `headroom_stats` with that existing
pattern.
<!-- maintainer-validation-2026-06-13 -->
## Testing
Describe the tests you ran to verify your changes:
- [x] Commit message validation passes (`commitlint --last --config
.commitlintrc.json`)
- [x] Manual review performed
## Test Output
```text
npx --yes -p @commitlint/cli -p @commitlint/config-conventional commitlint --last --config .commitlintrc.json
# passed with no output
```
## Real Behavior Proof
- Environment: Windows local maintainer checkout, PowerShell, GitHub CLI
authenticated as maintainer.
- Exact command / steps: Amended the single PR commit subject to
`fix(mcp): normalize headroom_stats input schema` with original author
`Praneet <praneetware@gmail.com>`, then validated with commitlint.
- Observed result: Commitlint passed locally; branch pushed with
`--force-with-lease` after confirming the remote still pointed at
`26ad25dcc0a3fd5bf37a6f10f1bbd782d034ba9a`.
- Not tested: Full project test suite was not rerun for this
commit-message-only maintenance update.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
<!-- maintainer-template-normalization-2026-06-13 -->
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Added an explicit empty `required` list to the `headroom_stats` MCP
input schema.
- Matched `headroom_stats` to the repository pattern for no-argument MCP
tool schemas.
- Rewrote the PR commit subject to satisfy the repository commitlint
rules while preserving the original author.
## Testing
- [x] Commit message validation passes (`commitlint --last --config
.commitlintrc.json`)
- [x] Manual review performed
```text
npx --yes -p @commitlint/cli -p @commitlint/config-conventional commitlint --last --config .commitlintrc.json
# passed with no output
```
|
||
|
Kumario
|
0c5c89d05c
|
fix(anthropic): strip styled Claude model ids (#651)
## Description Fixes #626 by normalizing Anthropic/Claude model ids that contain ANSI escape sequences or dangling style suffixes before provider lookups and upstream forwarding. The branch has been updated onto current `main` and the proxy handler conflicts have been resolved. ## Type of Change - [x] Bug fix - [ ] New feature - [ ] Documentation - [ ] Refactor - [x] Tests only ## Changes Made - Normalize Anthropic model ids before context/pricing lookup. - Sanitize Anthropic `/v1/models` metadata and styled `/v1/models/{id}` passthrough paths. - Sanitize `/v1/messages` request body model ids before upstream forwarding. - Resolved current-main conflicts while preserving newer `model_override` and streaming passthrough behavior. ## Testing - [x] Unit tests - [x] Route/proxy tests - [x] Lint/static checks - [ ] Manual testing ### Test Output ```text UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --with pytest --with fastapi --with httpx --with uvicorn --with h2 python -m pytest tests/test_providers/test_anthropic.py tests/test_provider_proxy_routes.py::test_anthropic_model_metadata_strips_ansi_model_ids tests/test_provider_proxy_routes.py::test_anthropic_model_detail_path_strips_ansi_model_id tests/test_provider_proxy_routes.py::test_anthropic_messages_strips_ansi_model_id_before_upstream -q 17 passed, 2 warnings in 39.91s UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --with ruff ruff check headroom/providers/anthropic.py headroom/proxy/handlers/openai.py headroom/proxy/handlers/anthropic.py tests/test_providers/test_anthropic.py tests/test_provider_proxy_routes.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13.3, focused local worktree for PR #651 after merging current `upstream/main`. - Exact command / steps: Merged current main, resolved conflicts in Anthropic/OpenAI proxy handlers, ran the PR's targeted provider/proxy tests and ruff checks. - Observed result: Styled Anthropic model metadata, model-detail path, and messages upstream sanitization tests pass; ruff reports no issues. - Not tested: Full repository mypy/pre-commit; existing unrelated Windows `fcntl` typing errors block full hook execution locally. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review <!-- headroom-maintainer-template-completion:start --> ## Description This PR prepares `fix(anthropic): strip styled Claude model ids` for review by documenting the intended change, validation evidence, and remaining merge-readiness context. Linked issues: #626 ## Type of Change - [x] Bug fix - [ ] New feature - [ ] Documentation - [ ] Refactor - [ ] Tests only ## Changes Made - Commit: fix(anthropic): normalize styled model ids - Commit: fix(proxy): strip styled Anthropic model ids - Commit: fix: format anthropic model sanitization - Commit: Merge remote-tracking branch 'upstream/main' into review/pr-651 - Touches `headroom/cache/dynamic_detector.py` - Touches `headroom/providers/anthropic.py` - Touches `headroom/proxy/handlers/anthropic.py` - Touches `headroom/proxy/handlers/openai.py` - Touches `tests/test_provider_proxy_routes.py` - Touches `tests/test_providers/test_anthropic.py` ## Testing - [x] GitHub checks reviewed - [x] Metadata/template validation - [ ] Local functional testing ### Test Output ```text gh pr view 651 --repo chopratejas/headroom --json statusCheckRollup - PR Governance / template: FAILURE - CI / changes: SUCCESS - Init E2E / docker-init-e2e: SUCCESS - Wrap E2E / docker-wrap-e2e: SUCCESS - Wrap Native E2E / wrap-native (ubuntu-latest): SUCCESS - Wrap Native E2E / wrap-native (macos-latest): SUCCESS - CI / commitlint: SUCCESS - PR Governance / label: SUCCESS - CI / lint: SUCCESS - CI / build-wheel: SUCCESS - CI / prefetch-model: SUCCESS - CI / build: SUCCESS ``` ## Real Behavior Proof - Environment: GitHub PR metadata and checks for `chopratejas/headroom` PR #651. - Exact command / steps: Reviewed PR title, commits, changed files, linked issues, labels, and check rollup; appended this maintainer template completion block without replacing the author's original description. - Observed result: PR body now contains all required governance sections, checked readiness fields, and a non-placeholder validation evidence block. - Not tested: This pass updated PR metadata only; code validation remains represented by the linked GitHub checks and any author-provided evidence above. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review <!-- headroom-maintainer-template-completion:end --> --------- Co-authored-by: Tejas Chopra <chopratejas@gmail.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Matt Van Horn
|
9b7b436b04
|
fix: wire HEADROOM_EXCLUDE_TOOLS / HEADROOM_TOOL_PROFILES into Click proxy entrypoint (#943)
## Description The Click-based `headroom proxy` entrypoint (`headroom/cli/proxy.py`) constructed `ProxyConfig` without calling `_parse_exclude_tools` or `_parse_tool_profiles`, so `HEADROOM_EXCLUDE_TOOLS` and `HEADROOM_TOOL_PROFILES` were silently ignored for any service launched via `headroom proxy`. The argparse path in `headroom/proxy/server.py` already handled these correctly. This PR imports both helpers into the Click entrypoint and wires their output into `ProxyConfig`. Closes #825 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cli/proxy.py`: import `_parse_exclude_tools` and `_parse_tool_profiles` alongside `ProxyConfig`/`run_server`; pass their output into the `ProxyConfig(...)` construction (`or None` guard collapses empty set/dict to `None` so unset vars leave `DEFAULT_EXCLUDE_TOOLS` unchanged) - `tests/test_cli_proxy_env.py`: new `TestCLIProxyExcludeToolsEnvVar` class with 5 regression tests ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ### Paste relevant command output or artifact links here ```text ============================= test session starts ============================== platform darwin -- Python 3.13.12, pytest-9.0.3 collected 43 items tests/test_cli_proxy_env.py::TestCLIProxyExcludeToolsEnvVar::test_exclude_tools_single_name_from_env PASSED tests/test_cli_proxy_env.py::TestCLIProxyExcludeToolsEnvVar::test_exclude_tools_multi_name_from_env PASSED tests/test_cli_proxy_env.py::TestCLIProxyExcludeToolsEnvVar::test_exclude_tools_unset_leaves_none PASSED tests/test_cli_proxy_env.py::TestCLIProxyExcludeToolsEnvVar::test_tool_profiles_from_env PASSED tests/test_cli_proxy_env.py::TestCLIProxyExcludeToolsEnvVar::test_tool_profiles_unset_leaves_none PASSED ============================== 43 passed in 8.95s ============================== ruff check headroom/cli/proxy.py tests/test_cli_proxy_env.py All checks passed! ``` ## Real Behavior Proof - Environment: Python 3.13.12, headroom-ai dev install - Exact command / steps: `HEADROOM_EXCLUDE_TOOLS=WebSearch headroom proxy` before fix silently built `ProxyConfig(exclude_tools=None)` despite the env var being set - Observed result: After fix, `ProxyConfig.exclude_tools` contains `{"WebSearch", "websearch"}` as verified by the new unit tests - Not tested: end-to-end proxy run with a live Anthropic endpoint ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes The fix mirrors the exact pattern already used in the argparse path (`_main()` in `headroom/proxy/server.py` lines 3920-3922). The `or None` guard is intentional: `_parse_exclude_tools(None)` returns `set()` when the env var is unset, and `ProxyConfig.exclude_tools=None` means "use `DEFAULT_EXCLUDE_TOOLS` unchanged" — passing an empty set would instead replace the defaults with nothing. Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Zbl1007
|
e0a9fdb62c
|
chore(imports): lazy-load dynamic detector ML imports (#597)
## Description Avoid importing optional ML dependencies when `headroom.cache.dynamic_detector` is imported during wrap/proxy startup. This keeps the dynamic detector module cheap to import while preserving the existing NER and semantic detector behavior when those tiers are actually used. Fixes #195 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Replaced eager `spacy`, `sentence_transformers`, and `numpy` imports in `dynamic_detector.py` with `find_spec` availability checks. - Kept NER and semantic model loading on the existing first-use detector initialization paths. - Moved `numpy` import to the semantic similarity calculation path where it is actually needed. - Added regression coverage proving `dynamic_detector` import does not load optional ML modules even when stub versions of those modules are importable. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text .venv/bin/pytest tests/test_package_init_lazy.py 7 passed in 2.82s .venv/bin/pytest tests/test_cache/test_dynamic_detector.py tests/test_package_init_lazy.py 43 passed, 2 skipped in 19.48s .venv/bin/python -m ruff check headroom/cache/dynamic_detector.py tests/test_package_init_lazy.py All checks passed! .venv/bin/python -m ruff format --check headroom/cache/dynamic_detector.py tests/test_package_init_lazy.py 2 files already formatted git diff --check # no output ``` ## Real Behavior Proof - Environment: macOS local checkout, Python 3.11 virtualenv at `.venv`. - Exact command / steps: Added a subprocess regression test that creates importable stub `spacy`, `numpy`, `torch`, and `sentence_transformers` modules, imports `headroom.cache.dynamic_detector`, and inspects `sys.modules`. - Observed result: Before the implementation change, the new regression test failed because `spacy` was loaded during module import. After the change, `spacy`, `sentence_transformers`, and `torch` remain unloaded and the dynamic detector test suite still passes. - Not tested: Full repository-wide `mypy headroom`; full CI requires maintainer approval for fork workflows. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) Not applicable. ## Additional Notes This branch has been narrowed after review feedback. It now only contains the lazy `dynamic_detector` import fix. The wrapper startup-timeout behavior was removed from this PR so it can be reviewed separately from the existing timeout work. Signed-off-by: Zbl1007 <1399853961@qq.com> |
||
|
Focused Instability
|
5939004185
|
feat(evals): adversarial-input robustness grid for compressors (#918)
## Description Closes #916. CompressionAttack (arXiv:2510.22963) showed that prompt compressors are an attack surface for LLM middleware: adversarial text in compressible content can preferentially survive compression (amplifying injection density) or abuse compressor control surfaces. Headroom has a concrete instance of the latter — content carrying a CCR retrieval marker is pinned as already-compressed, so a spoofed marker string in tool output could make content compression-immune. This adds an offline, deterministic eval grid measuring both, with no LLM, no API key, and no model download (Kompress disabled by default). Closes #916. ## Type of Change - [x] New feature (non-breaking change which adds functionality) ## Changes Made - `headroom/evals/adversarial_grid.py`: payload corpus (instruction override, fake system tag, fake tool directive, CCR marker spoof in block + inline forms, steering imperative, benign control), realistic + synthetic carriers (60-record JSON array, 150-line worker log), and a payload-class × carrier × splice-position grid. - Per-cell metrics: payload survival (normalization-tolerant containment), benign-line survival baseline, and compression suppression (payload-ratio minus clean-ratio — the marker-spoof immunity signal), plus per-class aggregates. - `headroom/cli/evals.py`: wire the grid into the evals CLI command. - Tests in `tests/test_adversarial_grid.py`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] New tests added for new functionality - [x] New and existing unit tests pass locally with my changes ### Test Output ```text $ pytest tests/test_adversarial_grid.py -q 12 passed in 1.10s ``` ## Real Behavior Proof - Environment: local macOS, repo .venv, Python 3.11.9; offline (no API key, Kompress disabled) - Exact command / steps: rebased onto current main (dropping the now-superseded codecov-upload commit — main already uploads per-shard coverage via codecov-action@v5), then `pytest tests/test_adversarial_grid.py -q` - Observed result: 12/12 pass; grid runs deterministically with no network/model access and reports survival + suppression metrics per cell. - Not tested: LLM-in-the-loop attack realism — out of scope by design; this grid is the offline deterministic layer. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes Force-pushed after a rebase onto current main to resolve a `.github/workflows/ci.yml` conflict introduced by #921: the standalone codecov-upload commit was dropped because main now performs per-shard coverage upload globally. PR payload is unchanged (adversarial grid + tests). --------- Co-authored-by: integration-check <integration@local> |
||
|
Focused Instability
|
553ade4ec6
|
feat(policy): consume net-cost mutation gate in ContentRouter (#856 P2) (#905)
## Description Fixes #907. Part of #904 — the **P2 (consume, flag-gated)** item from #856's phased plan. (#903, which this was stacked on, has merged; this is now a clean diff.) `HEADROOM_NET_COST_POLICY=1` (default **off** — flag absent restores byte-identical current behavior) routes every ContentRouter mutation candidate through `CompressionPolicy.net_mutation_gain` before compression is applied, at both decision sites: the result-cache-hit path and the fresh-compression merge (pass 3). v1 estimators (as specced in #856): **ΔT** exact (compressed form already computed); **S** = token total after the slot, precomputed once as a reverse cumulative sum (O(1) per candidate); **R / P_alive** env-tunable (`HEADROOM_NET_COST_EXPECTED_READS`=10, `HEADROOM_NET_COST_P_ALIVE`=1.0). Every decision logs all inputs at INFO and increments `netcost_allowed`/`netcost_skipped` counters so the flag can be validated from telemetry before any default-on. Closes #907. ## Type of Change - [x] New feature (non-breaking change which adds functionality) ## Changes Made - Add flag-gated net-cost mutation gate to `ContentRouter` at both mutation sites (cache-hit + fresh-compress merge). - Precompute reverse-cumulative suffix token sums once per request for O(1) S lookups. - Emit INFO telemetry, `netcost_allowed`/`netcost_skipped` counters, and a `netcost:skip:<band>` transform marker on blocked slots. - **Review-response (4eb2307):** reject non-finite env values (`math.isfinite` guard), count suffix tokens block-aware via `_netcost_message_tokens()` (was `str(content)`, which miscounted Anthropic block lists), and bucket the skip marker via `_gain_bucket()` to bound dashboard cardinality. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] New tests added for new functionality - [x] New and existing unit tests pass locally with my changes ### Test Output ```text $ pytest tests/test_netcost_gate.py -q 11 passed in 0.82s $ pytest tests/ -k "content_router or netcost or router" -q 133 passed, 8 skipped, 6120 deselected in 23.26s $ ruff check headroom/transforms/content_router.py All checks passed! ``` ## Real Behavior Proof - Environment: local macOS, repo .venv, Python 3.11.9, gpt-4o tokenizer fixture - Exact command / steps: `pytest tests/test_netcost_gate.py -q` then the router-suite selector above; gate exercised end-to-end through the real tokenizer + compression path (flag on via monkeypatch) - Observed result: with R=10/P=1 defaults, a 300-row tool result followed by a 40k-word suffix is left uncompressed (gate skips, `netcost:skip:` marker emitted); a 2000-row result with a 5-word suffix compresses (gate allows). Non-finite env (`inf`/`nan`) falls back to defaults and still skips. - Not tested: live proxy traffic / real dashboard validation — deferred to the default-on milestone per #904 (this ships default-off precisely to gather that telemetry first). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes Cache-hit re-tokenization (`:2312`) and the large integration fixtures are tracked as follow-ups in the PR review thread; both are intentional given the flag is default-off. Known v1 limitations (whole-suffix S, no batch awareness, static P_alive) are tracked in #904 as P2b/P3a/P3b. PR body updated to satisfy the new PR-governance template gate (#914-era governance workflow). --------- Co-authored-by: integration-check <integration@local> |
||
|
Focused Instability
|
f9285766dd
|
feat: attribute reread waste to over-compression via marker check (#901)
## Description Fixes #899. The `reread` signal (#853/#854) counts re-served tool results but cannot answer the question that motivated it: **did Headroom cause the re-read?** A re-read after an intact first serve is agent behavior; a re-read after Headroom markerized the first serve is over-compression cost. This PR splits the signal so the actionable part is visible. Request-local, no store lookups: the client resends full history each turn and the pipeline recompresses it deterministically, so the current request already holds the evidence. `TransformPipeline.apply` passes `current_messages` into `parse_messages(compressed_messages=...)`. For each counted reread group, if the transformed copy of the **first serve** carries a CCR retrieval marker and its original text is gone, the group's counted repeats go into `reread_compressed_tokens`. Lossless reshaping (no marker) is deliberately not attributed. Closes #899. ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - `parser.py`: `parse_messages` gains an optional `compressed_messages` param; the content-hash reread loop accumulates per-group `counted_tokens` and attributes them to `reread_compressed_tokens` when the first serve's transformed copy carries a CCR marker (`CCR_RETRIEVAL_MARKER_RE`, kept local to avoid a transforms import cycle). - `transforms/pipeline.py`: pass `current_messages` (post-transform copy) into the existing waste-detection `parse_messages` call. - `config.py`: new `reread_compressed_tokens` WasteSignals field; `dashboard.html` + `reporting/generator.py` surface it. - Tests: `tests/test_reread_attribution.py` + WasteSignals contract update. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] New tests added for new functionality - [x] New and existing unit tests pass locally with my changes ### Test Output ```text $ pytest tests/test_reread_attribution.py tests/test_parser.py tests/test_gemini_function_response_waste.py tests/test_codex_responses_waste_signals.py -q 122 passed in 1.50s $ pytest tests/ -k "waste or pipeline or reporting or config or reread" -q 348 passed, 33 skipped, 6010 deselected # (1 unrelated env-dependent failure: test_proxy_gemini_native_integration::test_generation_config — 404, reproduces on main without these changes; needs a Gemini key locally) $ ruff check headroom/parser.py headroom/transforms/pipeline.py All checks passed! ``` ## Real Behavior Proof - Environment: local macOS, repo .venv, Python 3.11.9 - Exact command / steps: rebased onto current main to resolve conflicts with #909 (merged), then ran the reread + parser + waste suites above - Observed result: a reread whose first serve is markerized attributes to `reread_compressed_tokens`; an intact first serve and a lossless (no-marker) reshape do not. #909's re-issued-call detection (same call, different bytes) continues to count and dedup correctly alongside it — all 122 targeted tests pass. - Not tested: live proxy traffic; the one gemini-native route test above (environmental 404, not introduced here). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes **Rebased onto current main after #909 merged.** #909 added a re-issued-call reread pass *after* the original content-hash loop this PR modifies — the conflict was textual/adjacent, not a re-architecture. Resolution preserves #909's `counted_results` dedup contract and leaves its new pass unchanged; #901's attribution stays scoped to the content-hash groups it was reviewed against (attributing #909's call-key pass too would be a separate follow-up). The diff differs from the prior approval only by this reshape — worth a quick re-glance. |
||
|
Focused Instability
|
2a4d300841
|
feat(dashboard): surface compression-vs-cache net impact in Prefix Cache panel (#913)
## Summary Closes #911, and delivers the dashboard half of the question in #855 ("how do I understand the cache impact — can we add it to the dashboard?"). `GET /stats` already exposes `prefix_cache.compression_vs_cache` (`tokens_saved_by_compression`, `tokens_lost_to_cache_bust`, `cache_bust_count`, `net_tokens`) and `prefix_cache.prefix_freeze` (`busts_avoided`, `tokens_preserved`, `compression_foregone_tokens`, `net_benefit_tokens`) — built in `headroom/proxy/cost.py` — but the dashboard never rendered them. This adds a **Compression vs Cache** section to the existing Prefix Cache Impact panel: - **Saved by Compression** — tokens removed before send - **Lost to Cache Busts** — tokens lost, with observed bust count - **Net** — color-coded emerald when positive, red when negative, with a matching "Net positive / Net negative" headline pill - **Prefix Freeze Net** — net benefit of freeze decisions (preserved minus compression foregone), with busts avoided The section is gated on data presence (hidden until any underlying counter is non-zero), styled to match the existing TTL bucket cards, works in light and dark mode, and carries `data-testid` hooks. Frontend-only: no backend changes; the stats endpoint already serves every field. ## Testing New `tests/test_dashboard_cache_net_playwright.py` (mirrors the TTL playwright test): pins the rendered values, the negative-net red styling, and the hidden-when-empty behavior. 3/3 pass locally under chromium. Note for reviewers: the harness matches stubbed routes on URL **path**, because the dashboard now fetches `/stats?cached=1` — full-URL `endswith("/stats")` checks miss it and the request escapes to the real network. The pre-existing `test_dashboard_cache_ttl_playwright.py` has this exact bitrot (plus stale text assertions) and currently fails when playwright is actually installed — playwright isn't installed in CI so it silently skips. Left out of scope here; can file separately. `ruff check` and `ruff format --check` clean. Refs #855. Co-authored-by: integration-check <integration@local> |
||
|
dependabot[bot]
|
4f59097045
|
ci: bump esbuild from 0.27.7 to 0.28.1 in /docs in the npm_and_yarn group across 1 directory (#936)
Bumps the npm_and_yarn group with 1 update in the /docs directory: [esbuild](https://github.com/evanw/esbuild). Updates `esbuild` from 0.27.7 to 0.28.1 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/evanw/esbuild/releases">esbuild's releases</a>.</em></p> <blockquote> <h2>v0.28.1</h2> <ul> <li> <p>Disallow <code>\</code> in local development server HTTP requests (<a href="https://github.com/evanw/esbuild/security/advisories/GHSA-g7r4-m6w7-qqqr">GHSA-g7r4-m6w7-qqqr</a>)</p> <p>This release fixes a security issue where HTTP requests to esbuild's local development server could traverse outside of the serve directory on Windows using a <code>\</code> backslash character. It happened due to the use of Go's <code>path.Clean()</code> function, which only handles Unix-style <code>/</code> characters. HTTP requests with paths containing <code>\</code> are no longer allowed.</p> <p>Thanks to <a href="https://github.com/dellalibera"><code>@dellalibera</code></a> for reporting this issue.</p> </li> <li> <p>Add integrity checks to the Deno API (<a href="https://github.com/evanw/esbuild/security/advisories/GHSA-gv7w-rqvm-qjhr">GHSA-gv7w-rqvm-qjhr</a>)</p> <p>The previous release of esbuild added integrity checks to esbuild's npm install script. This release also adds integrity checks to esbuild's Deno install script. Now esbuild's Deno API will also fail with an error if the downloaded esbuild binary contains something other than the expected content.</p> <p>Note that esbuild's Deno API installs from <code>registry.npmjs.org</code> by default, but allows the <code>NPM_CONFIG_REGISTRY</code> environment variable to override this with a custom package registry. This change means that the esbuild executable served by <code>NPM_CONFIG_REGISTRY</code> must now match the expected content.</p> <p>Thanks to <a href="https://github.com/sondt99"><code>@sondt99</code></a> for reporting this issue.</p> </li> <li> <p>Avoid inlining <code>using</code> and <code>await using</code> declarations (<a href="https://redirect.github.com/evanw/esbuild/issues/4482">#4482</a>)</p> <p>Previously esbuild's minifier sometimes incorrectly inlined <code>using</code> and <code>await using</code> declarations into subsequent uses of that declaration, which then fails to dispose of the resource correctly. This bug happened because inlining was done for <code>let</code> and <code>const</code> declarations by avoiding doing it for <code>var</code> declarations, which no longer worked when more declaration types were added. Here's an example:</p> <pre lang="js"><code>// Original code { using x = new Resource() x.activate() } <p>// Old output (with --minify)<br /> new Resource().activate();</p> <p>// New output (with --minify)<br /> {using e=new Resource;e.activate()}<br /> </code></pre></p> </li> <li> <p>Fix module evaluation when an error is thrown (<a href="https://redirect.github.com/evanw/esbuild/issues/4461">#4461</a>, <a href="https://redirect.github.com/evanw/esbuild/pull/4467">#4467</a>)</p> <p>If an error is thrown during module evaluation, esbuild previously didn't preserve the state of the module for subsequent module references. This was observable if <code>import()</code> or <code>require()</code> is used to import a module multiple times. The thrown error is supposed to be thrown by every call to <code>import()</code> or <code>require()</code>, not just the first. With this release, esbuild will now throw the same error every time you call <code>import()</code> or <code>require()</code> on a module that throws during its evaluation.</p> </li> <li> <p>Fix some edge cases around the <code>new</code> operator (<a href="https://redirect.github.com/evanw/esbuild/issues/4477">#4477</a>)</p> <p>Previously esbuild incorrectly printed certain edge cases involving complex expressions inside the target of a <code>new</code> expression (specifically an optional chain and/or a tagged template literal). The generated code for the <code>new</code> target was not correctly wrapped with parentheses, and either contained a syntax error or had different semantics. These edge cases have been fixed so that they now correctly wrap the <code>new</code> target in parentheses. Here is an example of some affected code:</p> <pre lang="js"><code>// Original code new (foo()`bar`)() new (foo()?.bar)() <p>// Old output<br /> new foo()<code>bar</code>();<br /> new (foo())?.bar();</p> <p></code></pre></p> </li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/evanw/esbuild/blob/main/CHANGELOG.md">esbuild's changelog</a>.</em></p> <blockquote> <h2>0.28.1</h2> <ul> <li> <p>Disallow <code>\</code> in local development server HTTP requests (<a href="https://github.com/evanw/esbuild/security/advisories/GHSA-g7r4-m6w7-qqqr">GHSA-g7r4-m6w7-qqqr</a>)</p> <p>This release fixes a security issue where HTTP requests to esbuild's local development server could traverse outside of the serve directory on Windows using a <code>\</code> backslash character. It happened due to the use of Go's <code>path.Clean()</code> function, which only handles Unix-style <code>/</code> characters. HTTP requests with paths containing <code>\</code> are no longer allowed.</p> <p>Thanks to <a href="https://github.com/dellalibera"><code>@dellalibera</code></a> for reporting this issue.</p> </li> <li> <p>Add integrity checks to the Deno API (<a href="https://github.com/evanw/esbuild/security/advisories/GHSA-gv7w-rqvm-qjhr">GHSA-gv7w-rqvm-qjhr</a>)</p> <p>The previous release of esbuild added integrity checks to esbuild's npm install script. This release also adds integrity checks to esbuild's Deno install script. Now esbuild's Deno API will also fail with an error if the downloaded esbuild binary contains something other than the expected content.</p> <p>Note that esbuild's Deno API installs from <code>registry.npmjs.org</code> by default, but allows the <code>NPM_CONFIG_REGISTRY</code> environment variable to override this with a custom package registry. This change means that the esbuild executable served by <code>NPM_CONFIG_REGISTRY</code> must now match the expected content.</p> <p>Thanks to <a href="https://github.com/sondt99"><code>@sondt99</code></a> for reporting this issue.</p> </li> <li> <p>Avoid inlining <code>using</code> and <code>await using</code> declarations (<a href="https://redirect.github.com/evanw/esbuild/issues/4482">#4482</a>)</p> <p>Previously esbuild's minifier sometimes incorrectly inlined <code>using</code> and <code>await using</code> declarations into subsequent uses of that declaration, which then fails to dispose of the resource correctly. This bug happened because inlining was done for <code>let</code> and <code>const</code> declarations by avoiding doing it for <code>var</code> declarations, which no longer worked when more declaration types were added. Here's an example:</p> <pre lang="js"><code>// Original code { using x = new Resource() x.activate() } <p>// Old output (with --minify)<br /> new Resource().activate();</p> <p>// New output (with --minify)<br /> {using e=new Resource;e.activate()}<br /> </code></pre></p> </li> <li> <p>Fix module evaluation when an error is thrown (<a href="https://redirect.github.com/evanw/esbuild/issues/4461">#4461</a>, <a href="https://redirect.github.com/evanw/esbuild/pull/4467">#4467</a>)</p> <p>If an error is thrown during module evaluation, esbuild previously didn't preserve the state of the module for subsequent module references. This was observable if <code>import()</code> or <code>require()</code> is used to import a module multiple times. The thrown error is supposed to be thrown by every call to <code>import()</code> or <code>require()</code>, not just the first. With this release, esbuild will now throw the same error every time you call <code>import()</code> or <code>require()</code> on a module that throws during its evaluation.</p> </li> <li> <p>Fix some edge cases around the <code>new</code> operator (<a href="https://redirect.github.com/evanw/esbuild/issues/4477">#4477</a>)</p> <p>Previously esbuild incorrectly printed certain edge cases involving complex expressions inside the target of a <code>new</code> expression (specifically an optional chain and/or a tagged template literal). The generated code for the <code>new</code> target was not correctly wrapped with parentheses, and either contained a syntax error or had different semantics. These edge cases have been fixed so that they now correctly wrap the <code>new</code> target in parentheses. Here is an example of some affected code:</p> <pre lang="js"><code>// Original code new (foo()`bar`)() new (foo()?.bar)() <p>// Old output<br /> new foo()<code>bar</code>();<br /> new (foo())?.bar();<br /> </code></pre></p> </li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
Joel Belanger
|
0b4a4bd483
|
fix: support Copilot Business subscription auth (#641)
## Description Adds a first-party `headroom copilot-auth login` flow for Copilot subscription mode and uses the resulting Copilot OAuth token to perform GitHub's Copilot token exchange before launching the wrapped Copilot CLI. This fixes Business/Enterprise Cloud accounts where a generic GitHub/Copilot token can read Copilot account metadata but is rejected by the Copilot token exchange endpoint. It also avoids treating GitHub.com Enterprise Cloud account URLs such as `github.com/enterprises/acme` as API hostnames. Fixes #635 Related: #488, #610 Builds on #576 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [x] Documentation update - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Adds `headroom copilot-auth login` and `headroom copilot-auth status`. - Stores a Headroom-specific Copilot OAuth token under Headroom's state dir. - Exchanges reusable Copilot OAuth tokens with Copilot Chat-compatible headers before subscription-mode launch. - Carries the resolved Copilot API endpoint into `headroom wrap copilot --subscription`. - Handles GitHub.com Enterprise Cloud URLs without synthesizing invalid `api.github.com/enterprises/...` hosts. - Adds focused unit tests and README guidance for subscription login. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```console ruff check headroom/copilot_auth.py headroom/cli/copilot_auth.py headroom/cli/main.py headroom/cli/__init__.py headroom/cli/wrap.py tests/test_copilot_auth.py tests/test_cli/test_copilot_auth.py tests/test_cli/test_wrap_copilot.py tests/test_copilot_subscription_smoke.py # All checks passed! ruff format --check headroom/copilot_auth.py headroom/cli/copilot_auth.py headroom/cli/main.py headroom/cli/__init__.py headroom/cli/wrap.py tests/test_copilot_auth.py tests/test_cli/test_copilot_auth.py tests/test_cli/test_wrap_copilot.py tests/test_copilot_subscription_smoke.py # 9 files already formatted python -m py_compile headroom/copilot_auth.py headroom/cli/copilot_auth.py headroom/cli/main.py headroom/cli/__init__.py headroom/cli/wrap.py tests/test_copilot_auth.py tests/test_cli/test_copilot_auth.py tests/test_cli/test_wrap_copilot.py tests/test_copilot_subscription_smoke.py uv run --no-project --with pytest --with pytest-asyncio --with click --with rich --with opentelemetry-api --with pydantic --with tiktoken --with 'litellm==1.82.3' --with fastapi --with uvicorn --with 'httpx[http2]' --with openai --with mcp --with magika --with zstandard --with websockets --with onnxruntime --with transformers --with watchdog --with sqlite-vec pytest tests/test_copilot_auth.py tests/test_cli/test_copilot_auth.py tests/test_cli/test_wrap_copilot.py tests/test_cli_proxy_env.py tests/test_copilot_subscription_smoke.py # 127 passed ``` Local note: `uv run pytest ...` against the project currently fails before running tests because `uv.lock` has an unrelated `gitpython` wheel/version mismatch. ## Manual Validation I tested this with an existing GitHub Copilot Business subscription associated with a GitHub.com Enterprise Cloud account. The Enterprise Cloud value I tested was in the form: ```text github.com/enterprises/<enterprise> ``` The tested flow was: ```text headroom copilot-auth login headroom wrap copilot --subscription -- --model gpt-5.4 ``` This validated that Headroom does not treat github.com/enterprises/<enterprise> as a Copilot API hostname. Instead, token exchange uses GitHub.com and Headroom routes subscription-mode traffic to the Copilot API endpoint returned by GitHub for the signed-in account. I did not test this with GitHub Enterprise Server or a custom enterprise domain such as ghe.example.com. No tokens, request IDs, or organization-specific identifiers are included in this PR. ## Real Behavior Proof - Environment: macOS Darwin, Python 3.12.7, local checkout on `codex/copilot-business-auth`. - Exact command / steps: Ran `headroom copilot-auth login`, then launched `headroom wrap copilot --subscription -- --model gpt-5.4` with a GitHub Copilot Business subscription tied to a GitHub.com Enterprise Cloud account. - Observed result: Headroom did not treat `github.com/enterprises/<enterprise>` as a Copilot API hostname; token exchange used GitHub.com and subscription traffic was routed to the Copilot API endpoint returned for the signed-in account. The latest focused Copilot auth/proxy tests pass locally (`127 passed`). - Not tested: GitHub Enterprise Server or custom enterprise domains such as `ghe.example.com`; Windows Credential Manager integration still needs confirmation from someone on Windows. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing targeted unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes Acknowledgement: the OAuth/token-exchange behavior was informed by `anomalyco/opencode-copilot-auth` by Aiden Cline. No tokens are printed by the new login/status commands; only a short SHA-256 fingerprint is displayed for troubleshooting. The interactive login is included because the missing piece is not just an Enterprise URL or routing hint. For GitHub.com Enterprise Cloud accounts, URLs like `github.com/enterprises/acme` identify the enterprise account but are not Copilot API hostnames; token exchange still happens through GitHub.com and then returns the account-specific Copilot API endpoint. A command-line enterprise argument can help for true GitHub Enterprise Server/custom-domain deployments, but it cannot produce the Copilot OAuth token class that the token-exchange endpoint accepts. Ideally, Headroom would avoid an extra interactive login and reuse an existing GitHub/Copilot CLI session everywhere. In practice, some reusable-looking tokens can read Copilot account metadata but are rejected by Copilot token exchange, which leaves Business/Enterprise Cloud users with missing model catalogs. The explicit login command is the smallest independent way to obtain and persist the token needed for that exchange without asking users to pass a secret on the command line. --------- Co-authored-by: jbelanger <your-username@users.noreply.github.com> |
||
|
Yasser Sheikh
|
b08ec15b0d
|
fix(proxy): add native Bedrock converse-stream route (#917)
## Description
Adds native Bedrock `POST /model/{model_id}/converse-stream` routing in
`headroom-proxy` by reusing the existing streaming handler and
preserving route-specific upstream action forwarding.
This addresses a gap where native Bedrock streaming support existed for
`invoke-with-response-stream` but not `converse-stream`, even though
both share the same EventStream transport and SSE translation path in
this proxy.
Fixes #919
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring
## Changes Made
- Add route mount in `crates/headroom-proxy/src/proxy.rs`:
- `POST /model/:model_id/converse-stream` ->
`bedrock::invoke_streaming::handle_invoke_streaming`
- Update streaming handler URL construction in
`crates/headroom-proxy/src/bedrock/invoke_streaming.rs`:
- infer action from inbound path (`invoke-with-response-stream` or
`converse-stream`)
- build upstream URL with the resolved action
- return structured `400` for unsupported streaming action paths
- Add unit tests in
`crates/headroom-proxy/src/bedrock/invoke_streaming.rs`:
- action extraction coverage for both streaming paths
- upstream URL construction coverage for `converse-stream`
- Add integration coverage in
`crates/headroom-proxy/tests/integration_bedrock_streaming.rs`:
- `converse_stream_route_translates_to_sse`
- Add changelog entry under `Unreleased` bug fixes in `CHANGELOG.md`.
## Testing
- `cargo fmt --all`
- `cargo test -p headroom-proxy --test integration_bedrock_streaming --
--nocapture`
- `cargo test -p headroom-proxy --test integration_bedrock_metrics --
--nocapture`
## Real behavior proof
- **Setup tested on**
- macOS (darwin)
- Rust workspace local dev build
- `headroom-proxy` integration tests using wiremock upstream (no AWS
dependency)
- **Exact commands run after patch**
- `cargo test -p headroom-proxy --test integration_bedrock_streaming --
--nocapture`
- `cargo test -p headroom-proxy --test integration_bedrock_metrics --
--nocapture`
- **After-fix evidence + observed result**
- New integration test `converse_stream_route_translates_to_sse` passes.
- Streaming suite result: `10 passed; 0 failed`.
- Metrics suite result: `4 passed; 0 failed`.
- Logs show requests reaching `/model/.../converse-stream` and flowing
through Bedrock streaming path.
- **What I did not test**
- Live AWS Bedrock calls against real credentials/models.
- End-to-end CLI/runtime behavior outside Rust integration test harness.
|
||
|
Focused Instability
|
7d4ae86ec0
|
feat(parser): detect re-issued identical tool calls as reread waste (#909)
Fixes #908 ## Problem Reread waste detection matches `tool_result` blocks by exact `content_hash` only. Two gaps hide a common waste pattern — the agent re-issuing the *same tool call* and paying full price for a near-identical result: 1. **Byte-different results escape matching.** Same tool, same arguments, but the second result differs trivially (embedded mtimes, timestamps, ordering) → different hash, zero reread counted. 2. **Anthropic `tool_use` parts were dropped entirely** in `parse_message_to_blocks` — only OpenAI-style `message.tool_calls` produced `tool_call` blocks, so Anthropic/Strands traffic had no call-side record at all. ## Fix - Parse Anthropic `tool_use` / Strands `toolUse` content parts into `tool_call` blocks (same shape as the OpenAI path: `function_name`, `tool_call_id` flags). - Tag every `tool_call` block with a canonical `call_key` = hash(name + arguments re-serialized with sorted keys), so `'{"path": "a.py", "lines": 100}'` (OpenAI JSON string) and `{"lines": 100, "path": "a.py"}` (Anthropic dict) hash equal — covered by a cross-format parity test. - Second reread pass in `parse_messages` groups calls by `call_key`: repeat invocations past the existing `REREAD_ADJACENT_GAP` polling guard count their **result** tokens into `reread_tokens`, subject to the existing `REREAD_MIN_TOKENS` floor. Results already counted by the content-hash pass are skipped, so byte-identical repeats are never double-counted. No new `WasteSignals` field — a byte-different re-fetch of an identical call is reread waste by the existing definition. Detection is Python-only (`parser.py`); no Rust parity surface. ## Proof Re-reading the same file twice, 7 messages apart, second serve differing only by an mtime line: ``` main: tool_call blocks: 2, reread_tokens: 0 this branch: tool_call blocks: 2, reread_tokens: 381 ``` ## Testing - 11 new tests (`TestCallArgMatchReread`): changed-result repeat counted (OpenAI + Anthropic + Strands formats), byte-identical repeat counted exactly once, polling gap skipped, different args not matched, sub-floor results skipped, repeat without result ignored, canonical-key normalization, cross-format call_key parity. - Full `tests/test_parser.py` suite: 87 passed. Consumer regression sweep (reporting, config, request outcome, read lifecycle, observability, storage): 188 passed. - `ruff check` + `ruff format --check` + `mypy headroom/parser.py` clean. Co-authored-by: integration-check <integration@local> |
||
|
Focused Instability
|
0632eba6c3
|
fix(policy): correct warm-cache penalty in net_mutation_gain to (S + dT) (#903)
Fixes #906. ## What Part of #904 (net-cost policy completion tracking). Follows up #856 / #857 with the corrected gain term raised in [this #856 comment](https://github.com/chopratejas/headroom/issues/856#issuecomment-4679706939) — prerequisite for P2 (pipeline consumption), which would otherwise wire in a formula that is always-pro-mutation by exactly `P_alive·(w−r)·ΔT`. ## Why the corrected form is right With a live cache, the ΔT tokens a mutation removes are **already cache-written** — keeping them costs only reads (`ΔT·r·R`), so a mutation cannot avoid a fresh write of them. Blending alive (`ΔT·r·R − (w−r)·S`) and dead (`ΔT·(w + r·(R−1))`, no suffix penalty) cases over `P_alive`: ``` gain = ΔT·(w + r·(R−1)) − P_alive·(w−r)·(S + ΔT) ``` Three independent confirmations: 1. **Direct cost check** (w=1.25, r=0.1, warm, ΔT=50K, S=10K, R=2): keeping costs 60K·0.1·2 = 12,000 in reads; mutating costs 10K·1.25 (suffix rewrite, the first of the R touches) + 10K·0.1 (remaining read) = 13,500 — mutation loses 1,500, matching the corrected gain of −1,500. The old form said +56,000. 2. **The issue's own anchors**: corrected break-even is exactly `R = 11.5·S/ΔT` → 2K/50K = 287.5 (~290, as the issue says) and 50K/10K = 2.3 — the spec text's anchor numbers can only be derived from the corrected penalty. The implemented form gave 276 and *negative*. 3. **Internal consistency**: `break_even_reads` already shipped with the ~11.5·S/ΔT shape; this PR reconciles `net_mutation_gain` with it (and drops break_even's stray −1 term). ## Behavior changes (formula is still dead code — nothing consumes it yet) - 50K-shave/10K-suffix/R=3 golden: +61,000 → **+3,500** (tight win, consistent with 2.3-read break-even). - 2K-shave/50K-suffix/R=10 golden: −53,200 → **−55,500**. - S=0 boundary: an edit of already-cached content with no suffix is profitable whenever ≥1 read remains (`gain = ΔT·r·R`), and exactly 0 at R=0 warm. Not-yet-cached (live-zone) content should bypass the formula — now documented on both implementations. Rust + Python goldens updated in lockstep: 13 Rust + 19 Python tests green. ## Next (separate PRs) - **P2**: flag-gated consumption (`HEADROOM_NET_COST_POLICY=1`) with decision telemetry. - **P3**: batch deep edits (reclaim threshold), idle-timer compaction near TTL lapse. Co-authored-by: integration-check <integration@local> |
||
|
Copilot
|
96a7d7cbbe
|
Fix CI lint failure by formatting PR governance scripts (#933)
`CI / lint (pull_request)` failed because `ruff format --check` detected
formatting drift in the new PR governance script and its tests. This PR
aligns those files with repository formatting rules so the lint job can
pass.
- **Root cause**
- `ruff format --check .` reported two files as non-canonical:
- `scripts/pr-governance.py`
- `scripts/tests/test_pr_governance.py`
- **Change set**
- Applied `ruff` formatting to only the two flagged files.
- No behavioral or logic changes; edits are line-wrap/format
normalization only.
- **Representative update**
```python
parser.add_argument(
"--event", type=Path, required=True, help="Path to the GitHub event
payload JSON."
)
```
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
|
||
|
Focused Instability
|
b9e27614c6
|
fix(codex): compute waste signals on the OpenAI Responses path (#898)
## Problem Fixes #820. `headroom codex` traffic through `handle_openai_responses` never produced waste signals: the path compresses via CompressionUnits (not `TransformPipeline`, which is where waste detection lives), and the minimal `messages` list it synthesises only covers `instructions` + string-typed `input` — list-typed `input` (every real multi-turn Codex session) is dropped entirely. Tool output never reached `parse_messages`, so the dashboard "What Headroom Removed" stayed empty and the new `reread` signal (#853/#854) was blind for Codex. ## Fix (telemetry-only) 1. **`_responses_input_to_waste_messages(instructions, input_data)`** — converts a Responses payload to OpenAI-style messages for waste parsing only. Tool output items (`function_call_output`, `custom_tool_call_output`, `local_shell_call_output`, `apply_patch_call_output`) become `role="tool"` messages (with `tool_call_id`); `message` items keep their role and joined part text; string/part-list `output` and `content` both handled. 2. **`handle_openai_responses`** parses that list behind the same >100 saved-token gate `TransformPipeline.apply` uses, fail-open, and threads the result into the non-streaming `RequestOutcome` and the streaming branch. 3. **`_stream_response` / `_finalize_stream_response`** gain an optional `waste_signals` param passed through to `RequestOutcome.from_stream` (which already supported it). Default `None` — the other callers are unaffected. 4. `OPENAI_RESPONSES_OUTPUT_TYPES` now aliases the module-level frozenset the converter uses (single source; usage is membership-only, no behavior change). The existing `role="tool"` parsing from #815 handles the rest: tool_result blocks, waste flags, and reread grouping all apply. ## Tests `tests/test_codex_responses_waste_signals.py` — 13 new tests covering part-text extraction (string/part-list/non-text), conversion (roles preserved, all four output item types, tool_call_id, skipped unusable items, non-list input), and parsing (tool_result blocks + `json_bloat` from `function_call_output`; identical outputs far apart count as `reread`). Local regression sweep: responses compression units, codex routing/aliases/contract parity, responses bypass/compaction/T3-replay, request outcome, all streaming suites — 168 tests green. ## Live proof Mock `/v1/responses` upstream on a real port, proxy with `optimize=True`; list-typed `input` with a large `function_call_output` served twice (5 messages apart) plus compressible assistant bulk: ``` waste_signals: { "json_bloat": 20448, "reread": 8525, ... } PROOF OK: codex responses waste visible ``` ## Notes - Sibling of #897 (Gemini functionResponse waste signals) — same bug class from #813's matrix, independent code paths, no conflicts. - The WS Responses path (`handle_openai_responses_ws`) still computes no waste signals; left as a follow-up since its outcome plumbing differs. Co-authored-by: integration-check <integration@local> |
||
|
Focused Instability
|
9b0c840dd7
|
fix(gemini): surface functionResponse payloads to waste-signal detection (#897)
## Problem Fixes #819. Gemini `functionResponse` parts are preserved verbatim on the wire (by design — they are never compressed), but their payloads never reached `parse_messages`: `_gemini_contents_to_messages` only extracts `text` parts. Tool output — where most waste lives — contributed nothing to waste detection on either Gemini path, so `json_bloat`, `repetition`, and the new `reread` signal (#853/#854) were all blind to it. ## Fix (telemetry-only) 1. **`_gemini_contents_to_messages(..., include_function_responses=True)`** — new keyword-only flag. When set, each `functionResponse` payload is additionally emitted as a `role="tool"` message (dict payloads JSON-serialized, strings passed through, missing/`None` responses skipped). `preserved_indices` semantics are unchanged: the entries are still restored verbatim on the wire. 2. **`TransformPipeline.apply(..., waste_messages=...)`** — new optional kwarg (popped before transforms, like `record_metrics`). When provided, the waste-signal parse runs over this richer list instead of the transform input. Transforms, token accounting, and savings deltas are untouched — this is why the richer list is not simply fed to the pipeline: compressed copies of preserved entries are discarded on rebuild, which would corrupt savings reporting. 3. Both Gemini `generateContent` paths (native + Cloud Code Assist) build the enriched list and pass it through. The existing `role="tool"` parsing from #815 handles the rest: tool_result blocks, waste flags, and reread grouping all apply. ## Tests `tests/test_gemini_function_response_waste.py` — 11 new tests: - conversion: default unchanged (regression), dict/string payloads, missing response skipped, text-before-tool ordering, preserved_indices unchanged, circular-reference fallback - parsing: functionResponse payload produces tool_result blocks + `json_bloat`; identical payloads far apart count as `reread` - pipeline: `waste_messages` overrides the waste source, does not affect transform output/token counts, falls back to transform input when absent Full local sweep of touched suites: gemini multimodal, parser, safety rails, canonical pipeline — green. The 13 failures in `test_proxy_gemini_*_integration.py` are credential-dependent and identical on clean `main`. ## Live proof Mock Gemini upstream on a real port, proxy with `optimize=True`; conversation with a large functionResponse payload served twice (5 messages apart) plus compressible model text: ``` waste_signals: { "json_bloat": 35003, "reread": 11673, ... } PROOF OK: waste visible, wire verbatim ``` Upstream received both `functionResponse` entries byte-identical to the client request. ## Known limitations / follow-ups - The Cloud Code Assist path passes `waste_messages` but does not yet consume `result.waste_signals` into a recorded outcome (pre-existing gap; the native path records it). - Requests where **all** content entries are preserved (pure functionResponse/media conversations) early-exit before the pipeline and still produce no waste signals. - Codex/Responses-API counterpart is #820 (separate PR). Co-authored-by: integration-check <integration@local> |
||
|
gglucass
|
8c00f7103c
|
fix(codex): poll /wham/usage for subscription limits (handshake no longer sends x-codex-* headers) (#924)
## Description Codex's subscription usage window (primary/secondary rate-limit gauges) stopped populating for ChatGPT-OAuth sessions. This PR restores it by polling Codex's dedicated usage endpoint instead of relying on response headers that are no longer sent. ### Why the previous approach no longer works The existing code populates `CodexRateLimitState` from `x-codex-*` rate-limit headers captured on the `/v1/responses` WebSocket handshake (`update_from_headers` at WS accept). That worked when OpenAI returned `x-codex-primary-used-percent`, `x-codex-primary-window-minutes`, etc. on the handshake response. OpenAI has since stopped sending those headers on the ChatGPT WebSocket handshake. I confirmed this by faithfully replaying a real Plus-account handshake (both `prewarm` and regular `request_kind`): no `x-codex-*` headers come back on either. This matches OpenAI's own move to a dedicated usage endpoint (`GET /backend-api/codex/usage` in CodexApi mode) and reports such as openai/codex#14728. So `update_from_headers` now runs on every accept but finds nothing to parse, and the window silently stays empty. The headers aren't coming back, so there is nothing to fix in the parsing path. The data now lives behind a request we have to make ourselves. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `subscription/codex_rate_limits.py`: - `parse_codex_usage_payload()` / `update_from_usage_payload()` — map the `GET /backend-api/wham/usage` JSON (`rate_limit.primary_window` / `secondary_window` with `used_percent`, `limit_window_seconds`, `reset_at`; `credits`; `rate_limit_reached_type`) into the existing `CodexRateLimitState`. `limit_window_seconds` is converted to window-minutes with the same round-up codex-rs uses (`(secs + 59) // 60`). - `maybe_schedule_usage_poll()` — fire-and-forget, throttled to one request per 60s, scoped to ChatGPT sessions (requires both a Bearer token and `ChatGPT-Account-Id`; API-key traffic is skipped). Uses an in-flight guard so concurrent accepts don't stack polls. Endpoint is overridable via `HEADROOM_CODEX_USAGE_URL`. - `proxy/handlers/openai.py`: - At the Codex WS accept site, after the now-usually-empty `update_from_headers` block, schedule the usage poll. Wrapped in `contextlib.suppress` and fully non-blocking so it can never delay or fail the WebSocket accept. The old header-capture path is intentionally left in place as a no-cost fallback in case OpenAI restores the headers. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed (live `/wham/usage` replay against a Plus account returned HTTP 200 with the expected schema; payload fixture in tests mirrors that real shape) ## Test Output ``` $ uv run pytest tests/test_codex_rate_limits.py -q ........................................ [100%] 41 passed in 0.16s $ uv run ruff check headroom/subscription/codex_rate_limits.py headroom/proxy/handlers/openai.py tests/test_codex_rate_limits.py All checks passed! $ uv run mypy headroom/subscription/codex_rate_limits.py Success: no issues found in 1 source file ``` ## Additional Notes - New tests cover: full-payload mapping, window-minutes round-up, credits balance kept only when `has_credits`, promo object vs string, empty payload returns `None`, missing `used_percent` skipped, header-gating (requires Bearer + account-id), poll throttling, and no-event-loop safety. - Scoping to `ChatGPT-Account-Id` keeps the poll off API-key traffic, and the 60s throttle plus in-flight guard bound it to at most one lightweight GET per minute per running proxy. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Focused Instability
|
7ced77b6e7
|
docs: fix dead contact links in issue templates and troubleshooting guide (#910)
## Summary The side note in #855 reports that the **Issues → Question** contact link points to a non-existing page. Confirmed, plus two more dead links of the same class: - `.github/ISSUE_TEMPLATE/config.yml` — "Questions & Discussions" points at `github.com/headroom-sdk/headroom/discussions` (the `headroom-sdk` org 404s); now points at this repo's Discussions (live, Discussions are enabled here). - `.github/ISSUE_TEMPLATE/config.yml` — "Documentation" points at `headroom.dev/docs` (404); now points at the repo homepage docs site `headroom-docs.vercel.app/docs` (200). - `docs/content/docs/troubleshooting.mdx` — "File an issue at github.com/headroom-sdk/headroom" same dead org; now points at this repo. `.github/FUNDING.yml` also references `headroom-sdk` but that's a sponsorship target choice, so left untouched. ## Testing Link targets verified by HTTP status: old URLs return 404, new URLs return 200. Docs-only change, no code paths affected. Fixes the side note in #855. Co-authored-by: integration-check <integration@local> |
||
|
Michael Sam
|
6d3f39f213
|
feat: add dashboard agent usage stats (#814)
## Description Add a clear dashboard view for per-agent token usage so end users can see Cursor, Claude, Codex, and other detected clients with before/after token counts, tokens saved, and savings percentages. The stats API now exposes a stable `agent_usage` object that the dashboard renders near the top of the session view. Fixes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made ### New Files **Tests:** - `tests/test_dashboard_agent_usage.py` — Covers agent classification, exact per-request aggregation, and aggregate fallback behavior. ### Modified Files - `headroom/proxy/server.py` — Adds per-agent usage aggregation to `/stats` with before tokens, after tokens, output tokens, saved tokens, savings percentage, source, providers, and models. - `headroom/dashboard/templates/dashboard.html` — Adds a prominent Agent Usage panel with totals, coverage status, per-agent token-flow bars, request counts, before/after tokens, saved tokens, and share of savings. ## Testing - [x] Unit tests pass: `.venv312/bin/pytest tests/test_dashboard_agent_usage.py` - [x] Linting passes: `.venv312/bin/ruff check headroom/proxy/server.py tests/test_dashboard_agent_usage.py` - [x] Diff whitespace check passes: `git diff --check origin/main...HEAD` - [x] Dashboard smoke render: local proxy on `127.0.0.1:8790`, captured Chrome headless screenshot of `/dashboard` - [x] New tests added for new functionality ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing relevant unit tests pass locally with my changes - [ ] I have made corresponding changes to the documentation - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes The agent usage panel uses exact request-log data when available. If detailed request logs are empty, it falls back to aggregate provider/model request counts and labels the coverage as aggregate fallback so users are not misled. |
||
|
Logan Kang
|
dff6a19946
|
fix(codex): write canonical hooks feature flag and migrate deprecated codex_hooks (#743)
## Description `headroom init codex` writes the hooks feature flag into `.codex/config.toml` under the key `codex_hooks`. Codex renamed the canonical key to `hooks` and kept `codex_hooks` as a legacy alias (openai/codex#20522). Current Codex builds warn about `[features].codex_hooks` and tell users to use `[features].hooks` instead, so configs written by headroom should stop emitting the deprecated key. This PR switches headroom to write the canonical `hooks` key and **migrates existing configs in place**. The migration is the tricky part: a config can already contain `codex_hooks`, `hooks`, or both, in any order, inside or outside headroom's marker block — and a naive replace can emit a *duplicate* `hooks` key, which is invalid TOML that Codex rejects outright. The fix strips every `codex_hooks` line up front (any value, anywhere) — mirroring the existing top-level key cleanup in `_ensure_codex_provider` (#260) — then guarantees `hooks` is present without ever duplicating it, and respects a user-managed `hooks` value that lives outside our marker block. Fixes: N/A (no tracking issue — surfaced while aligning with Codex >= 0.129; related upstream context: openai/codex#20522 and the warning behavior discussed in openai/codex#22148) ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Write the canonical `hooks` key instead of the deprecated `codex_hooks` in `_ensure_codex_feature_flag` (`headroom/cli/init.py`). - Strip any `codex_hooks` line (any value, inside or outside the marker block) before ensuring the flag, so re-running `init` migrates a legacy config instead of leaving a stale key or producing a duplicate `hooks` key (invalid TOML). - Respect a user-managed `hooks` value found outside headroom's marker block (e.g. `hooks = false`); only the deprecated alias is removed. - Make the insert/create paths match `_replace_marker_block`'s normalisation so re-running `init` is byte-idempotent. - Extract a `_codex_feature_block()` helper to remove the 4x duplicated marker block assembly. - Add regression tests for the previously-broken edge cases. ## Testing - [x] Unit tests pass (`pytest`) — affected module fully green (see output) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom/cli/init.py`) - [x] New tests added for new functionality - [x] Manual testing performed (reproduced each edge case against the patched function via `tomllib.loads`) ## Test Output ``` $ pytest -v tests/test_cli/test_init_cli.py -k "feature_flag or hooks_feature" tests/test_cli/test_init_cli.py::test_init_codex_merges_feature_flag_into_existing_table PASSED tests/test_cli/test_init_cli.py::test_init_codex_creates_hooks_feature_flag_on_first_init PASSED tests/test_cli/test_init_cli.py::test_ensure_codex_feature_flag_replaces_existing_marker PASSED tests/test_cli/test_init_cli.py::test_ensure_codex_feature_flag_migrates_legacy_codex_hooks_key PASSED tests/test_cli/test_init_cli.py::test_ensure_codex_feature_flag_migrates_when_both_keys_present PASSED tests/test_cli/test_init_cli.py::test_ensure_codex_feature_flag_migrates_when_keys_reversed PASSED tests/test_cli/test_init_cli.py::test_ensure_codex_feature_flag_drops_legacy_key_outside_marker PASSED tests/test_cli/test_init_cli.py::test_ensure_codex_feature_flag_is_idempotent PASSED tests/test_cli/test_init_cli.py::test_ensure_codex_feature_flag_creates_features_section_when_missing PASSED ======================= 9 passed, 45 deselected in 0.24s ======================= $ pytest -q tests/test_cli/test_init_cli.py 54 passed $ ruff check . All checks passed! $ mypy headroom/cli/init.py Success: no issues found in 1 source file ``` Note: the broader `tests/test_cli/` run has one unrelated failure (`test_wrap_copilot_auto_detects_running_proxy_backend`) caused by a real proxy already bound to port 8787 in the local environment — it fails identically on a clean checkout without this change. ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (none required) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable (managed by release-please; generated from the conventional commit, not edited by hand) ## Screenshots (if applicable) N/A ## Additional Notes - **Why the duplicate-key path matters:** TOML forbids duplicate keys, so a `[features]` table containing both `codex_hooks` and `hooks` (which the old in-place migration could produce) makes Codex reject `config.toml` entirely. The new "strip then ensure" approach can never emit two `hooks` lines. - **Version provenance:** the `codex_hooks` -> `hooks` rename landed in openai/codex#20522, first shipped in Codex `rust-v0.129.0`. `codex_hooks` remains a working legacy alias, but current Codex builds can warn users to move to `[features].hooks`. - **Idempotency:** running `headroom init codex` repeatedly now produces a byte-stable `config.toml`, so there is no churn on re-init. |
||
|
Focused Instability
|
b7350aa29c
|
ci: run dashboard playwright tests in a dedicated job (#921)
## Summary Closes #920. Follow-up noted in #915. The dashboard Playwright tests guard on `pytest.importorskip("playwright...")` and no CI job installs playwright, so they have skipped on every CI run since they were added — which is how the bitrot fixed in #915 went unnoticed. This adds a `test-dashboard-ui` job to `ci.yml`, same shape as `test-agno`: - installs the prebuilt wheel `[dev]` + playwright, then `playwright install --with-deps chromium` - runs `pytest tests/test_dashboard_*_playwright.py` — the stub-based tests only (all routes mocked via `page.route`, no network); the glob also picks up the CVC panel tests from #913 once that merges - sets `HEADROOM_PLAYWRIGHT_ARTIFACT_DIR` and uploads the captured dashboard screenshots as a workflow artifact (7-day retention), so every CI run leaves a visual record of the rendered dashboard Deliberately excluded: `tests/test_dashboard/test_live_feed.py` — it navigates to a live proxy on `localhost:8787` and would fail on a runner with nothing listening. The main test shards keep skipping playwright tests (playwright stays uninstalled there), so nothing double-runs. ## Testing - `yaml.safe_load` parses the workflow; the `workflow-validation` CI job (actionlint + act) runs on this PR since it touches `ci.yml` - The test this job will run passes locally: `tests/test_dashboard_cache_ttl_playwright.py` — 1 passed (chromium) - This PR's own CI run exercises the new job end-to-end |
||
|
github-actions[bot]
|
8a53c8ec3b
|
chore: release main (#891)
🤖 I have created a release *beep* *boop* --- <details><summary>0.25.0</summary> ## [0.25.0](https://github.com/chopratejas/headroom/compare/v0.24.0...v0.25.0) (2026-06-12) ### Features * add differential network capture harness ([#761](https://github.com/chopratejas/headroom/issues/761)) ([ |
||
|
Copilot
|
b716c8c2ee
|
fix(ci): correct comments, timeouts, and pip reliability in native e2e workflows (#878)
Review feedback on PR #837 identified several issues in the newly added `wrap-native-e2e.yml` and `install-native-e2e.yml` workflows. ## Changes **`wrap-native-e2e.yml`** - Header comment claimed "linux / macos / windows" coverage — Windows is matrix-excluded; updated to reflect actual runners and note Windows is pending CRT fix - Removed Windows-specific wording ("Windows path handling") from the workflow description; made OS-agnostic - `timeout-minutes`: `15` → `25` to match `init-native-e2e.yml` and avoid maturin build flakes on macOS **Both `wrap-native-e2e.yml` and `install-native-e2e.yml`** - pip install made more resilient on macOS runners, matching the pattern already used in `ci.yml`: ```yaml - name: Install pytest shell: bash run: | python -m pip install --upgrade pip python -m pip install --retries 10 --timeout 60 pytest pytest-cov ``` - `timeout-minutes`: `15` → `25` in `install-native-e2e.yml` for the same reason --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
e408012c2b
|
ci: bump dtolnay/rust-toolchain from 1.95.0 to 1.100.0 in the actions-minor-patch group (#849)
Bumps the actions-minor-patch group with 1 update:
[dtolnay/rust-toolchain](https://github.com/dtolnay/rust-toolchain).
Updates `dtolnay/rust-toolchain` from 1.95.0 to 1.100.0
<details>
<summary>Commits</summary>
<ul>
<li><a
href="
|
||
|
Devanshi Vyas
|
05bd56bcb6
|
fix(wrap): track shared proxy clients with markers (#877)
## Description Replace argv-based proxy client detection with per-port wrap client markers so cleanup and ephemeral restarts do not tear down a shared proxy while another wrapped session is still attached. Also prune stale markers, guard against PID reuse when process identity is available, and add coverage for the marker-based lifecycle behavior. Fixes #804 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Testing Describe the tests you ran to verify your changes: - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
gglucass
|
2269e40bde
|
feat(proxy): log compressed messages alongside original request (#261)
## Description Expose the post-compression message list that was actually sent upstream as a new `compressed_messages` field on `RequestLog`, paired with the existing (now consistently pre-compression) `request_messages`. Consumers of `/transformations/feed` — dashboards and any downstream observability — can now diff the two sides of a compression to see exactly what the pipeline stripped, replaced, or kept. Turns an abstract "saved N tokens" into a legible before/after. Gated by the same `log_full_messages` flag as `request_messages` so the two sides stay in sync; it's pointless to store one without the other. Also fixes a latent correctness bug: today's `request_messages` field is inconsistent across the four `RequestLog` construction sites — sometimes it's the pre-compression snapshot, sometimes it's the mutated `body["messages"]` (which is the compressed list, because the proxy mutates `body` in place before the log call). After this change, `request_messages` always means pre-compression and `compressed_messages` always means what went upstream. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) Note on "breaking": strictly speaking this is a semantic correction of an inconsistently-populated field, not a schema break. The field name `request_messages` is unchanged and the JSON shape is unchanged; what changes is that the field now consistently holds the pre-compression list. Consumers that treated it as "whatever messages we have" continue to work. Consumers that depended on the accidental post-compression value (if any existed) would shift to `compressed_messages`. ## Changes Made - **`headroom/proxy/models.py`**: `RequestLog` gains `compressed_messages: list[dict] | None = None`. Doc comment explains it's paired with `request_messages` and gated by the same `log_full_messages` flag. - **`headroom/proxy/handlers/anthropic.py`** (2 sites — Bedrock non-streaming and main non-streaming): `request_messages` now consistently sources from `original_messages` (the pre-compression snapshot at line 724), `compressed_messages` sources from `body["messages"]` (the compressed list after in-place mutation at line 1189). Both gated symmetrically. - **`headroom/proxy/handlers/streaming.py`** (2 sites — main streaming in `_finalize_stream_response`, Bedrock streaming in `_stream_response_bedrock`): same treatment. `_stream_response_bedrock` gains a new `original_messages: list[dict] | None = None` parameter so it has access to the pre-compression snapshot; the sole caller in `anthropic.py` now threads it through. - **`headroom/proxy/server.py`**: `/transformations/feed` adds `compressed_messages` to the JSON payload alongside the existing `request_messages` / `response_content`. *Split into a separate preceding commit is a one-time EOL normalization to LF — the file blob in history carries CRLF but `.gitattributes` declares `*.py text eol=lf`, so any contributor editing `server.py` triggers the same whole-file renormalization. Separating the two commits keeps this feature commit's diff at a single line.* - **`headroom/proxy/request_logger.py`**: `compressed_messages` is stripped from the JSONL file log and from `get_recent()` alongside the existing `request_messages` / `response_content` stripping. `get_memory_stats()` also counts it toward the deque's byte budget. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .` and `ruff format --check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed (via the Headroom Desktop client that consumes `/transformations/feed` — confirmed both fields arrive and render) Test coverage added/extended: - `tests/test_proxy/test_request_logger.py` (new file): round-trip unit tests for `RequestLogger`. Confirms `get_recent` strips both sides (pre + post), `get_recent_with_messages` exposes both, and the JSONL file log drops both when `log_full_messages=False`. - `tests/test_proxy/test_transformations_feed.py`: extended to assert `compressed_messages` appears in the endpoint payload alongside `request_messages` / `response_content`. - `tests/test_proxy_streaming_request_logger.py`: existing include/omit tests updated to assert both sides populate when the flag is on and both are `None` when it's off. ## Test Output ``` $ uv run ruff check headroom tests All checks passed! $ uv run ruff format --check headroom tests 614 files already formatted $ uv run pytest tests/test_proxy/test_request_logger.py tests/test_proxy_streaming_request_logger.py tests/test_proxy/test_transformations_feed.py -v ... tests/test_proxy/test_request_logger.py::test_get_recent_strips_compressed_messages_alongside_request_and_response PASSED tests/test_proxy/test_request_logger.py::test_get_recent_with_messages_returns_compressed_messages PASSED tests/test_proxy/test_request_logger.py::test_jsonl_file_strips_both_sides_when_log_full_messages_disabled PASSED tests/test_proxy_streaming_request_logger.py::test_finalize_stream_response_logs_original_and_compressed_messages PASSED tests/test_proxy_streaming_request_logger.py::test_finalize_stream_response_omits_messages_when_log_full_messages_disabled PASSED tests/test_proxy/test_transformations_feed.py::test_transformations_feed_returns_messages PASSED ... ============================== 11 passed in 5.36s ============================== ``` ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas (the two-sided gating at each log site, the `_stream_response_bedrock` parameter addition, and the `get_memory_stats` accounting) - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes ### Non-Anthropic backends `handlers/openai.py` and `handlers/gemini.py` do not currently emit `RequestLog` entries at all — only Anthropic and the shared streaming paths do. This PR therefore only populates `compressed_messages` on Anthropic traffic (which is what `/transformations/feed` shows today). Wiring OpenAI and Gemini into `RequestLogger` end-to-end is a separate, larger gap worth its own PR. ### `server.py` EOL normalization The feature change in `server.py` is a single line. To keep the diff readable, the preceding commit is a whitespace-only `chore(proxy): normalize server.py to LF per .gitattributes` — the file blob was stored with CRLF terminators but `.gitattributes` declares `*.py text eol=lf`. Any contributor touching `server.py` triggers this renormalization; isolating it here keeps the feature commit reviewable. Happy to rebase / drop / reshape as preferred. ### Downstream desktop compatibility The Headroom Desktop client I work on now consumes `compressed_messages` and renders the pre/post pair side-by-side on the "Recent large compression" card. The desktop was updated to handle both shapes: proxies without the field render the legacy single "Request" block; proxies with the field render "Request (original, N tokens)" + "Request (compressed, M tokens)" where N/M come from `input_tokens_original` / `input_tokens_optimized`. No changes needed downstream if this PR lands. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Chris Yau
|
b4395993ae
|
fix(init): suppress hook recovery output (#760)
## Summary - silence best-effort profile recovery while `headroom init hook ensure` runs from installed hooks - suppress both Python-level stdout/stderr and child process file-descriptor output so SessionStart hooks do not emit invalid JSON - add a regression test for noisy supervisor recovery failures ## Verification - `python3 -m py_compile headroom/cli/init.py` - live local hook probe: `headroom init hook ensure --profile default --marker headroom-init-codex` exits 0 with empty output - targeted pytest was not runnable locally because `uv.lock` currently fails to parse due to an inconsistent GitPython wheel version entry |
||
|
Patrick A
|
d76a7296df
|
fix(proxy): make CCR multi-worker warning conditional on backend (#770)
## Problem The multi-worker startup warning always mentioned CCR retrieval failures, even when the operator had already configured a cross-worker backend via `HEADROOM_CCR_BACKEND`. That's noise — if they've set `HEADROOM_CCR_BACKEND=sqlite` or `redis`, CCR fragmentation is already resolved. This was surfaced during review of #628 (now closed): the reviewer correctly noted that Python `CompressionStore` defaults to `InMemoryBackend`, which is per-process — each uvicorn worker has its own singleton, so CCR markers written on worker A are invisible to worker B unless a shared backend is configured. ## Changes ### `headroom/proxy/server.py` The `workers > 1` warning is now conditional on `HEADROOM_CCR_BACKEND`: - **Backend unset (default `InMemoryBackend`, per-process):** warning includes CCR retrieval failures and suggests `HEADROOM_CCR_BACKEND=sqlite` to use a shared cross-worker store. - **Backend configured (`sqlite`/`redis`):** warning covers only the remaining per-worker stores (compression cache, prefix tracker, TOIN, CostTracker) — CCR fragmentation is already resolved. ### `RUST_DEV.md` Updated the multi-worker fragmentation section: - Removed the incorrect parenthetical claiming this only applies when the operator *explicitly* chooses `CcrBackendConfig::InMemory` (Python defaults to InMemory) - Added Python `CompressionStore` as item 1 in the fragmented-state list, with a note that setting `HEADROOM_CCR_BACKEND=sqlite` resolves it - Restored TOIN to the fragmented list with a note that its file-backed snapshots do not make it coherently shared across workers - Updated "Detecting it in the wild" to document the conditional warning behaviour ## Files changed | File | Change | |---|---| | `headroom/proxy/server.py` | Conditional two-branch warning based on `HEADROOM_CCR_BACKEND` | | `RUST_DEV.md` | Accurate per-process description of Python `CompressionStore`; restored TOIN | | `CHANGELOG.md` | Entry under `[Unreleased]` | Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
Michael Sam
|
d2cdab268d
|
feat(proxy): add agent-90 savings profile (#830)
## Summary - add an `agent-90` savings profile with cross-agent proxy env exports - wire the profile into proxy/router runtime kwargs, including force-Kompress routing and a smaller read-protection window - expose effective savings-profile config in `/stats` and add focused regression coverage ## Type of change - [x] feat (non-breaking change which adds functionality) - [ ] fix (non-breaking change which fixes an issue) - [ ] docs - [ ] test/CI-only - [ ] refactor-only ## Testing - [x] `python3 -m py_compile headroom/agent_savings.py headroom/cli/agent_savings.py headroom/cli/main.py headroom/cli/proxy.py headroom/proxy/models.py headroom/proxy/server.py headroom/transforms/content_router.py tests/test_agent_savings.py tests/test_proxy_healthchecks.py tests/test_cli/test_wrap_persistent.py tests/test_transforms/test_content_router.py` - [x] `git diff --check` - [x] manual smoke: `agent-savings --profile agent-90 --format json` returns `HEADROOM_TARGET_RATIO=0.10` - [x] manual smoke: `proxy_pipeline_kwargs(ProxyConfig(savings_profile="agent-90"))` enables `force_kompress`, system/user compression, and `read_protection_window=2` - [x] manual smoke: Anthropic-style `tool_result` routes through Kompress with `target_ratio=0.10` - [ ] `pytest` suite not run: pytest is not installed in the available local Python environments ## Notes This keeps agent-90 as an opt-in profile. Existing defaults remain unchanged unless `HEADROOM_SAVINGS_PROFILE=agent-90` or `ProxyConfig(savings_profile="agent-90")` is set. |
||
|
jimu
|
058bcedab8
|
feat(plugins): Hermes agent headroom_retrieve plugin (#824)
## Summary Implements the Hermes-side retrieval plugin proposed in #796 (as invited — thanks for the quick response!). When Hermes routes traffic through `headroom proxy`, compressed markers are a one-way street: Hermes registers its own tools, so it never gets the `headroom_retrieve` CCR tool that Claude Code receives via MCP injection. In practice the model either re-runs the original command or — observed in the wild — treats `ccr:abc123` as a file path and tries to `cat` it. This plugin uses Hermes's user-plugin system (`~/.hermes/plugins/`) to register a native `headroom_retrieve` tool that calls the proxy's `POST /v1/retrieve` endpoint. ## What's included - `plugins/hermes/headroom_retrieve/` — `plugin.yaml` + `__init__.py` (single-file, httpx, ~100 lines) - `plugins/hermes/README.md` — install steps and proxy-side recommendations ## Design notes - **Both marker formats covered**: Kompress emits `[N items compressed ... hash=KEY]`, SmartCrusher's opaque-blob walker emits `<<ccr:HASH[,KIND,SIZE]>>`. The tool description teaches both and explicitly says markers are NOT file paths; the handler normalizes whole-marker input (`<<ccr:abc,base64,4.5KB>>` → `abc`). - **Re-compression loop guard**: retrieved originals travel back through the proxy on the next request and get re-compressed into a fresh marker, looping forever. README documents `HEADROOM_EXCLUDE_TOOLS=read_file,headroom_retrieve` as the fix (Hermes tool names don't match `DEFAULT_EXCLUDE_TOOLS`, which targets Claude Code's `Read`/`Grep`/...). - **Actionable failure modes**: 404 (TTL expired / proxy restarted) and connection-refused both return guidance to re-run the original command rather than retry. ## Relationship to existing PRs Complementary to #707 / #556 (`headroom wrap hermes`, proxy-side): those launch/route Hermes through the proxy; this gives the agent the retrieval capability once it's routed. Notably #707 disables CCR tool injection in Hermes mode precisely because Hermes must register its own tool — this plugin is that registration. ## Testing Running in production on macOS (headroom 0.23.0, pipx) and Linux (0.22.4, systemd) for a day. Verified: fresh-marker retrieval roundtrip, whole-marker hash normalization (6 input shapes), expired-hash 404 messaging, proxy-down messaging, and end-to-end via live Hermes sessions (fresh ≥500B `read_file` returns original with the documented exclude config). Closes #796 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: akb4q <zhunyunjiang@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Devanshi Vyas
|
249af6cc7b
|
fix(ccr): use shared compression store (#875)
## Description Use shared get_compression_store() singleton in MCP _get_local_store so headroom_retrieve sees proxy-compressed content. Fixes #860 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Testing Describe the tests you ran to verify your changes: - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality |
||
|
Federico Rao
|
f9384ef4b7
|
fix: suppress LiteLLM provider banner before import (#874)
## Summary - set `LITELLM_SUPPRESS_DEBUG_INFO` before importing `litellm` in the LiteLLM provider - keep the existing post-import suppression flags as a fallback - add a regression test that verifies the env flag exists before `litellm` import Fixes #613 ## Tests - `uv run pytest tests/test_startup_log_noise.py::TestLiteLLMLogSuppression -q` - `uv run ruff check headroom/providers/litellm.py tests/test_startup_log_noise.py` - `python3 -m py_compile headroom/providers/litellm.py tests/test_startup_log_noise.py` |
||
|
Focused Instability
|
bfcb07d78e
|
fix(ccr): key Rust search/diff/log markers with explicit_hash (#852)
Fixes #816 ## What The three `_persist_to_python_ccr` shims (`search_compressor.py`, `diff_compressor.py`, `log_compressor.py`) called `store.store(original, compressed)` with the default key — `SHA-256(original)[:24]` since PR #395 — while the Rust side embeds `MD5(original)[:24]` in the emitted `Retrieve more: hash=...` marker. Marker key and storage key never matched, so **every retrieval of a Rust search/diff/log marker returned "Entry not found or expired"** (inside any TTL — the symptom class reported in #714). Fix is exactly what #816 proposed: pass the marker's key via `explicit_hash=cache_key` at all three call sites, the same contract SmartCrusher has used since PR #395. No store changes needed — `store()` already validates and honors `explicit_hash`. Also corrected the stale comment in `search_compressor.py` that still claimed "both use MD5(original)[:24]". ## Tests `tests/test_ccr_rust_marker_hash_bridge.py` (companion to `test_ccr_row_drop_store_bridge.py`, which pinned the same bug class for SmartCrusher in #389): for each shim, the store entry must be retrievable under the Rust marker key AND absent under the SHA-256 default key. Verified red→green: all 3 tests fail on main with the exact issue symptom ("store has no entry under the Rust marker key ...; the marker dangles") and pass with the fix. `test_ccr_row_drop_store_bridge.py` still green. `ruff check` + `ruff format --check` clean. --------- Co-authored-by: Ash Rhodes <ashley.rhodes@king.com> |
||
|
Focused Instability
|
5f1d88ad27
|
feat: detect re-served tool results as over-compression waste signal (#854)
Closes #853 ## What Adds a `reread` waste signal: identical `tool_result` content appearing at more than one message position means the agent re-fetched something already in context — the dominant failure signature of over-compression (Manus context-engineering; JetBrains "Complexity Trap", arXiv:2508.21433). Per-request savings can't see this cost; this signal makes it visible. - `WasteSignals.reread_tokens` — new field, in `total()`, exported as `"reread"` in `to_dict()`. - `parse_messages()` groups `tool_result` blocks by their **existing** `content_hash` and counts every repeat beyond the first serve. No new hashing or tokenization; one O(blocks) dict pass. - `REREAD_MIN_TOKENS = 50` guard: short outputs ("ok", empty diffs) legitimately repeat and are skipped. Duplicates within a single message (same `source_index`) are not counted. - Works across all formats the parser already normalizes to `tool_result` blocks: OpenAI `role=tool`, Anthropic `tool_result`, Strands/Bedrock `toolResult` (#813/#815). - Flows through existing generic plumbing with zero handler changes: pipeline → `RequestOutcome.waste_signals` → Prometheus `headroom_waste_signal_tokens_total{signal="reread"}` → dashboard "Waste Detected" panel. Dashboard gains label/color entries for the new key. ## Tests 7 new tests in `tests/test_parser.py::TestRereadDetection` (red before, green after): OpenAI + Anthropic format detection, repeat-counting semantics (first serve free), single-occurrence, short-duplicate guard, same-message guard, `total()`/`to_dict()` participation. Updated 2 exact-shape assertions in `tests/test_config.py`. Local runs: `tests/test_parser.py` (72 passed), `tests/test_config.py` + outcome/reporting/observability/storage/proxy-hooks suites (190 passed), `tests/test_canonical_pipeline.py` + `tests/test_proxy_pipeline_lifecycle.py` (11 passed). `ruff check` + `ruff format --check` clean. ## Real behavior proof **Setup:** macOS (Darwin 25.5), Python 3.11.9, this branch, real proxy server (`python -m headroom.proxy.server --port 18970 --anthropic-api-url http://127.0.0.1:18971`) with a local mock Anthropic upstream returning a canned `/v1/messages` response (no real key needed). **Steps:** POSTed an Anthropic-format conversation to the live proxy: agent fetches a 14 KB JSON log array via `get_logs` tool, then fetches the identical content again under a different `tool_use_id` (the re-read). **Observed result** — `curl http://127.0.0.1:18970/metrics` after the request: ``` # HELP headroom_waste_signal_tokens_total Tokens attributed to detected waste signals # TYPE headroom_waste_signal_tokens_total counter headroom_waste_signal_tokens_total{signal="json_bloat"} 9858 headroom_waste_signal_tokens_total{signal="reread"} 4935 ``` `reread` = 4935 tokens, exactly the second serve of the ~4.9k-token tool result (json_bloat counts both occurrences ≈ 2×). `/stats` shows the same: `"waste_signals": {"json_bloat": 9858, "reread": 4935, ...}` — which is what the dashboard panel renders. Also verified the negative path live: a conversation whose tool results contain non-compressible plain code text produced no waste-signal entries (the pipeline only attributes waste when compression actually engaged, unchanged behavior). **Not tested:** Gemini `functionResponse` path (parser doesn't produce `tool_result` blocks for it — pre-existing gap tracked in #819); dashboard rendering only verified via the `/stats` payload the panel binds to, not a browser screenshot. ## Out of scope (per #853) Tool-call argument matching, compression-marker attribution, tokens-per-task metric, cache hit-rate panel. --------- Co-authored-by: Ash Rhodes <ashley.rhodes@king.com> |
||
|
Focused Instability
|
d5f58026e2
|
feat: net-cost cache mutation formula on CompressionPolicy (#856 P1) (#857)
Closes #856 **P1 of the #856 phased plan** — pure functions, zero behavior change. (Closing keyword links the issue; if P2 hasn't started when this merges, reopen #856 or it remains the design record for the P2/P3 follow-up PRs.) ## What Adds the break-even decision rule for deep (pre-cache-marker) edits to `CompressionPolicy`: ``` gain = ΔT · (w + r·(R−1)) − P_alive · (w − r) · S ``` - `net_mutation_gain()`, `should_mutate_deep()` (gain > 0), `break_even_reads()` (R = ((w−r)/r)·(S/ΔT−1) ≈ 11.5·S/ΔT) on the Rust struct (source of truth) and the Python hand-mirror, following the existing F2.1/F2.2 parity pattern. - `CACHE_WRITE_MULTIPLIER = 1.25` / `CACHE_READ_MULTIPLIER = 0.1` public constants (Anthropic 5-minute tier). - Inputs clamped (`expected_reads ≥ 0`, `p_alive ∈ [0,1]`); methods take `&self`/`self` so a follow-up can add per-mode margins. - The formula derives the existing Subscription live-zone policy as its S=0 special case rather than contradicting it. **No callers yet.** P2 (consuming this in `TransformPipeline` behind `HEADROOM_NET_COST_POLICY`, replacing the binary `live_zone_only` gate, with decision telemetry) is specified in #856 and awaits maintainer direction — this PR just lands the audited arithmetic both dispatchers will share. ## Tests Golden-value parity: 6 new Rust unit tests and 7 new Python tests assert the **identical scenario numbers** (loss −53 200 for a 2K shave under a 50K warm suffix at R=10; win +61 000 for a 50K shave under a 10K suffix at R=3; S=0 always profitable; P_alive=0 always profitable — the idle-timer window; clamping; break-even 276 reads for the 2K/50K anchor). A drift on either side trips the pair loudly, same contract as the existing field-map parity test. - `cargo test -p headroom-core --lib compression_policy`: 12 passed (6 existing + 6 new) - `pytest tests/test_compression_policy.py`: 17 passed (10 existing + 7 new) - `cargo fmt --check`, `cargo clippy -p headroom-core` clean; `ruff check` + `ruff format --check` clean ## Real behavior proof Not applicable in the runtime sense — this PR intentionally adds **no runtime behavior** (pure functions, no call sites). The arithmetic is validated against the research anchors above in both languages' test suites; live decision telemetry arrives with P2 where the formula first gates real traffic. ## Out of scope P2 (flag-gated pipeline consumption + telemetry), P3 (deep-edit batching, idle-timer compaction near TTL lapse), retiring the deprecated `volatile_token_threshold`/`max_lossy_ratio` fields — all tracked in #856. --------- Co-authored-by: Ash Rhodes <ashley.rhodes@king.com> |
||
|
Focused Instability
|
06b2625b17
|
feat: gated Markdown-KV compaction formatter (serialization-aware output) (#859)
Closes #858. ## What Adds an opt-in **Markdown-KV** renderer to the lossless-first compaction stage, plus the plumbing to pick a compaction formatter by name. Default behavior is unchanged (`csv-schema`). Format-comprehension benchmarks show models retrieve values from Markdown-KV substantially more reliably than from CSV (~60.7% vs ~44.3%) — token-cheapest is not the same as most comprehensible. This makes the trade-off selectable per workload. ## How - **`MarkdownKvFormatter`** (`compaction/formatter.rs`): keeps the `[N]{cols}` declaration line, renders each row as a Markdown list item with `key: value` lines. - Missing cells omitted entirely (the KV advantage over positional CSV). - Strings ambiguous on a line (newlines, leading/trailing whitespace, empty) render JSON-quoted; everything else raw — commas and quotes need no escaping. - Nested cells inline compact JSON; opaque cells keep the fixed `<<ccr:HASH,KIND,SIZE>>` marker contract shared by all formatters. - **`CompactionStage::from_format_name`** maps `"csv-schema" | "json" | "markdown-kv"` to presets. - **Core**: `SmartCrusher::with_compaction_format(config, name)` — standard OSS composition with the named formatter. - **PyO3 bridge**: `SmartCrusher.with_compaction_format(config, format_name)` staticmethod; `ValueError` on unknown names (loud, no silent fallback). - **Python**: `SmartCrusher(compaction_format=...)` kwarg, falling back to the `HEADROOM_COMPACTION_FORMAT` env var, default `"csv-schema"`. ## Safety - **Default-off**: the default constructor path still calls the Rust `new()` constructor, so byte-parity coverage stays on the exact production codepath. A test asserts default output is byte-identical to an explicit `csv-schema` opt-in. - The existing `lossless_min_savings_ratio` gate (0.30) still applies. Markdown-KV repeats field names per row, so it clears the gate less often than CSV and falls through to the lossy path — we never inline a "lossless" rendering that isn't actually smaller. - CCR marker format unchanged across formatters; downstream retrieval pattern-matching keeps working. - No user/assistant content dropped — the formatter is a pure rendering of the same Compaction IR. ## Tests - Rust: 10 new unit tests in `compaction/formatter.rs` (table/buckets rendering, missing-cell omission, string quoting, CCR markers, drop summary, byte-size sanity vs raw JSON). `cargo test -p headroom-core`: 894 passed. Clippy + fmt clean. - Python: `tests/test_compaction_markdown_kv.py` (10 tests) — bridge rendering end-to-end, name→preset parity with the default constructor, kwarg/env knob precedence, loud failure on unknown names, default-output-unchanged guarantee. Existing smart_crusher suite: 38 passed. --------- Co-authored-by: Ash Rhodes <ashley.rhodes@king.com> |
||
|
Focused Instability
|
c2106cbdab
|
feat: probe-based retention scoring of recorded compression events (#862)
Closes #861 ## What First piece of compression quality measurement on **real proxied sessions** (vs the existing public-benchmark evals): an opt-in recorder captures (original, compressed) message pairs at each compression event, and a deterministic offline prober scores what survived. **Recorder** (`headroom/proxy/probe_recorder.py`) - `CompressionEventRecorder` implements the existing `PipelineExtension` protocol, subscribed to `INPUT_COMPRESSED`. Registered ONLY when `HEADROOM_PROBE_RECORD_DIR` is set — off means not even constructed, zero request-path overhead. - One JSONL line per compression event that changed tokens: `{ts, request_id, provider, model, tokens_before, tokens_after, transforms_applied, original_messages, compressed_messages}`. One file per PID (no interleaving), directory mode 0700. - Fail-open everywhere: construction failure logs a warning and disables recording; runtime exceptions are already swallowed by `PipelineExtensionManager.emit`. - Enabling handler change: the two `INPUT_COMPRESSED` emit sites (anthropic + openai) add a read-only `original_messages` reference to event metadata. No copies, no behavior change for other consumers. **Probes** (`headroom/evals/session_probes.py` + `headroom evals probes` CLI) - Probe targets extracted from ORIGINAL tool-result content across three dimensions: **exact numerics** (number + key context, incl. JSON-quoted keys), **artifact trail** (paths, URLs, hex hashes, UUIDs), **error evidence** (lines matching the existing `is_error_content` heuristic). - Each target classified as **retained** (verbatim, or surviving a legitimate format conversion — punctuation-normalized match; numerics require key AND value to survive; error lines tolerate dropped JSON key prefixes), **recoverable** (absent but a CCR retrieval marker is present), or **lost** (gone with no retrieval path). - Report: aggregate retention per dimension, bucketed by compression ratio (the quality-per-ratio curve), and grouped per transform. `--json-output` for machine-readable results. - Fully offline: no LLM, no API key. The recording format is designed to feed an LLM-judge pass later (out of scope per #861). ## Tests 33 new tests (red before, green after): `tests/test_probe_recorder.py` (11 — event filtering, JSONL shape, env activation, fail-open on unusable path, 0700 dir mode) and `tests/test_session_probes.py` (22 — extraction per dimension incl. JSON-quoted numerics, retained/recoverable/lost classification, format-change survival for numerics and error lines, ratio bucketing, transform dedup, malformed-line skipping, report rendering/serialization). Both proxy lifecycle tests additionally assert the INPUT_COMPRESSED `original_messages` metadata contract end-to-end through the real anthropic/openai handlers, so a refactor cannot silently disable the recorder. Local runs: new tests + `tests/test_proxy_pipeline_lifecycle.py` + `tests/test_canonical_pipeline.py` + `tests/test_pipeline.py` — 45 passed. `ruff check` + `uvx ruff format --check` clean. ### Self-review hardening (second commit) - Hex artifact regex now requires at least one `a-f`, so bare decimal runs (timestamps, counters) no longer inflate the artifact dimension. - Inflation events (ratio > 1, the #847 territory) get an explicit `1.00+ (inflated)` ratio bucket instead of silently dropping out of the bucketed view. - `run_probes` streams recording files line by line instead of slurping them. - Documented honestly: marker recoverability is event-scoped (comparative metric, not absolute); recorder writes synchronously on the request path (diagnostic sessions, not always-on). - `headroom/evals/README.md` gained a Session Probes usage section. ## Real behavior proof **Setup:** macOS (Darwin 25.5), Python 3.11.9, this branch, real proxy (`python -m headroom.proxy.server --port 18994 --anthropic-api-url http://127.0.0.1:18995`) with a local mock Anthropic upstream (no key), `HEADROOM_PROBE_RECORD_DIR=/tmp/headroom-probe-proof/recordings`. **Steps:** POSTed three Anthropic-format conversations whose tool results carry large JSON arrays (220-row uniform logs, 3000-row logs, 800 heterogeneous events), each containing known numerics, paths, trace hashes, and one error line. **Observed** — recorder wrote `compression-events-<pid>.jsonl` (one line per event); `headroom evals probes --recordings ...`: ``` Probed 3 compression events Aggregate retention: numerics 97.7% retained, 2.3% recoverable, 0.0% lost (527 targets) artifacts 100.0% retained, 0.0% recoverable, 0.0% lost (6555 targets) errors 100.0% retained, 0.0% recoverable, 0.0% lost (3 targets) By compression ratio (tokens_after / tokens_before): ratio 0.50-0.75: numerics 100.0% retained (CSV compaction — lossless, correctly recognized) ratio 0.75-1.00: numerics 95.7% retained, 4.3% recoverable (SmartCrusher sampling — dropped values carried a CCR marker) ``` All three classifications exercised: verbatim/format-change retention on the CSV-compacted events, **recoverable** on the heterogeneous event where SmartCrusher sampled rows out behind a `Retrieve more: hash=` marker, and the injected error lines retained in every event (the error-protection gate held). The `lost` path is covered by unit tests. The first iteration of this proof exposed two real bugs — naive verbatim matching misreported lossless JSON→CSV compaction as 100% lost, and duplicated transform markers double-counted tallies — both fixed with regression tests. **Not tested live:** Gemini path (no `INPUT_COMPRESSED` emit parity — pre-existing, same gap as #819); LLM-judge scoring (out of scope per #861). ## Security Recordings contain full conversation content in plaintext: opt-in env var only, local disk only, dir mode 0700, documented in CLI help. ## Out of scope (per #861) LLM-judge dimensions (decisions/intent, next steps), ACON-style counterfactual replay, automatic rule revision. --------- Co-authored-by: Ash Rhodes <ashley.rhodes@king.com> |
||
|
Logan Kang
|
c71592d421
|
feat(memory): add opt-in Apple-GPU (MPS) embedding runtime (#766)
## Description
On Apple-Silicon Macs — especially fanless models like the MacBook Air
(M5) — running the proxy with memory context injection can pin the CPU
while embedding. The embedding work runs an uncapped session on the CPU,
saturating multiple cores, which starves the proxy's asyncio loop and
leads to request timeouts.
This PR adds an **opt-in** runtime that offloads the memory embedder to
the Apple GPU (MPS). Setting `HEADROOM_EMBEDDER_RUNTIME=pytorch_mps`
routes embedding through the torch `sentence-transformers` backend on
MPS instead of the default ONNX CPU embedder, moving the work off the
CPU and keeping the proxy responsive.
The default behavior is unchanged — the feature is strictly opt-in,
env-var only, and falls through to the existing default embedder
selection (with a warning) whenever MPS or the torch dependencies are
unavailable.
Fixes: N/A — no tracking issue (surfaced while running codex auto-review
through
the proxy on a fanless MacBook Air (M5)).
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **Runtime selection** (`headroom/proxy/memory_handler.py`): read
`HEADROOM_EMBEDDER_RUNTIME`; when set to `pytorch_mps` **and** MPS is
actually available, route the memory embedder to the torch
`sentence-transformers` backend (Apple GPU).
- If MPS is unavailable or torch/sentence-transformers is not installed,
log a warning and fall through to the existing default embedder
selection (ONNX when available, else the pre-existing local
sentence-transformers fallback) — no crash. The default (env var unset)
is unchanged. Env-var only
- **MPS serialization** (`headroom/memory/adapters/embedders.py`):
`LocalEmbedder` now funnels every `encode()` through a dedicated
single-worker `ThreadPoolExecutor` when the resolved device is MPS.
torch-MPS is not thread-safe, and the existing `run_in_executor(None,
...)` dispatch would otherwise let concurrent proxy requests call MPS
from multiple threads. CPU/CUDA keep the shared default executor
(behavior unchanged). `close()` also drops the cached model so re-use
after close re-initializes cleanly.
- **Packaging** (`pyproject.toml`): new `pytorch-mps` extra (`torch` +
`sentence-transformers`), **platform-gated to macOS** (`; sys_platform
== 'darwin'`) since MPS is Apple-Silicon-only. Deliberately left out of
`[all]` (its deps already arrive via `[ml]`/`[memory]`).
- **Tests** (`tests/test_memory/test_embedder_mps_serialization.py`):
regression coverage for the serialized executor, concurrency safety (no
SIGABRT), CPU-path default behavior, and close/re-use re-initialization.
- **Docs**: `wiki/{configuration,memory,macos-deployment}.md`,
`docs/content/docs/{configuration,installation,memory}.mdx`,
`README.md`, `CHANGELOG.md`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed (CPU-offload + concurrency profiling on
Apple Silicon)
## Test Output
```
$ pytest -v tests/test_memory/test_embedder_mps_serialization.py
collected 4 items
tests/test_memory/test_embedder_mps_serialization.py::test_cpu_uses_shared_executor PASSED [ 25%]
tests/test_memory/test_embedder_mps_serialization.py::test_mps_creates_single_worker_executor PASSED [ 50%]
tests/test_memory/test_embedder_mps_serialization.py::test_mps_concurrent_embeds_do_not_crash PASSED [ 75%]
tests/test_memory/test_embedder_mps_serialization.py::test_mps_reembed_after_close_recreates_executor PASSED [100%]
============================== 4 passed in 6.92s ===============================
$ ruff check headroom/memory/adapters/embedders.py headroom/proxy/memory_handler.py tests/test_memory/test_embedder_mps_serialization.py
All checks passed!
$ mypy headroom/memory/adapters/embedders.py headroom/proxy/memory_handler.py
Success: no issues found in 2 source files
$ pytest -q tests/test_memory/ tests/test_memory_handler_concurrent_init.py tests/test_memory_handler_native_ops.py
553 passed, 1 skipped in 13.51s
```
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
**Why MPS (and not CoreML or a thread cap):** measured on an
Apple-Silicon Mac, the default uncapped CPU embedding session saturates
the cores; the same model on MPS runs at a fraction of the CPU (≈8x
lower sustained CPU utilization in profiling) while producing
**byte-identical embeddings** (cosine distance ≈ 0 across runtimes), so
relevance/ranking is unchanged. A CoreML execution-provider path was
evaluated and rejected: the default optimized ONNX model uses fused ops
that fall back to CPU under CoreML (no offload), and a full-precision
re-export was impractical (very low throughput + multi-GB memory). MPS
via `sentence-transformers` was the only practical GPU offload.
**Why serialization is mandatory:** torch-MPS is not thread-safe —
concurrent encode calls from a multi-worker executor abort with
`-[IOGPUMetalCommandBuffer validate]: failed assertion 'commit an
already committed command buffer'` (reproduced deterministically; a
single-worker executor resolves it).
Under concurrent load the serialized single-GPU-stream throughput meets
or exceeds the parallel CPU path while using a fraction of the cores.
**Scope / boundary:** this targets the Python **memory** embedder, which
is live on the proxy request path (memory context injection). The
Rust-backed SmartCrusher compression path is unaffected and remains
non-configurable from Python by design.
**Safety:** default behavior is unchanged (ONNX, no torch). The feature
is opt-in, env-var only, macOS-gated at the packaging layer, and
degrades gracefully (warn + the existing default embedder selection)
when MPS or the dependencies are unavailable.
|
||
|
@aaronjmars
|
b4b50253f1
|
fix(security): block DNS-rebinding on /debug/* and /stats/reset via Host-header allowlist (#605)
## Summary `require_loopback` in `headroom/proxy/loopback_guard.py` guards `/debug/tasks`, `/debug/ws-sessions`, `/debug/warmup`, and `/stats/reset` by checking `request.client.host` only. A malicious website can use DNS rebinding to make a victim's browser send requests to `127.0.0.1` while the page origin (and inbound `Host:` header) still reads `attacker.com`. The IP check passes — the browser IS on loopback — and `app.add_middleware(CORSMiddleware, allow_origins=["*"], ...)` (`headroom/proxy/server.py:1678`) lets the attacker's JS read the response. This PR adds the canonical second gate: the `Host:` header must also name loopback (`127.0.0.1[:port]`, `[::1][:port]`, or `localhost[:port]`). ## Impact While the proxy binds to `127.0.0.1` by default (`headroom/proxy/server.py:2956`), any browser the user opens can be used by an unrelated tab to reach the proxy via DNS rebinding. Concretely: - `GET /debug/tasks` → leaks asyncio task names, coroutine qualnames, and (with `?stack=true`) stack depths — reveals which coroutines are mid-flight and which codepaths are warm. Useful for fingerprinting + targeting follow-on attacks. - `GET /debug/ws-sessions` → leaks live WebSocket session metadata including `upstream_url` per session. - `GET /debug/warmup` → leaks warmup registry shape (lower risk). - `POST /stats/reset` → resets in-memory proxy stats (state mutation; minor DoS of dashboards/observability). The fix path matches the OWASP DNS-rebinding mitigation and Starlette's `TrustedHostMiddleware` posture. ## Location - `headroom/proxy/loopback_guard.py:73` — guarded the IP check only, no `Host:` validation - `headroom/proxy/server.py:1819-1845`, `:2312` — endpoints reachable via DNS rebinding under the old gate ## Fix `require_loopback` now runs two gates: (1) the existing `request.client.host` loopback IP check, and (2) a new `Host:` header allowlist via `is_loopback_host_header(...)`. The header helper accepts `127.0.0.1[:port]`, `[::1][:port]`, `localhost[:port]`, and IPv6-mapped IPv4 (`::ffff:127.0.0.1`) — strips brackets and ports, then delegates to the existing `is_loopback_host` logic. Cross-origin browser fetches always carry the attacker's hostname in `Host:`, so rebinding requests now 404 alongside any other external attempt. Same 404 (not 403) semantics, so debug endpoints remain invisible to external scanners. The new helper is in `__all__` for explicit re-use. The existing manual-`Request`-stub unit test path (no `.headers` attribute) is preserved by an `if headers is None: return` fallback so older callers that pass a bare stub still work. Test fixtures in `tests/test_proxy_debug_endpoints.py` were updated to pin `base_url="http://127.0.0.1"` so the loopback-`Host:` invariant holds in the green-path tests, plus a new `app_and_rebinding_client` fixture and `test_debug_endpoints_block_dns_rebinding` exercising the gate at the HTTP level. Six new unit tests cover `is_loopback_host_header` (canonical accept, external reject, malformed reject, and rebinding signature). Net: +219 lines, -4 lines, in two files. **Compatibility note for operators:** a deployment that fronts the proxy behind a reverse proxy with a non-loopback Host (e.g. `headroom.local` mapped to `127.0.0.1`) and still wants `/debug/*` exposed would need to either point that reverse proxy at the upstream API endpoints only, or extend `is_loopback_host_header` with an env-var allowlist in a follow-up. The default `HEADROOM_HOST=127.0.0.1` path is unaffected. ## Detected by Aeon (manual review — no scanner rule for this; pattern matches the threat-model-claims + local-HTTP-server axes from [`skills/vuln-scanner`](https://github.com/aaronjmars/aeon-aaron/blob/main/skills/vuln-scanner/SKILL.md), priors that have surfaced 10+ similar finds across 5 languages over the last 6 weeks). - Severity: medium - CWE-350 (Reliance on Reverse DNS Resolution for a Security-Critical Action) - CWE-352 (Cross-Site Request Forgery) — adjacent - Class match: DNS-rebinding-via-loopback-bind (recurring axis in the tracker) ## Verification - `python3 -m py_compile headroom/proxy/loopback_guard.py` — parses. - The new unit tests in `tests/test_proxy_debug_endpoints.py` cover: canonical loopback Host headers (accept), external Host headers (reject), malformed/empty (reject), an HTTP-level DNS-rebinding simulation (`app_and_rebinding_client` fixture with `base_url="http://attacker.com"` + loopback client tuple), and the unchanged IP-only stub path. - Other test files that use `require_loopback` (`test_proxy_openai_responses_integration.py`, `test_proxy_openai_responses_bypass.py`, `test_proxy_dashboard_stats_cache.py`) all override it via `app.dependency_overrides[require_loopback] = lambda: None`, so they are not affected by the new gate. - I was unable to run `pytest` locally in this sandbox; please confirm the new fixtures slot in cleanly when the CI matrix runs. --- Filed by [Aeon](https://github.com/aaronjmars/aeon-aaron). --------- Co-authored-by: aeonframework <noreply@anthropic.com> |
||
|
Abhinav Kaurav
|
6b0c09ffd5
|
fix: harden Copilot API auth token handling (#557)
## Description Improve Copilot API authentication behavior by correctly handling incoming bearer tokens and ensuring required Copilot headers are present. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added token classification logic to distinguish short-lived Copilot API tokens (`tid_`) from GitHub OAuth tokens. - Updated auth flow to pass through valid existing Copilot API bearer tokens and replace unsuitable bearer tokens. - Added default `Copilot-Integration-Id` and `editor-version` headers when missing. - Improved Windows credential lookup to consider both GitHub CLI (`gh:`) and Copilot CLI credential target prefixes. - Added regression tests for pass-through, replacement, header injection, and token prefix classification. ## Testing Describe the tests you ran to verify your changes: - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ## Test Output ```text pytest -q tests/test_copilot_auth.py 29 passed in 0.40s ``` ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes This change is intentionally scoped to auth behavior and tests in: - `headroom/copilot_auth.py` - `tests/test_copilot_auth.py` --------- Co-authored-by: Abhinav Kaurav <abhinav.kaurav@e2open.com> |
||
|
Focused Instability
|
c0cadccff9
|
feat: compression safety rails — error-output protection, pipeline circuit breaker, library inflation guard (#851)
Closes #847 ## What Three safety rails, each of which only ever makes compression LESS aggressive — zero behavior change for content that compresses normally: 1. **Error-output protection** (`ContentRouter`) — failed tool calls pass through verbatim on both the OpenAI `role=tool` string path and the Anthropic `tool_result` block path. Triggered by the explicit `is_error: true` flag or the existing Rust error-indicator detector (`headroom._core.content_has_error_indicators`, previously only used for TOIN signatures). Capped by `error_protection_max_chars` (8000, ~2K tokens) so big error-laden CI logs still reach `LogCompressor`, which preserves error lines — the two features stay complementary. `protect_error_outputs=False` disables. 2. **Pipeline circuit breaker** (`TransformPipeline`) — after 3 consecutive transform failures, `apply()` passes messages through untouched for a 60s cooldown instead of re-running (and re-failing) transforms on every request. Env-tunable: `HEADROOM_PIPELINE_BREAKER_THRESHOLD` (0 disables), `HEADROOM_PIPELINE_BREAKER_COOLDOWN_S`. Passthrough results tagged `pipeline:circuit_open`; a clean run closes the breaker. Thread-safe (lock-guarded counters, `time.monotonic`). 3. **Library inflation guard** (`compress()`) — all four proxy handlers already revert when "optimization" inflates tokens; the public library path returned inflated messages as-is. Now mirrors the proxy guard and tags `inflation_guard:reverted`. ## Why Production agent research backs each rail: keeping error outputs verbatim measurably improves agent recovery (Manus context-engineering; JetBrains "Complexity Trap", arXiv:2508.21433); Claude Code added its consecutive-compaction-failure cap after telemetry showed failure loops; the inflation guard closes a library/proxy asymmetry. All three follow CONTRIBUTING's "Safety first: never drop user/assistant content, prefer false negatives." ## Tests `tests/test_compression_safety_rails.py` — 10 tests: - error protection: string path, `is_error` flag (with neutral text proving the flag alone triggers), indicator scan, size-cap fall-through, config-disable - circuit breaker: opens after threshold + passthrough, success resets count, cooldown expiry closes, env-disable - inflation guard: inflated result reverts to originals Regression: `test_transforms_content_router`, `test_pipeline`, `test_compress_api`, `test_compress_failure`, `test_canonical_pipeline`, `test_proxy_pipeline_lifecycle`, `test_observability_*`, `test_compression_policy` — 69 passed. `ruff check` + `ruff format --check` clean. --------- Co-authored-by: Ash Rhodes <ashley.rhodes@king.com> |
||
|
dependabot[bot]
|
dc95c6bb00
|
ci: bump actions/stale from 9 to 10 (#850)
Bumps [actions/stale](https://github.com/actions/stale) from 9 to 10. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/actions/stale/releases">actions/stale's releases</a>.</em></p> <blockquote> <h2>v10.0.0</h2> <h2>What's Changed</h2> <h3>Breaking Changes</h3> <ul> <li>Upgrade to node 24 by <a href="https://github.com/salmanmkc"><code>@salmanmkc</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1279">actions/stale#1279</a> Make sure your runner is on version v2.327.1 or later to ensure compatibility with this release. <a href="https://github.com/actions/runner/releases/tag/v2.327.1">Release Notes</a></li> </ul> <h3>Enhancement</h3> <ul> <li>Introducing sort-by option by <a href="https://github.com/suyashgaonkar"><code>@suyashgaonkar</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1254">actions/stale#1254</a></li> </ul> <h3>Dependency Upgrades</h3> <ul> <li>Upgrade actions/publish-immutable-action from 0.0.3 to 0.0.4 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/actions/stale/pull/1186">actions/stale#1186</a></li> <li>Upgrade undici from 5.28.4 to 5.28.5 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/actions/stale/pull/1201">actions/stale#1201</a></li> <li>Upgrade <code>@action/cache</code> from 4.0.0 to 4.0.2 by <a href="https://github.com/aparnajyothi-y"><code>@aparnajyothi-y</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1226">actions/stale#1226</a></li> <li>Upgrade <code>@action/cache</code> from 4.0.2 to 4.0.3 by <a href="https://github.com/suyashgaonkar"><code>@suyashgaonkar</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1233">actions/stale#1233</a></li> <li>Upgrade undici from 5.28.5 to 5.29.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/actions/stale/pull/1251">actions/stale#1251</a></li> <li>Upgrade form-data to bring in fix for critical vulnerability by <a href="https://github.com/gowridurgad"><code>@gowridurgad</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1277">actions/stale#1277</a></li> </ul> <h3>Documentation changes</h3> <ul> <li>Changelog update for recent releases by <a href="https://github.com/suyashgaonkar"><code>@suyashgaonkar</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1224">actions/stale#1224</a></li> <li>Permissions update in Readme by <a href="https://github.com/ghadimir"><code>@ghadimir</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1248">actions/stale#1248</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/suyashgaonkar"><code>@suyashgaonkar</code></a> made their first contribution in <a href="https://redirect.github.com/actions/stale/pull/1224">actions/stale#1224</a></li> <li><a href="https://github.com/GhadimiR"><code>@GhadimiR</code></a> made their first contribution in <a href="https://redirect.github.com/actions/stale/pull/1248">actions/stale#1248</a></li> <li><a href="https://github.com/gowridurgad"><code>@gowridurgad</code></a> made their first contribution in <a href="https://redirect.github.com/actions/stale/pull/1277">actions/stale#1277</a></li> <li><a href="https://github.com/salmanmkc"><code>@salmanmkc</code></a> made their first contribution in <a href="https://redirect.github.com/actions/stale/pull/1279">actions/stale#1279</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/actions/stale/compare/v9...v10.0.0">https://github.com/actions/stale/compare/v9...v10.0.0</a></p> <h2>v9.1.0</h2> <h2>What's Changed</h2> <ul> <li>Documentation update by <a href="https://github.com/Marukome0743"><code>@Marukome0743</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1116">actions/stale#1116</a></li> <li>Add workflow file for publishing releases to immutable action package by <a href="https://github.com/Jcambass"><code>@Jcambass</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1179">actions/stale#1179</a></li> <li>Update undici from 5.28.2 to 5.28.4 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1150">actions/stale#1150</a></li> <li>Update actions/checkout from 3 to 4 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1091">actions/stale#1091</a></li> <li>Update actions/publish-action from 0.2.2 to 0.3.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1147">actions/stale#1147</a></li> <li>Update ts-jest from 29.1.1 to 29.2.5 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1175">actions/stale#1175</a></li> <li>Update <code>@actions/core</code> from 1.10.1 to 1.11.1 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1191">actions/stale#1191</a></li> <li>Update <code>@types/jest</code> from 29.5.11 to 29.5.14 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1193">actions/stale#1193</a></li> <li>Update <code>@actions/cache</code> from 3.2.2 to 4.0.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1194">actions/stale#1194</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/Marukome0743"><code>@Marukome0743</code></a> made their first contribution in <a href="https://redirect.github.com/actions/stale/pull/1116">actions/stale#1116</a></li> <li><a href="https://github.com/Jcambass"><code>@Jcambass</code></a> made their first contribution in <a href="https://redirect.github.com/actions/stale/pull/1179">actions/stale#1179</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/actions/stale/compare/v9...v9.1.0">https://github.com/actions/stale/compare/v9...v9.1.0</a></p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/actions/stale/blob/main/CHANGELOG.md">actions/stale's changelog</a>.</em></p> <blockquote> <h1>Changelog</h1> <h1>[10.1.0]</h1> <h2>What's Changed</h2> <ul> <li>Add only-issue-types option to filter issues by type by <a href="https://github.com/Bibo-Joshi"><code>@Bibo-Joshi</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1255">actions/stale#1255</a></li> </ul> <h1>[10.0.0]</h1> <h2>What's Changed</h2> <h2>Breaking Changes</h2> <ul> <li>Upgrade to node 24 by <a href="https://github.com/salmanmkc"><code>@salmanmkc</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1279">actions/stale#1279</a> Make sure your runner is on version v2.327.1 or later to ensure compatibility with this release. <a href="https://github.com/actions/runner/releases/tag/v2.327.1">Release Notes</a></li> </ul> <h2>Enhancement</h2> <ul> <li>Introducing sort-by option by <a href="https://github.com/suyashgaonkar"><code>@suyashgaonkar</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1254">actions/stale#1254</a></li> </ul> <h2>Dependency Upgrades</h2> <ul> <li>Upgrade actions/publish-immutable-action from 0.0.3 to 0.0.4 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/actions/stale/pull/1186">actions/stale#1186</a></li> <li>Upgrade undici from 5.28.4 to 5.28.5 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/actions/stale/pull/1201">actions/stale#1201</a></li> <li>Upgrade <code>@action/cache</code> from 4.0.0 to 4.0.2 by <a href="https://github.com/aparnajyothi-y"><code>@aparnajyothi-y</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1226">actions/stale#1226</a></li> <li>Upgrade <code>@action/cache</code> from 4.0.2 to 4.0.3 by <a href="https://github.com/suyashgaonkar"><code>@suyashgaonkar</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1233">actions/stale#1233</a></li> <li>Upgrade undici from 5.28.5 to 5.29.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/actions/stale/pull/1251">actions/stale#1251</a></li> <li>Upgrade form-data to bring in fix for critical vulnerability by <a href="https://github.com/gowridurgad"><code>@gowridurgad</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1277">actions/stale#1277</a></li> </ul> <h2>Documentation changes</h2> <ul> <li>Changelog update for recent releases by <a href="https://github.com/suyashgaonkar"><code>@suyashgaonkar</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1224">actions/stale#1224</a></li> <li>Permissions update in Readme by <a href="https://github.com/ghadimir"><code>@ghadimir</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1248">actions/stale#1248</a></li> </ul> <h1>[9.1.0]</h1> <h2>What's Changed</h2> <ul> <li>Documentation update by <a href="https://github.com/Marukome0743"><code>@Marukome0743</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1116">actions/stale#1116</a></li> <li>Add workflow file for publishing releases to immutable action package by <a href="https://github.com/Jcambass"><code>@Jcambass</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1179">actions/stale#1179</a></li> <li>Update undici from 5.28.2 to 5.28.4 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1150">actions/stale#1150</a></li> <li>Update actions/checkout from 3 to 4 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1091">actions/stale#1091</a></li> <li>Update actions/publish-action from 0.2.2 to 0.3.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1147">actions/stale#1147</a></li> <li>Update ts-jest from 29.1.1 to 29.2.5 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1175">actions/stale#1175</a></li> <li>Update <code>@actions/core</code> from 1.10.1 to 1.11.1 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1191">actions/stale#1191</a></li> <li>Update <code>@types/jest</code> from 29.5.11 to 29.5.14 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1193">actions/stale#1193</a></li> <li>Update <code>@actions/cache</code> from 3.2.2 to 4.0.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/stale/pull/1194">actions/stale#1194</a></li> </ul> <h1>[9.0.0]</h1> <h2>Breaking Changes</h2> <ol> <li>Action is now stateful: If the action ends because of <a href="https://github.com/actions/stale#operations-per-run">operations-per-run</a> then the next run will start from the first unprocessed issue skipping the issues processed during the previous run(s). The state is reset when all the issues are processed. This should be considered for scheduling workflow runs.</li> <li>Version 9 of this action updated the runtime to Node.js 20. All scripts are now run with Node.js 20 instead of Node.js 16 and are affected by any breaking changes between Node.js 16 and 20.</li> </ol> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
gglucass
|
841663da16
|
fix(proxy): make Kompress eager preload cache-only so a cold cache can't block startup (#783)
## Description `ContentRouter.eager_load_compressors()` runs a network `hf_hub_download` of the Kompress ONNX model on the **blocking startup/lifespan path**, before the proxy binds its port. On a cold cache this is unsafe: - the download can hang long enough to blow the supervisor's bind timeout, or - a native crash in the download/ML stack (an **uncatchable `Fatal Python error: Aborted` / SIGABRT**) kills the interpreter before it ever `listen()`s. Either way the supervisor sees "proxy never opened its port" and gives up. We observed this in the field from the desktop app (process aborted during `eager_load_compressors -> _load_kompress_onnx -> hf_hub_download` of `onnx/kompress-int8.onnx`, while the only Python thread was parked in the HuggingFace download file-lock; the abort came from a native thread, so `try/except` at the call site cannot catch it). The eager preload is a latency optimization and must never be able to block — or kill — startup. This change makes startup preload **cache-only**: if the model isn't already cached, we defer the download to first use (off the startup path) and bind the port normally. Warm starts are unchanged. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `onnx_runtime.hf_hub_download_local_first(...)`: added `allow_network` (default `True`). When `False`, a cache miss re-raises the local-lookup error instead of falling back to a network download. - `kompress_compressor`: added `allow_download` (default `True`) threaded through `preload()` -> `_load_kompress()` -> `_load_kompress_onnx()` / `_load_kompress_pytorch()` and the ModernBERT tokenizer load. Added `KompressModelNotCached`, raised when a cache-only load misses. Auto-mode no longer falls back to a PyTorch network download on a cache-only miss — it propagates so the caller can defer. - `content_router.eager_load_compressors()`: calls `preload(allow_download=False)`. On `KompressModelNotCached` it logs and reports the component as `"deferred"` (a status `warmup.merge_transform_status` already handles gracefully) instead of letting a cold download run on the startup path. Default (first-request) loading behavior and warm-start preload are unchanged. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed New tests in `tests/test_kompress_preload_deferral.py` cover: cache-only `hf_hub_download_local_first` never hits the network; default still falls back; cache-only ONNX load raises `KompressModelNotCached`; auto-mode does **not** trigger a PyTorch download on a cache-only miss; and `eager_load_compressors` reports `deferred` (cold) / `enabled` (warm). Existing `_load_kompress` dispatch tests updated for the new keyword-only param. > Note on environment: I do not have a clean reproduction of the native SIGABRT itself (it depends on a specific machine's HF download/ML native stack), so the "Manual testing performed" box is left unchecked. The tests target the structural fix — that startup preload can no longer perform a network download — which is the precondition for the crash. ## Test Output ``` $ uv run pytest -v tests/test_kompress_preload_deferral.py tests/test_kompress_preload_deferral.py::test_local_first_no_network_when_disallowed PASSED tests/test_kompress_preload_deferral.py::test_local_first_falls_back_to_network_by_default PASSED tests/test_kompress_preload_deferral.py::test_load_kompress_onnx_cache_miss_raises_not_cached PASSED tests/test_kompress_preload_deferral.py::test_load_kompress_auto_does_not_pytorch_download_on_cache_miss PASSED tests/test_kompress_preload_deferral.py::test_eager_load_defers_when_model_not_cached PASSED tests/test_kompress_preload_deferral.py::test_eager_load_enabled_when_model_cached PASSED 6 passed in 4.82s $ uv run pytest tests/test_transforms/test_kompress_compressor.py tests/test_transforms_content_router.py tests/test_onnx_runtime.py tests/test_proxy_warmup.py 63 passed $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/onnx_runtime.py headroom/transforms/kompress_compressor.py headroom/transforms/content_router.py Success: no issues found ``` ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable (auto-generated from conventional commits) ## Additional Notes This contains the cold-start case. A native crash in onnxruntime *session init* (as opposed to the download) on first request would still be a separate issue; it is not what was observed here (the abort was during the HF download), and isolating it would be a larger, separate change. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
gglucass
|
9bff5752bb
|
fix(learn): claude-cli streams output with idle timeout (#373)
## Description `headroom learn` with the claude-cli backend used `subprocess.run` with a hard 120s wall-clock cap and no liveness signal. A successful long analysis and a hung connection looked identical — exit 0 with "0 recommendations" was the only user-visible signal when the LLM call timed out, which silently hides genuine learnings. This PR makes the CLI backend timeout-aware, with progress detection for claude-cli and configurable wall-clock caps for every backend. Fixes #(issue number) ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Streaming claude-cli with idle timeout**: invoke `claude -p --output-format stream-json --verbose` and run a watchdog loop that drains stdout/stderr via reader threads. Each stream-json event resets an idle deadline. Kill the process if no output for `HEADROOM_LEARN_CLI_IDLE_TIMEOUT_SECS` (default 60s) or if total elapsed exceeds `HEADROOM_LEARN_CLI_TIMEOUT_SECS` (default 300s, was 120s). The final `type:"result"` event carries the assistant response, which is then parsed as JSON. Reader threads (rather than `select`) are used so the watchdog works on Windows where `select` does not support pipe handles. - **Bumped default `_CLI_TIMEOUT` from 120s to 300s** as the hard cap for all CLI backends. The previous 120s was too tight for large digests on slower networks. - **Env-var overrides** via new helper `_resolve_timeout_secs(env_var, default)`: - `HEADROOM_LEARN_CLI_TIMEOUT_SECS` — hard wall-clock cap (all CLI backends) - `HEADROOM_LEARN_CLI_IDLE_TIMEOUT_SECS` — idle cap (streaming claude-cli only) - Non-positive or non-integer values log a warning and fall back to defaults, so a typo can't disable the timeout. - **gemini-cli and codex-cli** keep `subprocess.run(timeout=hard_cap)` since they do not emit progress events. They benefit from the bumped default and the env-var override. - **CHANGELOG.md** updated under `[Unreleased]` → `### Fixed`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed (existing repro: 16k-call digest that previously timed out at 120s) New test coverage in `tests/test_learn/test_analyzer.py`: - `test_claude_cli_streams_and_parses_result_event` — happy path, fake Popen yields system/assistant/result events - `test_claude_cli_parses_fenced_result` — markdown fences in the result event still parse - `test_claude_cli_idle_timeout_kills_hang` — `HEADROOM_LEARN_CLI_IDLE_TIMEOUT_SECS=1` + a hanging stdout iterator triggers the idle watchdog - `test_claude_cli_hard_cap_kills_continuous_chatter` — continuous events with a low hard cap fire the wall-clock kill (proves idle reset alone can't keep a runaway alive) - `test_claude_cli_missing_result_event_raises` — graceful failure when no `result` event is emitted - `test_claude_cli_nonzero_exit_raises` / `test_claude_cli_unparseable_result_raises_with_context` / `test_claude_cli_not_installed_raises` — error paths - Parallel codex-cli error coverage (timeout-honors-env-override included) so the wall-clock path is exercised - `TestResolveTimeoutSecs` — unset / empty / non-integer / non-positive / valid override ## Test Output ``` $ uv run pytest tests/test_learn/test_analyzer.py ============================== 67 passed in 2.14s ============================== $ uv run ruff check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py All checks passed! $ uv run ruff format --check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py 2 files already formatted $ uv run mypy headroom/learn/analyzer.py Success: no issues found in 1 source file ``` ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - The contract assumed for claude-cli stream-json output is: each line is a JSON object with a `type` field; the final event has `type:"result"` with a string `result` field carrying the assistant text. This matches the documented Anthropic CLI behavior. If the contract changes upstream, `_call_claude_cli_streaming` raises a clear "did not emit a final \`result\` event" error rather than silently succeeding. - Backwards-compatible for users without env-var configuration: behavior just becomes "longer hard cap, plus idle watchdog for claude-cli", neither of which can falsely succeed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
gglucass
|
8f374263d3
|
feat(transforms): attribute read_lifecycle + smart_crush tags (#249)
## What Enrich the `transforms_applied` tags emitted by `ReadLifecycleManager` and `SmartCrusher` so each tag carries the specific target it acted on, instead of being an opaque counter: - `read_lifecycle:<state>` -> `read_lifecycle:<state>:<file_path>` - `smart_crush:<n>` -> `smart_crush:<n>:<tool1,tool2,...>` (tool names resolved from the assistant's `tool_calls` / `tool_use` metadata; falls back to `smart_crush:<n>` when no name resolves) Downstream UIs can then show *what* a compression acted on (which file was a stale read, which tools had their output crushed), not just that it happened. ## Note on the rebase The original revision targeted `ToolCrusher` / `tool_crush:<n>`. That transform has since been retired and replaced by the Rust-backed `SmartCrusher` (which emits `smart_crush:<n>`), so the tool-name attribution moved to `smart_crusher.py`. The `read_lifecycle` half is unchanged. ## Response-header compatibility `x-headroom-transforms` is built as `",".join(transforms_applied)`. A tag containing a comma (tool-name lists; file paths) would make that header ambiguous to split back into tags. To keep the header backward compatible, `header_safe_transforms` (`headroom/proxy/cost.py`) collapses the enriched tags back to their legacy counter shape **for the header only** -- the full enriched detail still flows through the structured `transforms_applied` list (dashboards, request logs, activity feed). Applied at all three header sites (openai / anthropic / gemini handlers). Paths containing `:` survive in `transforms_applied` because consumers bound their split to 3 parts. ## Tests - `tests/test_transforms/test_read_lifecycle.py` -- OpenAI + Anthropic tag shape, colon-in-path preservation - `tests/test_transforms/test_smart_crusher_attribution.py` -- OpenAI + Anthropic tool-name resolution, dedup, no-name fallback, id/name-missing skips - `tests/test_proxy/test_header_safe_transforms.py` -- header normalization keeps the joined header unambiguous (incl. comma-in-path) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Khalid Shaikh
|
eb2e50feb2
|
feat: add OAuth2 client-credentials upstream-auth proxy extension (#778) (#784)
## What & why
Adds **`headroom-oauth2`** under `plugins/` — a generic, vendor-neutral
proxy extension that mints an OAuth2 **client-credentials** (RFC 6749
§4.4) bearer from a configured token endpoint and injects it as the
upstream `Authorization` on each proxied request, via the opt-in
`headroom.proxy_extension` seam. **No core changes.**
This lets headroom front any gateway that requires a *minted,
short-lived machine token* rather than a static API key. It complements
**#510** (env-var/static-key auth) rather than replacing it.
Implements **#778** (feature request). Opening the implementation
alongside the issue so there's something concrete to react to — **happy
to hold/rework pending a 👍 from a maintainer**, per CONTRIBUTING.
## Spec
Full spec in
[`plugins/headroom-oauth2/SPEC.md`](plugins/headroom-oauth2/SPEC.md)
(API surface, behavior/compat, user stories, failure modes, resilience
incl. multi-process, security, observability, rollback). Highlights:
- **Opt-in & no-op by default:** dormant until `--proxy-extension
oauth2`, and a no-op unless `HEADROOM_OAUTH2_TOKEN_URL` is set. No
change to defaults, body, routing, or compression.
- **Config is 100% env** (no new CLI flags):
token_url/client_id/secret/scopes/audience, RFC 8707 `resource`,
`post`|`basic` auth style, static upstream headers, timeout/skew.
- **Token caching + single-flight refresh**; `expires_in` clamped to a
positive TTL.
- **Fails closed** on misconfig; returns `502 upstream_auth_error` on
mint failure **without leaking the IdP error body**; `token_url` is
**https-enforced** (loopback exempt for tests).
- **Standard-library only** (token minted via `urllib` → system cert
store, so it works behind corporate SSL inspection). `litellm` is
touched only for static headers and is an optional extra, not a core
dep.
- **Effective for** OpenAI-compatible / passthrough litellm backends.
`bedrock`/`vertex`/`sagemaker` auth from env and ignore a forwarded
bearer → the extension **warns loudly** and is a no-op there.
## Tests
37 tests covering behavior **and** failure modes (`ruff check`/`format`
clean, **98% coverage**): post/basic mint, caching, single-flight (cold
+ on-refresh, exact mint counts under concurrency), https enforcement +
`localhost` rejection + `::1`, `expires_in`
clamp/float/missing/non-numeric, `extra_params` cannot clobber canonical
fields, bad-status/non-JSON/no-token/unreachable (asserting no
secret/body leak), ASGI middleware (inject, non-http passthrough, 502 +
`no-store`, missing `headers` key), and `install()`
(no-op/fail-closed/bad-timeout/env-auth-backend-warning/static-headers).
## Real behavior proof
- **Setup:** Linux aarch64, Python 3.13.5, `headroom-ai` 0.23.0, real
`headroom proxy` process.
- **Steps:** started `headroom proxy --backend litellm-openai
--proxy-extension oauth2` with `HEADROOM_OAUTH2_*` env pointed at a
local OAuth2 token endpoint; an upstream echo server captured what the
backend received; sent two `/v1/messages` requests through the proxy.
- **Observed (copied output):**
```
PROXY: headroom-oauth2: client-credentials auth installed
(token_url=…/token, style=post)
MINTS (across 2 requests): 1 # token cached + reused -> 1 mint for 2
requests
UPSTREAM RECEIVED: auth=Bearer MINTED-FROM-IDP-…
static=generic-static-header
SECRET LEAK CHECK (client_secret in proxy logs): 0
```
→ The minted bearer (not the placeholder backend key) and the configured
static header reached the upstream; the client's inbound credential was
replaced; the client secret never appeared in logs; caching worked.
- **What I did *not* test:** a live commercial IdP
(Entra/Okta/Auth0/etc.) and a live cloud gateway — the token endpoint
and upstream here are local stand-ins. Also not tested: multi-worker
(gunicorn) deployment, and Python 3.10/3.11 (developed on 3.13).
## Placement
Proposed as a standalone installable package under
`plugins/headroom-oauth2/` (registers via the entry-point seam; `pip
install -e plugins/headroom-oauth2`). Open to baking it into core or
publishing it separately — maintainer's call.
|
||
|
yehsuf
|
5dfb446da1
|
fix(health): readyz verifies upstream connectivity, not just process liveness (#744)
Closes #740 ## What `/readyz` and `/health` previously reported healthy even when the upstream API was completely unreachable (e.g. SSL certificate errors, wrong URL, network failure). The proxy would accept traffic and return 502 on every `/v1/messages` request. ## Changes - Added `_check_upstream()` async function that probes the configured upstream base URL with a HEAD request (5s timeout, result cached 30s) to verify TLS + TCP reachability without triggering an inference call - `/readyz` now calls `_check_upstream()` before building its response; returns HTTP 503 if the upstream is unreachable - `/health` exposes an `upstream` sub-check entry with `enabled`, `ready`, `status`, and `error` fields - `HEADROOM_SKIP_UPSTREAM_CHECK=1` opts out (for air-gapped or test environments) - Existing tests updated to set `HEADROOM_SKIP_UPSTREAM_CHECK=1` so unit tests don't make live network calls - Three new tests covering: opt-out via env var, 503 on upstream failure, `/health` includes upstream check ## Behaviour | Endpoint | Before | After | |---|---|---| | `/livez` | process alive | unchanged | | `/readyz` | process alive | process alive AND upstream reachable | | `/health` | no upstream info | includes `checks.upstream` with status + error | --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |