mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
99 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
github-actions[bot]
|
aea3c35177
|
chore: release main (#1441)
🤖 I have created a release *beep* *boop* --- <details><summary>0.28.0</summary> ## [0.28.0](https://github.com/headroomlabs-ai/headroom/compare/v0.27.0...v0.28.0) (2026-06-29) ### Features * add --disable-kompress-fallback to restore legacy PASSTHROUGH fallback ([#1185](https://github.com/headroomlabs-ai/headroom/issues/1185)) ([ |
||
|
Tejas Chopra
|
c2fc4d3753
|
fix(ccr): make headroom_retrieve a hash-only full-content lookup (#1532)
The optional `query` parameter on headroom_retrieve routed retrieval through CompressionStore.search(), which BM25-scored the items inside a single cached blob and dropped everything below a 0.3 relevance floor. On small per-blob corpora with conversational queries this returned an empty result the large majority of the time, so the LLM saw "nothing found" for content that was actually present — pushing users to turn compression off entirely. Retrieval is fundamentally a hash lookup (this already matches the Rust proxy's CCR store, which is put/get only — "no BM25 search"). Remove the query/search path end to end and always return the full original content: Core (Python proxy): - tool schemas (anthropic/openai/google) drop the `query` property - parse_tool_call returns the hash (str | None) instead of (hash, query) - response handler, proxy POST/GET/tool-call handlers, the MCP retrieve tool, and the streaming feedback recorders retrieve by hash only - proactive context-tracker expansion always restores full content - delete CompressionStore.search() and its BM25 machinery (the bm25 module stays — it is still used by relevance/) - CCRToolCall.query, CCRToolResult.was_search, and ExpansionRecommendation.expand_full/search_query are removed Plugins (advertised a now-defunct query param to the LLM): - hermes (Python), openclaw + opencode (TypeScript) retrieve tools drop `query` from their schemas, signatures, request URLs, and tests Benchmarks/docs: - ccr_regression + adversarial benchmarks switch from store.search() to full hash retrieval (search input-injection tests repurposed to the hash, the only remaining input surface) - wiki/ARCHITECTURE.md, wiki/ccr.md, docs/content/docs/ccr.mdx, config.py and store docstrings updated to describe hash-only retrieval Tests updated to assert full-content retrieval and guard the removed surface; the full CCR/proxy/store/TOIN suite passes. ruff + mypy clean. ## Description <!-- Briefly explain the change and why it is needed. --> Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # Paste relevant command output or artifact links here ``` ## Real Behavior Proof - Environment: - Exact command / steps: - Observed result: - Not tested: ## Review Readiness - [ ] I have performed a self-review - [ ] This PR is ready for human review ## Checklist - [ ] My code follows the project's style guidelines - [ ] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) Add screenshots to help explain your changes. ## Additional Notes <!-- Mention any N/A checklist items, tradeoffs, follow-ups, or maintainer context. --> |
||
|
Tejas Chopra
|
a639540959
|
chore: remove committed node_modules + stray/internal markdown (repo hygiene) (#1528)
## Description Repo hygiene for a public OSS project: removes committed `node_modules`, stray/internal/draft markdown, and commercial-surface references — keeping every real doc (the published docs site, the wiki guides, and all component READMEs) intact. Every file was content-audited before removal, and load-bearing files were verified against the code/CI and kept. Net: **1,695 files changed, +23 / −266,409** (the deletions are dominated by a committed `node_modules` tree). Closes # (no tracking issue) ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [x] Code refactoring (no functional changes) ## Changes Made **Removed (verified to have no code/CI dependencies):** - `examples/vercel-ai-sdk-pr/` — 1,649 committed `node_modules` files (zero example source); `node_modules/` added to `.gitignore`. - `docs/spec/` (23 draft "Living Specification" files — orphaned, `1.0.0-draft`, drifted from the code), `docs/superpowers/` (2 agent plans), `docs/proposals/` (2 internal/commercial memos). - 6 orphan `docs/*.md` (auth-modes, bedrock, claude-code-vertex-headroom, cortex-code, output-token-reduction-guide, rtk-loop-weighting). - `PR.md` (committed PR draft), `ENTERPRISE.md`, `.github/FUNDING.yml`. **Content scrubs:** - Removed unreleased "Headroom Cloud" / `api.headroom.ai` / `hr_` references from `configuration.mdx`, `wiki/configuration.md`, `wiki/typescript-sdk.md`, `sdk/typescript/README.md` (reworded to neutral, accurate phrasing). - Dropped a stale "awaiting maintainer before merge" line from `plugins/headroom-oauth2/SPEC.md`; tidied `.gitignore` comments (kept the protective `headroom-managed/` ignore rule). - Fixed the now-dangling links into removed files (README nav/`output-token-reduction` link, `scripts/README`, `wiki/vertex`). **Explicitly KEPT (load-bearing — would orphan in-code citations if removed):** - `.changelog.md` — consumed by `.github/workflows/release.yml` (read as the release-notes file). - `REALIGNMENT/`, `docs/observability.md`, `docs/rtk-architecture.md`, `wiki/plans/`, `TESTING-copilot-subscription.md` — referenced by the Rust core / Python / tests as design docs. ## Testing - [ ] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [ ] Type checking passes (`mypy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # Docs/markdown + .gitignore only — no Python/Rust source changed, so the # behavioral test suite is unaffected. Verified the cleanup did not orphan # references or break the published docs site: $ git ls-files 'docs/content/docs/*.mdx' | wc -l # published site intact 42 $ # meta.json nav unchanged; no published page removed. $ grep -rnI "Headroom Cloud|api.headroom.ai|'hr_" $(git ls-files '*.md' '*.mdx') >>> none $ # dangling refs to removed files (excl pre-existing P0/P2 spec stubs that $ # never existed in git): none remaining. ``` ## Real Behavior Proof - Environment: macOS, local git clone of the repo (markdown/.gitignore changes only — no runtime). - Exact command / steps: 4 read-only content-audit agents classified every `.md`/`.mdx` file; each removal candidate was cross-checked against the codebase (`grep` for citations in `.rs`/`.py`/tests, workflows, and configs); only files with no dependents were removed; the tree was re-grepped after removal to confirm no new dangling references; verified the published docs site page count (`git ls-files 'docs/content/docs/*.mdx' | wc -l` = 42, unchanged). - Observed result: the 42-page published docs site and all wiki guides are untouched; no source or workflow references a removed file; `.changelog.md` (consumed by release.yml) and the code-cited design docs were detected as dependencies and kept; the committed `node_modules` tree is removed and `node_modules/` is gitignored so it can't be re-committed; zero "Headroom Cloud"/`headroom.dev` references remain. - Not tested: N/A — no executable code changed (only markdown, `.mdx`, and `.gitignore`), so the behavioral test suite is unaffected. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - This branch deletes `.github/FUNDING.yml` while PR #1526 edits it — the two will be sequenced at merge (delete wins). - A follow-up option (not in this PR): also remove the internal design docs that are currently cited by the code (`REALIGNMENT/`, `docs/observability.md`, `docs/rtk-architecture.md`, `wiki/plans/`) — that requires scrubbing ~15–20 in-code citations so nothing dangles, so it's deliberately deferred. - Untracked local working files (`benchmarks/hf_pilot/`, `tools/copilot-test/`) are intentionally left out of git (not committed). |
||
|
Tejas Chopra
|
5771a8020e
|
fix(deps): remediate dependency CVEs and publish SBOM (#1509)
## Description
Supply-chain hardening: takes the **shipped** dependency surface from
**26 known CVEs to 0**. `pip install headroom-ai[all]` now resolves with
no known vulnerabilities (verified with Anchore syft + grype). Also
publishes a checked-in SBOM package (`sbom/`) so any user — especially
pilots running their own security review — can verify what's inside and
that we track it.
This addresses the Dependabot alerts on `main` (9 high / 4 moderate / 7
low at time of writing).
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
**Rust**
- `pyo3` 0.24 → 0.29 (GHSA-36hh-v3qg-5jq4 High, GHSA-chgr-c6px-7xpp
Med). Migrated `Python::allow_threads` → `Python::detach` (10 sites) and
added `from_py_object` to the `Clone`-deriving `#[pyclass]` types (both
required by the 0.25+ API).
- `pyo3-log` 0.12 → 0.13; `lru` 0.12 → 0.18 (GHSA-rhfx-m35p-ff5j).
**Python**
- `torch` → 2.12.1, `mem0ai` → 2.x.
- Floor-pinned transitive CVE deps via `[tool.uv]
constraint-dependencies`: `pygments>=2.20.0`,
`pydantic-settings>=2.14.2`, `gitpython>=3.1.50`, `langsmith>=0.9.0`.
- **Removed `benchmark` from the `[all]` aggregate** so the default
install is CVE-free. `lm-eval` is invoked as an external subprocess
(`python -m lm_eval`) and never imported, so it is not a true runtime
dep — it remains available via the opt-in `[benchmark]` extra. See
[Accepted Risks](#additional-notes).
**npm (build/test tooling — never shipped in the
wheel/container/published SDK)**
- `esbuild` override `>=0.28.1` in `sdk/typescript` + `plugins/openclaw`
(GHSA-g7r4-m6w7-qqqr).
- `docs/`: `@anthropic-ai/sdk` → `^0.106.0` (GHSA-p7fg-763f-g4gf),
`postcss` override to force Next.js's bundled copy ≥8.5.10
(GHSA-qx2v-qp2m-jg93); regenerated a stale `bun.lock` that carried a
**Critical** vitest/vite.
**CI**
- Pinned `pypa/gh-action-pypi-publish` `@release/v1` → `@v1.13.0`
(GHSA-vxmw-7h4f-hqxh) in `release.yml` + `publish.yml`.
**SBOM**
- New `sbom/` directory: CycloneDX 1.7 + SPDX 2.3 SBOMs, grype scan
evidence, 330-package license inventory, and a regeneration guide.
## Testing
- [ ] Unit tests pass (`pytest`) — N/A, no Python source changed
(deps/config only)
- [x] Linting passes — `cargo fmt --check` + `cargo clippy` clean on the
changed crate; 0 `.py` files changed so `ruff`/`mypy` scope is
unaffected
- [x] Type checking passes — `cargo check --workspace` (0 errors)
- [ ] New tests added — N/A (dependency bumps; covered by existing
suites)
- [x] Manual testing performed — see Real Behavior Proof
### Test Output
```text
# headroom-ai[all] product surface — the number that matters
$ grype sbom:sbom/headroom-sbom-all-extra.cdx.json
No vulnerabilities found
# full repo scan (universal lock incl. opt-in [benchmark] + dev)
$ grype sbom:sbom/headroom-sbom.cdx.json
NAME INSTALLED TYPE VULNERABILITY SEVERITY
sqlitedict 2.1.0 python GHSA-g4r7-86gm-pgqc High # [benchmark]-only, unpatchable, accepted
nltk 3.9.4 python GHSA-p4gq-832x-fm9v High # [benchmark]-only, unpatchable, accepted
# pyo3 0.29 migration — extension builds + imports + runs
$ cargo check --workspace
Finished `dev` profile [unoptimized + debuginfo] target(s)
$ maturin develop && python -c "from headroom._core import DiffCompressor, SmartCrusher; ..."
extension OK — detach + from_py_object paths exercised
# lru 0.18 — eviction path
$ cargo test -p headroom-proxy --lib drift
14 passed, 213 filtered out
# per-ecosystem npm audits
$ (cd sdk/typescript && npm audit) -> found 0 vulnerabilities
$ (cd plugins/openclaw && npm audit) -> found 0 vulnerabilities
$ (cd docs && npm audit && bun audit) -> found 0 vulnerabilities / No vulnerabilities found
```
## Real Behavior Proof
- Environment: macOS (darwin 25.4.0, arm64), Python 3.12 `.venv`, Rust
1.95 toolchain, syft 1.46.0, grype 0.115.0, bun 1.3.14, maturin 1.13.3.
- Exact command / steps: (1) `uv export --extra all --no-dev
--no-emit-project | syft → grype` for the product surface; (2) `cargo
check --workspace` + `maturin develop` + extension import/compress smoke
test; (3) `cargo test -p headroom-proxy --lib drift`; (4) `cargo fmt
--check` + `cargo clippy -p headroom-py`; (5) `npm audit` in
sdk/openclaw/docs + `bun audit` in docs.
- Observed result: `headroom-ai[all]` resolution scans clean — "No
vulnerabilities found" (179 pkgs); full/prod SBOM shows only the 2
documented accepted CVEs; pyo3 0.29 extension imports and runs (detach +
from_py_object paths exercised); drift tests 14/14 pass; cargo fmt +
clippy clean; all npm/bun audits report 0.
- Not tested: full `pytest` suite (no Python source changed);
release-profile wheel build (used dev-profile `maturin develop` for the
import proof — the extension is semantically identical).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
(`sbom/README.md`)
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective — N/A
(dependency bumps; existing suites + scans cover it)
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md — N/A (Release Please
auto-generates from the conventional commit)
## Additional Notes
**Accepted risks (the 2 residual CVEs).** Both originate solely from the
EleutherAI `lm-evaluation-harness` under the **opt-in `[benchmark]`
extra**, which Headroom invokes as a subprocess (never imports):
- `sqlitedict` CVE-2024-35515 (High) — pickle deserialization; package
abandoned (last release 2021), **no upstream fix exists**.
- `nltk` CVE-2026-54293 (High) — path traversal in `nltk.data.load()`;
affects ≤3.9.4 (current latest), **no patched release**.
Neither is in `[all]`, the published wheel, or the container. They are
documented in `sbom/README.md` and will be picked up automatically once
upstream ships fixes.
**Release/CHANGELOG:** N/A items above are because this is a
dependency/security PR with no Python source changes; CHANGELOG is
Release-Please-managed via the conventional commit message.
|
||
|
Rudimar Ronsoni
|
fa05ebc849
|
docs: clarify OpenCode integration (#1317)
## Description Clarifies the OpenCode documentation follow-up for PR #1105 so users can install `headroom-opencode`, configure provider routing, use the native plugin, and copy working retrieve/compression helper examples. ## Type of Change - [x] Documentation update - [ ] Bug fix - [ ] New feature - [ ] Breaking change ## Changes Made - Documented how `headroom wrap opencode` wires provider config, MCP tools, and runtime environment. - Documented the native `HeadroomPlugin` path, `HEADROOM_PROXY_URL`, retrieve tooling, and programmatic config helpers. - Fixed `plugins/opencode/README.md` examples so `compressWithHeadroom` uses the exported options-object API and `headroom_retrieve` uses `hash`. ## Testing - [x] Type checks pass. - [x] Unit tests pass. - [x] Whitespace check passes. ### Test Output ```text plugins/opencode: npm run typecheck > tsc --noEmit plugins/opencode: npm test Test Files 2 passed (2) Tests 9 passed (9) docs: npm run types:check ✓ Types generated successfully repo: git diff --check (no output) ``` ## Real Behavior Proof - Environment: Local macOS worktree at `docs/pr-1105-documentation-followup`, Node/npm project commands run from `plugins/opencode` and `docs`. - Exact command / steps: Updated the README snippets, ran `npm run typecheck`, reran `npm test` with elevated permissions after the sandbox blocked a local `127.0.0.1` listener, ran `npm run types:check` in `docs`, and ran `git diff --check`. - Observed result: Typecheck completed with `tsc --noEmit`; the OpenCode package test suite reported 2 files and 9 tests passed; docs type generation completed successfully; `git diff --check` produced no output. - Not tested: Browser-rendered documentation preview. `docs: npm run build` was started locally but produced no output for roughly 90 seconds and was stopped, so this follow-up does not claim a fresh local docs build result. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes - The linked review comment asked for README examples to match `compressWithHeadroom(messages, options)` and `createHeadroomRetrieveTool` requiring `hash`; both snippets now match the exported API. |
||
|
felixboenkost-droid
|
6d116b15f1
|
Harden OpenClaw plugin proxy routing (#1074)
## Description Hardens the bundled OpenClaw plugin so configured proxy routing is fail-closed and `autoStart` is opt-in. Closes: N/A This follow-up is intentionally separate from the ContentRouter cache fix because it changes plugin/gateway behavior rather than core compression routing. The plugin should not mutate upstream provider routing unless a configured proxy URL is reachable and looks like Headroom. It should also avoid unhandled startup promise rejections when proxy startup is fire-and-forget. Why this shape: - `autoStart: false` by default matches deployments where Headroom is supervised externally, for example by systemd. The plugin should not silently start or assume ownership of a proxy unless the operator opted in. - Provider routing is fail-closed: a configured URL must first respond like Headroom, not merely expose a generic liveness endpoint. This prevents accidentally routing model traffic through the wrong local service. - `/readyz` is treated as liveness, not identity. Identity comes from Headroom-shaped stats endpoints (`/v1/retrieve/stats` or `/stats`) because those are harder for unrelated services to satisfy by accident. - Startup remains asynchronous, but errors are captured and exposed instead of becoming unhandled promise rejections. - This is a separate PR because the core cache fix is about compression correctness, while this patch is about integration safety around OpenClaw gateway routing. ## Type of Change - [x] Bug fix (non-breaking change fixes issue) - [ ] New feature (non-breaking change adds functionality) - [ ] Breaking change (fix or feature would cause existing functionality change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Make proxy `autoStart` opt-in (`default: false`). - Probe configured `proxyUrl` before applying provider routing. - Treat `/readyz` as liveness only; require Headroom-shaped `/v1/retrieve/stats` or `/stats` for identity. - Observe fire-and-forget startup promise rejection and expose startup error for callers. - Isolate proxy-ready listener failures. - Keep provider routing deferred when no active/probed Headroom proxy exists. - Register retrieve tool with explicit `headroom_retrieve` name. - Extend plugin/unit tests for configured proxy failures, generic non-Headroom endpoints, path collisions, and routing behavior. Changed files: - `plugins/openclaw/README.md` - `plugins/openclaw/openclaw.plugin.json` - `plugins/openclaw/src/engine.ts` - `plugins/openclaw/src/plugin/index.ts` - `plugins/openclaw/src/proxy-manager.ts` - `plugins/openclaw/test/engine.test.ts` - `plugins/openclaw/test/gateway-config.test.ts` - `plugins/openclaw/test/plugin-runtime-routing.test.ts` - `plugins/openclaw/test/proxy-manager.test.ts` ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added new functionality - [x] Manual testing performed ### Test Output ```text $ npm test Test Files 6 passed (6) Tests 74 passed (74) $ npm run typecheck tsc --noEmit $ npm run build tsup && node prepare-dist.mjs Build success ``` ## Real Behavior Proof - Environment: local OpenClaw plugin package in the Headroom repo. - Exact command / steps: - Run plugin test suite. - Run TypeScript typecheck. - Run plugin build. - Observed result: - Tests passed: `74/74`. - Typecheck passed. - Build passed. - Not tested: - Full OpenClaw Gateway integration as part of this standalone PR prep. ## Review Readiness - [x] I performed self-review - [x] This PR ready for human review ## Checklist - [x] My code follows project's style guidelines - [x] I performed self-review my code - [ ] I commented my code, particularly in hard-to-understand areas - [x] I made corresponding changes documentation - [x] My changes generate no new warnings - [x] I added tests prove fix is effective or feature works - [x] New and existing unit tests pass locally my changes - [ ] I updated CHANGELOG.md if applicable ## Screenshots (if applicable) N/A. ## Additional Notes Checklist items left unchecked intentionally: - No CHANGELOG update included. - No extra comments were needed beyond existing code structure. Co-authored-by: Björn-Christian Bönkost <bjoern@v2202603344248440850.hotsrv.de> |
||
|
Rudimar Ronsoni
|
b4571cc346
|
feat: headroom wrap opencode / unwrap opencode CLI (#1105)
## Summary This PR implements transparent `headroom wrap opencode` support without asking users to edit OpenCode provider URLs, choose an extra CLI flag, or maintain a static provider list. The wrapper now lives at the runtime transport boundary: OpenCode keeps its user/provider config, while Headroom intercepts outbound provider traffic in-process and routes it through the local Headroom proxy. ## What changed ### Transparent OpenCode wrapping - `headroom wrap opencode` injects the `headroom-opencode` plugin through `OPENCODE_CONFIG_CONTENT`. - Existing OpenCode provider URLs are preserved. We do not rewrite user config URLs to point at Headroom. - Existing `OPENAI_BASE_URL` and `ANTHROPIC_BASE_URL` env vars are preserved. - Local OpenCode traffic, localhost traffic, and Headroom proxy traffic bypass the shim to avoid loops. ### Runtime transport interception - Added an OpenCode plugin transport shim that wraps: - `globalThis.fetch` - `http.request` / `http.get` - `https.request` / `https.get` - External provider calls are routed to the local Headroom proxy. - The original upstream origin is passed through `x-headroom-base-url`, so the proxy can forward to the real provider without changing OpenCode config. - External `http2.connect` is blocked loudly instead of allowing direct provider traffic to leak outside Headroom. ### Live provider additions Provider coverage is no longer based on a static config scan. Because routing happens at outbound request time, providers added mid-session are routed through Headroom automatically as long as they use the covered Node transport paths. ### Subagent and child-process coverage - The parent OpenCode plugin sets a packaged Node preload shim through `NODE_OPTIONS=--import=.../hook-shim/handler.js`. - The transport shim patches `child_process.spawn`, `exec`, `execFile`, and `fork` so child Node processes receive the Headroom preload even when OpenCode passes a custom `env`. - The child-process shim fails closed if it loads without `HEADROOM_OPENCODE_TRANSPORT_PROXY_URL`. - This closes the subagent leak path where a child Node process could otherwise start without Headroom transport interception. ## Why this goes beyond PR #1089 PR #1089 improves OpenCode provider registration, but it still focuses on provider config shape. This PR moves the enforcement boundary to runtime transport interception. This PR goes further because: - No provider URL rewriting is required. - New providers added mid-session are covered automatically. - Subagents and child Node processes inherit the Headroom transport shim. - Direct external HTTP/2 paths fail loudly instead of leaking. - The wrap remains transparent to the user's OpenCode provider config. - The wrapper is fail-closed for unsupported child-process preload state. ## Additional robustness fixes While validating the change in Docker, the full Python suite exposed unrelated Linux/container robustness issues. These are fixed in this PR so the suite is green: - Binary cache handling now treats cache paths under a non-writable existing parent as unavailable, including when tests run as root in Docker. - `release_version.py` honors `MANUAL_VER` before git calls so direct script execution works outside a `.git` checkout. - Test logger isolation now resets relevant Headroom child loggers so proxy logging setup cannot poison later `caplog` tests. - The scanner missing-path test now uses a guaranteed missing `tmp_path` child instead of relying on `/nonexistent/path`. ## Validation All implementation validation was run inside Docker. - Full Python suite from a fresh Docker copy: `6605 passed, 523 skipped`. - Ruff on changed Python/OpenCode paths: passed. - OpenCode plugin typecheck: passed. - OpenCode plugin tests: `9 passed`. - OpenCode plugin build: passed. - Hook shim preload smoke test: passed. ## Notes This PR intentionally does not add a CLI option. `headroom wrap opencode` means full wrap. Either Headroom wraps OpenCode transparently, or the path fails loudly instead of silently leaking provider traffic. --------- Co-authored-by: Rudimar Ronsoni <6081613+rudironsoni@users.noreply.github.com> |
||
|
github-actions[bot]
|
95b2333ee5
|
chore: release main (#1274)
🤖 I have created a release *beep* *boop* --- <details><summary>0.27.0</summary> ## [0.27.0](https://github.com/chopratejas/headroom/compare/v0.26.0...v0.27.0) (2026-06-22) ### Features * **cli:** add headroom doctor setup diagnostics ([#926](https://github.com/chopratejas/headroom/issues/926)) ([ |
||
|
problemsolverai2026-svg
|
2e6c442dc8
|
fix(openclaw): wrap plugin export as {register} object for OpenClaw 2026.x compatibility (#1218)
## Description
The `headroom-openclaw` plugin silently fails to load on OpenClaw
2026.x. OpenClaw's plugin loader calls `setupRegistration.register(api)`
on the plugin's default export. The current plugin exports a **bare
function** as its default — which has no `.register()` method — so the
loader skips it silently and the plugin never initializes. No error is
thrown, no warning logged. The plugin appears "enabled" in the registry
but does nothing.
Fix: wrap the function in a `{ register: headroomPlugin }` object. One
structural change, plugin body unchanged.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Wrapped `export default function headroomPlugin(api)` in a `{
register: headroomPlugin }` object so OpenClaw's loader can call
`.register(api)` on it
## Testing
- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ node -e "
const mod = require('./plugins/openclaw/dist/index.js');
const plugin = mod.default;
console.log('type:', typeof plugin);
console.log('has register:', typeof plugin.register);
const mockApi = {
config: { plugins: { entries: { headroom: { config: { proxyUrl: 'http://127.0.0.1:8787' } } } } },
logger: { info: (m) => console.log('INFO:', m), warn: console.warn, error: console.error },
registerContextEngine: (id) => console.log('registerContextEngine:', id),
registerTool: () => console.log('registerTool called'),
on: (e) => console.log('on:', e),
};
plugin.register(mockApi);
"
type: object
has register: function
registerContextEngine: headroom
registerTool called
on: gateway_start
INFO: [headroom] Plugin registered
INFO: Headroom proxy ready at http://127.0.0.1:8787
```
## Real Behavior Proof
- Environment: macOS 15.x arm64, OpenClaw 2026.6.8, Node.js v25.8.0,
headroom-openclaw 0.1.0
- Exact command / steps: `openclaw plugins install headroom-ai/openclaw`
→ restart OpenClaw → check gateway logs for `[headroom]` entries → check
`curl http://localhost:8787/stats` for `api_requests > 0`
- Observed result: Before fix — no `[headroom]` log entries, proxy never
started, `api_requests: 0`. After fix — `[headroom] Plugin registered`
in gateway log, proxy starts, `api_requests: 135`, `$10.11` saved in one
session.
- Observed result (after fix): `[headroom] Plugin registered` in gateway
log. Proxy starts on port 8787. Real Anthropic API calls intercepted —
`/stats` showed `api_requests: 135`, `total_saved_usd: 10.11` after one
session.
- Not tested: Windows, Linux
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
Found while integrating headroom into
[Vespera](https://github.com/problemsolverai2026-svg/vespera), a
persistent local AI system. The workaround was manually setting
`models.providers.anthropic.baseUrl = "http://127.0.0.1:8787"` in
OpenClaw config — which works but bypasses the plugin entirely. The
proper plugin path should work out of the box.
|
||
|
dependabot[bot]
|
75f81cd19f
|
ci: bump the npm_and_yarn group across 3 directories with 3 updates (#1056)
[//]: # (dependabot-start) ⚠️ **Dependabot is rebasing this PR** ⚠️ Rebasing might not happen immediately, so don't worry if this takes some time. Note: if you make any changes to this PR yourself, they will take precedence over the rebase. --- [//]: # (dependabot-end) Bumps the npm_and_yarn group with 1 update in the /docs directory: [js-yaml](https://github.com/nodeca/js-yaml). Bumps the npm_and_yarn group with 1 update in the /plugins/openclaw directory: [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite). Bumps the npm_and_yarn group with 2 updates in the /sdk/typescript directory: [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) and [form-data](https://github.com/form-data/form-data). Updates `js-yaml` from 4.1.1 to 4.2.0 <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md">js-yaml's changelog</a>.</em></p> <blockquote> <h2>[4.2.0] - 2026-06-01</h2> <h3>Added</h3> <ul> <li>Added <code>docs/safety.md</code> with notes about processing untrusted YAML.</li> <li>Added <code>maxDepth</code> (100) loader option. Not a problem, but gives a better exception instead of RangeError on stack overflow.</li> <li>Added <code>maxMergeSeqLength</code> (20) loader option. Not a problem after <code>merge</code> fix, but an additional restriction for safety.</li> <li>Added sourcemaps to <code>dist/</code> builds.</li> </ul> <h3>Changed</h3> <ul> <li>Stop resolving numbers with underscores as numeric scalars, <a href="https://redirect.github.com/nodeca/js-yaml/issues/627">#627</a>.</li> <li>Switched dev toolchains to Vite / neostandard.</li> <li>Updated demo.</li> <li>Reorganized tests.</li> <li><code>dist/</code> files are no longer kept in the repository.</li> </ul> <h3>Fixed</h3> <ul> <li>Fix parsing of properties on the first implicit block mapping key, <a href="https://redirect.github.com/nodeca/js-yaml/issues/62">#62</a>.</li> <li>Fix trailing whitespace handling when folding flow scalar lines, <a href="https://redirect.github.com/nodeca/js-yaml/issues/307">#307</a>.</li> <li>Reject top-level block scalars without content indentation, <a href="https://redirect.github.com/nodeca/js-yaml/issues/280">#280</a>.</li> <li>Ensure numbers survive round-trip, <a href="https://redirect.github.com/nodeca/js-yaml/issues/737">#737</a>.</li> <li>Fix test coverage for issue <a href="https://redirect.github.com/nodeca/js-yaml/issues/221">#221</a>.</li> <li>Fix flow scalar trailing whitespace folding, <a href="https://redirect.github.com/nodeca/js-yaml/issues/307">#307</a>.</li> <li>Fix digits in YAML named tag handles.</li> </ul> <h3>Security</h3> <ul> <li>Fix potential DoS via quadratic complexity in merge - deduplicate repeated elements (makes sense for malformed files > 10K).</li> </ul> <h2>[3.14.2] - 2025-11-15</h2> <h3>Security</h3> <ul> <li>Backported v4.1.1 fix to v3</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/nodeca/js-yaml/commits">compare view</a></li> </ul> </details> <br /> Updates `vite` from 8.0.10 to 8.0.16 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/vitejs/vite/releases">vite's releases</a>.</em></p> <blockquote> <h2>v8.0.16</h2> <p>Please refer to <a href="https://github.com/vitejs/vite/blob/v8.0.16/packages/vite/CHANGELOG.md">CHANGELOG.md</a> for details.</p> <h2>v8.0.15</h2> <p>Please refer to <a href="https://github.com/vitejs/vite/blob/v8.0.15/packages/vite/CHANGELOG.md">CHANGELOG.md</a> for details.</p> <h2>v8.0.14</h2> <p>Please refer to <a href="https://github.com/vitejs/vite/blob/v8.0.14/packages/vite/CHANGELOG.md">CHANGELOG.md</a> for details.</p> <h2>v8.0.13</h2> <p>Please refer to <a href="https://github.com/vitejs/vite/blob/v8.0.13/packages/vite/CHANGELOG.md">CHANGELOG.md</a> for details.</p> <h2>v8.0.12</h2> <p>Please refer to <a href="https://github.com/vitejs/vite/blob/v8.0.12/packages/vite/CHANGELOG.md">CHANGELOG.md</a> for details.</p> <h2>v8.0.11</h2> <p>Please refer to <a href="https://github.com/vitejs/vite/blob/v8.0.11/packages/vite/CHANGELOG.md">CHANGELOG.md</a> for details.</p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/vitejs/vite/blob/main/packages/vite/CHANGELOG.md">vite's changelog</a>.</em></p> <blockquote> <h2><!-- raw HTML omitted --><a href="https://github.com/vitejs/vite/compare/v8.0.15...v8.0.16">8.0.16</a> (2026-06-01)<!-- raw HTML omitted --></h2> <h3>Bug Fixes</h3> <ul> <li><strong>deps:</strong> reject UNC paths for launch-editor-middleware (<a href="https://redirect.github.com/vitejs/vite/issues/22571">#22571</a>) (<a href=" |
||
|
github-actions[bot]
|
b81a4a7a16
|
chore: release main (#931)
🤖 I have created a release *beep* *boop* --- <details><summary>0.26.0</summary> ## [0.26.0](https://github.com/chopratejas/headroom/compare/v0.25.0...v0.26.0) (2026-06-16) ### Features * add Copilot BYOK provider wrapper utilities and CLI support ([#1041](https://github.com/chopratejas/headroom/issues/1041)) ([ |
||
|
Tim Poppe
|
7c8c909c85
|
fix(openclaw): declare headroom_retrieve tool contract (#947)
## Summary - add `contracts.tools` to the OpenClaw plugin manifest - declare `headroom_retrieve` so the manifest matches the tool registered at runtime - remove the OpenClaw `contracts.tools` warning during startup ## Testing - npm test - npm run typecheck <!-- headroom-maintainer-template-completion:start --> ## Description This PR prepares `fix(openclaw): declare headroom_retrieve tool contract` for review by documenting the intended change, validation evidence, and remaining merge-readiness context. Linked issues: None declared. ## Type of Change - [x] Bug fix - [ ] New feature - [ ] Documentation - [ ] Refactor - [ ] Tests only ## Changes Made - Commit: fix(openclaw): declare headroom_retrieve tool contract - Touches `plugins/openclaw/openclaw.plugin.json` ## Testing - [x] GitHub checks reviewed - [x] Metadata/template validation - [ ] Local functional testing ### Test Output ```text gh pr view 947 --repo chopratejas/headroom --json statusCheckRollup - PR Governance / template: FAILURE - PR Governance / label: SUCCESS - external / GitGuardian Security Checks: SUCCESS ``` ## Real Behavior Proof - Environment: GitHub PR metadata and checks for `chopratejas/headroom` PR #947. - Exact command / steps: Reviewed PR title, commits, changed files, linked issues, labels, and check rollup; appended this maintainer template completion block without replacing the author's original description. - Observed result: PR body now contains all required governance sections, checked readiness fields, and a non-placeholder validation evidence block. - Not tested: This pass updated PR metadata only; code validation remains represented by the linked GitHub checks and any author-provided evidence above. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review <!-- headroom-maintainer-template-completion:end --> |
||
|
dependabot[bot]
|
4f59097045
|
ci: bump esbuild from 0.27.7 to 0.28.1 in /docs in the npm_and_yarn group across 1 directory (#936)
Bumps the npm_and_yarn group with 1 update in the /docs directory: [esbuild](https://github.com/evanw/esbuild). Updates `esbuild` from 0.27.7 to 0.28.1 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/evanw/esbuild/releases">esbuild's releases</a>.</em></p> <blockquote> <h2>v0.28.1</h2> <ul> <li> <p>Disallow <code>\</code> in local development server HTTP requests (<a href="https://github.com/evanw/esbuild/security/advisories/GHSA-g7r4-m6w7-qqqr">GHSA-g7r4-m6w7-qqqr</a>)</p> <p>This release fixes a security issue where HTTP requests to esbuild's local development server could traverse outside of the serve directory on Windows using a <code>\</code> backslash character. It happened due to the use of Go's <code>path.Clean()</code> function, which only handles Unix-style <code>/</code> characters. HTTP requests with paths containing <code>\</code> are no longer allowed.</p> <p>Thanks to <a href="https://github.com/dellalibera"><code>@dellalibera</code></a> for reporting this issue.</p> </li> <li> <p>Add integrity checks to the Deno API (<a href="https://github.com/evanw/esbuild/security/advisories/GHSA-gv7w-rqvm-qjhr">GHSA-gv7w-rqvm-qjhr</a>)</p> <p>The previous release of esbuild added integrity checks to esbuild's npm install script. This release also adds integrity checks to esbuild's Deno install script. Now esbuild's Deno API will also fail with an error if the downloaded esbuild binary contains something other than the expected content.</p> <p>Note that esbuild's Deno API installs from <code>registry.npmjs.org</code> by default, but allows the <code>NPM_CONFIG_REGISTRY</code> environment variable to override this with a custom package registry. This change means that the esbuild executable served by <code>NPM_CONFIG_REGISTRY</code> must now match the expected content.</p> <p>Thanks to <a href="https://github.com/sondt99"><code>@sondt99</code></a> for reporting this issue.</p> </li> <li> <p>Avoid inlining <code>using</code> and <code>await using</code> declarations (<a href="https://redirect.github.com/evanw/esbuild/issues/4482">#4482</a>)</p> <p>Previously esbuild's minifier sometimes incorrectly inlined <code>using</code> and <code>await using</code> declarations into subsequent uses of that declaration, which then fails to dispose of the resource correctly. This bug happened because inlining was done for <code>let</code> and <code>const</code> declarations by avoiding doing it for <code>var</code> declarations, which no longer worked when more declaration types were added. Here's an example:</p> <pre lang="js"><code>// Original code { using x = new Resource() x.activate() } <p>// Old output (with --minify)<br /> new Resource().activate();</p> <p>// New output (with --minify)<br /> {using e=new Resource;e.activate()}<br /> </code></pre></p> </li> <li> <p>Fix module evaluation when an error is thrown (<a href="https://redirect.github.com/evanw/esbuild/issues/4461">#4461</a>, <a href="https://redirect.github.com/evanw/esbuild/pull/4467">#4467</a>)</p> <p>If an error is thrown during module evaluation, esbuild previously didn't preserve the state of the module for subsequent module references. This was observable if <code>import()</code> or <code>require()</code> is used to import a module multiple times. The thrown error is supposed to be thrown by every call to <code>import()</code> or <code>require()</code>, not just the first. With this release, esbuild will now throw the same error every time you call <code>import()</code> or <code>require()</code> on a module that throws during its evaluation.</p> </li> <li> <p>Fix some edge cases around the <code>new</code> operator (<a href="https://redirect.github.com/evanw/esbuild/issues/4477">#4477</a>)</p> <p>Previously esbuild incorrectly printed certain edge cases involving complex expressions inside the target of a <code>new</code> expression (specifically an optional chain and/or a tagged template literal). The generated code for the <code>new</code> target was not correctly wrapped with parentheses, and either contained a syntax error or had different semantics. These edge cases have been fixed so that they now correctly wrap the <code>new</code> target in parentheses. Here is an example of some affected code:</p> <pre lang="js"><code>// Original code new (foo()`bar`)() new (foo()?.bar)() <p>// Old output<br /> new foo()<code>bar</code>();<br /> new (foo())?.bar();</p> <p></code></pre></p> </li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/evanw/esbuild/blob/main/CHANGELOG.md">esbuild's changelog</a>.</em></p> <blockquote> <h2>0.28.1</h2> <ul> <li> <p>Disallow <code>\</code> in local development server HTTP requests (<a href="https://github.com/evanw/esbuild/security/advisories/GHSA-g7r4-m6w7-qqqr">GHSA-g7r4-m6w7-qqqr</a>)</p> <p>This release fixes a security issue where HTTP requests to esbuild's local development server could traverse outside of the serve directory on Windows using a <code>\</code> backslash character. It happened due to the use of Go's <code>path.Clean()</code> function, which only handles Unix-style <code>/</code> characters. HTTP requests with paths containing <code>\</code> are no longer allowed.</p> <p>Thanks to <a href="https://github.com/dellalibera"><code>@dellalibera</code></a> for reporting this issue.</p> </li> <li> <p>Add integrity checks to the Deno API (<a href="https://github.com/evanw/esbuild/security/advisories/GHSA-gv7w-rqvm-qjhr">GHSA-gv7w-rqvm-qjhr</a>)</p> <p>The previous release of esbuild added integrity checks to esbuild's npm install script. This release also adds integrity checks to esbuild's Deno install script. Now esbuild's Deno API will also fail with an error if the downloaded esbuild binary contains something other than the expected content.</p> <p>Note that esbuild's Deno API installs from <code>registry.npmjs.org</code> by default, but allows the <code>NPM_CONFIG_REGISTRY</code> environment variable to override this with a custom package registry. This change means that the esbuild executable served by <code>NPM_CONFIG_REGISTRY</code> must now match the expected content.</p> <p>Thanks to <a href="https://github.com/sondt99"><code>@sondt99</code></a> for reporting this issue.</p> </li> <li> <p>Avoid inlining <code>using</code> and <code>await using</code> declarations (<a href="https://redirect.github.com/evanw/esbuild/issues/4482">#4482</a>)</p> <p>Previously esbuild's minifier sometimes incorrectly inlined <code>using</code> and <code>await using</code> declarations into subsequent uses of that declaration, which then fails to dispose of the resource correctly. This bug happened because inlining was done for <code>let</code> and <code>const</code> declarations by avoiding doing it for <code>var</code> declarations, which no longer worked when more declaration types were added. Here's an example:</p> <pre lang="js"><code>// Original code { using x = new Resource() x.activate() } <p>// Old output (with --minify)<br /> new Resource().activate();</p> <p>// New output (with --minify)<br /> {using e=new Resource;e.activate()}<br /> </code></pre></p> </li> <li> <p>Fix module evaluation when an error is thrown (<a href="https://redirect.github.com/evanw/esbuild/issues/4461">#4461</a>, <a href="https://redirect.github.com/evanw/esbuild/pull/4467">#4467</a>)</p> <p>If an error is thrown during module evaluation, esbuild previously didn't preserve the state of the module for subsequent module references. This was observable if <code>import()</code> or <code>require()</code> is used to import a module multiple times. The thrown error is supposed to be thrown by every call to <code>import()</code> or <code>require()</code>, not just the first. With this release, esbuild will now throw the same error every time you call <code>import()</code> or <code>require()</code> on a module that throws during its evaluation.</p> </li> <li> <p>Fix some edge cases around the <code>new</code> operator (<a href="https://redirect.github.com/evanw/esbuild/issues/4477">#4477</a>)</p> <p>Previously esbuild incorrectly printed certain edge cases involving complex expressions inside the target of a <code>new</code> expression (specifically an optional chain and/or a tagged template literal). The generated code for the <code>new</code> target was not correctly wrapped with parentheses, and either contained a syntax error or had different semantics. These edge cases have been fixed so that they now correctly wrap the <code>new</code> target in parentheses. Here is an example of some affected code:</p> <pre lang="js"><code>// Original code new (foo()`bar`)() new (foo()?.bar)() <p>// Old output<br /> new foo()<code>bar</code>();<br /> new (foo())?.bar();<br /> </code></pre></p> </li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
github-actions[bot]
|
8a53c8ec3b
|
chore: release main (#891)
🤖 I have created a release *beep* *boop* --- <details><summary>0.25.0</summary> ## [0.25.0](https://github.com/chopratejas/headroom/compare/v0.24.0...v0.25.0) (2026-06-12) ### Features * add differential network capture harness ([#761](https://github.com/chopratejas/headroom/issues/761)) ([ |
||
|
jimu
|
058bcedab8
|
feat(plugins): Hermes agent headroom_retrieve plugin (#824)
## Summary Implements the Hermes-side retrieval plugin proposed in #796 (as invited — thanks for the quick response!). When Hermes routes traffic through `headroom proxy`, compressed markers are a one-way street: Hermes registers its own tools, so it never gets the `headroom_retrieve` CCR tool that Claude Code receives via MCP injection. In practice the model either re-runs the original command or — observed in the wild — treats `ccr:abc123` as a file path and tries to `cat` it. This plugin uses Hermes's user-plugin system (`~/.hermes/plugins/`) to register a native `headroom_retrieve` tool that calls the proxy's `POST /v1/retrieve` endpoint. ## What's included - `plugins/hermes/headroom_retrieve/` — `plugin.yaml` + `__init__.py` (single-file, httpx, ~100 lines) - `plugins/hermes/README.md` — install steps and proxy-side recommendations ## Design notes - **Both marker formats covered**: Kompress emits `[N items compressed ... hash=KEY]`, SmartCrusher's opaque-blob walker emits `<<ccr:HASH[,KIND,SIZE]>>`. The tool description teaches both and explicitly says markers are NOT file paths; the handler normalizes whole-marker input (`<<ccr:abc,base64,4.5KB>>` → `abc`). - **Re-compression loop guard**: retrieved originals travel back through the proxy on the next request and get re-compressed into a fresh marker, looping forever. README documents `HEADROOM_EXCLUDE_TOOLS=read_file,headroom_retrieve` as the fix (Hermes tool names don't match `DEFAULT_EXCLUDE_TOOLS`, which targets Claude Code's `Read`/`Grep`/...). - **Actionable failure modes**: 404 (TTL expired / proxy restarted) and connection-refused both return guidance to re-run the original command rather than retry. ## Relationship to existing PRs Complementary to #707 / #556 (`headroom wrap hermes`, proxy-side): those launch/route Hermes through the proxy; this gives the agent the retrieval capability once it's routed. Notably #707 disables CCR tool injection in Hermes mode precisely because Hermes must register its own tool — this plugin is that registration. ## Testing Running in production on macOS (headroom 0.23.0, pipx) and Linux (0.22.4, systemd) for a day. Verified: fresh-marker retrieval roundtrip, whole-marker hash normalization (6 input shapes), expired-hash 404 messaging, proxy-down messaging, and end-to-end via live Hermes sessions (fresh ≥500B `read_file` returns original with the documented exclude config). Closes #796 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: akb4q <zhunyunjiang@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Khalid Shaikh
|
eb2e50feb2
|
feat: add OAuth2 client-credentials upstream-auth proxy extension (#778) (#784)
## What & why
Adds **`headroom-oauth2`** under `plugins/` — a generic, vendor-neutral
proxy extension that mints an OAuth2 **client-credentials** (RFC 6749
§4.4) bearer from a configured token endpoint and injects it as the
upstream `Authorization` on each proxied request, via the opt-in
`headroom.proxy_extension` seam. **No core changes.**
This lets headroom front any gateway that requires a *minted,
short-lived machine token* rather than a static API key. It complements
**#510** (env-var/static-key auth) rather than replacing it.
Implements **#778** (feature request). Opening the implementation
alongside the issue so there's something concrete to react to — **happy
to hold/rework pending a 👍 from a maintainer**, per CONTRIBUTING.
## Spec
Full spec in
[`plugins/headroom-oauth2/SPEC.md`](plugins/headroom-oauth2/SPEC.md)
(API surface, behavior/compat, user stories, failure modes, resilience
incl. multi-process, security, observability, rollback). Highlights:
- **Opt-in & no-op by default:** dormant until `--proxy-extension
oauth2`, and a no-op unless `HEADROOM_OAUTH2_TOKEN_URL` is set. No
change to defaults, body, routing, or compression.
- **Config is 100% env** (no new CLI flags):
token_url/client_id/secret/scopes/audience, RFC 8707 `resource`,
`post`|`basic` auth style, static upstream headers, timeout/skew.
- **Token caching + single-flight refresh**; `expires_in` clamped to a
positive TTL.
- **Fails closed** on misconfig; returns `502 upstream_auth_error` on
mint failure **without leaking the IdP error body**; `token_url` is
**https-enforced** (loopback exempt for tests).
- **Standard-library only** (token minted via `urllib` → system cert
store, so it works behind corporate SSL inspection). `litellm` is
touched only for static headers and is an optional extra, not a core
dep.
- **Effective for** OpenAI-compatible / passthrough litellm backends.
`bedrock`/`vertex`/`sagemaker` auth from env and ignore a forwarded
bearer → the extension **warns loudly** and is a no-op there.
## Tests
37 tests covering behavior **and** failure modes (`ruff check`/`format`
clean, **98% coverage**): post/basic mint, caching, single-flight (cold
+ on-refresh, exact mint counts under concurrency), https enforcement +
`localhost` rejection + `::1`, `expires_in`
clamp/float/missing/non-numeric, `extra_params` cannot clobber canonical
fields, bad-status/non-JSON/no-token/unreachable (asserting no
secret/body leak), ASGI middleware (inject, non-http passthrough, 502 +
`no-store`, missing `headers` key), and `install()`
(no-op/fail-closed/bad-timeout/env-auth-backend-warning/static-headers).
## Real behavior proof
- **Setup:** Linux aarch64, Python 3.13.5, `headroom-ai` 0.23.0, real
`headroom proxy` process.
- **Steps:** started `headroom proxy --backend litellm-openai
--proxy-extension oauth2` with `HEADROOM_OAUTH2_*` env pointed at a
local OAuth2 token endpoint; an upstream echo server captured what the
backend received; sent two `/v1/messages` requests through the proxy.
- **Observed (copied output):**
```
PROXY: headroom-oauth2: client-credentials auth installed
(token_url=…/token, style=post)
MINTS (across 2 requests): 1 # token cached + reused -> 1 mint for 2
requests
UPSTREAM RECEIVED: auth=Bearer MINTED-FROM-IDP-…
static=generic-static-header
SECRET LEAK CHECK (client_secret in proxy logs): 0
```
→ The minted bearer (not the placeholder backend key) and the configured
static header reached the upstream; the client's inbound credential was
replaced; the client secret never appeared in logs; caching worked.
- **What I did *not* test:** a live commercial IdP
(Entra/Okta/Auth0/etc.) and a live cloud gateway — the token endpoint
and upstream here are local stand-ins. Also not tested: multi-worker
(gunicorn) deployment, and Python 3.10/3.11 (developed on 3.13).
## Placement
Proposed as a standalone installable package under
`plugins/headroom-oauth2/` (registers via the entry-point seam; `pip
install -e plugins/headroom-oauth2`). Open to baking it into core or
publishing it separately — maintainer's call.
|
||
|
github-actions[bot]
|
01762b1ec7
|
chore: release main (#607)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
github-actions[bot]
|
f7c2552264
|
chore: release main | ||
|
github-actions[bot]
|
cc71e07d01
|
chore: release main | ||
|
chopratejas
|
8e4ab187e9 |
ci(release): align manifest + pyproject + package.json to 0.22.3
The repo had drifted: pyproject.toml said 0.9.1 but PyPI's latest
published headroom-ai was 0.22.3. release_version.py papered over
this by taking max(canonical, latest_tag) at release time;
release-please does NOT do that — it trusts the manifest verbatim.
Left as-is, release-please would propose 0.9.2 on the next merge
and PyPI would reject it ("400 Cannot publish version lower than
latest"), looping the bot forever.
Fix: align every version-bearing file to 0.22.3 (the truth on
PyPI). Done via `scripts/version-sync.py --version 0.22.3`:
- .release-please-manifest.json
- pyproject.toml
- sdk/typescript/package.json
- plugins/openclaw/package.json (+ headroom-ai dep range -> ^0.22.3)
- .claude-plugin/marketplace.json
- .github/plugin/marketplace.json
- plugins/headroom-agent-hooks/.claude-plugin/plugin.json
- plugins/headroom-agent-hooks/.github/plugin/plugin.json
After this lands, the bot's next release PR will propose 0.22.4
(patch) or 0.23.0 (minor) depending on conventional-commit traffic
since v0.22.3.
|
||
|
chopratejas
|
3ec549288a |
fix(proxy): thread tags into 13 outcome sites + synth /v1/models + free-fn _extract_tags
Three fixes bundled; all in admin / cache-hit paths where tests didn't catch the regression. ## (A) 13 RequestOutcome sites missing tags= An AST audit found that 13 of 21 ``RequestOutcome(...)`` construction sites across the four handler files emitted outcomes without threading ``tags=``. Affected paths: * ``handle_anthropic_messages`` — the ``from_response_cache=True`` early-return outcome (Claude Code cache-hit turns dashboard-blind) * ``handle_openai_chat`` — same cache-hit early-return (Codex + Cursor + Continue cache-hit turns dashboard-blind) * ``handle_openai_responses_ws`` — the per-turn outcome inside the Codex WS session. The stale comment that said "ws_session_tags is not yet bound" was wrong — ``ws_tags`` was already extracted at handler entry * ``handle_anthropic_batch_create / batch_passthrough / batch_results`` * ``handle_passthrough`` (OpenAI Models / Files / List-Batches) * ``handle_google_batch_create / batch_passthrough / batch_results`` * ``_google_batch_passthrough`` (internal helper) * ``handle_batch_create`` (OpenAI batch entry) * ``handle_gemini_count_tokens`` (also fixed in #479; identical) Pattern of the fix is uniform: pull tags from headers and thread them into the ``RequestOutcome`` construction. New contract test ``test_handler_outcome_tag_invariant.py`` walks each handler file's AST and asserts every ``RequestOutcome`` site inside any ``handle_*`` or ``*_passthrough`` method passes both ``tags=`` and ``client=``. Future handlers get a clear test failure with file + line + method name if they regress. ## (B) Issue #478 — /v1/models 403 under Codex ChatGPT auth Codex Desktop with ChatGPT-subscription OAuth polls ``/v1/models`` to populate its model picker. Forwarding to ``chatgpt.com/backend-api/ models`` returned 403 to OAuth tokens. Fix: synthesize an OpenAI- compatible payload locally from a known-supported model set (``gpt-5.5`` through ``gpt-5``). All other ChatGPT-auth paths still forward as before — only model-metadata gets the local response. ## (C) Move _extract_tags to free function (mixin-isolation test compat) Handlers called ``self._extract_tags(headers)``. That worked in production where ``HeadroomProxy`` composes every mixin and defines the method, but broke tests that instantiate a single mixin via ``object.__new__(OpenAIHandlerMixin)``. The free-function form removes that coupling — handlers import ``extract_tags`` from ``headroom.proxy.helpers`` and call directly. ``HeadroomProxy. _extract_tags`` is kept as a thin wrapper for any external caller still using the method form. 17 call sites migrated. ## Zero behavior change for existing users Claude Code, Codex, Cursor, Continue, Aider, Gemini-routed harnesses all hit handlers that already extracted tags. Their wire bytes to upstream LLMs are byte-identical. Only the dashboard view gains tags on previously-blind paths. Closes #478. |
||
|
chopratejas
|
e4e28b65f4 |
fix(proxy): surface CompressionDecision.passthrough_reason in tags
Adds ``CompressionDecision.apply_to_tags(tags)`` — a one-liner mutator that stamps the passthrough reason into a tags dict for downstream observability. Each migrated handler now calls ``_decision.apply_to_tags(tags)`` immediately after ``CompressionDecision.decide(...)``. The tags dict flows unchanged into every downstream ``RequestOutcome(tags=tags, ...)`` construction, which the funnel surfaces in ``RequestLog.tags`` — same mechanism the funnel already uses for ``client``. Dashboards can now slice passthrough traffic by cause: * tags["passthrough_reason"] == "bypass_header" * tags["passthrough_reason"] == "compression_disabled" * tags["passthrough_reason"] == "no_messages" * tags["passthrough_reason"] == "license_denied" No-op when ``should_compress=True`` — compressing requests don't carry the tag, so absence vs presence is itself the signal. Bonus fix: ``handle_gemini_count_tokens`` was the one Gemini handler that never pulled tags out of headers, so its emitted ``RequestOutcome`` reached the dashboard without any of the per- request slicing keys. Added the missing ``tags = self._extract_tags (request.headers)`` and threaded ``tags=tags`` into its outcome. Closes the observability loop opened by PR #477: the four Gemini- bypass-bug fixes are now visible in the request-log feed the moment they fire. |
||
|
chopratejas
|
694589fec4 |
refactor(proxy): collapse 3 stream finalizers onto RequestOutcome.from_stream
Three streaming finalizers — ``_finalize_stream_response``,
``_stream_response_bedrock``, ``_stream_openai_via_backend`` — each
duplicated the same set of body- and config-derived fields when
constructing a ``RequestOutcome``:
* ``attempted_input_tokens = optimized_tokens + tokens_saved``
* ``num_messages = len(body.get("messages", []))``
* ``request_messages`` conditional on ``config.log_full_messages``
* ``transforms_applied`` list → tuple (frozen-dataclass contract)
* ``tags or {}`` normalization
* ``turn_id`` via ``compute_turn_id``
The last one was a real bug. Only the Bedrock site computed
``turn_id`` — sites 1 and 3 silently dropped it, breaking the
dashboard's multi-turn-session grouping for every Anthropic-SSE and
OpenAI-via-backend request. The new ``RequestOutcome.from_stream``
classmethod computes it uniformly so the three finalizers cannot
drift apart on derivation logic again.
Each call site now hands ``from_stream`` the body + provider-specific
cache/timing fields and gets a fully-constructed outcome back. The
funnel call after it stays identical (``await
self._record_request_outcome(outcome)``).
|
||
|
chopratejas
|
d73cbd6b0a |
fix(build): shrink Rust extension wheels — strip + thin-LTO + single codegen unit
PyPI rejected the v0.21.37 release publish with:
HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/
Project size too large. Limit for project 'headroom-ai' total size is 10 GB.
PyPI inventory check confirmed: **191 versions × ~213 MB/release =
10.00 GB exactly** — at the cumulative project storage ceiling. Each
recent release ships 12 wheels × ~16-18 MB each.
Post-mortem inspection of a production wheel
(``headroom_ai-0.21.36-cp311-cp311-manylinux_2_28_x86_64.whl``)
showed the binary was ``not stripped``:
.text 18.3 MB (code)
.rodata 11.4 MB (Magika model + ONNX runtime data)
.strtab 4.9 MB (debug strings — strippable)
.eh_frame 1.9 MB (unwind tables)
.symtab 1.5 MB (debug symbols — strippable)
.gcc_except_table 1.2 MB
This commit adds a release profile:
[profile.release]
strip = "symbols"
lto = "thin"
codegen-units = 1
That:
* Strips ``.symtab`` + ``.strtab`` (~6.4 MB direct savings per wheel)
* Enables thin link-time optimization for cross-crate dead-code
elimination (~5-10% ``.text`` savings)
* Single codegen unit for better inlining + DCE at the cost of
~30-50% slower release builds (acceptable for CI)
Deliberately NOT setting ``panic = "abort"``:
* The proxy is a long-lived async process. A panic on one bad
request triggering process abort would disconnect every concurrent
client. Accept the smaller savings; keep unwind behaviour.
Estimated impact
* Per wheel: ~16-18 MB → ~10-11 MB (40% smaller)
* Per release (12 wheels): ~213 MB → ~130 MB
* PyPI capacity: ~30+ more releases before hitting 10 GB again
Verification
* Local build of ``headroom._core`` with new profile:
``.so`` size 29 MB on macOS arm64 (was ~45 MB pre-fix; final wheel
compressed will be smaller on Linux which also benefits from the
``strip`` directive).
* 77 Rust-parity tests pass — extension still functional.
* Single-codegen-unit slows build by ~30-50% but maturin/cibuildwheel
build time was never the bottleneck.
Forward strategy (separate work)
* Submit a PyPI project-size-limit-increase request to unblock the
immediate release.
* Adopt a release-deprecation policy: yank versions older than N
patches per minor; consider dropping Python 3.10 wheels (EOL'd
October 2026) and manylinux_2_28_aarch64 wheels (niche audience,
largest at 18.75 MB).
* Investigate runtime-download for Magika model (~10 MB further
savings) — same pattern Kompress already uses.
|
||
|
chopratejas
|
07bcda2796 | chore: sync plugin versions to 0.21.33 | ||
|
chopratejas
|
3432ee3a96 | docs: add README redesign spec (lean-ctx parity + Headroom-first blend) | ||
|
Tejas Chopra
|
2b331e297a | fix: add Codex wire debug and WS usage metrics | ||
|
chopratejas
|
5dd2ac5e51 |
fix(cli): resolve duplicate --code-aware flag breaking proxy import
PR #411 reintroduced an older `--code-aware` is_flag option and a duplicate `code_aware_enabled=` kwarg in the ProxyConfig call, which collided with the canonical tristate `--code-aware/--no-code-aware` introduced in #260. The result: every CLI entry point (`headroom proxy`, `headroom wrap codex`, `headroom wrap claude`, etc.) raised at import time: File ".../headroom/cli/proxy.py", line 575 code_aware_enabled=code_aware or _get_env_bool(...) SyntaxError: keyword argument repeated: code_aware_enabled Removes: - The legacy `@click.option("--code-aware", is_flag=True, ...)` - The legacy `code_aware: bool` function parameter - The duplicate `code_aware_enabled=` kwarg Keeps the tristate `--code-aware/--no-code-aware` > env-var > default-off resolver. Behavior is unchanged for all flag combinations covered by tests/test_cli_proxy_env.py. Test mocks for `run_server` updated to accept `**kwargs` to match the real signature (config plus run-time options like print_banner). Without this the four code-aware tests added in #411 raised TypeError on each invocation. Plugin marketplace/manifest version bump 0.21.5 → 0.21.7 carried in this commit by the sync-plugin-versions pre-commit hook. |
||
|
chopratejas
|
89f7b6c2dd |
fix: complete /v1/responses compression telemetry, multi-frame WS, frozen-count cap
Builds on PR #406 (HTTP /v1/responses PyO3) and PR #410 (WS first-frame PyO3) which landed the binding for `compress_openai_responses_live_zone`. This change closes the remaining gaps so every (provider × endpoint × auth-mode × streaming) combination compresses AND surfaces in the dashboard. Telemetry: extend the PyO3 binding return tuple from `(bytes, modified)` to `(bytes, modified, tokens_saved, transforms_applied)` by adding `CompressionManifest::tokens_saved()` and `transforms_applied()` accessors on the existing manifest. The Python proxy populates request-log telemetry from the binding output instead of recounting tokens. Updates the existing 2-tuple call sites in HTTP and WS first-frame, plus the unpacks in tests. WebSocket multi-frame compression: subscription Codex users keep a long-lived WS open and send multiple `response.create` events per session. PR #410 only compressed the first frame; subsequent frames went raw. Added `_maybe_compress_response_create_frame` closure inside `_client_to_upstream` that runs the same Rust dispatcher on every client→upstream `response.create` text frame, passes other event types (response.cancel, session.update, etc.) through unchanged, and accumulates `tokens_saved` / `transforms_applied` / `ws_frames_compressed` counters across the session. Pre-existing dashboard gap: `streaming.py` and `anthropic.py` write `RequestLog` entries; the non-streaming OpenAI HTTP and WS handlers did not. Result: /transformations/feed was invisible for every Codex turn and every Cline / OpenClaude / Aider turn. Added the same wiring in `handle_openai_chat` (non-streaming), `handle_openai_responses` (non-streaming HTTP), and `handle_openai_responses_ws` (session-end). All three populate `auth_mode` + `endpoint` tags so the dashboard can break compression activity down by client class (PAYG / OAuth / Subscription) and surface (`chat_completions` / `responses_http` / `responses_ws`). The WS metric record is now unconditional — was previously gated on `tokens_saved > 0`, so first-frame no-changes never registered. compute_frozen_count over-freeze for prose-format clients: `compute_frozen_count` walked until it found an unstable `tool_result` / `role: "tool"` block. Cline / OpenClaude / Aider — clients that embed tool calls as XML inside plain text — never produce such a boundary, so the function returned `len(messages)` and the pipeline froze 100% of messages including the brand-new user turn. Live zone empty → `Transform content_router: 16414 → 16414 tokens (saved 0)`. Reported on Discord 2026-05-07 with Cline+DeepSeek. Fix: cap at `max(0, len(messages) - 1)`. Updates 3 existing test assertions whose expected values encoded the old over-freeze. Adds 6 new prose-format invariant tests. CodeQL "clear-text logging of sensitive information" fix: `tests/e2e_real_compression.py` previously stored API keys in local variables in the same scope as diagnostic prints, which CodeQL flagged via data-flow analysis. Refactored to read keys from `os.environ` inside the request helper — the credentials never enter the runner's main scope, so the taint flow never reaches the print. End-to-end verification with real keys (.env): /v1/messages (PAYG, non-stream) tok 14109 → 969 saved 13140 /v1/messages (PAYG, stream) tok 14109 → 969 saved 13140 /v1/chat/completions (PAYG, non-stream) tok 18460 → 1374 saved 17086 /v1/chat/completions (PAYG, stream) tok 18460 → 1374 saved 17086 (cache_hit=100%) /v1/responses HTTP (PAYG, non-stream) bytes 50138 → 597 saved 18391 /v1/responses WS (frame 1) bytes 46429 → 488 saved 16791 /v1/responses WS (frame 2 multi) bytes 46429 → 488 saved 16791 /v1/responses WS (response.cancel) passthrough untouched Tests: cargo workspace + pytest (4846 pass, 0 fail), make ci-precheck passed, two E2E scripts (multi-turn HTTP, WS fake-upstream) all green. |
||
|
chopratejas
|
c48735d029 |
fix(core): expose compress_openai_responses_live_zone via PyO3 (hot-fix c1/2)
PR-C5 (May 3) retired the Python `/v1/responses` compression pipeline with the comment "Rust handles item-aware compression natively" — but the standalone `crates/headroom-proxy` binary that was supposed to do that compression is not deployed by the CLI today (`headroom proxy` and `headroom wrap codex` both run only the Python proxy via uvicorn). Result: every `/v1/responses` request since v0.20.16 has been forwarded uncompressed. Codex CLI is the flagship consumer of this endpoint; this is the regression users have been reporting. Closes Bug 1 of the Codex regression by exposing the existing `headroom_core::transforms::compress_openai_responses_live_zone` as a PyO3 binding so the Python proxy can call the live-zone dispatcher in-process. The `headroom._core` extension is already loaded at proxy startup (PR-A0 verifies), so adding one more callable is mechanical. Why PyO3 inline (Layer 1) vs originally-intended two-process chain (Layer 2): the inline call requires zero deployment changes — the wheel already ships `headroom._core`. Layer 2 (build + ship the standalone `headroom-proxy` binary, teach CLI to spawn both processes) is the right long-term move; Layer 1 restores v0.5.21 functional behaviour today. # Returns `(body, modified)`. On change → `(new_body_bytes, True)`; on passthrough → `(input_bytes, False)`. # Failure mode Never raises. The dispatcher's `LiveZoneError` cases (body not JSON, no input array) are passthrough conditions matching the Rust proxy's `compress_openai_responses_request` contract. # Tests 14 new tests in `tests/test_responses_pyo3_compression.py`: binding exposed, passthrough cases, every F1 AuthMode variant, empty-model default, no-raise on garbage bytes. |
||
|
chopratejas
|
c090b617df |
chore(ci): drop ubuntu:20.04 + python 3.10 from smoke-import matrix
PR #396's first dry-run on its own changes failed three smoke matrix entries — two fixed by PR #397's __libc_single_threaded shim (now on main, this PR is rebased on top), one orthogonal: ubuntu:20.04 + 3.10. Failure mode on ubuntu:20.04 + 3.10: deadsnakes PPA install path stopped reliably provisioning python3.10-venv on focal once Ubuntu 20.04 hit End of Standard Support in May 2025. The error is from apt-get, NOT from `import headroom._core` — the wheel never gets a chance to load: E: Unable to locate package python3.10-venv Continuing to promise wheel-runtime correctness on glibc 2.31 in CI would require either pulling ESM-tier ubuntu:focal images (paid / auth-gated) or pinning a specific deadsnakes snapshot URL — neither of which we want owning long-term. Floor coverage we keep: - manylinux_2_28_x86_64 (glibc 2.28 — the floor we promise) - manylinux_2_28_aarch64 (glibc 2.28 — same) - ubuntu:22.04 + 3.12 x86_64 (glibc 2.35 — issue #355's reporter env) - ubuntu:22.04 + 3.12 aarch64 (glibc 2.35 — arm equivalent) - macos-14 + 3.13 (Apple Silicon native) Test pin in tests/test_release_workflows.py (test_release_workflow_has_smoke_import_wheel_gate) only requires the manylinux floors, ubuntu:22.04, and macos-14, so this drop doesn't break the structural invariant. |
||
|
chopratejas
|
af9e6282f8 |
fix(crusher): switch compression_store cache key from MD5 to SHA-256 (CodeQL #395)
CodeQL flagged `compression_store.store()`'s default MD5 cache key as `py/weak-sensitive-data-hashing` after the explicit_hash refactor in PR #395 brought the line into a new diff context. First attempt: `usedforsecurity=False` + `# lgtm[...]` comment to silence the alert without changing the hash. Both failed — CodeQL ignores the hashlib parameter and our LGTM marker, the alert stayed open on the PR. Second attempt (this commit): drop MD5 entirely. The cache key is for deduplication / lookup, not security or integrity, so any deterministic function works. SHA-256[:24] gives the same 96-bit collision space as MD5[:24] (~280 trillion entries for 50% collision under birthday bound), is FIPS-clean, and CodeQL won't flag it. Behaviour impact: zero. The cache is in-memory (no disk persistence), so the same content always hashes deterministically under whichever function is in use — there is no upgrade-time mismatch to manage. Drive-by: pre-commit's sync-plugin-versions hook bumped marketplace + plugin manifests to 0.20.28 since v0.20.27 was tagged on main. |
||
|
chopratejas
|
1314842b19 |
fix(ci): vendor OpenSSL via cargo + drop x86_64 macOS from wheel matrix
The previous hot-fix (#363) addressed npm artifact downloads and added openssl-devel installs in the manylinux container, but the wheel build still fails on three of four matrix entries with three distinct errors: 1. ubuntu-x86_64 with `manylinux: auto` resolved to manylinux2014 (CentOS 7 / OpenSSL 1.0.2k). `openssl-sys 0.9` requires OpenSSL 1.1.0+ — "different version of OpenSSL was found". 2. ubuntu-aarch64 cross-compiles via `aarch64-unknown-linux-gnu-gcc` from an x86_64 manylinux container. The `yum install openssl-devel` we added installs x86_64 headers; `/usr/aarch64-unknown-linux-gnu/ include/` has no OpenSSL — "openssl/opensslv.h: No such file or directory". 3. macos-15-intel fails on `ort-sys` (transitive via the ML compression backend), which has no prebuilt ONNX Runtime binaries for `x86_64-apple-darwin`. Unrelated to OpenSSL; an upstream limitation. Why the workspace pulls openssl-sys at all: `hf-hub` (transitive via `fastembed`) hard-codes `native-tls` as a default feature. Cargo's feature unification then enables openssl-sys for the whole workspace despite our `reqwest`/`tokio-tungstenite`/`tokio-rustls` preferences. # Fix 1: vendored OpenSSL Add `openssl = { version = "0.10", features = ["vendored"] }` to `crates/headroom-proxy/Cargo.toml`. The `vendored` feature compiles OpenSSL from source as part of the cargo build — works on every target uniformly. Local build verified: cargo now pulls `openssl-src v300.6.0+3.6.2` and compiles it. ~30s extra one-time build cost. The `openssl/vendored` feature DEFEATS `OPENSSL_DIR`. We therefore remove the previous hot-fix's "Install OpenSSL (macOS)" step that exported `OPENSSL_DIR` — leaving it would silently regress to the system-OpenSSL path that broke originally. # Fix 2: pin manylinux floor to 2_28 Change x86_64-unknown-linux-gnu from `manylinux: auto` to `manylinux: 2_28` (matching aarch64 + the e2e Dockerfiles). This isn't strictly required with vendored OpenSSL — the floor is now glibc 2.28 / AlmaLinux 8 which has modern toolchain — but it removes the CentOS-7 surface entirely and matches our runtime container target. # Fix 3: drop x86_64-apple-darwin from the matrix `ort-sys 2.0.0-rc.12` has no prebuilt ONNX Runtime binaries for that target. Building ORT from source would add CMake + ~5 minutes per build. Apple Silicon macOS (`aarch64-apple-darwin`) is fully covered; Intel-mac users install from the platform-independent sdist this matrix also produces. Tracked as a follow-up: switch the ML backend to `ort-tract` or upstream a request for x86_64 macOS prebuilts. # before-script-linux: keep perl-IPC-Cmd, drop openssl-devel OpenSSL's vendored `Configure` script needs `IPC::Cmd` (without it the build fails with "Can't locate IPC/Cmd.pm"). System openssl-devel is no longer needed. # Tests 4 new regression tests gate this: - `test_headroom_proxy_vendors_openssl` - `test_build_wheels_installs_perl_ipc_cmd_for_vendored_openssl` - `test_build_wheels_does_not_set_openssl_dir` - `test_build_wheels_matrix_excludes_intel_macos` Plus the previous 7. All 11 release-workflow tests pass. `make ci-precheck` PASSED. Local `cargo build --release -p headroom-py` green. |
||
|
chopratejas
|
2a91cbb4b4 |
refactor: single-wheel maturin build backend (fixes #355)
Eliminates the dual-package architecture that was the root cause of #355. `pip install headroom-ai` now produces ONE wheel containing both the Python source (headroom/*.py) and the compiled Rust extension (headroom/_core.so). No more separate `headroom-core-py` package, no more chicken-and-egg with PyPI publication, no more wheelhouse / PIP_FIND_LINKS / composite-action plumbing in CI. This is the canonical pattern used by cryptography, polars, ruff, pydantic-core, and other Rust-as-core Python packages. Honors the "Rust as core engine" direction. ## What changed - pyproject.toml: `[build-system]` swapped from hatchling to maturin. `[tool.hatch.*]` deleted; `[tool.maturin]` added pointing at `crates/headroom-py/Cargo.toml` for the cdylib. `python-source = "."` picks up the root `headroom/` package directly (dashboard HTML templates and other non-Python files included automatically). - crates/headroom-py/pyproject.toml: deleted. The crate is no longer a separate published package; its Cargo.toml stays as the cdylib build target invoked via `[tool.maturin] manifest-path`. - crates/headroom-py/python/: deleted (placeholder layout for the old separate package). ## CI updates - ci.yml: `test` / `test-extras` / `test-agno` jobs simplified — Rust toolchain set up before `pip install -e .` (which now invokes maturin via build-system). Removed the "build wheel + symlink .so" dance. `build` job swapped from `python -m build` (hatch) to `maturin build` + `maturin sdist`. - release.yml: collapsed dual-package matrix into one. New `build-wheels` matrix produces cross-platform wheels for cp310/11/12/13 × {linux x86_64, linux aarch64, macos x86_64, macos aarch64}. New `collect-dist` aggregator merges artifacts. publish-pypi consumes the merged dist. - init-native-e2e.yml: dropped windows-latest from the matrix — upstream `esaxx-rs` (/MT) and `ort-sys` (/MD) link with conflicting MSVC C runtime libraries, so the Rust extension cannot build for win_amd64 today. Tracked as a follow-up; not a blocker for Linux+macOS. - headroom-e2e-setup: composite action now sets up Rust toolchain + Swatinem/rust-cache before `pip install -e .[proxy]`. - eval.yml, publish.yml, rust.yml: same pattern — rust toolchain before install. rust.yml's wheels job builds from root pyproject.toml (no more `-m crates/headroom-py/Cargo.toml`). - e2e/init/Dockerfile, e2e/wrap/Dockerfile: install rust + maturin in the build stage; copy `crates/` + workspace `Cargo.toml/lock` so the install can build the extension. Dropped `HEADROOM_REQUIRE_RUST_CORE=false` from wrap-e2e — the image now ships the full Rust core. - Dockerfile (main): simplified — no more Layer 2/3 dance with `headroom-core-py` install + symlink. Single `uv pip install` builds + installs everything. - .devcontainer/Dockerfile: rust toolchain + libssl-dev + maturin added so `uv sync` builds the extension inside the devcontainer. ## Lockfile + script - uv.lock: regenerated. No `headroom-core-py` entries remain. - scripts/build_rust_extension.sh: simplified from a symlink-into-tree workaround to a thin wrapper around `pip install -e .`. The maturin build-backend handles placement automatically. ## Local validation (all green on macOS aarch64) 1. Clean venv `pip install -e .` → `from headroom._core import …` works. 2. `maturin build --release` → 13.8 MB wheel, 336 files including `headroom/_core.cpython-311-darwin.so` (32 MB cdylib) and `headroom/dashboard/templates/dashboard.html`. 3. `pip install <wheel>` in fresh venv → import works. 4. Wheel contents verified via `unzip -l`. 5. `pytest tests/test_transforms/test_diff_compressor.py` — 29 passed. 6. `pytest tests/test_relevance.py` — 30 passed. 7. `cargo build --workspace` + `cargo test --workspace` — all green. 8. `make ci-precheck` — 176 Python tests + Rust + commitlint green. ## Migration notes Users on `pip install headroom-ai` get the Rust core automatically (linux + macos wheels). sdist installs require rust toolchain available locally — pip will build via maturin. Closes #355 Supersedes #357 (workarounds-based fix abandoned in favor of architectural fix) |
||
|
chopratejas
|
b3b3feff6f |
fix: B4 — token validation gate + per-content-type byte thresholds
Eliminate P3-33 / P3-34. Wraps every per-block compression in
the live-zone dispatcher with two new gates:
1. Per-content-type byte thresholds — pinned as `const` at the top
of `live_zone.rs` so the table is grep-able and reviewable in
one place. No magic numbers anywhere in the dispatch logic; a
`threshold_for(ContentType)` helper returns the value. Below
threshold → no compressor invoked, recorded as
`BlockAction::BelowByteThreshold { content_type, byte_count,
threshold_bytes }`. Thresholds:
- JSON-array tool_results: 1 KiB
- Build / log output: 512 B
- Search-result blocks: 1 KiB
- Git-diff blocks: 1 KiB
- Source code: 2 KiB (pinned for the future
Rust code-compressor port)
- Plain text: 5 KiB (pinned for Kompress wiring)
- HTML: 5 KiB (no compressor today)
2. Tokenizer-validated rejection — the byte-length proxy
(`compressed_bytes >= original_bytes`) is replaced with a
token-count check using `headroom_core::tokenizer::get_tokenizer`.
The dispatcher creates one tokenizer per request (model-aware
via the new `model: &str` parameter to
`compress_anthropic_live_zone`) and counts both the original
and compressed text. When `compressed_tokens >= original_tokens`
the candidate is rejected and the original bytes are kept.
`BlockAction::Compressed` and `BlockAction::RejectedNotSmaller`
gain `original_tokens` and `compressed_tokens` fields so the
proxy can log token-savings (the currency that actually matters
for prompt cache + provider billing) instead of bytes.
The proxy `live_zone_anthropic.rs` extracts `body["model"]` (or
falls back to `DEFAULT_MODEL = "claude-3-5-sonnet-20241022"` when
the field is missing — the chars-per-token estimator is calibrated
for the Claude family at 3.5 cpt) and threads it through. The
`Compressed` outcome now reports token counts from the manifest,
not byte counts, so the existing
`tokens_before / tokens_after` plumbing is suddenly accurate.
Tests added:
- `live_zone_thresholds.rs::below_threshold_no_compression_attempted`
— 200 B JSON array → `BelowByteThreshold` and `NoChange`.
- `live_zone_thresholds.rs::above_threshold_compression_attempted`
— 10 KB JSON array → byte-threshold gate clears and a compressor
runs (either `Compressed` or `RejectedNotSmaller`).
- `live_zone_token_validation.rs::compressed_more_tokens_falls_back`
— pathological input must not produce `Compressed` with
`compressed_tokens >= original_tokens`.
- `live_zone_token_validation.rs::compressed_fewer_tokens_accepted`
— well-formed JSON array of dicts → `Compressed` with strict
token shrinkage.
- Property test `live_zone_compression_token_count_non_increasing`
— for any well-formed body generated by `proptest`, the
dispatcher's emitted body has token-count <= input's token-count.
Pins the central PR-B4 invariant: the dispatcher never inflates
tokens.
Existing 12 unit tests in `live_zone.rs` and 6 integration tests
in `tests/live_zone_dispatch.rs` updated for the new field shape
and the `model` parameter; all pass. The diff-routing test's
fixture grew to 1.3 KiB so it clears the new GitDiff threshold
gate, exercising the dispatch path rather than short-circuiting.
Per-PR-B4 plan: REALIGNMENT/04-phase-B-live-zone.md.
|
||
|
chopratejas
|
aec5ba3253 |
fix: A6 — anthropic-beta and openai-beta deterministic merge + session-sticky
PR-A6 of the Phase A cache-safety lockdown. Eliminates P5-50 and preps
P0-6 (memory tool injection toggling).
Two cache-killer patterns the merge + tracker defeat:
1. Mid-session mutation: when memory was enabled the proxy did an
ad-hoc concat of `context-management-2025-06-27` onto the client
value (anthropic.py:1244-1248). The order varied with the client
value, breaking byte-stable headers across turns.
2. Token drop-out across turns: clients (Claude Code, Codex CLI) MAY
drop a beta token between turn N and turn N+1 even when the proxy
mutated turn N to add it. The cache hot zone is positional, so the
next turn's prefix bytes hash differently and the prefix-cache
read misses.
Changes
-------
`headroom/proxy/helpers.py`
* `merge_anthropic_beta` / `merge_openai_beta`: pure, deterministic,
order-preserving merge. Client tokens first (in their original
order), then Headroom-required tokens (in the order passed). Dedupe
is case-insensitive but preserves the original casing of the first
occurrence. No regex.
* `SessionBetaTracker`: bounded LRU keyed by (provider, session_id),
unioning client tokens with previously-seen tokens. OrderedDict
LRU; threading.RLock for thread safety (mirrors the
CompressionCache pattern from compression_cache.py).
* `get_session_beta_tracker` / `_reset_session_beta_tracker_for_test`
process-wide singleton with test reset.
* `log_beta_header_merge`: structured log per cache-affecting merge.
* Env-var knobs (NO HARDCODES):
- HEADROOM_BETA_HEADER_STICKY=enabled|disabled (default enabled).
- HEADROOM_BETA_TRACKER_MAX_SESSIONS (default 1000).
`headroom/proxy/handlers/anthropic.py`
* After `compute_session_id` (line ~744): record client
`anthropic-beta` against the session tracker, write the sticky
value back into `headers` if changed. Order matters: sticky-merge
FIRST so memory-injection has the canonical baseline.
* Memory-injection site (line ~1244): replace the ad-hoc concat with
`merge_anthropic_beta(headers["anthropic-beta"], required_tokens)`.
`headroom/proxy/handlers/openai.py`
* Chat-completions (line ~360): record/merge `openai-beta`.
* /v1/responses HTTP (line ~1213): compute `_responses_session_id`
and record/merge `openai-beta`.
* /v1/responses WS (line ~1711): replace the ad-hoc absent-only
inject with `merge_openai_beta(sticky, ["responses_websockets=
2026-02-06"])`. Replaces any case-variants of the existing key.
Tests
-----
`tests/test_anthropic_beta_session_sticky.py` (26 tests):
* Pure helper: empty inputs, only-client, only-headroom, ordering,
dedupe casing, deterministic memory-injection order, no-double-
inject when token already present.
* Tracker: sticky-on across turns even when client drops, casing
preservation, provider namespace independence, LRU eviction at
max_sessions, env-var validation (loud failures), thread safety
under 16-thread concurrent access, blank-input rejection.
`tests/test_openai_beta_session_sticky.py` (17 tests):
* Mirror of the anthropic suite for `OpenAI-Beta`.
* Plus WS-specific coverage: sticky-then-merge of
`responses_websockets=2026-02-06` against client baseline.
`tests/test_openai_codex_routing.py`
* Add `session_tracker_store` stub to `_DummyOpenAIHandler` so the
routing tests still exercise the responses HTTP handler now that
it computes a session_id for beta-merge.
Notes
-----
Build constraints honored:
* Configurable: HEADROOM_BETA_HEADER_STICKY,
HEADROOM_BETA_TRACKER_MAX_SESSIONS.
* No regex, no hardcodes (env-var bounds), no fallbacks (disabled
mode is operator opt-in for diagnostics, loud failures on invalid
values).
* Structured tracing log via `log_beta_header_merge`.
Acceptance:
* 43 new tests pass.
* `cargo test --workspace` green (no Rust changes).
* `make ci-precheck` green.
|
||
|
chopratejas
|
0ce2243dfb |
docs: add Realignment plan (40 PRs, 9 phases)
Comprehensive PR-by-PR plan to realign Headroom around live-zone-only compression with prefix-cache safety as a non-negotiable invariant. Drafted from a 10-agent deep audit against the LLM-proxy compression guide. - 14 documents under REALIGNMENT/ - 72 ranked bugs (P0 cache-killers through P6 test-infra) - 40 feature PRs + 10 test-infra PRs across 9 phases - ~25K LOC retirement (ICM + scoring + relevance + rolling-window + summarizer + tool-crusher + LiteLLM-fake-Bedrock) - Preserves TOIN, CCR, Kompress-base per user direction - Auth-mode policy gates (PAYG / OAuth / subscription) - Phase 3 cache stabilization surface (tool-sort, schema-sort, cache_control auto-place, prompt_cache_key) - Native Bedrock SigV4 + Vertex ADC handlers - Test infrastructure: SHA-256 byte-faithful gate, SSE corner cases, property tests, real-traffic shadow |
||
|
chopratejas
|
fa5fbfabf4 |
fix(rust): wire ICM compressor into Rust proxy on /v1/messages
Adds an opt-in compression interceptor that buffers Anthropic
/v1/messages requests, runs IntelligentContextManager over the
messages array, and forwards the (possibly trimmed) body upstream.
All other paths, methods, and content-types stay on the original
streaming passthrough — so existing operators see zero change.
Behaviour gates ALL must be true to buffer + compress:
- --compression flag (or HEADROOM_PROXY_COMPRESSION=1)
- method == POST
- path == /v1/messages
- Content-Type: application/json
- ICM constructed successfully at startup
Falls through to streaming on any failure: parse, missing fields,
unknown model, body-too-large. Compression must never break a
request — that's the safety contract.
Model context windows come from a vendored LiteLLM snapshot at
crates/headroom-proxy/data/model_prices_and_context_window.json
parsed once into an OnceLock<HashMap>. Refresh via
scripts/refresh_model_limits.sh. Rationale documented inline:
hardcoded tables silently rot; LiteLLM is the canonical source
the entire LLM-tooling ecosystem relies on.
New tests:
- 16 unit tests across compression::{anthropic, icm, model_limits}
- 5 integration tests: off-passthrough, on-short-passthrough,
on-oversized-trim, on-non-json-skip, on-non-llm-path-skip
Verification:
- cargo test --workspace -> 884 passed, 0 failed
- cargo clippy --workspace -- -D warnings -> clean
- cargo fmt --check -> clean
|
||
|
chopratejas
|
01a423a316 |
fix(rust): reformat/offload pipeline + log templates + diff noise (Phase 3g rework)
Replaces PR1's lossless/lossy split with ReformatTransform (pack denser, no info lost) and OffloadTransform (drop bytes, CCR-stash original via required cache_key). With CCR every transform is information-preserving end-to-end, so the lossless/lossy distinction misnamed the architecture. OffloadTransform carries a cheap, structural estimate_bloat() method scoped to its domain — generic byte-redundancy heuristics miss domain semantics. The orchestrator runs reformat phase + per-offload bloat estimation in parallel via rayon::join + par_iter, then runs offload iff bloat clears threshold OR reformat underwhelmed. Transforms shipped: REFORMATS (lossless): - JsonMinifier: serde_json round-trip whitespace stripping. - LogTemplate: Drain-inspired order-preserving template miner. Collapses consecutive runs of same-template lines into [Template Tn: ...] (Nx) + variant table. Win comes from emitting the constant-token prefix once instead of N times. Lossless: every original line reconstructible from template + variants. OFFLOADS (drop bytes, stash original via CCR): - LogOffload: wraps existing LogCompressor; bloat = repetition x uniqueness_weight + dilution x priority_dilution_weight. - DiffOffload: wraps existing DiffCompressor; bloat = context-to- change ratio. Bug-fix-on-port — persists original under the cache_key the parity-bound DiffCompressor mints (closes a leak). - DiffNoise: drops lockfile hunks (Cargo.lock, package-lock.json, yarn.lock, etc., suffix list configurable in TOML) and whitespace-only hunks. Stashes original via CCR for retrieval. Search offload exists but is not in default re-exports — modern agents (Claude Code, Codex) use scoped rg/grep, the marginal value didn't justify default registration. Reach via the explicit module path if opting in. JSON Offload is intentionally absent from this PR — already lives at SmartCrusher; Phase 3g PR3 wraps it in the OffloadTransform contract. Thresholds and weights live in config/pipeline.toml, embedded via include_str!; PipelineConfig::from_toml_str loads runtime overrides. 98 new pipeline tests; full headroom-core suite (714) and workspace tests green; cargo fmt clean. No regex, per project convention. |
||
|
chopratejas
|
45720301e5 |
feat(rust): port log_compressor to Rust + bug fixes (Phase 3e.5)
Ports `headroom.transforms.log_compressor` to Rust. The biggest-by- impact remaining compressor port: build/test logs are where the 10-50x compression wins live. * Stack-trace state machine: per-flavor dispatcher (Python Traceback, JS, Java, Rust error, Go); each flavor has its own termination rule. Python terminated on any blank line, dropping mid-trace lines from chained-exception traces. * Conservative dedupe: preserves message prefix (everything before first `:` or `=`); only trailing region is tokenised. Python's blanket normalisation collapsed segfaults at different addresses. * Loud CCR failures: `tracing::warn!` + `logger.warning` instead of bare `except: pass`. * `LogLevel::FAIL` documented as cosmetic-equivalent to ERROR. Same shape as search_compressor port. Rust `LogCompressor` orchestrates format detect -> classify -> score -> select -> format -> CCR. Inline static-table format detector (YAGNI), aho-corasick level classifier with word-boundary post-filter (`signals::keyword_detector` technique), hand-rolled per-flavor stack-trace state machine. `signals::LineImportanceDetector` NOT consumed -- log levels are structural, not prose-style importance. `headroom.transforms.log_compressor` becomes a thin shim: `compress()` delegates to Rust end-to-end; internal helpers preserved for the existing 50-test surface. Two existing tests updated for new dedupe semantics + new compress orchestration. * 17 Rust unit tests * 50 Python tests pass * `make ci-precheck` clean |
||
|
chopratejas
|
12c2665531 |
feat(rust): signals trait module + KeywordDetector (Phase 3e.1)
Establish `crates/headroom-core/src/signals/` as a top-level module holding cross-cutting detection traits. Phase 3e.1 ports `error_detection.py` to a `LineImportanceDetector` trait + a `Tiered<T>` combinator + a single concrete `KeywordDetector` impl backed by aho-corasick. Three traits at three granularities are sketched (line / blob / item); only line-importance is implemented today. Two bug fixes from the Python source bake into both the Rust impl and the Python regex shim: 1. `ERROR_KEYWORDS` listed `timeout|abort|denied|rejected` but `ERROR_PATTERN` regex omitted them. Lines like `"Connection timeout"` were silently neutral despite the keyword being canonical. Both surfaces now flag them. 2. `SECURITY_KEYWORDS` carried `token`, which false-positived on every reference to LLM tokens (`input_tokens`, `tokens_saved`, ...) in our own product. Dropped from the security set. The Python `error_detection.py` shim now reflects keyword data out of Rust via `keyword_registry_snapshot()` and recompiles the legacy `re.Pattern` objects on the fly. Existing callers (text_compressor, search_compressor, intelligent_context) continue to import the same names with no source changes; caller migration to the trait API happens in their own port PRs. The trait architecture is the seam where a future ML detector slots in without touching `KeywordDetector` or any caller. The canonical extension is documented in `signals/README.md` as a classifier head on the existing `bge-small-en-v1.5` embedder loaded by `relevance::EmbeddingScorer` -- 384-dim -> 4-class softmax, ~1.5 KB head, ~1 ms inference, no extra model file. Two alternatives (distilled tinyBERT in ONNX, logistic regression on lexical features) are kept open in case BGE-head underfits. Per the no-silent-fallbacks rule: only `KeywordDetector` lands as a concrete impl. No NoOp, no MockDetector, no stub-ML -- those will arrive with their real implementations. Phase 3g (Compression Pipeline Formalization, issue #315) is queued as the cross-cutting follow-up that will make lossless-then-lossy- then-CCR ordering an explicit, observable architecture rather than implicit per-compressor logic. Trait shapes there will reuse the signals primitive landed in this PR. |
||
|
chopratejas
|
3f8de4e117 |
fix(smart_crusher): re-land orphaned audit close-out — CCR knob + scorer fail-loud
Re-lands two audit fixes that were marked "merged" on GitHub but never reached main: squash-merging the parent stack changed its commit SHA, which silently dropped the contents of the stacked PRs (#301, #305). Single PR this time — no stacking risk. What lands: 1. **`enable_ccr_marker` Rust gate** — new field on `SmartCrusherConfig` (default `true`). `crush_array` checks it before emitting the `<<ccr:HASH>>` marker text and the CCR store write. PyO3 surface + parity-fixture tolerance updated; recorded fixtures predate the field and inherit the `true` default. 2. **Python shim collapses both flags to the gate** — both `ccr_config.enabled=False` and `ccr_config.inject_retrieval_marker =False` now flip the Rust gate off. Storing a payload nothing in the prompt can reference is pointless, and storing under `enabled=False` would be a surprise side effect the user explicitly opted out of. 3. **Custom `scorer` / `relevance_config` fails loud** — replaces the prior WARNING-and-drop. Silently dropping a user-supplied scorer is a textbook silent fallback. `NotImplementedError` instead. Verified zero production callers pass these args; full plumbing arrives with Stage-3c.2's relevance-crate Python bridge. Tests: - 2 new Rust unit tests in `crusher.rs::tests` - 6 new Python tests in `test_smart_crusher_toin_attachment.py` (3 CCR marker-knob behaviors + 3 scorer fail-loud) - Removed `test_ccr_inject_marker_false_logs_warning` (the WARNING is gone now that the flag is honored) - `make ci-precheck` green; eval suite + observability tests run twice consecutively to verify no TOIN file pollution leaks into the regular+coverage double-run on Python 3.11 RUST_DEV.md audit table reflects both gaps closed. |
||
|
chopratejas
|
b8fc7eee19 |
fix(integrations): filter CCR-dropped sentinel in test iteration
The PR8 marker injection appends a sentinel object
{"_ccr_dropped": "<<ccr:HASH N_rows_offloaded>>"} to the kept-items
array on the lossy path so the LLM sees the retrieval pointer in the
prompt. Tests that iterate compressed arrays via subscript access
(e["level"], r["status"], i["labels"], m["text"]) hit KeyError on
the sentinel because it doesn't share the record schema.
Same root cause as the test_quality_retention fixes in PR8 -- these
integration tests were left out of that pass.
Ship a public helper headroom.transforms.smart_crusher.strip_ccr_sentinels
so tests can use it cleanly: `for e in strip_ccr_sentinels(entries):`
and production callers iterating compressed output get a single
canonical filter instead of inlining the _ccr_dropped check.
The 7 previously-failing tests in PR #292 CI now pass:
- langchain test_100_percent_errors_preserved_logs
- langchain test_errors_preserved_with_many_errors
- langchain test_search_results_with_query_term
- mcp test_all_log_errors_preserved
- mcp test_slack_significant_compression_with_content
- mcp test_database_error_status_preserved
- mcp test_github_bugs_partial_preservation
753 tests across the integration + transforms + retention suites pass
locally. Plugin manifests auto-bumped 0.13.3 -> 0.13.4 by the
sync-plugin-versions hook (unrelated to this fix).
|
||
|
chopratejas
|
beec0789ed |
fix(ci): pin dtolnay/rust-toolchain to 1.95.0 to match rust-toolchain.toml
The action was set to @stable, which installs whatever the latest stable is (1.95.0 right now). Then maturin invokes cargo, which reads rust-toolchain.toml and re-resolves to "1.95.0 + clippy + rustfmt". rustup treats stable and 1.95.0 as distinct toolchain identities and refuses the second install with: failed to install component 'clippy-preview-x86_64-unknown-linux-gnu', detected conflict: 'bin/cargo-clippy' This was intermittent across the matrix (only test (3.10) tripped on the most recent run; others got lucky on cache state). Pinning the action ref to 1.95.0 makes both sides ask for the exact same toolchain identity, so the second install is a no-op and the conflict can't fire. Bump procedure stays the same: when rust-toolchain.toml's channel changes, update these refs in lock-step. Plugin manifests auto-bumped 0.11.0 -> 0.13.2 by sync-plugin-versions hook (unrelated to the workflow fix). |
||
|
chopratejas
|
e640e18f37 |
feat(rust): SmartCrusher PR2 — lossless-first tabular compaction
Stage 3c.2 PR2. Adds an opt-in compaction stage that runs BEFORE the
existing lossy pipeline. When configured, it tries to losslessly
re-shape arrays of objects into a recursive Compaction IR and renders
that to bytes via a pluggable Formatter trait. When not configured
(default OSS), behavior is byte-equal with the pre-PR2 path — all 17
SmartCrusher parity fixtures stay green.
# What lands
- Recursive Compaction IR (`compaction/ir.rs`): Table / Buckets /
OpaqueRef / Untouched. CellValue can hold a nested Compaction so
multi-level cases (stringified-JSON inside cells, heterogeneous
arrays bucketed by discriminator, opaque blobs CCR-substituted)
share one tree shape.
- Cell classifier (`compaction/classifier.rs`): per-cell decision —
Scalar / JsonObject / JsonArray / StringifiedJson(parsed) /
Opaque(kind). Conservative: in doubt, return Scalar.
- TabularCompactor (`compaction/compactor.rs`): array → IR. Handles
uniform-nested flattening into dotted columns ("meta.region",
"meta.tier"), stringified-JSON parsing + recursion, opaque-blob
CCR-substitution (12-char SHA-256 prefix), and heterogeneous
bucketing by discriminator. Falls through to a sparse Table when
no clean discriminator exists, so we always do better than the
lossy path for object arrays.
- Formatter trait (`compaction/formatter.rs`) + two impls:
- JsonFormatter: structured JSON for debugging / programmatic use.
- CsvSchemaFormatter: [N]{col:type,col:type} declaration + CSV
rows. Steals TOON's row-count-and-shape declaration without
adopting TOON's bespoke escaping. CSV is the format LLMs are
strongest at — every model has seen millions of examples in
training. >30% smaller than raw JSON serialization on tabular
fixtures.
- Wiring (`crusher.rs`, `builder.rs`): SmartCrusher gains an optional
compaction stage. Builder methods with_compaction(stage) and
with_default_compaction() opt in. CrushArrayResult gets two new
fields (compacted, compaction_kind) populated only when the stage
runs. strategy_info becomes compaction kind when compaction won.
# Why this design
- Three-trait extension surface preserved. PR1 added Constraint /
Observer / Scorer; PR2 adds Formatter as the fourth pluggable
seam. Enterprise plug-ins land cleanly without forking core.
- Empty default builder rule held. SmartCrusherBuilder::new() still
produces a no-compaction crusher. with_default_compaction() is
the explicit OSS preset. No silent fallbacks.
- Recursive IR was the unlock. A flat table-of-scalars IR would have
collapsed the moment a cell held nested JSON. Making
CellValue::Nested hold another Compaction made stringified-JSON
parsing + heterogeneous bucketing + opaque substitution all share
one renderer pass.
- CCR substitution for opaque cells. Strings classified as
base64/HTML/long-opaque become structured markers keyed by 12-char
SHA-256 prefix. The full bytes round-trip via the CCR store (PyO3
bridge owns actual storage; this PR emits the marker and computes
the hash).
# Tests
- 60 new unit tests across IR / classifier / compactor / formatter /
wiring (448 total in headroom-core, was 388).
- 17/17 SmartCrusher parity fixtures byte-equal — default-config
path completely unchanged.
- 21/21 Python parity tests pass via PyO3 bridge.
- make ci-precheck green: ruff, mypy, cargo fmt/clippy/test
(1.95.0), commitlint.
# Deferred to follow-up PRs
- ToonFormatter (small; ship after eval harness compares formats)
- Diff/code detection in cells → routes to DiffCompressor /
CodeCompressor (coupled to ContentRouter Phase 4)
- Budget-aware row dropping (Constraint-respecting) when rendered
size exceeds budget
- Format A/B eval harness
- ContentRouter unification (Phase 4)
Modules: crates/headroom-core/src/transforms/smart_crusher/compaction/*, builder.rs, crusher.rs, mod.rs
|
||
|
chopratejas
|
f3d5392cc8 |
ci(docker): fix Argument list too long when signing bake outputs
The cosign signing step passed bake metadata via env var:
env:
BAKE_META: ${{ steps.bake.outputs.metadata }}
run: echo "$BAKE_META" | jq ...
For large bake targets (code-nonroot, runtime-code-nonroot) the
metadata JSON is large enough that combined argv+env at bash spawn
exceeds Linux ARG_MAX (~128 KiB on ubuntu-latest), so bash dies with
E2BIG before the script even runs.
Switch to writing metadata into a heredoc-backed temp file, then read
it via jq file input. Heredocs put the JSON in the script body itself,
which bash reads from a temp file (no ARG_MAX limit), bypassing the
env-size ceiling entirely.
Module: .github/workflows/docker.yml
|
||
|
chopratejas
|
cb80bf69fe | chore: sync plugin versions to 0.11.0 | ||
|
chopratejas
|
a22a7277da | chore: sync plugin versions to 0.10.13 | ||
|
chopratejas
|
4429a11166 |
Merge remote-tracking branch 'origin/main' into rust-rewrite
# Conflicts: # headroom/proxy/server.py |
||
|
dependabot[bot]
|
2f659535d2
|
chore(deps): bump the npm_and_yarn group across 3 directories with 4 updates
Bumps the npm_and_yarn group with 1 update in the /sdk/typescript directory: [esbuild](https://github.com/evanw/esbuild). Bumps the npm_and_yarn group with 1 update in the /plugins/openclaw directory: [esbuild](https://github.com/evanw/esbuild). Bumps the npm_and_yarn group with 2 updates in the /docs directory: [postcss](https://github.com/postcss/postcss) and [next](https://github.com/vercel/next.js). Updates `esbuild` from 0.21.5 to 0.27.4 - [Release notes](https://github.com/evanw/esbuild/releases) - [Changelog](https://github.com/evanw/esbuild/blob/main/CHANGELOG-2024.md) - [Commits](https://github.com/evanw/esbuild/compare/v0.21.5...v0.27.4) Updates `postcss` from 8.5.8 to 8.5.10 - [Release notes](https://github.com/postcss/postcss/releases) - [Changelog](https://github.com/postcss/postcss/blob/main/CHANGELOG.md) - [Commits](https://github.com/postcss/postcss/compare/8.5.8...8.5.10) Updates `vite` from 5.4.21 to 8.0.10 - [Release notes](https://github.com/vitejs/vite/releases) - [Changelog](https://github.com/vitejs/vite/blob/main/packages/vite/CHANGELOG.md) - [Commits](https://github.com/vitejs/vite/commits/v8.0.10/packages/vite) Updates `esbuild` from 0.21.5 to 0.27.4 - [Release notes](https://github.com/evanw/esbuild/releases) - [Changelog](https://github.com/evanw/esbuild/blob/main/CHANGELOG-2024.md) - [Commits](https://github.com/evanw/esbuild/compare/v0.21.5...v0.27.4) Updates `postcss` from 8.5.8 to 8.5.10 - [Release notes](https://github.com/postcss/postcss/releases) - [Changelog](https://github.com/postcss/postcss/blob/main/CHANGELOG.md) - [Commits](https://github.com/postcss/postcss/compare/8.5.8...8.5.10) Updates `vite` from 5.4.21 to 8.0.10 - [Release notes](https://github.com/vitejs/vite/releases) - [Changelog](https://github.com/vitejs/vite/blob/main/packages/vite/CHANGELOG.md) - [Commits](https://github.com/vitejs/vite/commits/v8.0.10/packages/vite) Updates `postcss` from 8.5.8 to 8.5.10 - [Release notes](https://github.com/postcss/postcss/releases) - [Changelog](https://github.com/postcss/postcss/blob/main/CHANGELOG.md) - [Commits](https://github.com/postcss/postcss/compare/8.5.8...8.5.10) Updates `next` from 16.2.2 to 16.2.4 - [Release notes](https://github.com/vercel/next.js/releases) - [Changelog](https://github.com/vercel/next.js/blob/canary/release.js) - [Commits](https://github.com/vercel/next.js/compare/v16.2.2...v16.2.4) --- updated-dependencies: - dependency-name: esbuild dependency-version: 0.27.4 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: postcss dependency-version: 8.5.10 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: vite dependency-version: 8.0.10 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: esbuild dependency-version: 0.27.4 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: postcss dependency-version: 8.5.10 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: vite dependency-version: 8.0.10 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: postcss dependency-version: 8.5.10 dependency-type: direct:development dependency-group: npm_and_yarn - dependency-name: next dependency-version: 16.2.4 dependency-type: direct:production dependency-group: npm_and_yarn ... Signed-off-by: dependabot[bot] <support@github.com> |