## Description
The 0.33.0 release PR (#2339) has sat in `changes-requested` since
2026-07-17. Root cause: **release-please only rewrites `pyproject.toml`
and its configured `extra-files`**, but other tracked files also carry
the version — and `server.json` is asserted byte-for-byte against
`render_server_json()`, which derives its version from `pyproject.toml`.
So the bump alone fails
`tests/test_mcp_registry/test_server_json.py::test_root_server_json_matches_builder`
(the `test (2)` shard) on every regenerated release PR.
Nothing in the repo regenerated `server.json` at all, so it fell behind
every release.
Unblocks #2339.
### Why the release *build* passes but the release PR does not
`release.yml` already runs `scripts/version-sync.py` immediately before
its own `verify-versions.py` gate (lines 145 and 278). That is why
`build` and `build-wheels` are green on #2339 despite the drift — it
syncs in the workspace, uncommitted. The regular CI test job does
**not** sync, so the fix has to be committed to the branch.
This also explains why reviewers kept seeing `verify-versions.py` fail
locally while CI's build jobs passed: the verifier is never run
un-synced inside `release.yml`.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **`scripts/version-sync.py`**: also write `server.json`. It was the
one version-carrying file with no writer anywhere. Values are rewritten
in place so key order and formatting keep matching the builder's
byte-for-byte output (verified: the file is pure ASCII and round-trips
exactly through `json.dumps(..., indent=2) + "\n"`).
- **`.github/workflows/release-metadata-sync.yml`** (new): on a push to
`release-please--branches--**`, run version-sync → gate on
verify-versions → commit if changed.
- **Keyed off the branch push** because release-please force-regenerates
that branch on every merge to main. That is precisely what wiped the
hand-pushed metadata fixes on #2339 (`2a86c8ff`, `d5ea4dc5`) — a push
trigger re-heals after every regeneration instead of being lost.
- **Uses the same PAT as `release-please.yml`**: a `GITHUB_TOKEN` push
does not trigger workflows, so the release PR's checks would never
re-run against the synced commit and would stay red.
- **Idempotent**: the self-triggered rerun finds no diff and exits
before pushing, so the loop terminates after one no-op run.
- **Corrected pre-existing drift on `main`**: the agent-hooks plugin
manifests, both marketplace manifests, and `.releasemetadata` were
stranded at **0.31.0** — never bumped for 0.32.0 either.
`verify-versions.py` now passes on `main`.
### Why not more `extra-files` entries
That would need ~13 jsonpath entries restating what `version-sync.py`
already knows, and a jsonpath that fails to match **fails silently** —
the same class of failure this PR removes, discoverable only after a
real release PR regenerates. There is also no precedent for nested
jsonpath (`$.packages[0].version`, `$.metadata.version`) in the config
today; both existing entries are plain `$.version`. Running the script
keeps one source of truth, and files added to it later are covered with
no change here.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — no `headroom/` sources
touched
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest scripts/tests/ tests/test_release_workflows.py tests/test_mcp_registry/ -q
207 passed in 2.69s
$ ruff check scripts/version-sync.py scripts/tests/test_version_sync.py tests/test_release_workflows.py
All checks passed!
$ ruff format --check <same>
3 files already formatted
$ actionlint .github/workflows/release-metadata-sync.yml
(clean)
```
New tests:
- `test_server_json_version_is_synchronized` — version-sync moves both
`server.json` version fields and preserves the other keys.
- `test_release_metadata_sync_runs_on_release_please_branch` — asserts
the trigger, the sync→verify→commit ordering, the no-op guard, and the
PAT.
- `test_version_sync_covers_every_file_the_verifier_gates` — guards
`version-sync.py` and `verify-versions.py` against drifting apart again,
which is the root cause here.
## Real Behavior Proof
- **Environment:** macOS (Darwin arm64), Python 3.12, repo venv.
- **Exact command / steps:** reproduced the CI failure locally by
simulating release-please's partial bump, then applying the fix.
**Reproducing the exact `test (2)` failure** — set `pyproject` to 0.33.0
while `server.json` stays at 0.32.0, as release-please leaves it:
```text
$ python -m pytest tests/test_mcp_registry/test_server_json.py -q
FAILED tests/test_mcp_registry/test_server_json.py::test_root_server_json_matches_builder
1 failed, 3 passed
```
**After `version-sync.py`:**
```text
$ python scripts/version-sync.py && python -m pytest tests/test_mcp_registry/test_server_json.py -q
4 passed
```
**Both gates green on a simulated 0.33.0 bump:**
```text
$ python scripts/version-sync.py --version 0.33.0
Version synchronized to 0.33.0
$ python scripts/verify-versions.py
All versions aligned at 0.33.0
$ python -m pytest tests/test_mcp_registry/test_server_json.py -q
4 passed
```
**Idempotency** (the property the workflow's loop-termination relies
on): re-running against an already-synced tree leaves `pyproject.toml`,
`server.json`, `openclaw`, and `sdk/typescript` untouched.
- **Not tested:** the workflow has not executed on a real release-please
branch regeneration — that can only be exercised once this is on `main`
and release-please next updates #2339. The PAT push path and the
self-trigger no-op are reasoned from `release-please.yml`'s existing
token comment and from local idempotency, not observed in CI.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
**Context on the v0.32.0 release failure, since it is easy to misread as
"images never build".** Every artifact built for v0.32.0 — all 5 wheel
platforms including Windows, all 16 Docker builds + 8 manifests +
`promote-latest`, npm, and GitHub Packages. Only `publish-pypi` failed
(PyPI attestations, already fixed by `f9cbdd6e` / #2405), and
`create-release` was skipped because it depends on it. That is why the
release looked like it produced nothing.
**Separate, approaching blocker — not addressed here.** PyPI is at
**9.69 GB of its 10 GB project cap (96.9%)**, leaving ~305 MB against
~68 MB per release, so roughly 4 more releases fit. The `0.21.x` series
alone holds **6.58 GB across 31 releases**, from the old
every-push-is-a-release era; pruning it would reclaim two thirds of the
quota. Worth a separate issue.
**`.releasemetadata` is written but never read** by anything outside
`version-sync.py` and its test. It is kept in sync here for internal
consistency, but it may be a deletion candidate.
## Description
Current `main` has advanced the core package versions to `0.31.0`, but
the plugin marketplace manifests and hook plugin manifests were still
left at `0.30.0`. This PR now keeps the original version-sync intent
while updating the remaining metadata to the current release line.
It also preserves the previously-added `lxml-html-clean>=0.4.5` security
floor in `pyproject.toml` / `uv.lock` so the security audit remains
unblocked.
Closes#1872
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Synced `pyproject.toml`, SDK/package manifests, plugin manifests,
marketplaces, `.release-please-manifest.json`, and editable package lock
metadata to `0.31.0`.
- Updated the OpenClaw plugin dependency on `headroom-ai` to `^0.31.0`.
- Merged current `main` and resolved the version metadata conflicts in
favor of current `0.31.0` alignment.
## Testing
- [x] Unit tests pass (`pytest tests/test_plugin_manifests.py
scripts/tests/test_version_sync.py -q`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed (`python scripts/verify-versions.py`)
### Test Output
```text
python scripts/verify-versions.py
All versions aligned at 0.31.0
pytest tests/test_plugin_manifests.py scripts/tests/test_version_sync.py -q
10 passed in 0.42s
```
## Real Behavior Proof
- Environment: Windows, Python 3.13.13, PR head after merging current
`main`.
- Exact command / steps: Ran `python scripts/verify-versions.py` and
`pytest tests/test_plugin_manifests.py
scripts/tests/test_version_sync.py -q`.
- Observed result: Version verification exits successfully with `All
versions aligned at 0.31.0`; focused manifest/version-sync tests pass.
- Not tested: Full wheel/build matrix; this is metadata-only version
alignment and CI will cover the broader matrix.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] My changes generate no new warnings
- [x] New and existing focused tests pass locally with my changes
- [ ] I have made corresponding changes to the documentation
- [ ] I have updated the CHANGELOG.md if applicable
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
## Description
**Fixes the Release Please pipeline**, which stopped opening the release
PR, leaving pip / Docker / npm out of sync.
### Root cause
Recent releases (0.28 → 0.30) were cut **out-of-band** (manual tag +
release), so Release Please's state drifted from reality:
- `.release-please-manifest.json` was frozen at **0.29.0**, and the
in-repo version files at **0.29.0** — even the `v0.30.0` tag commit has
`pyproject.toml = 0.29.0`. The plugin/marketplace manifests were frozen
even further back, at **0.22.3**.
- `v0.30.0` was tagged and published to PyPI (CI stamps the version from
the tag at build time via `version-sync.py`, which is why PyPI got
0.30.0 despite the committed 0.29.0).
- With the manifest at 0.29.0, RP kept computing the next version as
**0.30.0**, saw that tag already exists, and produced **no PR** — so
nothing new could ship, and Docker/npm fell behind.
### Fix
Realign the repo with the last real release (0.30.0) so RP can drive the
next one:
- `.release-please-manifest.json` → `0.30.0`
- `pyproject.toml`, `sdk/typescript/package.json`,
`plugins/openclaw/package.json`, both `plugin.json`, both
`marketplace.json` → `0.30.0` (via `scripts/version-sync.py --version
0.30.0`, which also fixes the 0.22.3 drift).
### What happens after merge
1. Release Please runs on `main`, sees manifest = 0.30.0 + 69 releasable
commits since `v0.30.0`, and opens a clean **`chore: release 0.31.0`**
PR (bumping every version file).
2. Merging that PR tags `v0.31.0` and fires `release: published`, which
publishes **PyPI + npm (SDK + openclaw) + Docker** at 0.31.0 in one shot
— bringing all three registries back in sync.
No behavior/code change — versions only.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Advance the Release Please manifest to the last released version
(0.30.0).
- Sync all 7 version-tracked files to 0.30.0 with the repo's own
`version-sync.py`.
## Testing
- [x] Linting passes (`ruff check .`)
- [x] Manual testing performed (`scripts/verify-versions.py`)
### Test Output
```text
$ python scripts/version-sync.py --version 0.30.0
Version synchronized to 0.30.0
$ python scripts/verify-versions.py
All versions aligned at 0.30.0
Packages: pyproject.toml, plugins/openclaw/package.json, sdk/typescript/package.json,
plugins/headroom-agent-hooks/.claude-plugin/plugin.json,
plugins/headroom-agent-hooks/.github/plugin/plugin.json,
.claude-plugin/marketplace.json, .github/plugin/marketplace.json
```
## Real Behavior Proof
- Environment: local macOS, project `.venv` (Python 3.12.6).
- Exact command / steps: ran `scripts/version-sync.py --version 0.30.0`,
set the RP manifest to 0.30.0, then `scripts/verify-versions.py`.
- Observed result: `verify-versions.py` reports all seven version
locations aligned at 0.30.0; `git diff` shows version-field changes only
(no code). Confirmed PyPI latest is 0.30.0 and a `v0.30.0` tag/release
exists, while the manifest was 0.29.0 — the drift this PR corrects.
- Not tested: the downstream release itself (that runs when the
follow-up `chore: release 0.31.0` PR is merged); npm registry state
could not be read from this environment (network), but the `publish-npm`
job in `release.yml` publishes both npm packages on release.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] My changes generate no new warnings
- [x] New and existing unit tests pass locally with my changes
The repo had drifted: pyproject.toml said 0.9.1 but PyPI's latest
published headroom-ai was 0.22.3. release_version.py papered over
this by taking max(canonical, latest_tag) at release time;
release-please does NOT do that — it trusts the manifest verbatim.
Left as-is, release-please would propose 0.9.2 on the next merge
and PyPI would reject it ("400 Cannot publish version lower than
latest"), looping the bot forever.
Fix: align every version-bearing file to 0.22.3 (the truth on
PyPI). Done via `scripts/version-sync.py --version 0.22.3`:
- .release-please-manifest.json
- pyproject.toml
- sdk/typescript/package.json
- plugins/openclaw/package.json (+ headroom-ai dep range -> ^0.22.3)
- .claude-plugin/marketplace.json
- .github/plugin/marketplace.json
- plugins/headroom-agent-hooks/.claude-plugin/plugin.json
- plugins/headroom-agent-hooks/.github/plugin/plugin.json
After this lands, the bot's next release PR will propose 0.22.4
(patch) or 0.23.0 (minor) depending on conventional-commit traffic
since v0.22.3.
Three fixes bundled; all in admin / cache-hit paths where tests didn't
catch the regression.
## (A) 13 RequestOutcome sites missing tags=
An AST audit found that 13 of 21 ``RequestOutcome(...)`` construction
sites across the four handler files emitted outcomes without threading
``tags=``. Affected paths:
* ``handle_anthropic_messages`` — the ``from_response_cache=True``
early-return outcome (Claude Code cache-hit turns dashboard-blind)
* ``handle_openai_chat`` — same cache-hit early-return (Codex +
Cursor + Continue cache-hit turns dashboard-blind)
* ``handle_openai_responses_ws`` — the per-turn outcome inside the
Codex WS session. The stale comment that said "ws_session_tags is
not yet bound" was wrong — ``ws_tags`` was already extracted at
handler entry
* ``handle_anthropic_batch_create / batch_passthrough / batch_results``
* ``handle_passthrough`` (OpenAI Models / Files / List-Batches)
* ``handle_google_batch_create / batch_passthrough / batch_results``
* ``_google_batch_passthrough`` (internal helper)
* ``handle_batch_create`` (OpenAI batch entry)
* ``handle_gemini_count_tokens`` (also fixed in #479; identical)
Pattern of the fix is uniform: pull tags from headers and thread
them into the ``RequestOutcome`` construction.
New contract test ``test_handler_outcome_tag_invariant.py`` walks each
handler file's AST and asserts every ``RequestOutcome`` site inside any
``handle_*`` or ``*_passthrough`` method passes both ``tags=`` and
``client=``. Future handlers get a clear test failure with file +
line + method name if they regress.
## (B) Issue #478 — /v1/models 403 under Codex ChatGPT auth
Codex Desktop with ChatGPT-subscription OAuth polls ``/v1/models`` to
populate its model picker. Forwarding to ``chatgpt.com/backend-api/
models`` returned 403 to OAuth tokens. Fix: synthesize an OpenAI-
compatible payload locally from a known-supported model set
(``gpt-5.5`` through ``gpt-5``). All other ChatGPT-auth paths still
forward as before — only model-metadata gets the local response.
## (C) Move _extract_tags to free function (mixin-isolation test compat)
Handlers called ``self._extract_tags(headers)``. That worked in
production where ``HeadroomProxy`` composes every mixin and defines
the method, but broke tests that instantiate a single mixin via
``object.__new__(OpenAIHandlerMixin)``. The free-function form
removes that coupling — handlers import ``extract_tags`` from
``headroom.proxy.helpers`` and call directly. ``HeadroomProxy.
_extract_tags`` is kept as a thin wrapper for any external caller
still using the method form. 17 call sites migrated.
## Zero behavior change for existing users
Claude Code, Codex, Cursor, Continue, Aider, Gemini-routed harnesses
all hit handlers that already extracted tags. Their wire bytes to
upstream LLMs are byte-identical. Only the dashboard view gains tags
on previously-blind paths.
Closes#478.
Adds ``CompressionDecision.apply_to_tags(tags)`` — a one-liner mutator
that stamps the passthrough reason into a tags dict for downstream
observability. Each migrated handler now calls
``_decision.apply_to_tags(tags)`` immediately after
``CompressionDecision.decide(...)``. The tags dict flows unchanged
into every downstream ``RequestOutcome(tags=tags, ...)`` construction,
which the funnel surfaces in ``RequestLog.tags`` — same mechanism the
funnel already uses for ``client``.
Dashboards can now slice passthrough traffic by cause:
* tags["passthrough_reason"] == "bypass_header"
* tags["passthrough_reason"] == "compression_disabled"
* tags["passthrough_reason"] == "no_messages"
* tags["passthrough_reason"] == "license_denied"
No-op when ``should_compress=True`` — compressing requests don't
carry the tag, so absence vs presence is itself the signal.
Bonus fix: ``handle_gemini_count_tokens`` was the one Gemini handler
that never pulled tags out of headers, so its emitted
``RequestOutcome`` reached the dashboard without any of the per-
request slicing keys. Added the missing ``tags = self._extract_tags
(request.headers)`` and threaded ``tags=tags`` into its outcome.
Closes the observability loop opened by PR #477: the four Gemini-
bypass-bug fixes are now visible in the request-log feed the moment
they fire.
Three streaming finalizers — ``_finalize_stream_response``,
``_stream_response_bedrock``, ``_stream_openai_via_backend`` — each
duplicated the same set of body- and config-derived fields when
constructing a ``RequestOutcome``:
* ``attempted_input_tokens = optimized_tokens + tokens_saved``
* ``num_messages = len(body.get("messages", []))``
* ``request_messages`` conditional on ``config.log_full_messages``
* ``transforms_applied`` list → tuple (frozen-dataclass contract)
* ``tags or {}`` normalization
* ``turn_id`` via ``compute_turn_id``
The last one was a real bug. Only the Bedrock site computed
``turn_id`` — sites 1 and 3 silently dropped it, breaking the
dashboard's multi-turn-session grouping for every Anthropic-SSE and
OpenAI-via-backend request. The new ``RequestOutcome.from_stream``
classmethod computes it uniformly so the three finalizers cannot
drift apart on derivation logic again.
Each call site now hands ``from_stream`` the body + provider-specific
cache/timing fields and gets a fully-constructed outcome back. The
funnel call after it stays identical (``await
self._record_request_outcome(outcome)``).
PyPI rejected the v0.21.37 release publish with:
HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/
Project size too large. Limit for project 'headroom-ai' total size is 10 GB.
PyPI inventory check confirmed: **191 versions × ~213 MB/release =
10.00 GB exactly** — at the cumulative project storage ceiling. Each
recent release ships 12 wheels × ~16-18 MB each.
Post-mortem inspection of a production wheel
(``headroom_ai-0.21.36-cp311-cp311-manylinux_2_28_x86_64.whl``)
showed the binary was ``not stripped``:
.text 18.3 MB (code)
.rodata 11.4 MB (Magika model + ONNX runtime data)
.strtab 4.9 MB (debug strings — strippable)
.eh_frame 1.9 MB (unwind tables)
.symtab 1.5 MB (debug symbols — strippable)
.gcc_except_table 1.2 MB
This commit adds a release profile:
[profile.release]
strip = "symbols"
lto = "thin"
codegen-units = 1
That:
* Strips ``.symtab`` + ``.strtab`` (~6.4 MB direct savings per wheel)
* Enables thin link-time optimization for cross-crate dead-code
elimination (~5-10% ``.text`` savings)
* Single codegen unit for better inlining + DCE at the cost of
~30-50% slower release builds (acceptable for CI)
Deliberately NOT setting ``panic = "abort"``:
* The proxy is a long-lived async process. A panic on one bad
request triggering process abort would disconnect every concurrent
client. Accept the smaller savings; keep unwind behaviour.
Estimated impact
* Per wheel: ~16-18 MB → ~10-11 MB (40% smaller)
* Per release (12 wheels): ~213 MB → ~130 MB
* PyPI capacity: ~30+ more releases before hitting 10 GB again
Verification
* Local build of ``headroom._core`` with new profile:
``.so`` size 29 MB on macOS arm64 (was ~45 MB pre-fix; final wheel
compressed will be smaller on Linux which also benefits from the
``strip`` directive).
* 77 Rust-parity tests pass — extension still functional.
* Single-codegen-unit slows build by ~30-50% but maturin/cibuildwheel
build time was never the bottleneck.
Forward strategy (separate work)
* Submit a PyPI project-size-limit-increase request to unblock the
immediate release.
* Adopt a release-deprecation policy: yank versions older than N
patches per minor; consider dropping Python 3.10 wheels (EOL'd
October 2026) and manylinux_2_28_aarch64 wheels (niche audience,
largest at 18.75 MB).
* Investigate runtime-download for Magika model (~10 MB further
savings) — same pattern Kompress already uses.
PR #411 reintroduced an older `--code-aware` is_flag option and a
duplicate `code_aware_enabled=` kwarg in the ProxyConfig call, which
collided with the canonical tristate `--code-aware/--no-code-aware`
introduced in #260. The result: every CLI entry point (`headroom proxy`,
`headroom wrap codex`, `headroom wrap claude`, etc.) raised at import
time:
File ".../headroom/cli/proxy.py", line 575
code_aware_enabled=code_aware or _get_env_bool(...)
SyntaxError: keyword argument repeated: code_aware_enabled
Removes:
- The legacy `@click.option("--code-aware", is_flag=True, ...)`
- The legacy `code_aware: bool` function parameter
- The duplicate `code_aware_enabled=` kwarg
Keeps the tristate `--code-aware/--no-code-aware` > env-var >
default-off resolver. Behavior is unchanged for all flag combinations
covered by tests/test_cli_proxy_env.py.
Test mocks for `run_server` updated to accept `**kwargs` to match
the real signature (config plus run-time options like print_banner).
Without this the four code-aware tests added in #411 raised
TypeError on each invocation.
Plugin marketplace/manifest version bump 0.21.5 → 0.21.7 carried in
this commit by the sync-plugin-versions pre-commit hook.
Builds on PR #406 (HTTP /v1/responses PyO3) and PR #410 (WS first-frame
PyO3) which landed the binding for `compress_openai_responses_live_zone`.
This change closes the remaining gaps so every (provider × endpoint ×
auth-mode × streaming) combination compresses AND surfaces in the
dashboard.
Telemetry: extend the PyO3 binding return tuple from `(bytes, modified)`
to `(bytes, modified, tokens_saved, transforms_applied)` by adding
`CompressionManifest::tokens_saved()` and `transforms_applied()`
accessors on the existing manifest. The Python proxy populates
request-log telemetry from the binding output instead of recounting
tokens. Updates the existing 2-tuple call sites in HTTP and WS
first-frame, plus the unpacks in tests.
WebSocket multi-frame compression: subscription Codex users keep a
long-lived WS open and send multiple `response.create` events per
session. PR #410 only compressed the first frame; subsequent frames
went raw. Added `_maybe_compress_response_create_frame` closure inside
`_client_to_upstream` that runs the same Rust dispatcher on every
client→upstream `response.create` text frame, passes other event
types (response.cancel, session.update, etc.) through unchanged, and
accumulates `tokens_saved` / `transforms_applied` /
`ws_frames_compressed` counters across the session.
Pre-existing dashboard gap: `streaming.py` and `anthropic.py` write
`RequestLog` entries; the non-streaming OpenAI HTTP and WS handlers
did not. Result: /transformations/feed was invisible for every Codex
turn and every Cline / OpenClaude / Aider turn. Added the same wiring
in `handle_openai_chat` (non-streaming), `handle_openai_responses`
(non-streaming HTTP), and `handle_openai_responses_ws` (session-end).
All three populate `auth_mode` + `endpoint` tags so the dashboard can
break compression activity down by client class (PAYG / OAuth /
Subscription) and surface (`chat_completions` / `responses_http` /
`responses_ws`). The WS metric record is now unconditional — was
previously gated on `tokens_saved > 0`, so first-frame no-changes
never registered.
compute_frozen_count over-freeze for prose-format clients:
`compute_frozen_count` walked until it found an unstable
`tool_result` / `role: "tool"` block. Cline / OpenClaude / Aider —
clients that embed tool calls as XML inside plain text — never
produce such a boundary, so the function returned `len(messages)` and
the pipeline froze 100% of messages including the brand-new user
turn. Live zone empty → `Transform content_router: 16414 → 16414
tokens (saved 0)`. Reported on Discord 2026-05-07 with Cline+DeepSeek.
Fix: cap at `max(0, len(messages) - 1)`. Updates 3 existing test
assertions whose expected values encoded the old over-freeze. Adds 6
new prose-format invariant tests.
CodeQL "clear-text logging of sensitive information" fix:
`tests/e2e_real_compression.py` previously stored API keys in local
variables in the same scope as diagnostic prints, which CodeQL flagged
via data-flow analysis. Refactored to read keys from `os.environ`
inside the request helper — the credentials never enter the runner's
main scope, so the taint flow never reaches the print.
End-to-end verification with real keys (.env):
/v1/messages (PAYG, non-stream) tok 14109 → 969 saved 13140
/v1/messages (PAYG, stream) tok 14109 → 969 saved 13140
/v1/chat/completions (PAYG, non-stream) tok 18460 → 1374 saved 17086
/v1/chat/completions (PAYG, stream) tok 18460 → 1374 saved 17086 (cache_hit=100%)
/v1/responses HTTP (PAYG, non-stream) bytes 50138 → 597 saved 18391
/v1/responses WS (frame 1) bytes 46429 → 488 saved 16791
/v1/responses WS (frame 2 multi) bytes 46429 → 488 saved 16791
/v1/responses WS (response.cancel) passthrough untouched
Tests: cargo workspace + pytest (4846 pass, 0 fail), make ci-precheck
passed, two E2E scripts (multi-turn HTTP, WS fake-upstream) all green.
PR-C5 (May 3) retired the Python `/v1/responses` compression pipeline
with the comment "Rust handles item-aware compression natively" — but
the standalone `crates/headroom-proxy` binary that was supposed to do
that compression is not deployed by the CLI today (`headroom proxy`
and `headroom wrap codex` both run only the Python proxy via uvicorn).
Result: every `/v1/responses` request since v0.20.16 has been
forwarded uncompressed. Codex CLI is the flagship consumer of this
endpoint; this is the regression users have been reporting.
Closes Bug 1 of the Codex regression by exposing the existing
`headroom_core::transforms::compress_openai_responses_live_zone` as
a PyO3 binding so the Python proxy can call the live-zone dispatcher
in-process. The `headroom._core` extension is already loaded at
proxy startup (PR-A0 verifies), so adding one more callable is
mechanical.
Why PyO3 inline (Layer 1) vs originally-intended two-process chain
(Layer 2): the inline call requires zero deployment changes — the
wheel already ships `headroom._core`. Layer 2 (build + ship the
standalone `headroom-proxy` binary, teach CLI to spawn both
processes) is the right long-term move; Layer 1 restores v0.5.21
functional behaviour today.
# Returns
`(body, modified)`. On change → `(new_body_bytes, True)`; on
passthrough → `(input_bytes, False)`.
# Failure mode
Never raises. The dispatcher's `LiveZoneError` cases (body not JSON,
no input array) are passthrough conditions matching the Rust proxy's
`compress_openai_responses_request` contract.
# Tests
14 new tests in `tests/test_responses_pyo3_compression.py`:
binding exposed, passthrough cases, every F1 AuthMode variant,
empty-model default, no-raise on garbage bytes.
PR #396's first dry-run on its own changes failed three smoke matrix
entries — two fixed by PR #397's __libc_single_threaded shim (now on
main, this PR is rebased on top), one orthogonal: ubuntu:20.04 + 3.10.
Failure mode on ubuntu:20.04 + 3.10: deadsnakes PPA install path
stopped reliably provisioning python3.10-venv on focal once Ubuntu
20.04 hit End of Standard Support in May 2025. The error is from
apt-get, NOT from `import headroom._core` — the wheel never gets a
chance to load:
E: Unable to locate package python3.10-venv
Continuing to promise wheel-runtime correctness on glibc 2.31 in CI
would require either pulling ESM-tier ubuntu:focal images (paid /
auth-gated) or pinning a specific deadsnakes snapshot URL — neither
of which we want owning long-term.
Floor coverage we keep:
- manylinux_2_28_x86_64 (glibc 2.28 — the floor we promise)
- manylinux_2_28_aarch64 (glibc 2.28 — same)
- ubuntu:22.04 + 3.12 x86_64 (glibc 2.35 — issue #355's reporter env)
- ubuntu:22.04 + 3.12 aarch64 (glibc 2.35 — arm equivalent)
- macos-14 + 3.13 (Apple Silicon native)
Test pin in tests/test_release_workflows.py
(test_release_workflow_has_smoke_import_wheel_gate) only requires the
manylinux floors, ubuntu:22.04, and macos-14, so this drop doesn't
break the structural invariant.
CodeQL flagged `compression_store.store()`'s default MD5 cache key as
`py/weak-sensitive-data-hashing` after the explicit_hash refactor in
PR #395 brought the line into a new diff context.
First attempt: `usedforsecurity=False` + `# lgtm[...]` comment to
silence the alert without changing the hash. Both failed — CodeQL
ignores the hashlib parameter and our LGTM marker, the alert stayed
open on the PR.
Second attempt (this commit): drop MD5 entirely. The cache key is for
deduplication / lookup, not security or integrity, so any deterministic
function works. SHA-256[:24] gives the same 96-bit collision space as
MD5[:24] (~280 trillion entries for 50% collision under birthday
bound), is FIPS-clean, and CodeQL won't flag it.
Behaviour impact: zero. The cache is in-memory (no disk persistence),
so the same content always hashes deterministically under whichever
function is in use — there is no upgrade-time mismatch to manage.
Drive-by: pre-commit's sync-plugin-versions hook bumped marketplace +
plugin manifests to 0.20.28 since v0.20.27 was tagged on main.
The previous hot-fix (#363) addressed npm artifact downloads and added
openssl-devel installs in the manylinux container, but the wheel build
still fails on three of four matrix entries with three distinct errors:
1. ubuntu-x86_64 with `manylinux: auto` resolved to manylinux2014
(CentOS 7 / OpenSSL 1.0.2k). `openssl-sys 0.9` requires OpenSSL
1.1.0+ — "different version of OpenSSL was found".
2. ubuntu-aarch64 cross-compiles via `aarch64-unknown-linux-gnu-gcc`
from an x86_64 manylinux container. The `yum install openssl-devel`
we added installs x86_64 headers; `/usr/aarch64-unknown-linux-gnu/
include/` has no OpenSSL — "openssl/opensslv.h: No such file or
directory".
3. macos-15-intel fails on `ort-sys` (transitive via the ML compression
backend), which has no prebuilt ONNX Runtime binaries for
`x86_64-apple-darwin`. Unrelated to OpenSSL; an upstream limitation.
Why the workspace pulls openssl-sys at all: `hf-hub` (transitive via
`fastembed`) hard-codes `native-tls` as a default feature. Cargo's
feature unification then enables openssl-sys for the whole workspace
despite our `reqwest`/`tokio-tungstenite`/`tokio-rustls` preferences.
# Fix 1: vendored OpenSSL
Add `openssl = { version = "0.10", features = ["vendored"] }` to
`crates/headroom-proxy/Cargo.toml`. The `vendored` feature compiles
OpenSSL from source as part of the cargo build — works on every target
uniformly. Local build verified: cargo now pulls
`openssl-src v300.6.0+3.6.2` and compiles it. ~30s extra one-time
build cost.
The `openssl/vendored` feature DEFEATS `OPENSSL_DIR`. We therefore
remove the previous hot-fix's "Install OpenSSL (macOS)" step that
exported `OPENSSL_DIR` — leaving it would silently regress to the
system-OpenSSL path that broke originally.
# Fix 2: pin manylinux floor to 2_28
Change x86_64-unknown-linux-gnu from `manylinux: auto` to
`manylinux: 2_28` (matching aarch64 + the e2e Dockerfiles). This
isn't strictly required with vendored OpenSSL — the floor is now
glibc 2.28 / AlmaLinux 8 which has modern toolchain — but it removes
the CentOS-7 surface entirely and matches our runtime container
target.
# Fix 3: drop x86_64-apple-darwin from the matrix
`ort-sys 2.0.0-rc.12` has no prebuilt ONNX Runtime binaries for that
target. Building ORT from source would add CMake + ~5 minutes per
build. Apple Silicon macOS (`aarch64-apple-darwin`) is fully covered;
Intel-mac users install from the platform-independent sdist this
matrix also produces.
Tracked as a follow-up: switch the ML backend to `ort-tract` or
upstream a request for x86_64 macOS prebuilts.
# before-script-linux: keep perl-IPC-Cmd, drop openssl-devel
OpenSSL's vendored `Configure` script needs `IPC::Cmd` (without it
the build fails with "Can't locate IPC/Cmd.pm"). System
openssl-devel is no longer needed.
# Tests
4 new regression tests gate this:
- `test_headroom_proxy_vendors_openssl`
- `test_build_wheels_installs_perl_ipc_cmd_for_vendored_openssl`
- `test_build_wheels_does_not_set_openssl_dir`
- `test_build_wheels_matrix_excludes_intel_macos`
Plus the previous 7. All 11 release-workflow tests pass.
`make ci-precheck` PASSED. Local `cargo build --release -p headroom-py`
green.
Eliminates the dual-package architecture that was the root cause of #355.
`pip install headroom-ai` now produces ONE wheel containing both the Python
source (headroom/*.py) and the compiled Rust extension (headroom/_core.so).
No more separate `headroom-core-py` package, no more chicken-and-egg with
PyPI publication, no more wheelhouse / PIP_FIND_LINKS / composite-action
plumbing in CI.
This is the canonical pattern used by cryptography, polars, ruff,
pydantic-core, and other Rust-as-core Python packages. Honors the
"Rust as core engine" direction.
## What changed
- pyproject.toml: `[build-system]` swapped from hatchling to maturin.
`[tool.hatch.*]` deleted; `[tool.maturin]` added pointing at
`crates/headroom-py/Cargo.toml` for the cdylib. `python-source = "."`
picks up the root `headroom/` package directly (dashboard HTML
templates and other non-Python files included automatically).
- crates/headroom-py/pyproject.toml: deleted. The crate is no longer a
separate published package; its Cargo.toml stays as the cdylib build
target invoked via `[tool.maturin] manifest-path`.
- crates/headroom-py/python/: deleted (placeholder layout for the old
separate package).
## CI updates
- ci.yml: `test` / `test-extras` / `test-agno` jobs simplified — Rust
toolchain set up before `pip install -e .` (which now invokes maturin
via build-system). Removed the "build wheel + symlink .so" dance.
`build` job swapped from `python -m build` (hatch) to
`maturin build` + `maturin sdist`.
- release.yml: collapsed dual-package matrix into one. New `build-wheels`
matrix produces cross-platform wheels for cp310/11/12/13 ×
{linux x86_64, linux aarch64, macos x86_64, macos aarch64}. New
`collect-dist` aggregator merges artifacts. publish-pypi consumes the
merged dist.
- init-native-e2e.yml: dropped windows-latest from the matrix —
upstream `esaxx-rs` (/MT) and `ort-sys` (/MD) link with conflicting
MSVC C runtime libraries, so the Rust extension cannot build for
win_amd64 today. Tracked as a follow-up; not a blocker for Linux+macOS.
- headroom-e2e-setup: composite action now sets up Rust toolchain +
Swatinem/rust-cache before `pip install -e .[proxy]`.
- eval.yml, publish.yml, rust.yml: same pattern — rust toolchain before
install. rust.yml's wheels job builds from root pyproject.toml (no
more `-m crates/headroom-py/Cargo.toml`).
- e2e/init/Dockerfile, e2e/wrap/Dockerfile: install rust + maturin in
the build stage; copy `crates/` + workspace `Cargo.toml/lock` so the
install can build the extension. Dropped `HEADROOM_REQUIRE_RUST_CORE=false`
from wrap-e2e — the image now ships the full Rust core.
- Dockerfile (main): simplified — no more Layer 2/3 dance with
`headroom-core-py` install + symlink. Single `uv pip install` builds
+ installs everything.
- .devcontainer/Dockerfile: rust toolchain + libssl-dev + maturin
added so `uv sync` builds the extension inside the devcontainer.
## Lockfile + script
- uv.lock: regenerated. No `headroom-core-py` entries remain.
- scripts/build_rust_extension.sh: simplified from a symlink-into-tree
workaround to a thin wrapper around `pip install -e .`. The maturin
build-backend handles placement automatically.
## Local validation (all green on macOS aarch64)
1. Clean venv `pip install -e .` → `from headroom._core import …` works.
2. `maturin build --release` → 13.8 MB wheel, 336 files including
`headroom/_core.cpython-311-darwin.so` (32 MB cdylib) and
`headroom/dashboard/templates/dashboard.html`.
3. `pip install <wheel>` in fresh venv → import works.
4. Wheel contents verified via `unzip -l`.
5. `pytest tests/test_transforms/test_diff_compressor.py` — 29 passed.
6. `pytest tests/test_relevance.py` — 30 passed.
7. `cargo build --workspace` + `cargo test --workspace` — all green.
8. `make ci-precheck` — 176 Python tests + Rust + commitlint green.
## Migration notes
Users on `pip install headroom-ai` get the Rust core automatically
(linux + macos wheels). sdist installs require rust toolchain available
locally — pip will build via maturin.
Closes#355
Supersedes #357 (workarounds-based fix abandoned in favor of
architectural fix)
Eliminate P3-33 / P3-34. Wraps every per-block compression in
the live-zone dispatcher with two new gates:
1. Per-content-type byte thresholds — pinned as `const` at the top
of `live_zone.rs` so the table is grep-able and reviewable in
one place. No magic numbers anywhere in the dispatch logic; a
`threshold_for(ContentType)` helper returns the value. Below
threshold → no compressor invoked, recorded as
`BlockAction::BelowByteThreshold { content_type, byte_count,
threshold_bytes }`. Thresholds:
- JSON-array tool_results: 1 KiB
- Build / log output: 512 B
- Search-result blocks: 1 KiB
- Git-diff blocks: 1 KiB
- Source code: 2 KiB (pinned for the future
Rust code-compressor port)
- Plain text: 5 KiB (pinned for Kompress wiring)
- HTML: 5 KiB (no compressor today)
2. Tokenizer-validated rejection — the byte-length proxy
(`compressed_bytes >= original_bytes`) is replaced with a
token-count check using `headroom_core::tokenizer::get_tokenizer`.
The dispatcher creates one tokenizer per request (model-aware
via the new `model: &str` parameter to
`compress_anthropic_live_zone`) and counts both the original
and compressed text. When `compressed_tokens >= original_tokens`
the candidate is rejected and the original bytes are kept.
`BlockAction::Compressed` and `BlockAction::RejectedNotSmaller`
gain `original_tokens` and `compressed_tokens` fields so the
proxy can log token-savings (the currency that actually matters
for prompt cache + provider billing) instead of bytes.
The proxy `live_zone_anthropic.rs` extracts `body["model"]` (or
falls back to `DEFAULT_MODEL = "claude-3-5-sonnet-20241022"` when
the field is missing — the chars-per-token estimator is calibrated
for the Claude family at 3.5 cpt) and threads it through. The
`Compressed` outcome now reports token counts from the manifest,
not byte counts, so the existing
`tokens_before / tokens_after` plumbing is suddenly accurate.
Tests added:
- `live_zone_thresholds.rs::below_threshold_no_compression_attempted`
— 200 B JSON array → `BelowByteThreshold` and `NoChange`.
- `live_zone_thresholds.rs::above_threshold_compression_attempted`
— 10 KB JSON array → byte-threshold gate clears and a compressor
runs (either `Compressed` or `RejectedNotSmaller`).
- `live_zone_token_validation.rs::compressed_more_tokens_falls_back`
— pathological input must not produce `Compressed` with
`compressed_tokens >= original_tokens`.
- `live_zone_token_validation.rs::compressed_fewer_tokens_accepted`
— well-formed JSON array of dicts → `Compressed` with strict
token shrinkage.
- Property test `live_zone_compression_token_count_non_increasing`
— for any well-formed body generated by `proptest`, the
dispatcher's emitted body has token-count <= input's token-count.
Pins the central PR-B4 invariant: the dispatcher never inflates
tokens.
Existing 12 unit tests in `live_zone.rs` and 6 integration tests
in `tests/live_zone_dispatch.rs` updated for the new field shape
and the `model` parameter; all pass. The diff-routing test's
fixture grew to 1.3 KiB so it clears the new GitDiff threshold
gate, exercising the dispatch path rather than short-circuiting.
Per-PR-B4 plan: REALIGNMENT/04-phase-B-live-zone.md.
PR-A6 of the Phase A cache-safety lockdown. Eliminates P5-50 and preps
P0-6 (memory tool injection toggling).
Two cache-killer patterns the merge + tracker defeat:
1. Mid-session mutation: when memory was enabled the proxy did an
ad-hoc concat of `context-management-2025-06-27` onto the client
value (anthropic.py:1244-1248). The order varied with the client
value, breaking byte-stable headers across turns.
2. Token drop-out across turns: clients (Claude Code, Codex CLI) MAY
drop a beta token between turn N and turn N+1 even when the proxy
mutated turn N to add it. The cache hot zone is positional, so the
next turn's prefix bytes hash differently and the prefix-cache
read misses.
Changes
-------
`headroom/proxy/helpers.py`
* `merge_anthropic_beta` / `merge_openai_beta`: pure, deterministic,
order-preserving merge. Client tokens first (in their original
order), then Headroom-required tokens (in the order passed). Dedupe
is case-insensitive but preserves the original casing of the first
occurrence. No regex.
* `SessionBetaTracker`: bounded LRU keyed by (provider, session_id),
unioning client tokens with previously-seen tokens. OrderedDict
LRU; threading.RLock for thread safety (mirrors the
CompressionCache pattern from compression_cache.py).
* `get_session_beta_tracker` / `_reset_session_beta_tracker_for_test`
process-wide singleton with test reset.
* `log_beta_header_merge`: structured log per cache-affecting merge.
* Env-var knobs (NO HARDCODES):
- HEADROOM_BETA_HEADER_STICKY=enabled|disabled (default enabled).
- HEADROOM_BETA_TRACKER_MAX_SESSIONS (default 1000).
`headroom/proxy/handlers/anthropic.py`
* After `compute_session_id` (line ~744): record client
`anthropic-beta` against the session tracker, write the sticky
value back into `headers` if changed. Order matters: sticky-merge
FIRST so memory-injection has the canonical baseline.
* Memory-injection site (line ~1244): replace the ad-hoc concat with
`merge_anthropic_beta(headers["anthropic-beta"], required_tokens)`.
`headroom/proxy/handlers/openai.py`
* Chat-completions (line ~360): record/merge `openai-beta`.
* /v1/responses HTTP (line ~1213): compute `_responses_session_id`
and record/merge `openai-beta`.
* /v1/responses WS (line ~1711): replace the ad-hoc absent-only
inject with `merge_openai_beta(sticky, ["responses_websockets=
2026-02-06"])`. Replaces any case-variants of the existing key.
Tests
-----
`tests/test_anthropic_beta_session_sticky.py` (26 tests):
* Pure helper: empty inputs, only-client, only-headroom, ordering,
dedupe casing, deterministic memory-injection order, no-double-
inject when token already present.
* Tracker: sticky-on across turns even when client drops, casing
preservation, provider namespace independence, LRU eviction at
max_sessions, env-var validation (loud failures), thread safety
under 16-thread concurrent access, blank-input rejection.
`tests/test_openai_beta_session_sticky.py` (17 tests):
* Mirror of the anthropic suite for `OpenAI-Beta`.
* Plus WS-specific coverage: sticky-then-merge of
`responses_websockets=2026-02-06` against client baseline.
`tests/test_openai_codex_routing.py`
* Add `session_tracker_store` stub to `_DummyOpenAIHandler` so the
routing tests still exercise the responses HTTP handler now that
it computes a session_id for beta-merge.
Notes
-----
Build constraints honored:
* Configurable: HEADROOM_BETA_HEADER_STICKY,
HEADROOM_BETA_TRACKER_MAX_SESSIONS.
* No regex, no hardcodes (env-var bounds), no fallbacks (disabled
mode is operator opt-in for diagnostics, loud failures on invalid
values).
* Structured tracing log via `log_beta_header_merge`.
Acceptance:
* 43 new tests pass.
* `cargo test --workspace` green (no Rust changes).
* `make ci-precheck` green.
Adds an opt-in compression interceptor that buffers Anthropic
/v1/messages requests, runs IntelligentContextManager over the
messages array, and forwards the (possibly trimmed) body upstream.
All other paths, methods, and content-types stay on the original
streaming passthrough — so existing operators see zero change.
Behaviour gates ALL must be true to buffer + compress:
- --compression flag (or HEADROOM_PROXY_COMPRESSION=1)
- method == POST
- path == /v1/messages
- Content-Type: application/json
- ICM constructed successfully at startup
Falls through to streaming on any failure: parse, missing fields,
unknown model, body-too-large. Compression must never break a
request — that's the safety contract.
Model context windows come from a vendored LiteLLM snapshot at
crates/headroom-proxy/data/model_prices_and_context_window.json
parsed once into an OnceLock<HashMap>. Refresh via
scripts/refresh_model_limits.sh. Rationale documented inline:
hardcoded tables silently rot; LiteLLM is the canonical source
the entire LLM-tooling ecosystem relies on.
New tests:
- 16 unit tests across compression::{anthropic, icm, model_limits}
- 5 integration tests: off-passthrough, on-short-passthrough,
on-oversized-trim, on-non-json-skip, on-non-llm-path-skip
Verification:
- cargo test --workspace -> 884 passed, 0 failed
- cargo clippy --workspace -- -D warnings -> clean
- cargo fmt --check -> clean
Replaces PR1's lossless/lossy split with ReformatTransform (pack
denser, no info lost) and OffloadTransform (drop bytes, CCR-stash
original via required cache_key). With CCR every transform is
information-preserving end-to-end, so the lossless/lossy distinction
misnamed the architecture.
OffloadTransform carries a cheap, structural estimate_bloat() method
scoped to its domain — generic byte-redundancy heuristics miss domain
semantics. The orchestrator runs reformat phase + per-offload bloat
estimation in parallel via rayon::join + par_iter, then runs offload
iff bloat clears threshold OR reformat underwhelmed.
Transforms shipped:
REFORMATS (lossless):
- JsonMinifier: serde_json round-trip whitespace stripping.
- LogTemplate: Drain-inspired order-preserving template miner.
Collapses consecutive runs of same-template lines into
[Template Tn: ...] (Nx) + variant table. Win comes from emitting
the constant-token prefix once instead of N times. Lossless: every
original line reconstructible from template + variants.
OFFLOADS (drop bytes, stash original via CCR):
- LogOffload: wraps existing LogCompressor; bloat = repetition x
uniqueness_weight + dilution x priority_dilution_weight.
- DiffOffload: wraps existing DiffCompressor; bloat = context-to-
change ratio. Bug-fix-on-port — persists original under the
cache_key the parity-bound DiffCompressor mints (closes a leak).
- DiffNoise: drops lockfile hunks (Cargo.lock, package-lock.json,
yarn.lock, etc., suffix list configurable in TOML) and
whitespace-only hunks. Stashes original via CCR for retrieval.
Search offload exists but is not in default re-exports — modern
agents (Claude Code, Codex) use scoped rg/grep, the marginal value
didn't justify default registration. Reach via the explicit module
path if opting in.
JSON Offload is intentionally absent from this PR — already lives at
SmartCrusher; Phase 3g PR3 wraps it in the OffloadTransform contract.
Thresholds and weights live in config/pipeline.toml, embedded via
include_str!; PipelineConfig::from_toml_str loads runtime overrides.
98 new pipeline tests; full headroom-core suite (714) and workspace
tests green; cargo fmt clean. No regex, per project convention.
Ports `headroom.transforms.log_compressor` to Rust. The biggest-by-
impact remaining compressor port: build/test logs are where the
10-50x compression wins live.
* Stack-trace state machine: per-flavor dispatcher (Python Traceback,
JS, Java, Rust error, Go); each flavor has its own termination
rule. Python terminated on any blank line, dropping mid-trace
lines from chained-exception traces.
* Conservative dedupe: preserves message prefix (everything before
first `:` or `=`); only trailing region is tokenised. Python's
blanket normalisation collapsed segfaults at different addresses.
* Loud CCR failures: `tracing::warn!` + `logger.warning` instead of
bare `except: pass`.
* `LogLevel::FAIL` documented as cosmetic-equivalent to ERROR.
Same shape as search_compressor port. Rust `LogCompressor`
orchestrates format detect -> classify -> score -> select ->
format -> CCR. Inline static-table format detector (YAGNI),
aho-corasick level classifier with word-boundary post-filter
(`signals::keyword_detector` technique), hand-rolled per-flavor
stack-trace state machine. `signals::LineImportanceDetector` NOT
consumed -- log levels are structural, not prose-style importance.
`headroom.transforms.log_compressor` becomes a thin shim:
`compress()` delegates to Rust end-to-end; internal helpers
preserved for the existing 50-test surface. Two existing tests
updated for new dedupe semantics + new compress orchestration.
* 17 Rust unit tests
* 50 Python tests pass
* `make ci-precheck` clean
Establish `crates/headroom-core/src/signals/` as a top-level module
holding cross-cutting detection traits. Phase 3e.1 ports
`error_detection.py` to a `LineImportanceDetector` trait + a
`Tiered<T>` combinator + a single concrete `KeywordDetector` impl
backed by aho-corasick. Three traits at three granularities are
sketched (line / blob / item); only line-importance is implemented
today.
Two bug fixes from the Python source bake into both the Rust impl
and the Python regex shim:
1. `ERROR_KEYWORDS` listed `timeout|abort|denied|rejected` but
`ERROR_PATTERN` regex omitted them. Lines like `"Connection
timeout"` were silently neutral despite the keyword being canonical.
Both surfaces now flag them.
2. `SECURITY_KEYWORDS` carried `token`, which false-positived on
every reference to LLM tokens (`input_tokens`, `tokens_saved`, ...)
in our own product. Dropped from the security set.
The Python `error_detection.py` shim now reflects keyword data out of
Rust via `keyword_registry_snapshot()` and recompiles the legacy
`re.Pattern` objects on the fly. Existing callers (text_compressor,
search_compressor, intelligent_context) continue to import the same
names with no source changes; caller migration to the trait API
happens in their own port PRs.
The trait architecture is the seam where a future ML detector slots
in without touching `KeywordDetector` or any caller. The canonical
extension is documented in `signals/README.md` as a classifier head
on the existing `bge-small-en-v1.5` embedder loaded by
`relevance::EmbeddingScorer` -- 384-dim -> 4-class softmax,
~1.5 KB head, ~1 ms inference, no extra model file. Two alternatives
(distilled tinyBERT in ONNX, logistic regression on lexical
features) are kept open in case BGE-head underfits.
Per the no-silent-fallbacks rule: only `KeywordDetector` lands as a
concrete impl. No NoOp, no MockDetector, no stub-ML -- those will
arrive with their real implementations.
Phase 3g (Compression Pipeline Formalization, issue #315) is queued
as the cross-cutting follow-up that will make lossless-then-lossy-
then-CCR ordering an explicit, observable architecture rather than
implicit per-compressor logic. Trait shapes there will reuse the
signals primitive landed in this PR.
Re-lands two audit fixes that were marked "merged" on GitHub but never
reached main: squash-merging the parent stack changed its commit SHA,
which silently dropped the contents of the stacked PRs (#301, #305).
Single PR this time — no stacking risk.
What lands:
1. **`enable_ccr_marker` Rust gate** — new field on `SmartCrusherConfig`
(default `true`). `crush_array` checks it before emitting the
`<<ccr:HASH>>` marker text and the CCR store write. PyO3 surface +
parity-fixture tolerance updated; recorded fixtures predate the
field and inherit the `true` default.
2. **Python shim collapses both flags to the gate** — both
`ccr_config.enabled=False` and `ccr_config.inject_retrieval_marker
=False` now flip the Rust gate off. Storing a payload nothing in
the prompt can reference is pointless, and storing under
`enabled=False` would be a surprise side effect the user
explicitly opted out of.
3. **Custom `scorer` / `relevance_config` fails loud** — replaces the
prior WARNING-and-drop. Silently dropping a user-supplied scorer
is a textbook silent fallback. `NotImplementedError` instead.
Verified zero production callers pass these args; full plumbing
arrives with Stage-3c.2's relevance-crate Python bridge.
Tests:
- 2 new Rust unit tests in `crusher.rs::tests`
- 6 new Python tests in `test_smart_crusher_toin_attachment.py`
(3 CCR marker-knob behaviors + 3 scorer fail-loud)
- Removed `test_ccr_inject_marker_false_logs_warning` (the WARNING is
gone now that the flag is honored)
- `make ci-precheck` green; eval suite + observability tests run
twice consecutively to verify no TOIN file pollution leaks into the
regular+coverage double-run on Python 3.11
RUST_DEV.md audit table reflects both gaps closed.
The PR8 marker injection appends a sentinel object
{"_ccr_dropped": "<<ccr:HASH N_rows_offloaded>>"} to the kept-items
array on the lossy path so the LLM sees the retrieval pointer in the
prompt. Tests that iterate compressed arrays via subscript access
(e["level"], r["status"], i["labels"], m["text"]) hit KeyError on
the sentinel because it doesn't share the record schema.
Same root cause as the test_quality_retention fixes in PR8 -- these
integration tests were left out of that pass.
Ship a public helper headroom.transforms.smart_crusher.strip_ccr_sentinels
so tests can use it cleanly: `for e in strip_ccr_sentinels(entries):`
and production callers iterating compressed output get a single
canonical filter instead of inlining the _ccr_dropped check.
The 7 previously-failing tests in PR #292 CI now pass:
- langchain test_100_percent_errors_preserved_logs
- langchain test_errors_preserved_with_many_errors
- langchain test_search_results_with_query_term
- mcp test_all_log_errors_preserved
- mcp test_slack_significant_compression_with_content
- mcp test_database_error_status_preserved
- mcp test_github_bugs_partial_preservation
753 tests across the integration + transforms + retention suites pass
locally. Plugin manifests auto-bumped 0.13.3 -> 0.13.4 by the
sync-plugin-versions hook (unrelated to this fix).
The action was set to @stable, which installs whatever the latest
stable is (1.95.0 right now). Then maturin invokes cargo, which reads
rust-toolchain.toml and re-resolves to "1.95.0 + clippy + rustfmt".
rustup treats stable and 1.95.0 as distinct toolchain identities and
refuses the second install with:
failed to install component 'clippy-preview-x86_64-unknown-linux-gnu',
detected conflict: 'bin/cargo-clippy'
This was intermittent across the matrix (only test (3.10) tripped on
the most recent run; others got lucky on cache state). Pinning the
action ref to 1.95.0 makes both sides ask for the exact same toolchain
identity, so the second install is a no-op and the conflict can't fire.
Bump procedure stays the same: when rust-toolchain.toml's channel
changes, update these refs in lock-step.
Plugin manifests auto-bumped 0.11.0 -> 0.13.2 by sync-plugin-versions
hook (unrelated to the workflow fix).
Stage 3c.2 PR2. Adds an opt-in compaction stage that runs BEFORE the
existing lossy pipeline. When configured, it tries to losslessly
re-shape arrays of objects into a recursive Compaction IR and renders
that to bytes via a pluggable Formatter trait. When not configured
(default OSS), behavior is byte-equal with the pre-PR2 path — all 17
SmartCrusher parity fixtures stay green.
# What lands
- Recursive Compaction IR (`compaction/ir.rs`): Table / Buckets /
OpaqueRef / Untouched. CellValue can hold a nested Compaction so
multi-level cases (stringified-JSON inside cells, heterogeneous
arrays bucketed by discriminator, opaque blobs CCR-substituted)
share one tree shape.
- Cell classifier (`compaction/classifier.rs`): per-cell decision —
Scalar / JsonObject / JsonArray / StringifiedJson(parsed) /
Opaque(kind). Conservative: in doubt, return Scalar.
- TabularCompactor (`compaction/compactor.rs`): array → IR. Handles
uniform-nested flattening into dotted columns ("meta.region",
"meta.tier"), stringified-JSON parsing + recursion, opaque-blob
CCR-substitution (12-char SHA-256 prefix), and heterogeneous
bucketing by discriminator. Falls through to a sparse Table when
no clean discriminator exists, so we always do better than the
lossy path for object arrays.
- Formatter trait (`compaction/formatter.rs`) + two impls:
- JsonFormatter: structured JSON for debugging / programmatic use.
- CsvSchemaFormatter: [N]{col:type,col:type} declaration + CSV
rows. Steals TOON's row-count-and-shape declaration without
adopting TOON's bespoke escaping. CSV is the format LLMs are
strongest at — every model has seen millions of examples in
training. >30% smaller than raw JSON serialization on tabular
fixtures.
- Wiring (`crusher.rs`, `builder.rs`): SmartCrusher gains an optional
compaction stage. Builder methods with_compaction(stage) and
with_default_compaction() opt in. CrushArrayResult gets two new
fields (compacted, compaction_kind) populated only when the stage
runs. strategy_info becomes compaction kind when compaction won.
# Why this design
- Three-trait extension surface preserved. PR1 added Constraint /
Observer / Scorer; PR2 adds Formatter as the fourth pluggable
seam. Enterprise plug-ins land cleanly without forking core.
- Empty default builder rule held. SmartCrusherBuilder::new() still
produces a no-compaction crusher. with_default_compaction() is
the explicit OSS preset. No silent fallbacks.
- Recursive IR was the unlock. A flat table-of-scalars IR would have
collapsed the moment a cell held nested JSON. Making
CellValue::Nested hold another Compaction made stringified-JSON
parsing + heterogeneous bucketing + opaque substitution all share
one renderer pass.
- CCR substitution for opaque cells. Strings classified as
base64/HTML/long-opaque become structured markers keyed by 12-char
SHA-256 prefix. The full bytes round-trip via the CCR store (PyO3
bridge owns actual storage; this PR emits the marker and computes
the hash).
# Tests
- 60 new unit tests across IR / classifier / compactor / formatter /
wiring (448 total in headroom-core, was 388).
- 17/17 SmartCrusher parity fixtures byte-equal — default-config
path completely unchanged.
- 21/21 Python parity tests pass via PyO3 bridge.
- make ci-precheck green: ruff, mypy, cargo fmt/clippy/test
(1.95.0), commitlint.
# Deferred to follow-up PRs
- ToonFormatter (small; ship after eval harness compares formats)
- Diff/code detection in cells → routes to DiffCompressor /
CodeCompressor (coupled to ContentRouter Phase 4)
- Budget-aware row dropping (Constraint-respecting) when rendered
size exceeds budget
- Format A/B eval harness
- ContentRouter unification (Phase 4)
Modules: crates/headroom-core/src/transforms/smart_crusher/compaction/*, builder.rs, crusher.rs, mod.rs
The cosign signing step passed bake metadata via env var:
env:
BAKE_META: ${{ steps.bake.outputs.metadata }}
run: echo "$BAKE_META" | jq ...
For large bake targets (code-nonroot, runtime-code-nonroot) the
metadata JSON is large enough that combined argv+env at bash spawn
exceeds Linux ARG_MAX (~128 KiB on ubuntu-latest), so bash dies with
E2BIG before the script even runs.
Switch to writing metadata into a heredoc-backed temp file, then read
it via jq file input. Heredocs put the JSON in the script body itself,
which bash reads from a temp file (no ARG_MAX limit), bypassing the
env-size ceiling entirely.
Module: .github/workflows/docker.yml
