## Description
When a tool result is excluded from lossy compression (e.g. grep via
`HEADROOM_EXCLUDE_TOOLS`), the OpenAI Responses adapter performs a
byte-lossless fold on the output text. However, the excluded-tool fold
path joined all content-part text with `_responses_part_text()` and
recorded a `("output", None)` slot, which caused `_set_slot_text` to
replace the entire `output` with a plain string.
For content-part arrays (valid per OpenAI spec: `[{type: output_text,
text: "..."}, {type: input_image, ...}]`), this destroyed the array
structure — non-text parts like images and refusals were silently
dropped.
Closes#2235
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/openai.py` — In the excluded-tool lossless
fold path, detect list (content-part) outputs and fold each
`input_text`/`output_text` part individually using `("output_part",
index)` slots, matching the eligibility rule already used by
`_slot_texts()` in the normal compression path
- `tests/test_openai_responses_compression_units.py` — Strengthen
existing content-part test to assert output remains a list; add new test
with mixed parts (output_text + input_image + refusal) to verify
structure preservation and byte-identical non-text parts
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ uv run pytest tests/test_openai_responses_compression_units.py -q --no-header
26 passed in 1.04s
$ uv run pytest tests/test_openai_responses_compression_units.py tests/test_openai_responses_context_compaction.py tests/test_openai_responses_traffic_learner.py -q --no-header
39 passed in 6.43s
```
## Real Behavior Proof
- Environment: Linux 6.8.0-124-generic, Python 3.12.3, headroom main @
eac49656
- Exact command / steps: checkout branch, run `uv run pytest
tests/test_openai_responses_compression_units.py -x -q`, inspect output
structure of excluded-tool items with content-part arrays
- Observed result: All 26 tests pass. For content-part outputs with
mixed types, the compressed output remains a list with the same length
and part types — non-text parts are byte-identical, only
`output_text`/`input_text` parts are updated
- Not tested: Live Codex WS end-to-end (requires Codex Desktop with
content-part tool outputs)
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
Co-authored-by: lennney <lennney@users.noreply.github.com>
## Description
Batches small Codex/OpenAI Responses tool-output units through the
existing ContentRouter instead of skipping each unit individually below
the 512-byte floor. This fixes sessions where many small tool outputs
are collectively worth compressing, but no single output clears the
per-unit threshold.
The change keeps larger units on the existing independent compression
path, preserves CCR retrieval markers and protected tags across the
batch envelope, rejects structurally invalid batch output, and leaves
under-floor tails as size-floor passthroughs.
Fixes#2234
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Added `headroom/transforms/compression_batches.py` for bounded
compatible-unit batching, batch envelope parsing, tag/CCR marker
preservation, and per-entry result splitting.
- Updated the OpenAI Responses compression adapter to batch small
tool-output text slots while keeping larger units on the existing cached
per-unit path.
- Switched the unit size floor to UTF-8 bytes so CJK and other multibyte
text are measured consistently with the byte threshold.
- Added regression coverage for batching, CJK byte floors, CCR marker
preservation, malformed batch rejection, array output parts, and
under-floor tails.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uv run --with pytest --with fastapi --with httpx --with anyio --with uvicorn --with h2 pytest tests/test_compression_batches.py tests/test_compression_units.py tests/test_openai_responses_compression_units.py -q
47 passed, 1 warning
$ uvx ruff==0.15.17 check headroom/proxy/handlers/openai.py headroom/transforms/compression_batches.py headroom/transforms/compression_units.py tests/test_compression_batches.py tests/test_compression_units.py tests/test_openai_responses_compression_units.py --output-format concise
All checks passed!
$ uvx ruff==0.15.17 format --check headroom/proxy/handlers/openai.py headroom/transforms/compression_batches.py headroom/transforms/compression_units.py tests/test_compression_batches.py tests/test_compression_units.py tests/test_openai_responses_compression_units.py
6 files already formatted
$ uv run --with mypy mypy headroom/transforms/compression_batches.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.13.3, local checkout of this PR
branch.
- Exact command / steps: ran the focused batching/unit/OpenAI Responses
test suites above, including cases where four individually-small tool
outputs collectively exceed the shared floor and where output arrays
contain multiple text parts plus non-text parts.
- Observed result: small outputs are sent through one router call and
applied back to their original slots; under-floor tails remain
unmodified; non-text parts are preserved; CCR markers are retained or
the entire batch is rejected if moved/corrupted.
- Not tested: a live Codex Responses proxy session against an upstream
model; full-suite collection was not run locally.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
## Description
`WebSearch` and `WebFetch` tool results can be large reference payloads
whose exact formatting matters. This PR keeps those web-tool outputs
verbatim through both the chat/router path and the OpenAI Responses
path, including cross-turn dedup, while leaving ordinary compressible
tools such as `Bash` unchanged by default.
Closes#1810
## Changes Made
- Added `WebSearch`, `WebFetch`, `web_search`, and `web_fetch` to the
default excluded tools.
- Added a verbatim-only excluded-tool subset for web payloads so those
outputs bypass lossy compression, lossless JSON rewriting, and
cross-turn dedup folding.
- Updated the OpenAI Responses adapter to track protected call IDs for
verbatim web outputs.
- Added regressions for Anthropic-style tool results, OpenAI Responses
tool outputs, cross-turn dedup, and unchanged `Bash` compression
behavior.
- Merged current `main` and removed unrelated dependency floor changes
from the PR diff.
## Testing
```text
uv run --extra dev python -m pytest tests/test_websearch_tool_result_protection.py tests/test_content_router_exclude_tools.py tests/test_openai_responses_compression_units.py::test_openai_responses_adapter_keeps_websearch_output_verbatim tests/test_responses_cross_turn_dedup.py::test_protected_websearch_outputs_do_not_fold -q
13 passed
uv run --extra dev mypy headroom/transforms/content_router.py headroom/proxy/handlers/openai.py
Success: no issues found in 2 source files
git diff --check headroomlabs/main...HEAD
# no output
```
The local pre-commit hook also passed on the pushed cleanup/type-fix
commit.
## Review Readiness
- [x] Ready for review
- [x] Regression tests added
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
## Description
Wrapping Codex yielded **0% compression**: the OpenAI Responses path
extracts each `function_call_output` as its own `CompressionUnit` with a
512B per-item floor, so a session of many small tool outputs floored
every unit (reporter telemetry: 381 units, all
`size_floor`/`passthrough`, `tokens_saved=0`). The Anthropic path
compresses the whole message list in one batch and isn't subject to a
per-item floor.
Fix: evaluate the size floor once against the **aggregate** compressible
bytes of the extracted group (matching the batch path). Disable the
per-unit floor when the group clears the threshold; keep it when the
whole group is below it. Generic across any OpenAI-Responses caller — no
`codex` special-casing, reuses the existing
`OPENAI_RESPONSES_ROUTER_MIN_BYTES`.
Closes#2050
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/proxy/handlers/openai.py`: aggregate-then-floor for
Responses tool-output units.
- `tests/test_openai_responses_compression_units.py`: regression tests
(aggregate-clears → compresses all; aggregate-below → skips all).
## Testing
- [x] Linting passes (`ruff check .`)
- [x] New tests added for the fix
### Test Output
```text
$ ruff check headroom/proxy/handlers/openai.py tests/test_openai_responses_compression_units.py
All checks passed!
```
## Real Behavior Proof
- Environment: static fix; verified locally via `ruff` + `py_compile`
(native `_core` isn't built in the review worktree, so the pytest suite
runs in CI).
- Before: Codex session floored every tool-output unit → 0 tokens saved.
- After: units whose aggregate exceeds the floor reach the ContentRouter
and compress; trivially-small groups still skip.
- Not tested locally: live Codex end-to-end token-saved delta (needs a
real Codex session); CI unit tests cover the floor/routing logic.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Description
Fixes 0% savings when wrapping Codex (Closes#2050). `_slot_text` and
the
lossless-excluded fold in the OpenAI Responses compression path only
handled
`function_call_output` items whose `output` field is a plain string,
silently
skipping items whose `output` is an array of content parts (valid per
OpenAI
spec). Use `_responses_part_text()` — which already handles both — so
these
items reach the ContentRouter and accrue compression savings.
Also extend `_responses_input_item_text_bytes` to count text bytes
inside
content-part arrays in the `output` field, matching its existing
treatment of
the `content` field.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `_slot_text()` (openai.py:1260): use `_responses_part_text()` instead
of
`isinstance(output, str)` to extract text from both string and
content-part
outputs
- Lossless excluded fold (openai.py:1362): same change — use
`_responses_part_text()` so excluded-tool outputs with content parts can
still be losslessly compacted
- `_responses_input_item_text_bytes()` (openai.py:547): extend byte
counting
to handle content-part arrays in the `output` field, matching existing
`content` field handling
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Manual testing performed
### Test Output
```text
$ uv run pytest tests/test_openai_responses_compression_units.py -x -q
16 passed in 1.19s
$ uv run pytest tests/ -k "openai and responses and compress" -x -q
19 passed, 14 skipped in 18.17s
$ uv run ruff check headroom/proxy/handlers/openai.py
All checks passed!
```
## Real Behavior Proof
- Environment: Linux (6.8.0-124-generic), Python 3.12.3, headroom main @
868b88bc
- Exact command / steps: checkout branch, run `uv run pytest
tests/test_openai_responses_compression_units.py -x -q`, run `uv run
pytest tests/ -k "openai and responses and compress" -x -q`, run `uv run
ruff check headroom/proxy/handlers/openai.py`
- Observed result: All 35 tests pass (16 units + 19 integration), ruff
clean, no regressions
- Not tested: Live Codex WS end-to-end with actual content-part outputs
(requires Codex Desktop and a session that produces content-part tool
outputs)
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
---------
Co-authored-by: lennney <lennney@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
Builds on the now-merged personas (#1732). Two pieces:
### 1. Lossless compaction for EXCLUDED tool output
Excluded tools (Read/Grep/Glob/Write/Edit) stay out of *lossy*
compression, but their output is compacted by detected shape:
| shape | transform | guarantee |
|---|---|---|
| grep (SEARCH) | ripgrep --heading fold | **byte-lossless**
(`search_unheading` recovers) |
| log (BUILD_OUTPUT) | ANSI strip + run-collapse | **byte-lossless**
modulo non-semantic ANSI |
| json | whitespace-minify | **data-lossless** (`json.loads` equal), NOT
byte-exact |
Source code + glob path-lists → verbatim. grep gated on
`_try_detect_search` (the general/Magika classifier calls grep-over-code
SOURCE_CODE and would miss it). Off by default
(`compact_excluded_lossless`).
### 2. Enable it in the coding/general personas
`compact_excluded_lossless=True` on the coding + general profiles,
threaded via `proxy_env` + `proxy_pipeline_kwargs` + a per-request
`ContentRouter.apply` override. So `HEADROOM_SAVINGS_PROFILE=coding`
auto-folds excluded grep/log/json.
## Why
The coding persona was getting ~2.5% on OpenCode because its dominant
traffic (Grep/Read) is excluded, and RTK (shell-only, lossy) never sees
OpenCode's *native* tools. This recovers those savings losslessly.
## Measured (end-to-end via coding-persona kwargs, real `rg` output)
41,589 → 26,562 chars (**−36%**), `router:excluded:lossless_search`,
byte-recoverable.
## Accuracy
grep/log = byte-lossless → edit-safe. json = data-lossless (edit-caveat
for read-then-edit-JSON, documented). Read of source code → untouched
(tested).
47 tests (personas + all three tiers + persona-enablement + end-to-end).
ruff + mypy clean. **No personas duplication** — rebased onto main after
#1732 landed. Supersedes #1755.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Description
`HEADROOM_EXCLUDE_TOOLS` protects excluded tool outputs for Anthropic
`tool_result` blocks and OpenAI chat `role=tool` messages, but was
ignored on the Codex `/v1/responses` path. Large exact MCP outputs (e.g.
Serena `find_symbol` / `get_symbols_overview`) were compressed even when
the tool name was explicitly excluded, so the model saw summarized
output and fell back to raw file reads.
Closes#940
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
Root cause: `ContentRouter` consults `exclude_tools` via a `tool_call_id
-> name` map built from chat `tool_calls` / Anthropic `tool_use` blocks
(`_build_tool_name_map`). The Responses adapter
(`_compress_openai_responses_live_text_units_with_router`) extracted
every `function_call_output` as a compression unit without correlating
it to the originating `function_call`'s name, so `exclude_tools` was
never consulted for Responses tool outputs.
- `headroom/proxy/handlers/openai.py`:
- Build a `call_id -> tool name` map from the Responses `function_call`
items (the name lives on `function_call`, the originating `call_id` on
the matching `function_call_output`).
- Resolve the effective exclude set the same way `ContentRouter` does
(`router.config.exclude_tools`, falling back to `DEFAULT_EXCLUDE_TOOLS`
when `None`).
- Skip extraction of outputs whose originating tool is excluded,
mirroring the existing `headroom_retrieve` output guard. Name matching
also tests the lowercased name defensively for case-insensitivity.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
tests/test_openai_responses_compression_units.py::test_openai_responses_adapter_preserves_excluded_tool_outputs PASSED
tests/test_openai_responses_compression_units.py::test_openai_responses_adapter_compresses_non_excluded_tool_outputs PASSED
tests/test_openai_responses_compression_units.py::test_openai_responses_adapter_preserves_headroom_retrieve_outputs PASSED
tests/test_openai_responses_compression_units.py::test_openai_responses_adapter_compresses_custom_tool_call_output PASSED
4 passed
$ ruff check headroom/proxy/handlers/openai.py tests/test_openai_responses_compression_units.py
All checks passed!
```
## Real Behavior Proof
- Environment: macOS (ARM64), Python 3.13.
- Exact command / steps: ran the new and adjacent unit tests for the
Responses compression adapter. The native `headroom._core` extension
could not be compiled locally (macOS 26 C++ toolchain), so these tests
were executed with a stubbed `_core`; the changed code path is pure
Python and the tests override `router.compress`, so the stub does not
affect what is exercised. CI builds the real core.
- Observed result: outputs for an excluded tool (`serena.find_symbol`)
are left untouched (`modified=False`), while outputs for a non-excluded
tool still compress and are replaced with the routed summary.
- Not tested: full native build / live Codex end-to-end run; `mypy`.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
Documentation / CHANGELOG updates are N/A: this restores the documented
behavior of `HEADROOM_EXCLUDE_TOOLS` on a path where it was silently
dropped. `mypy` and a full native build were not run in this
environment; the change is pure Python.
