## Description
Batches small Codex/OpenAI Responses tool-output units through the
existing ContentRouter instead of skipping each unit individually below
the 512-byte floor. This fixes sessions where many small tool outputs
are collectively worth compressing, but no single output clears the
per-unit threshold.
The change keeps larger units on the existing independent compression
path, preserves CCR retrieval markers and protected tags across the
batch envelope, rejects structurally invalid batch output, and leaves
under-floor tails as size-floor passthroughs.
Fixes#2234
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Added `headroom/transforms/compression_batches.py` for bounded
compatible-unit batching, batch envelope parsing, tag/CCR marker
preservation, and per-entry result splitting.
- Updated the OpenAI Responses compression adapter to batch small
tool-output text slots while keeping larger units on the existing cached
per-unit path.
- Switched the unit size floor to UTF-8 bytes so CJK and other multibyte
text are measured consistently with the byte threshold.
- Added regression coverage for batching, CJK byte floors, CCR marker
preservation, malformed batch rejection, array output parts, and
under-floor tails.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uv run --with pytest --with fastapi --with httpx --with anyio --with uvicorn --with h2 pytest tests/test_compression_batches.py tests/test_compression_units.py tests/test_openai_responses_compression_units.py -q
47 passed, 1 warning
$ uvx ruff==0.15.17 check headroom/proxy/handlers/openai.py headroom/transforms/compression_batches.py headroom/transforms/compression_units.py tests/test_compression_batches.py tests/test_compression_units.py tests/test_openai_responses_compression_units.py --output-format concise
All checks passed!
$ uvx ruff==0.15.17 format --check headroom/proxy/handlers/openai.py headroom/transforms/compression_batches.py headroom/transforms/compression_units.py tests/test_compression_batches.py tests/test_compression_units.py tests/test_openai_responses_compression_units.py
6 files already formatted
$ uv run --with mypy mypy headroom/transforms/compression_batches.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.13.3, local checkout of this PR
branch.
- Exact command / steps: ran the focused batching/unit/OpenAI Responses
test suites above, including cases where four individually-small tool
outputs collectively exceed the shared floor and where output arrays
contain multiple text parts plus non-text parts.
- Observed result: small outputs are sent through one router call and
applied back to their original slots; under-floor tails remain
unmodified; non-text parts are preserved; CCR markers are retained or
the entire batch is rejected if moved/corrupted.
- Not tested: a live Codex Responses proxy session against an upstream
model; full-suite collection was not run locally.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
## Description
Closes#1342
Codex shell output currently goes through the unit-router compression
path as a plain `local_shell_call_output` string. When that path picks a
lossy strategy and the compressed text carries no CCR retrieval marker,
the agent gets a summary that can't be reversed back to the original
shell log. That breaks the point of showing command output at all.
This change keeps structured shell output verbatim unless the
replacement stays recoverable. Other tool-output paths stay unchanged.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/compression_units.py`: add a lossy-strategy set
and a structured-shell heuristic, then reject lossy unmarked
replacements for `role="tool"` plus
`item_type="local_shell_call_output"` by returning the original text
with `reason="lossy_unrecoverable_tool_output"`.
- `tests/test_compression_units.py`: add regression coverage for the
failing case, the recoverable-marker case, non-shell tool output, and
assistant text so the guard stays scoped.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
GitHub Actions on head 6790f03486:
- PR Governance: green
- CI: green, including lint, build-wheel, docker-native-e2e, test shards 1-4, test-agno, test-dashboard-ui, and test-extras
- Init E2E: green
- Wrap E2E: green
- Evaluation Suite smoke-test: green
```
## Real Behavior Proof
- Environment: current PR head
`6790f03486` in GitHub Actions.
- Exact command / steps: exercise `compress_unit_with_router` with
structured multi-line `local_shell_call_output`, return a lossy unmarked
replacement, and assert the original shell text is kept with
`reason="lossy_unrecoverable_tool_output"`. Paired tests prove that
CCR-marked replacements still compress, non-shell tool output still
compresses, and assistant text still compresses when explicitly allowed.
- Observed result: the new regression coverage passes on the PR head and
the full PR check set is green.
- Not tested: end-to-end live shell sessions through the Responses API;
intentionally unstructured shell output below this heuristic remains
compressible.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A, backend compression-path change.
## Additional Notes
`mypy headroom` was not run for this PR body refresh, so the type-check
box stays unchecked here. The changelog and docs boxes are N/A for this
targeted bug fix.
