akadmin/headroom
1
0
Fork 0
mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
2696 commits 410 branches 169 tags 142 MiB
Commit graph

3 commits

Author SHA1 Message Date
Tejas Chopra
8884d87378
fix(transforms): stop compression garbling mixed subagent output (#3286) 
## The report

A user's model called compressed subagent output "too garbled to use"
and burned CCR retrievals to reconstruct it — **not** because it needed
more context. One retrieval returned nothing but the Claude Code harness
sanitizer banner, reported as `original_item_count: 33,
compressed_item_count: 25`.

Root-cause chain (verified by reproduction): the harness prepends a
bracket-delimited banner (`[harness: ... you.]` — exactly 33
whitespace-delimited words) and neutralizes `<` → `<\`. Headroom's
mixed-content splitter typed the banner as JSON (bracket balance, no
validation) → SmartCrusher couldn't parse it → the fallback chain fed it
to lossy Kompress → Kompress word-dropped the banner 33→25 and stored it
behind a retrieval hash. Meanwhile tabular sections rendered as
quote-wrapped JSON-string blobs with `\n` as two-character escapes, and
`ensure_ascii=True` boundaries turned the output's unicode (`→ └ ✓`)
into `\uXXXX` soup. The model reasonably concluded the output was
garbled.

## Fixes

1. **`split_into_sections` validates JSON before typing a block
`JSON_ARRAY`** — same validation its own mixed-content gate
(`_has_valid_json_block_with_text`) has always used. Tag-protection
placeholders, which self-isolated only by accident of that bug
(`{{HEADROOM_TAG_N}}` bracket-balances), are now isolated explicitly via
a new `isolate=` parameter fed by the router; contiguous prose fragments
re-coalesce so the `\n\n` reassembly stops doubling newlines in
uncompressed prose.
2. **Kompress gets a real floor: `min_input_words = 64`**
(config-tunable, clamped at the historical 10), applied on the
in-process, batch, apply, and remote paths. Below it, lossy
word-dropping is a net loss — the retrieval marker alone is ~20 words —
and short blocks are disproportionately instruction-like.
3. **The mixed path unwraps SmartCrusher's whole-array CSV render** when
it comes back as a bare JSON string, splicing raw readable lines into
the text instead of a quoted escape blob.
4. **`ensure_ascii=False` at model-visible boundaries**: MCP
retrieve/stats responses and the audit-safe splice reserialization
(which now also matches serde_json's non-escaping behavior).
5. **Kompress honesty**: the marker says `N words compressed to M`
(shared `ccr_retrieval_marker` helper, unit-tested), and
`store_kompress_in_ccr` no longer writes word counts into the store's
*item count* fields — token counts already carry the size story.

The upstream trigger (the harness's `<` → `<\` neutralization corrupting
JSON semantics) is not Headroom's to fix, but with #1 and #2 the banner
now passes through byte-intact and nothing lossy touches it.

## Testing

- New `tests/test_garbled_compression_fixes.py` (12 tests) pins every
fix, including an end-to-end router pass over a reconstructed
harness-sanitized fixture asserting the banner survives byte-identical
and no `\uXXXX` appears.
- Existing small-fixture kompress/router tests updated to set
`min_input_words=10` explicitly (they test other mechanics; fixtures sit
under the new production floor by design).
- Affected sweep (`-k "compress or ccr or crusher or router or mixed or
kompress or hermes"`, ~2.9k tests): green apart from order-dependent
flakes that shift identity between runs (deepseek tokenizer `AutoConfig`
import, hermes/proxy-ccr) — each passes standalone and in direct
combination with the new tests; the full CI shards are the authoritative
check.
- ruff 0.16.3 `check` + `format --check` clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01EWKCmcH47hvvoQ35wftXhE

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-08-27 13:22:26 +05:30
Rod Boev
5bd2266f16
fix(kompress): raise the default execution-slot wait (#2456) 
## Description

Concurrent Kompress requests currently fail open after a 25 ms
execution-slot wait even though ordinary ONNX inference can hold the
single slot for hundreds of milliseconds. This raises the existing
default wait to 3000 ms while retaining concurrency one, the
`HEADROOM_KOMPRESS_EXECUTION_TIMEOUT_MS` override, the tighter acquire
and request budgets, and passthrough after a genuine timeout.

The reproduction and validated 3000 ms setting come from
https://github.com/headroomlabs-ai/headroom/issues/2451

Closes #2451

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Raise the default Kompress execution-slot wait from 25 ms to 3000 ms.
- Start the Kompress request deadline at call entry and carry it through
single-item acquire, single-to-batch delegation, and sequential-fallback
lineage.
- Cap the raised execution-slot wait by that live request deadline on
both single-item and batch acquire paths.
- Keep the per-backend default concurrency at one and preserve all
tighter time budgets.
- Add queued single-item, batch, request-deadline, carried-deadline
lineage, and router-watchdog lifecycle regressions at the same owner
layer that currently fails.
- Preserve the explicit short-timeout fail-open path.

## Testing

- [x] Unit tests pass (`uv run pytest tests/test_kompress_failsafe.py
tests/test_kompress_request_nonblocking.py
tests/test_content_router_single_item_deadline.py
tests/test_transforms/test_kompress_deadline.py -v`)
- [x] Linting passes (`uv run ruff check
headroom/transforms/kompress_compressor.py
tests/test_kompress_failsafe.py
tests/test_content_router_single_item_deadline.py
tests/test_transforms/test_kompress_deadline.py`)
- [x] Formatting passes (`uv run ruff format
headroom/transforms/kompress_compressor.py
tests/test_kompress_failsafe.py
tests/test_content_router_single_item_deadline.py
tests/test_transforms/test_kompress_deadline.py --check`)
- [x] New regression tests prove the saturation fix
- [ ] Manual testing performed

### Test Output

```text
uv run pytest tests/test_kompress_failsafe.py tests/test_kompress_request_nonblocking.py tests/test_content_router_single_item_deadline.py tests/test_transforms/test_kompress_deadline.py -v
37 passed in 4.22s

uv run ruff check headroom/transforms/kompress_compressor.py tests/test_kompress_failsafe.py tests/test_content_router_single_item_deadline.py tests/test_transforms/test_kompress_deadline.py
All checks passed!

uv run ruff format headroom/transforms/kompress_compressor.py tests/test_kompress_failsafe.py tests/test_content_router_single_item_deadline.py tests/test_transforms/test_kompress_deadline.py --check
4 files already formatted
```

## Real Behavior Proof

- Environment: worktree Python environment from `uv sync --extra dev`,
focused pytest with real Python threads and `threading.BoundedSemaphore`
- Exact command / steps: hold the sole execution slot with the
environment override unset, start queued single-item and batch
compression workers, wait until each worker proves it reached a blocked
acquire on the shared execution semaphore, release the slot, rerun the
explicit 1 ms timeout preservation case, then set
`HEADROOM_COMPRESSION_DEADLINE_MS=10` and repeat the held-slot
single-item and batch acquires plus a router single-cache-miss run whose
Kompress load sleeps past the request deadline.
- Observed result: The queued single-item and batch workers each proved
a real blocked acquire before release, then acquired after release and
compressed, while `HEADROOM_KOMPRESS_EXECUTION_TIMEOUT_MS=1` still
passed through promptly, the 10 ms request deadline capped the raised
default wait so both held-slot paths failed open before 200 ms without
reaching model inference, the single-to-batch and sequential-fallback
lineage regressions proved later branches inherit the original request
start instead of resetting it, and the router lifecycle proof showed the
carried deadline now allows slow Kompress load to start but still
expires before model inference after the outer request has already
failed open.
- Not tested: live ONNX proxy savings under sustained concurrent load

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

`CHANGELOG.md` stays unchanged because the release pipeline generates
changelog entries from conventional commits. The fail-open path from
#1430 stays intact; this change stops it from firing spuriously under
ordinary queueing.
 2026-07-22 06:17:33 -07:00
Rod Boev
e9000863fc
fix(kompress): fail-open wall-clock guard on single-cache-miss compression (#2114) 
## Description

The single-cache-miss branch in `ContentRouter` ran compression inline
on the request path without its own wall-clock guard, so a cooperative
stall waited for the full call even when
`HEADROOM_COMPRESSION_DEADLINE_MS` was meant to fail open. This change
adds a branch-level watchdog that returns `PASSTHROUGH` after the
deadline, scoped only to the one-pending-task path and not the native
GIL-hold root cause.

Closes #2046

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added `_compression_deadline_seconds()` and a watchdog around the
single-cache-miss inline compression branch in `ContentRouter`.
- Returned the original content with `PASSTHROUGH` and logged a
fail-open warning after the configured deadline, while preserving
under-deadline and deadline-disabled behavior.
- Added focused regressions for timeout, under-deadline output, and
disabled-deadline behavior, then kept the wider deadline suite green.
- Raised the locked production floors for `click` and `pillow` to clear
the current `pip-audit` findings that now fail external PR merge
snapshots.

## Testing

- [x] Unit tests pass (`uv run pytest
tests/test_content_router_single_item_deadline.py
tests/test_transforms/test_kompress_deadline.py -q`)
- [x] Linting passes (`uv run ruff check
headroom/transforms/content_router.py
tests/test_content_router_single_item_deadline.py
tests/test_transforms/test_kompress_deadline.py`)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
uv run pytest tests/test_content_router_single_item_deadline.py tests/test_transforms/test_kompress_deadline.py -q

============================= test session starts =============================
platform win32 -- Python 3.12.13, pytest-9.0.3, pluggy-1.6.0
rootdir: D:\Repos\headroom-pr-2046-compression-freeze
configfile: pyproject.toml
plugins: anyio-4.12.1, langsmith-0.9.3, asyncio-1.3.0, cov-7.0.0
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 5 items

tests\test_content_router_single_item_deadline.py ...                    [ 60%]
tests\test_transforms\test_kompress_deadline.py ..                       [100%]

============================== 5 passed in 0.42s ==============================

uv run ruff check headroom/transforms/content_router.py tests/test_content_router_single_item_deadline.py tests/test_transforms/test_kompress_deadline.py
All checks passed!

uv run ruff format headroom/transforms/content_router.py tests/test_content_router_single_item_deadline.py tests/test_transforms/test_kompress_deadline.py --check
3 files already formatted
```

## Real Behavior Proof

- Environment: Windows, Python 3.12.13, router-level harness with a
cooperative slow-compression stub
- Exact command / steps: `uv run pytest
tests/test_content_router_single_item_deadline.py
tests/test_transforms/test_kompress_deadline.py -q`, which forces one
frozen prefix and one cache miss, then sleeps past a 10 ms deadline
- Observed result: the guarded branch returns the original content
through `PASSTHROUGH` at the deadline, while under-deadline and
deadline-disabled behavior stay unchanged
- Not tested: native GIL-holding freeze

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

Not applicable.

## Additional Notes

- Security CI currently flags `click 8.3.1` and `pillow 12.2.0`, so this
PR carries the narrow floor bump to `click>=8.3.3` and `pillow>=12.3.0`
as a supply-chain unblock for the same final merge snapshot.
- `CHANGELOG.md` remains untouched because Headroom generates release
notes from conventional commits.
- This PR is a Python-side mitigation for the single-cache-miss branch
only; the native GIL-hold root cause remains a separate owner.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
 2026-07-13 23:50:49 -04:00