mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
9c30b62962
## Description
Cross-turn dedup (`HEADROOM_DEDUPE` / `enable_cross_turn_dedup`, plus
the cold-prefix recompaction router) folds a repeated tool-output span
into a one-line in-context pointer, `[↑NL same as msg M: 'anchor']`.
That pointer is only recoverable where the model can resolve the
reference. On the OpenAI chat-completions STREAMING path (what `headroom
wrap copilot` serves) it cannot, for two independent reasons:
1. The proxy itself logs `CCR: skipping retrieval-tool injection for
OpenAI chat streaming; this path cannot intercept tool calls`, so no
`headroom_retrieve` tool exists on this path and nothing can
mechanically resolve a fold.
2. The pointer names its source as `msg M`, Headroom's internal message
index. OpenAI-compatible chat clients never show the model numbered
messages, so the reference is unresolvable even though the original
bytes are technically still earlier in the same request.
Observed with Kimi k2.7-code / k3 via `wrap copilot`: the model treats
the pointer as deleted output, reports "the renderer is
deduplicating/compressing", and retry-loops near-identical reads (one
session burned ~200 turns; a folded conflicted-files listing hid 4 of 5
conflicted files and the agent committed unresolved `<<<<<<<` markers).
The router already keeps unrecoverable LOSSY output verbatim
(`lossy_unrecoverable_skipped`). Dedup folds are lossless in theory but
unrecoverable in practice on this path; this PR gives them the same
recoverability gate.
Closes #3190
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/transforms/content_router.py`: `ContentRouter.apply()`
accepts a per-request `cross_turn_dedup_recoverable` kwarg (default
`True`, so every existing caller is byte-identical). When `False`, the
cross-turn dedup pass is skipped and repeated spans stay verbatim,
mirroring the recoverability posture of the lossy
`lossy_unrecoverable_skipped` guard. Config comment on
`enable_cross_turn_dedup` documents the gate.
- `headroom/proxy/handlers/openai.py`: `handle_openai_chat` computes the
gate from the same predicate that already gates CCR retrieval-tool
injection, `_should_inject_openai_chat_ccr_tool(ccr_inject_tool,
stream)`, and threads it into both `openai_pipeline.apply(...)` call
sites (token-mode and non-token-mode branches). Streaming chat requests
skip the fold; buffered (non-streaming) chat, which can inject and
redeem the retrieval tool, keeps folding.
- `headroom/transforms/cold_prefix.py`: `cold_recompact_messages` no
longer hardcodes pointer emission; new keyword-only
`cross_turn_dedup_recoverable: bool = True` is forwarded to the router
gate. The only caller (Anthropic cache-mode cold turn) keeps the default
and is unchanged.
- `tests/test_cross_turn_dedup.py`: router-gate regression tests
(unrecoverable path keeps verbatim bytes for both the OpenAI `role:tool`
string shape and the Anthropic `tool_result` block shape;
default/explicit-`True` still folds).
- `tests/test_cold_prefix.py` (new): recompaction folds by default
(Anthropic path unchanged) and keeps verbatim bytes with
`cross_turn_dedup_recoverable=False`.
- `tests/test_openai_chat_dedup_recoverability.py` (new): end-to-end
through the real `/v1/chat/completions` handler with
`HEADROOM_DEDUPE=1`, capturing the exact upstream request body:
`stream=True` keeps both copies byte-verbatim with no `[↑` pointer;
`stream=False` still folds; `stream=False` under `--lossless` (which
forces `ccr_inject_tool=False`) also keeps verbatim bytes, locking the
intended coupling of "no retrieval tool" to "no bare pointer".
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
# BEFORE (branch base, fix reverted): the streaming regression test fails,
# the upstream body carries the unresolvable pointer and drops the bytes.
$ git stash push headroom/ && uv run pytest -q \
tests/test_openai_chat_dedup_recoverability.py::test_streaming_chat_keeps_verbatim_bytes_no_dedup_pointer
E assert '[↑' not in "fix the ove...t merge.py']"
E '[↑' is contained here:
E [↑14L same as msg 2: '$ cat merge.py']
FAILED tests/test_openai_chat_dedup_recoverability.py::test_streaming_chat_keeps_verbatim_bytes_no_dedup_pointer
(same run: test_cold_recompact_unrecoverable_path_keeps_verbatim_bytes also fails pre-fix;
both recoverable-path legs pass before and after)
# AFTER (full diff applied):
$ uv run pytest tests/test_cross_turn_dedup.py tests/test_cold_prefix.py \
tests/test_openai_chat_dedup_recoverability.py \
tests/test_proxy/test_openai_chat_ccr_injection.py tests/test_no_ccr_lossy.py \
tests/test_openai_chat_turn_hooks.py tests/test_openai_chat_tool_desc_compaction.py \
tests/test_responses_cross_turn_dedup.py -q
45 passed, 2 warnings in 8.75s
$ uv run pytest tests/test_proxy/ tests/test_openai_codex_routing.py \
tests/test_openai_chat_turn_hooks.py tests/test_openai_chat_tool_desc_compaction.py \
tests/test_openai_beta_session_sticky.py tests/test_openai_max_completion_tokens.py \
tests/test_no_ccr_lossy.py tests/test_netcost_gate.py tests/test_agent_savings.py \
tests/test_cross_turn_dedup.py tests/test_cold_prefix.py \
tests/test_openai_chat_dedup_recoverability.py -q
424 passed, 2 warnings in 73.52s
$ uv run ruff format --check <touched files> && uv run ruff check <touched files>
All checks passed!
$ uv run mypy headroom/transforms/cold_prefix.py headroom/transforms/content_router.py headroom/proxy/handlers/openai.py
Success: no issues found in 3 source files
$ cargo fmt --all -- --check # FMT_OK
$ cargo clippy --all-targets # 2 pre-existing warnings in untouched lib-test code, no errors
$ cargo test # all targets green; see Additional Notes for the one environmental exception
```
## Real Behavior Proof
- Environment: macOS (Darwin), Python 3.13, repo tip `upstream/main`
5e0ce242 (v0.36.2). No secrets, no external network: the proof drives
the real proxy handler in-process via FastAPI `TestClient` with the
upstream send stubbed, capturing the exact request body the provider
would receive.
- Exact command / steps (copy-pasteable, self-contained): next lines
```sh
# 1. The bug, on the branch base (pointer emitted on the streaming
path):
git stash push headroom/ # or check out upstream/main
uv run pytest -q tests/test_openai_chat_dedup_recoverability.py #
streaming leg FAILS
git stash pop
# 2. The fix:
uv run pytest -q tests/test_openai_chat_dedup_recoverability.py # both
legs pass
```
The test posts a chat-completions request whose history contains two
identical multi-line tool outputs (the shape that folds), with
`HEADROOM_DEDUPE=1`, and asserts on the captured upstream body:
- `stream=True` (the `wrap copilot` shape): both copies forwarded
byte-verbatim, no `[↑NL same as msg M]` pointer anywhere.
- `stream=False` (buffered, retrieval tool injectable): the repeated
span still folds to a pointer; the earliest copy stays verbatim as the
in-context original.
- Observed result: BEFORE, the streaming leg fails with the pointer
present in the upstream body (same
`transforms=router:cross_turn_dedup:N` evidence seen in proxy.log when
the bug bit). AFTER, streaming keeps verbatim bytes and buffered keeps
folding; the full touched-module suite (423 tests) is green.
- Not tested: a live `wrap copilot` session against the real Copilot API
(needs a subscription token; the in-process test captures the identical
upstream body the handler produces). The Responses API path
(`_dedup_responses_output_items`, Codex) is intentionally untouched:
Responses streaming has a separate buffered-CCR path that can intercept
tool calls. `/v1/compress` derived pipelines keep the default
(recoverable) behavior. Separately worth verifying in a follow-up:
whether `headroom_retrieve` resolves `msg M` dedup pointers on the paths
that keep folding, or only CCR `hash=` content markers (the
Anthropic-path fold is retained per the issue's scope, where it has not
been observed to cause retry loops).
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: only the OpenAI chat-completions
request path, and only when cross-turn dedup is active (opt-in
`HEADROOM_DEDUPE=1`, or cold-prefix recompaction): streaming chat now
keeps repeated tool-output bytes verbatim instead of emitting `[↑NL same
as msg M]` pointers, and (because `--lossless` forces
`ccr_inject_tool=False`) buffered chat in lossless mode does the same.
Buffered chat with CCR on, Anthropic, Responses, and `/v1/compress` are
byte-identical to before (default `cross_turn_dedup_recoverable=True`;
the Responses fold is covered by the untouched, still-green
`tests/test_responses_cross_turn_dedup.py`).
- Kill switch / disable path: dedup remains opt-in via
`HEADROOM_DEDUPE`; the gate itself can be overridden per request by
passing `cross_turn_dedup_recoverable=True`.
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the single commit; no state, schema, or config
migration involved.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation (docstrings
+ config comments)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
- Mirrors the existing recoverability precedent: the lossy path already
refuses to emit unrecoverable output (`lossy_unrecoverable_skipped`,
issue #1307); this extends the same posture to cross-turn dedup folds.
- The gate reuses `_should_inject_openai_chat_ccr_tool`, the predicate
that already decides whether the chat path can redeem an injected
retrieval tool, so the two can never drift apart.
- Prefer-false-negatives posture: a skipped fold only ever means bytes
stay verbatim; no content is dropped, reordered, or lossy-transformed by
this change.
- Secondary operational bug noticed while diagnosing (NOT fixed here,
separate issue candidate): all concurrent proxy processes write the same
`~/.headroom/logs/proxy.log` with independent rotating handlers, so
rotation stomps history across `wrap` instances on different ports.
- Local environment note: `cargo test` on this machine hangs inside
`crates/headroom-core/tests/kompress_parity.rs` (both tests stall in
`ort` ONNX-runtime environment init, reproducible on the untouched
branch base; this PR changes no Rust). With those two tests skipped, the
full Rust suite is green (all targets `ok`, 0 failed). `cargo clippy
--all-targets` and `cargo fmt --all -- --check` pass as-is.
69 lines
2.7 KiB
Python
69 lines
2.7 KiB
Python
"""Cold-prefix recompaction: the cross-turn dedup fold must be path-aware.
|
|
|
|
``cold_recompact_messages`` builds its own lossless ContentRouter with
|
|
``enable_cross_turn_dedup=True``. The fold rewrites repeated tool-output spans
|
|
to bare ``[↑NL same as msg M]`` in-context pointers — unresolvable on paths
|
|
where no CCR retrieval tool can be injected and the client never shows the
|
|
model numbered messages (OpenAI chat-completions streaming, wrap copilot).
|
|
The recompaction therefore takes ``cross_turn_dedup_recoverable`` and forwards
|
|
it to the router gate: unrecoverable paths keep the bytes verbatim, the
|
|
Anthropic cache-mode caller (default True) keeps folding.
|
|
"""
|
|
|
|
from headroom.transforms.cold_prefix import cold_recompact_messages
|
|
|
|
|
|
def _mk_tok():
|
|
from headroom.providers import OpenAIProvider
|
|
from headroom.tokenizer import Tokenizer
|
|
|
|
return Tokenizer(OpenAIProvider().get_token_counter("gpt-4o"), "gpt-4o")
|
|
|
|
|
|
def _toolmsg(text, tid):
|
|
return {
|
|
"role": "user",
|
|
"content": [{"type": "tool_result", "tool_use_id": tid, "content": text}],
|
|
}
|
|
|
|
|
|
def _conversation():
|
|
span = "\n".join(f" result_{i} = compute_overdraft(business_id={i})" for i in range(12))
|
|
return [
|
|
{"role": "user", "content": "fix the overdraft bug"},
|
|
{"role": "assistant", "content": "cat merge.py"},
|
|
_toolmsg(f"$ cat merge.py\n{span}\n# end", "t1"),
|
|
{"role": "assistant", "content": "sed -n range"},
|
|
_toolmsg(f"$ sed -n 1,20p merge.py\n{span}\n# more", "t2"),
|
|
]
|
|
|
|
|
|
def test_cold_recompact_folds_by_default():
|
|
# Anthropic cache-mode path (the only caller today): unchanged — the
|
|
# repeated span still folds to an in-context pointer.
|
|
msgs = _conversation()
|
|
out, transforms = cold_recompact_messages(msgs, tokenizer=_mk_tok())
|
|
later = out[-1]["content"][0]["content"]
|
|
assert "[↑" in later
|
|
assert any("cross_turn_dedup" in t for t in transforms)
|
|
|
|
|
|
def test_cold_recompact_unrecoverable_path_keeps_verbatim_bytes():
|
|
# Unresolvable-pointer path (OpenAI chat streaming shape): the fold is
|
|
# skipped, the repeated span stays byte-verbatim, and no pointer is
|
|
# emitted — while the recompaction itself still runs (message count and
|
|
# order unchanged).
|
|
msgs = _conversation()
|
|
out, transforms = cold_recompact_messages(
|
|
msgs, tokenizer=_mk_tok(), cross_turn_dedup_recoverable=False
|
|
)
|
|
later = out[-1]["content"][0]["content"]
|
|
assert "[↑" not in later
|
|
assert (
|
|
later
|
|
== "$ sed -n 1,20p merge.py\n"
|
|
+ "\n".join(f" result_{i} = compute_overdraft(business_id={i})" for i in range(12))
|
|
+ "\n# more"
|
|
)
|
|
assert not any("cross_turn_dedup" in t for t in transforms)
|
|
assert len(out) == len(msgs)
|