mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
fix(transforms): stop folding datetime-prefixed user messages as search results (#3221)
## Description Interactive `headroom wrap copilot` sessions intermittently lose the user's message: the model answers "How can I help you today?" to a real task prompt. Root cause: Copilot CLI prepends `<current_datetime>…</current_datetime>` to every interactive user turn; the ISO-8601 timestamp matches the grep `file:line:` detector, so a datetime + one-line prompt (1 match / 2 non-empty lines = 50% ≥ 30%) classifies as `SEARCH_RESULTS`, and `SearchCompressor` — which keeps only detector-matching lines — deletes the prompt before upstream. On the OpenAI chat streaming path there is no retrieval tool, so the loss is unrecoverable. Fix: `_try_detect_search` now (a) requires the pre-colon segment to look like a file path (no `<`, `>`, `=`), and (b) requires at least two matching lines, so one coincidental `word:digits:` line can no longer classify a whole payload. A genuine one-line grep result loses nothing: all its lines match, so the compressor would have kept it verbatim anyway. Closes #3220 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/transforms/content_detector.py`: new `_is_search_result_line` helper (path-like prefix gate); `_try_detect_search` gains a two-matching-line absolute floor. - `tests/test_transforms_content_detection.py`: regression tests — datetime-prefixed one-liner not search; two-line floor; tag-like / `key=value` prefixes rejected; genuine grep output still detected. - `tests/test_transforms_content_router.py`: router-level regression — the incident payload never routes to SEARCH and the prose survives `ContentRouter().compress()`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/python -m pytest tests/test_transforms_content_detection.py tests/test_transforms_content_router.py tests/test_mixed_content_sections.py tests/test_text_compressors.py tests/test_transforms_tabular.py -q 135 passed in 21.06s $ .venv/bin/ruff check headroom/transforms/content_detector.py tests/test_transforms_content_detection.py tests/test_transforms_content_router.py All checks passed! $ .venv/bin/mypy headroom/transforms/content_detector.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS 26.5 arm64, Python 3.13, editable source build 0.37.0-dev; upstream `api.githubcopilot.com`, cheapest subscription model `kimi-k2.7-code`. - Exact command / steps: standalone copilot-routed proxy (`OPENAI_TARGET_API_URL=https://api.githubcopilot.com headroom proxy --port 8899`) + `.overlay/e2e-copilot-content-probe.sh --port 8899 --model kimi-k2.7-code`, which sends the real interactive wire shape (`<current_datetime>…` + one-line sentinel prompt, streaming) and a multi-line control. - Observed result: BEFORE the fix, probe 1 FAIL — model replied "Hello! I see the current datetime is … How can I assist you today?" with proxy log `transforms=router:search:0.50` (prompt deleted). AFTER the fix, both probes PASS — the sentinel echoes verbatim, proving the user message reached upstream intact. - Not tested: other harnesses' interactive wrappers (claude/droid/auggie send different shapes; the detector fix is generic); the mixed-content section splitter has its own grep pattern (out of scope — its 1-line "search" sections are kept verbatim, no data loss). ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: N/A (no flag). - Stable/default behavior changed: content with exactly one `path:line:`-shaped line no longer classifies as search results (stays uncompressed instead — safe direction; compression only ever engages on ≥2 matching lines now). - Kill switch / disable path: N/A. - Unsafe override required: no. - Qualification impact: none. - Rollback path: revert; prior behavior restores (with the bug). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — proxy transform change; no UI. ## Additional Notes Detection-precision tradeoff is documented in code comments: single-line genuine grep output is no longer folded (no data loss either way — the compressor keeps all-matching content verbatim). A residual edge (prose with ≥2 coincidental `x:1:` lines in ≤6 lines) is accepted and documented in the issue.
This commit is contained in:
parent
7550efb68f
commit
7784bb1846
3 changed files with 88 additions and 2 deletions
|
|
@ -457,6 +457,23 @@ def _try_detect_html(content: str) -> DetectionResult | None:
|
|||
)
|
||||
|
||||
|
||||
def _is_search_result_line(line: str) -> bool:
|
||||
"""True when a line looks like ``path:line:content`` grep output.
|
||||
|
||||
The bare ``^[^\\s:]+:\\d+:`` shape also matches ISO-8601 timestamps
|
||||
(``…T09:57:59…``) and XML-ish wrappers harnesses prepend to user turns
|
||||
(Copilot CLI's ``<current_datetime>…`` line), which misroutes prose to
|
||||
the SearchCompressor — and that compressor keeps only matching lines,
|
||||
deleting the rest. So the pre-colon segment must additionally look like
|
||||
a file path: no angle brackets and no ``=`` (rules out markup tags and
|
||||
``key=value:12:`` log lines).
|
||||
"""
|
||||
if not _SEARCH_RESULT_PATTERN.match(line):
|
||||
return False
|
||||
prefix = line.split(":", 1)[0]
|
||||
return "<" not in prefix and ">" not in prefix and "=" not in prefix
|
||||
|
||||
|
||||
def _try_detect_search(content: str) -> DetectionResult | None:
|
||||
"""Try to detect grep/ripgrep search results."""
|
||||
lines = content.split("\n")[:100] # Check first 100 lines
|
||||
|
|
@ -465,10 +482,16 @@ def _try_detect_search(content: str) -> DetectionResult | None:
|
|||
|
||||
matching_lines = 0
|
||||
for line in lines:
|
||||
if line.strip() and _SEARCH_RESULT_PATTERN.match(line):
|
||||
if line.strip() and _is_search_result_line(line):
|
||||
matching_lines += 1
|
||||
|
||||
if matching_lines == 0:
|
||||
# Absolute floor: a single coincidental `word:digits:` line (a timestamp,
|
||||
# a URL, a time literal inside prose) must not classify a whole payload as
|
||||
# search results — the SearchCompressor drops every non-matching line, so
|
||||
# a false positive is data loss. A genuine one-line grep result loses
|
||||
# nothing by staying uncompressed: all of its lines match, so the
|
||||
# compressor would have kept it verbatim anyway.
|
||||
if matching_lines < 2:
|
||||
return None
|
||||
|
||||
# Calculate confidence based on proportion of matching lines
|
||||
|
|
|
|||
|
|
@ -170,6 +170,49 @@ def test_search_detection_uses_match_ratio() -> None:
|
|||
assert _try_detect_search("\n\n") is None
|
||||
|
||||
|
||||
def test_search_detection_rejects_datetime_prefixed_user_message() -> None:
|
||||
"""Regression: wrap-copilot ate one-line interactive prompts (2026-08-23).
|
||||
|
||||
Copilot CLI prepends ``<current_datetime>…</current_datetime>`` to every
|
||||
interactive user turn. The ISO-8601 ``T09:57:59`` matched the grep
|
||||
``file:line:`` pattern, so a datetime + one-line prompt classified as
|
||||
SEARCH_RESULTS (1 match / 2 lines = 50% ≥ 30%) and the SearchCompressor
|
||||
deleted the prompt line — the model received only the timestamp.
|
||||
"""
|
||||
incident = (
|
||||
"<current_datetime>2026-08-23T09:57:59.792+02:00</current_datetime>\n"
|
||||
"\n"
|
||||
"Please update the PR desc and check .overlay/ for hints."
|
||||
)
|
||||
assert _try_detect_search(incident) is None
|
||||
assert detect_content_type(incident).content_type is not ContentType.SEARCH_RESULTS
|
||||
|
||||
|
||||
def test_search_detection_requires_two_matching_lines() -> None:
|
||||
"""A single coincidental ``word:digits:`` line must not classify prose."""
|
||||
assert _try_detect_search("src/foo.py:12:def foo():") is None
|
||||
assert (
|
||||
_try_detect_search(
|
||||
"Meeting at 09:30:00 tomorrow.\nBring the reports.\nDo not forget coffee."
|
||||
)
|
||||
is None
|
||||
)
|
||||
# Two genuine grep lines still classify.
|
||||
two = "src/foo.py:12:def foo():\nsrc/bar.py:34: foo()"
|
||||
result = _try_detect_search(two)
|
||||
assert result is not None
|
||||
assert result.content_type is ContentType.SEARCH_RESULTS
|
||||
|
||||
|
||||
def test_search_detection_rejects_tag_like_and_key_value_prefixes() -> None:
|
||||
"""Markup / key=value lines are not file paths even with ``:\\d+:`` inside."""
|
||||
assert (
|
||||
_try_detect_search('<log time="10:00:00">started</log>\n<log time="10:00:01">stopped</log>')
|
||||
is None
|
||||
)
|
||||
assert _try_detect_search("timeout=30:12:retried\ntimeout=31:12:retried") is None
|
||||
|
||||
|
||||
def test_log_detection_prefers_build_output_patterns() -> None:
|
||||
log_output = "\n".join(
|
||||
[
|
||||
|
|
|
|||
|
|
@ -1785,3 +1785,23 @@ def test_detect_content_overrides_html_misroute_for_grep_and_logs(
|
|||
"</section></main></body></html>"
|
||||
)
|
||||
assert _detect_content(html).content_type is ContentType.HTML
|
||||
|
||||
|
||||
def test_datetime_prefixed_user_prompt_survives_router() -> None:
|
||||
"""Regression (2026-08-23): interactive wrap-copilot prompts were deleted.
|
||||
|
||||
Copilot CLI prepends ``<current_datetime>…</current_datetime>`` to every
|
||||
interactive user turn; the ISO timestamp matched the grep ``file:line:``
|
||||
detector, the one-line prompt classified as SEARCH_RESULTS, and
|
||||
SearchCompressor kept only the datetime line — the model received no
|
||||
request and answered "How can I help you today?". The router must never
|
||||
route this shape to the search line-filter and must keep the prose.
|
||||
"""
|
||||
prompt = (
|
||||
"<current_datetime>2026-08-23T09:57:59.792+02:00</current_datetime>\n"
|
||||
"\n"
|
||||
"Please update the PR desc and check .overlay/ for hints."
|
||||
)
|
||||
result = ContentRouter().compress(prompt)
|
||||
assert result.strategy_used is not CompressionStrategy.SEARCH
|
||||
assert "Please update the PR desc" in result.compressed
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue