mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
c9aaba3f5b
`headroom/transforms/tag_protector.py` was a regex-driven scan-and-
replace loop that ran on every kompress call from ContentRouter
(`content_router.py:1089`). The Python implementation had five real
bugs we now fix in the port — the most consequential being a
`str.replace(.., .., 1)` first-occurrence-replace bug that silently
collapsed two identical custom-tag blocks in the same input to a
single placeholder + a stray duplicate of the second block.
# Bug fixes (each pinned by a `fixed_in_3e4` test)
* **#1: O(n²) on nested custom tags.** Python's `while changed` loop
restarted a full regex scan after every replacement. Rust walks
once in linear time on input length.
* **#2: First-occurrence replace bug.** `result.replace(orig, ph, 1)`
replaces the FIRST textual match, not the matched offset. Two
identical custom-tag blocks collapsed to one placeholder + a stray
duplicate of the second block. The Rust walker stitches output by
offset so distinct blocks always get distinct placeholders.
* **#3: Silent 50-iteration cap.** Python had a hard `max_iterations
= 50` safety limit that quietly truncated tag protection on deeply
nested input. The Rust walker is bounded by input length only.
* **#4: Self-closing pass duplicate-replace risk.** Python ran a
second loop with the same `replace_first` bug for self-closers.
Rust handles self-closers in the same single pass.
* **#5: Placeholder collision.** If the input contained a literal
`{{HEADROOM_TAG_…}}` substring, Python silently let the collision
break restoration. Rust salts the prefix and reports it in stats.
# Architecture
Two-phase walker:
* Phase 1 (`identify_spans`): linear scan over input bytes, hand-
rolled tag-open / tag-close lexer (no regex). Maintains a stack of
open custom tags; on a matching close, collapses the inner span
into a single `Span { start, end, Block }`. Self-closing custom
tags become `Span { ..., SelfClosing }` immediately. Marker-only
mode (`compress_tagged_content=true`) emits Open/CloseMarker spans
instead. Orphan opens stay un-protected (matches Python behavior).
Orphan closes are emitted verbatim and counted in stats.
* Phase 2 (`emit_output`): walks `text` once, splicing placeholders
for span ranges and copying everything else verbatim. Offset-based,
never `str.replace`.
PyO3 surface: `protect_tags`, `restore_tags`, `is_html_tag`,
`known_html_tag_names`. The Python shim retires the regex internals
and re-exports `KNOWN_HTML_TAGS` (rebuilt from the Rust list) +
`_is_html_tag` for backwards compat with `content_router.py` and the
existing test surface.
# Test plan
* 25 Rust unit tests including 4 `fixed_in_3e4_*` bug-fix tests
* 27 Python tests (23 existing + 4 new `fixed_in_3e4` parity tests)
* 5 integration tests in `test_tag_protection_integration.py` pass
* `make ci-precheck` clean
|
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_cache_aligner.py | ||
| test_code_compressor.py | ||
| test_content_router.py | ||
| test_diff_compressor.py | ||
| test_diff_compressor_rust_parity.py | ||
| test_html_extractor.py | ||
| test_intelligent_context.py | ||
| test_kompress_compressor.py | ||
| test_progressive_summarizer.py | ||
| test_read_lifecycle.py | ||
| test_rolling_window.py | ||
| test_scoring.py | ||
| test_smart_crusher_bugs.py | ||
| test_smart_crusher_ccr_roundtrip.py | ||
| test_smart_crusher_lossless_default.py | ||
| test_smart_crusher_rust_parity.py | ||
| test_tag_protector.py | ||
| test_tool_crusher.py | ||