Phase B step 1 of the live-zone-only realignment. Removes ~10K LOC of
"drop messages from history" machinery that became unreachable after
PR-A1 made `/v1/messages` a passthrough on the proxy. Live-zone-only
compression (PR-B2..B7) operates on content blocks within messages;
message-list mutation no longer happens in the pipeline.
Python deletes:
- headroom/transforms/intelligent_context.py (1077 LOC)
- headroom/transforms/rolling_window.py (395 LOC)
- headroom/transforms/progressive_summarizer.py (508 LOC)
- headroom/transforms/scoring.py (459 LOC)
- headroom/transforms/tool_crusher.py (338 LOC)
- 5 corresponding tests/test_transforms/* and tests/test_proxy_intelligent_context.py
Rust deletes:
- crates/headroom-core/src/context/* (manager, config, workspace,
candidate, ccr_drop, strategy/, mod) + safety.rs replaced
- crates/headroom-core/src/scoring/* (mod, score, scorer, traits, weights)
- MessageScorerComparator from crates/headroom-parity (PR #338/#343
becomes deletable; sunk cost stays sunk)
- 13 message_scorer fixtures + record_message_scorer.py
Rust adds (move + rewrite):
- crates/headroom-core/src/transforms/safety.rs — `tool_pair_indices`
preserves the OpenAI/Anthropic tool_use ↔ tool_result pairing rule
the live-zone dispatcher (PR-B2) needs. No IcmConfig dependency.
Surface refactors:
- HeadroomConfig: drop `tool_crusher`, `rolling_window`,
`intelligent_context` fields; hoist `output_buffer_tokens` to top
level (used by client.py).
- ProxyConfig: drop `intelligent_context*` fields.
- `headroom wrap` proxy server: retire IntelligentContextManager
and RollingWindow imports + branch; pipeline is CacheAligner →
ContentRouter (smart_routing) or CacheAligner → SmartCrusher
(legacy).
- CLI: drop `--no-intelligent-context`, `--no-intelligent-scoring`,
`--no-compress-first` flags.
- LangChain memory integration: rename `_apply_rolling_window` →
`_apply_compression`, drop RollingWindowConfig dep. Threshold is
now advisory — B6 will rework the contract.
- TransformPipeline.create_pipeline now takes only cache_aligner_config.
- headroom/__init__.py + headroom/transforms/__init__.py: strip
exports of deleted symbols.
Bug fixes uncovered by full pytest sweep:
- providers/copilot/wrap.py: `environ or os.environ` collapsed
empty-dict to falsy → callers passing `environ={}` accidentally
pulled from os.environ. Use `environ if environ is not None else
os.environ`.
Test correctness fixes:
- _DummyAnthropicHandler._retry_request gains **_kwargs to match
the real handler signature post-A8.
- test_ws_http_fallback extracts JSON from `content=` (post-A3
byte-faithful) rather than the obsolete `json=` kwarg.
- test_ccr_response_handler_extra fixture joins SSE events with
`\n\n` per spec (post-A8 byte-buffer parser requirement).
- test_proxy_responses_phase_preservation: capture via direct
handler attached to the named logger, so the assertion is
order-independent (proxy `_setup_file_logging` flips
`headroom.propagate=False` once any earlier test triggers it).
- conftest.py autouse fixture resets `headroom.propagate=True`
before each test as a defensive measure for the same pollution.
- test_wrap_copilot_translated_backend_still_requires_byok:
monkeypatch.delenv every provider key so the BYOK error
actually fires.
- test_native_installers: skip when system bash < 4.3 (macOS ships 3.2).
- TestGeminiEmbedContent / TestGeminiBatchEmbedContents:
pytest.mark.skip — proxy currently has no :embedContent route;
feature gap, not regression.
Acceptance:
- cargo build --workspace + cargo clippy + cargo fmt --check: green.
- cargo test --workspace --exclude headroom-py: 777 passed.
- pytest: 4892 passed, 240 skipped, 0 failed.
- git grep returns only intentional comments referencing the deletion.
Per-PR-B1 plan: REALIGNMENT/04-phase-B-live-zone.md.
P0-1: Delete `_inject_system_context` from `proxy/server.py`. Memory
context now routes exclusively to the first text block of the latest
non-frozen user message via `_append_context_to_latest_non_frozen_user_turn`
(promoted to the canonical default in handlers/anthropic.py). Mirror
applied to OpenAI Responses API at handlers/openai.py: `body["instructions"]`
is no longer mutated; memory context appends to the latest user item in
`body["input"]`.
P2-23: Replace `headroom/transforms/cache_aligner.py` with a detector-only
implementation. The legacy rewrite path (~400 LOC) is removed. The volatile-
content detector uses no regex — UUIDs via `uuid.UUID`, ISO 8601 via
`datetime.fromisoformat`, JWT shape via base64url segment-count check, hex
hashes via length + `int(token, 16)` validation. Volatile findings surface
through `cache_metrics`/`warnings`/`logger.warning`; the prompt is never
mutated.
Configurability: new env var `HEADROOM_MEMORY_INJECTION_MODE` with values
`live_zone_tail` (default) and `disabled`. No `system_prompt` value — that
path is permanently retired.
Structured logs: every memory injection emits `event=memory_injection`
with `decision`, `bytes_injected`, `query_hash` (BLAKE2b, never raw query),
`session_id`, `request_id`. Auth is never logged.
Tests:
- Add `tests/test_proxy_system_prompt_immutable.py` (7 tests).
- Add `tests/test_cache_aligner_detector_only.py` (20 tests).
- Replace `tests/test_transforms/test_cache_aligner.py` (rewrite-path
tests, 58 cases) with detector-only behavior.
- Update `tests/test_acceptance.py::TestDateTrap` to pin the new
detector-only contract.
Acceptance:
- `git grep -n "_inject_system_context\|_inject_to_system_or_instructions" headroom/`
returns nothing.
- `git grep -n "import re\|from re import" headroom/transforms/cache_aligner.py`
returns nothing.
- Targeted suite (`test_proxy_system_prompt_immutable.py`,
`test_cache_aligner_detector_only.py`, `test_proxy_anthropic_cache_stability.py`,
`test_acceptance.py::TestDateTrap`, `test_memory*.py`, `test_cli/`) green.
Stage 3c.1b step 2 + cleanup. The python `SmartCrusher` (3669 lines)
is replaced by a thin pyo3-backed shim (~290 lines) that delegates
every byte to `headroom._core.SmartCrusher` (built from
`crates/headroom-py`, landed in the previous commit). There is no
python implementation and no env-var fallback — the wheel is a hard
import.
Why now: parity was already proven across 17 fixtures + the python-
side bridge test (1+17 in `test_smart_crusher_rust_parity.py`).
Keeping a shadow python impl behind a flag is a permanent maintenance
cost with no operational benefit. Stage 3c.1b deletes ~3380 lines of
python parser/scorer/analyzer/orchestrator code; the rust crate has
its own coverage (388 unit tests + property tests in headroom-core).
Surface preserved (drop-in for every production caller):
- `headroom.transforms.smart_crusher.SmartCrusher` — same class name,
same `__init__(config, relevance_config, scorer, ccr_config)`
signature (the latter three are accepted for source-compat and
silently dropped — rust port keeps those subsystems disabled in
Stage 3c.1, they re-attach in Stage 3c.2).
- `SmartCrusherConfig` and `CrushResult` dataclasses kept as python
dataclasses (callers use `asdict()` / dataclass matching on them).
- `crush(content, query, bias)`, `_smart_crush_content(content, ...)`,
`apply(messages, tokenizer, **kwargs)`, and
`_extract_context_from_messages(messages)` all preserved.
- `smart_crush_tool_output(content, config, ccr_config)` thin wrapper.
The transform-protocol `apply()` orchestration stays python (message
walking, digest-marker insertion, token counting); only the per-
message compression call delegates to rust.
Removed:
- Python parser / planner / scorer / analyzer / classifier (~3380 lines).
- Internal helpers `_classify_array`, `_detect_sequential_pattern`,
`_detect_rare_status_values`, `_detect_items_by_learned_semantics`,
`_percentile_linear`, `_compute_k_split`, `_crush_number_array`,
`_process_value`, etc. — rust crate has parallel coverage.
- `SmartAnalyzer`, `ArrayType`, `CompressionStrategy`,
`extract_query_anchors` — internals; not used by any production
caller (only tests probed them).
Tests deleted (probed deleted internals — same precedent as Stage 3b):
- `tests/test_transforms/test_smart_crusher.py` (40 tests)
- `tests/test_transforms/test_universal_json_crush.py` (45)
- `tests/test_transforms/test_anchor_selector.py` (49)
- `tests/test_toin_field_learning.py` (21)
- `tests/test_crushability.py` (20)
Tests trimmed (removed methods/classes that probe deferred subsystems
— scorer injection, CCR marker injection, TOIN feedback recording —
all of which re-attach in Stage 3c.2):
- `tests/test_transforms/test_smart_crusher_bugs.py`:
TestNumberArraySchemaPreservation, TestStage3c1BugFixes.
- `tests/test_relevance.py`: 2 scorer-injection tests.
- `tests/test_ccr.py`: TestSmartCrusherCCRIntegration class +
test_custom_marker_template.
- `tests/test_toin_integration.py`: TestTOINIntegration +
TestStoreToTOINHash classes.
- `tests/test_critical_fixes.py`: TestSmartCrusherTOINIntegration +
test_full_feedback_loop.
- `tests/test_acceptance.py::TestQueryAnchorExtraction`: dropped the
`extract_query_anchors` probe; kept the end-to-end "Alice
preserved" assertion.
Bug fixes from Stage 3c.1 (#1 percentile linear interp, #2 zero-
padded sequential, #3 rare-status pareto, #4 k-split overshoot) are
pinned by the rust crate and the parity fixtures
(`tests/parity/fixtures/smart_crusher/`).
Tests:
- 517 passed in the smart_crusher-adjacent file set
(test_transforms/, test_relevance*, test_ccr, test_toin_integration,
test_quality_retention, test_acceptance, test_critical_fixes).
- 18 in `test_smart_crusher_rust_parity.py` (1 sanity + 17 fixtures).
- 388 rust unit tests still green.
One stale-error-message regex in `test_relevance_extra.py` updated
from "requires sentence-transformers" → "requires fastembed".
