mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
Comprehensive PR-by-PR plan to realign Headroom around live-zone-only compression with prefix-cache safety as a non-negotiable invariant. Drafted from a 10-agent deep audit against the LLM-proxy compression guide. - 14 documents under REALIGNMENT/ - 72 ranked bugs (P0 cache-killers through P6 test-infra) - 40 feature PRs + 10 test-infra PRs across 9 phases - ~25K LOC retirement (ICM + scoring + relevance + rolling-window + summarizer + tool-crusher + LiteLLM-fake-Bedrock) - Preserves TOIN, CCR, Kompress-base per user direction - Auth-mode policy gates (PAYG / OAuth / subscription) - Phase 3 cache stabilization surface (tool-sort, schema-sort, cache_control auto-place, prompt_cache_key) - Native Bedrock SigV4 + Vertex ADC handlers - Test infrastructure: SHA-256 byte-faithful gate, SSE corner cases, property tests, real-traffic shadow
421 lines
26 KiB
Markdown
421 lines
26 KiB
Markdown
# 01 — Comprehensive Bug & Gap List
|
|
|
|
Ranked P0 (cache-killer) → P5 (long tail). Every entry has: title, file:line, evidence, guide §, fix, ROI estimate.
|
|
|
|
Sources: 10 parallel deep-audit subagents (Rust proxy passthrough; Rust compression correctness; Python proxy + bridges; prefix cache safety; streaming + wire-format; RTK + tests/parity; over-engineering; OpenAI long-tail + Bedrock; Headroom-side injections; auth-mode handling).
|
|
|
|
---
|
|
|
|
## P0 — Cache-killer smoking guns (every customer affected)
|
|
|
|
These bugs collapse Anthropic prompt-cache hit rate toward 0% for any traffic that triggers them. Fix in Phase A.
|
|
|
|
### P0-1. System prompt mutated by `.strip()` and memory-context append
|
|
- **File:** `headroom/proxy/server.py:1050-1058`; `headroom/proxy/handlers/openai.py:1212`
|
|
- **Evidence:** `body["system"] = (existing_system + "\n\n" + context).strip()` — strips whitespace and appends dynamic memory context to the cache hot zone on every memory-enabled call.
|
|
- **Guide:** §1.11 (whitespace fidelity), §6.3 #10 (tiny system prompt edits invalidate cache), §10.1 (system = always cache hot).
|
|
- **Fix:** Remove `_inject_system_context` path; route memory context to the **first block of the latest user message** (live zone). The existing `_append_context_to_latest_non_frozen_user_turn` already does this — make it the only path.
|
|
- **ROI:** Restores cache hits for ~all memory-enabled traffic.
|
|
- **Phase A → PR-A2.**
|
|
|
|
### P0-2. Every Python forwarder re-serializes JSON via `httpx ... json=body`
|
|
- **File:** `headroom/proxy/server.py:1088, 1090`; `headroom/proxy/handlers/streaming.py:651`; `headroom/proxy/handlers/openai.py:2392-2397`; `headroom/proxy/handlers/batch.py:344`
|
|
- **Evidence:** httpx default encoder calls `json.dumps(body, separators=(", ", ": "), ensure_ascii=True)`. Inbound bytes use `,`/`:` and raw UTF-8 in user content; outbound bytes use `, `/`: ` and `\uXXXX` escapes. Bytes never reach upstream byte-equal to bytes that arrived.
|
|
- **Guide:** §1.9 (the single most expensive proxy mistake), §1.10 (numeric precision), §1.11 (whitespace fidelity).
|
|
- **Fix:** Switch every forwarder to `httpx ... content=raw_bytes_modified_in_place`. Keep the original `await request.body()` bytes; if a transform mutated the body, re-serialize with `separators=(",", ":")` + `ensure_ascii=False`. Better: surgical byte-fragment replacement on `messages` only, leaving the envelope's bytes untouched.
|
|
- **ROI:** Restores cache hits for **all** Python-forwarded traffic.
|
|
- **Phase A → PR-A3.**
|
|
|
|
### P0-3. Rust proxy ignores customer `cache_control` markers
|
|
- **File:** `crates/headroom-proxy/src/compression/anthropic.rs:151-156`
|
|
- **Evidence:** `frozen_message_count: 0` hardcoded with `TODO: detect provider prefix-cached messages from the request. Until we wire that detection, we treat the whole list as droppable.` Combined with ICM, every compression event drops messages from index 0.
|
|
- **Guide:** §2.19 (up to 4 cache_control markers), §6.2 (cache breakpoints define the prefix).
|
|
- **Fix:** Walk `messages[*].content[*].cache_control`, `system[*].cache_control`, `tools[*].cache_control`; set `frozen_message_count` to the highest message index that contains a cache_control marker.
|
|
- **ROI:** Restores cache hits for **all** clients using Anthropic prompt caching (which is virtually all production Anthropic traffic).
|
|
- **Phase A → PR-A4.**
|
|
|
|
### P0-4. ICM compresses by dropping messages from cache hot zone (wrong scope)
|
|
- **File:** `crates/headroom-proxy/src/compression/anthropic.rs:146-157`; `crates/headroom-core/src/context/strategy/drop_by_score.rs:64-80`; `crates/headroom-core/src/context/manager.rs`; `headroom/transforms/intelligent_context.py:354-450`
|
|
- **Evidence:** ICM with default `keep_last_turns: 2` is allowed to drop any message older than the last two turns. Combined with P0-3, this is a 100%-likely cache-buster on any conversation with ≥3 user turns.
|
|
- **Guide:** §6.5 (live zone vs hot zone), §10.1 ("Old conversation turns ... never compress"), §6.3 #11 ("Truncation/summarization at the head"), §7.2 (append-only compression).
|
|
- **Fix:** Delete ICM. Replace with live-zone-only block-level compression. Phase B builds the replacement.
|
|
- **ROI:** Eliminates the largest single class of cache-bust events.
|
|
- **Phase A → PR-A1 (stop calling ICM); Phase B → PR-B1 (delete ICM).**
|
|
|
|
### P0-5. Numeric precision lost via `serde_json::Value` round-trip
|
|
- **File:** `crates/headroom-proxy/src/compression/anthropic.rs:91, 172`
|
|
- **Evidence:** Body parsed into `serde_json::Value` and re-serialized via `serde_json::to_vec(&parsed)`. `Value::Number` is `i64|u64|f64` so any `1.0` round-trips to `1`; large integers above 2^53 lose precision. `Cargo.toml:34` enables `preserve_order` only — no `arbitrary_precision`, no `RawValue`.
|
|
- **Guide:** §1.10.
|
|
- **Fix:** Add `arbitrary_precision` and `raw_value` features to `serde_json`. Use `&RawValue` for `messages[*]` so individual messages forward as exact byte copies. Strategy outputs only need to be "drop this index" or "replace this block's content."
|
|
- **ROI:** Closes the second-largest re-serialization byte-drift class.
|
|
- **Phase A → PR-A4 (jointly with P0-3).**
|
|
|
|
### P0-6. Memory tool injection toggles tools list and mutates `anthropic-beta`
|
|
- **File:** `headroom/proxy/memory_handler.py:389-398`; `headroom/proxy/handlers/anthropic.py:1147-1171`
|
|
- **Evidence:** Memory adds `memory_save`, `memory_search` tools to `body["tools"]` only when memory is enabled for the request. Mid-session config flicker → tool set changes → cache busts (§6.3 #2). Same code mutates `anthropic-beta` adding `context-management-2025-06-27` when injection happens (§6.3 #6).
|
|
- **Fix:** Make memory tool injection **session-sticky**: once injected, always inject for the lifetime of the session. Pin `anthropic-beta` order; never reorder tokens within the comma-list.
|
|
- **ROI:** Eliminates mid-session cache busts.
|
|
- **Phase A → PR-A6, PR-A7.**
|
|
|
|
### P0-7. `responses_converter.py` drops Codex `phase` field and corrupts multi-text-part rebuild
|
|
- **File:** `headroom/proxy/responses_converter.py:94, 221-256`
|
|
- **Evidence:** `phase` field is dropped on the Chat-Completions trip (line 94 maps `role` only); only `copy.copy(original)` accidentally retains it on the rebuild path. Multi-text-part input messages get corrupted: `_extract_text_from_parts` joins with `\n`, `_reconstruct_item:254-256` puts the concatenated text into the first part only and leaves parts 1..N as-is, doubling content.
|
|
- **Guide:** §4.5 (preserve `phase` exactly), §7.9 (position preservation).
|
|
- **Fix:** Stash `phase` and restore in `_reconstruct_item`. Rebuild text parts by index, replacing each part's text in place. Better: in Phase C, port `/v1/responses` to Rust and never decompose item structure for compression.
|
|
- **Phase A → PR-A8 (Python hotfix), Phase C → PR-C5 (full rebuild).**
|
|
|
|
---
|
|
|
|
## P1 — Wire-format / streaming corruption
|
|
|
|
### P1-8. SSE buffers decoded with `errors="ignore"` / `errors="replace"`
|
|
- **File:** `headroom/proxy/handlers/streaming.py:58, 772`; `headroom/ccr/response_handler.py:672`
|
|
- **Evidence:** `chunk.decode("utf-8", errors="ignore")` silently drops emoji/CJK bytes split across TCP reads. The wire passthrough at `streaming.py:788` is bytes (correct), but every `_parse_sse_usage_from_buffer` and `_parse_sse_to_response` decision is made on a string that may have lost bytes.
|
|
- **Guide:** §1.4 (UTF-8 multi-byte split across chunks).
|
|
- **Fix:** Bytes-level buffer; find `\n\n` boundary in bytes; decode each complete event after split.
|
|
- **Phase C → PR-C1 (Rust SSE parser); Phase A → PR-A8 includes a Python hotfix.**
|
|
|
|
### P1-9. SSE parser misses `thinking_delta`, `signature_delta`, `citations_delta`
|
|
- **File:** `headroom/proxy/handlers/streaming.py:213-298`
|
|
- **Evidence:** Only `text_delta` and `input_json_delta` are switched on (lines 268-271). Thinking blocks reconstructed without text or signature; signature-protected blocks rejected on replay.
|
|
- **Guide:** §2.5, §2.7, §5.1 transitions table.
|
|
- **Fix:** Add all delta-type arms. In Rust SSE parser (Phase C), implement guide §5.1 fully.
|
|
- **Phase A → PR-A8 (Python); Phase C → PR-C1 (Rust).**
|
|
|
|
### P1-10. Memory continuation re-emitter emits whole `partial_json` in one delta
|
|
- **File:** `headroom/proxy/handlers/streaming.py:300-391` (`_response_to_sse`)
|
|
- **Evidence:** `"partial_json": json.dumps(block["input"])` (line 366-374) emits the entire input JSON as a single delta — clients accumulating per-spec receive one giant fragment instead of an incremental stream. Tool IDs are fabricated as `f"toolu_{idx}"` (line 345). Thinking blocks dropped entirely.
|
|
- **Guide:** §2.6.
|
|
- **Fix:** Either delete this function (do memory continuation as non-streaming retry) or rewrite to spec.
|
|
- **Phase B → PR-B6 (memory injection refactor likely deletes it).**
|
|
|
|
### P1-11. LiteLLM bridge fabricates `toolu_<uuid>` when upstream `tc.id` missing
|
|
- **File:** `headroom/backends/litellm.py:860`
|
|
- **Evidence:** `tool_id = tc.id or f"toolu_{uuid.uuid4().hex[:24]}"`. If upstream omits `id` on chunk 1, fake ID is generated and the upstream tool_call_id is lost forever; next turn `tool_result` references the fake ID and pairing breaks.
|
|
- **Guide:** §3.5, §2.10.
|
|
- **Fix:** Drop the fallback; surface an error if `tc.id` is None on first appearance.
|
|
- **Phase D → PR-D1 deletes this whole file.**
|
|
|
|
### P1-12. OpenAI WS→HTTP fallback uses single-`\n` SSE split
|
|
- **File:** `headroom/proxy/handlers/openai.py:2422-2447`
|
|
- **Evidence:** `aiter_text()` decodes UTF-8 chunk-by-chunk → `buffer.split("\n", 1)` instead of `\n\n`. Multi-line `data:` payloads get wrong-split.
|
|
- **Fix:** Switch to `aiter_bytes()` + bytes-level `\n\n` boundary.
|
|
- **Phase C → PR-C3 ports this surface to Rust.**
|
|
|
|
### P1-13. Re-serialization in Rust path even when no body fields mutated
|
|
- **File:** `crates/headroom-proxy/src/compression/anthropic.rs:90-188`
|
|
- **Evidence:** When `should_apply` is true and ICM doesn't drop anything (`anthropic.rs:162-168`), the function correctly returns `NoCompression` and forwards original bytes. But the `Compressed` path always re-serializes via `serde_json::to_vec(&parsed)` — even if only one message changed, every retained message gets re-encoded through `Value`.
|
|
- **Guide:** §1.12.
|
|
- **Fix:** Use `RawValue` for retained `messages[*]` entries; only the modified message gets re-encoded.
|
|
- **Phase A → PR-A4 / Phase B → PR-B2 (live-zone replacement).**
|
|
|
|
### P1-14. Mid-stream `error` events not handled (Anthropic + OpenAI)
|
|
- **File:** `headroom/proxy/handlers/streaming.py:160-211, 213-298`
|
|
- **Evidence:** No `event_type == "error"` arm. The wire passthrough is byte-faithful (good) but Headroom's bookkeeping (`stream_state.input_tokens` etc.) silently doesn't reflect the failure; `_finalize_stream_response` reports a clean PERF line for an errored stream.
|
|
- **Guide:** §1.7, §2.21.
|
|
- **Fix:** Add `error` handling to telemetry.
|
|
- **Phase C → PR-C1 (Rust SSE).**
|
|
|
|
### P1-15. Connection drop without `message_stop`/`[DONE]` not surfaced
|
|
- **File:** `headroom/proxy/handlers/streaming.py:899` (`finally: await self._finalize_stream_response`)
|
|
- **Evidence:** The `finally` runs but there's no flag indicating the stream was truncated; logs report a PERF line as if it succeeded.
|
|
- **Guide:** §1.8.
|
|
- **Fix:** Track terminator-seen flag; emit truncation telemetry when missing.
|
|
- **Phase C → PR-C1.**
|
|
|
|
### P1-16. OpenAI `refusal` field on Chat assistant message not handled
|
|
- **File:** None (zero references)
|
|
- **Evidence:** Memory and tool-call extraction look only at `message.content` / `tool_calls`; refusal turns silently look like content==null with output_tokens=0.
|
|
- **Guide:** §3.7.
|
|
- **Fix:** Inspect `refusal` field; surface in telemetry.
|
|
- **Phase C → PR-C2 (Rust /v1/chat/completions).**
|
|
|
|
### P1-17. `current_block: Optional[dict]` instead of `blocks: HashMap<usize, BlockState>`
|
|
- **File:** `headroom/proxy/handlers/streaming.py:227, 700`
|
|
- **Evidence:** Anthropic emits one block at a time today, but the guide explicitly says "track blocks by `index`" — current code captures `index` and never uses it as a key.
|
|
- **Guide:** §2.4, §5.1.
|
|
- **Fix:** Index-keyed map.
|
|
- **Phase C → PR-C1.**
|
|
|
|
---
|
|
|
|
## P2 — Architectural over-build
|
|
|
|
### P2-18. ICM-as-history-dropper (the structural mismatch)
|
|
- **Files:** `headroom/transforms/intelligent_context.py`; `crates/headroom-core/src/context/manager.rs`; `crates/headroom-proxy/src/compression/icm.rs`
|
|
- **Status:** Delete in Phase B (PR-B1).
|
|
|
|
### P2-19. `RollingWindow`, `ProgressiveSummarizer` (head-truncation strategies)
|
|
- **Files:** `headroom/transforms/rolling_window.py` (395 LOC); `headroom/transforms/progressive_summarizer.py` (508 LOC)
|
|
- **Guide:** §6.3 #11, §6.4 (compaction is the explicit exception, intended to break cache once).
|
|
- **Status:** Delete in Phase B (PR-B1).
|
|
|
|
### P2-20. `MessageScorer`, `scoring/`, `relevance/` machinery
|
|
- **Files:** `crates/headroom-core/src/scoring/{scorer,score,weights,traits,mod}.rs` (~1500 LOC); `crates/headroom-core/src/relevance/{embedding,bm25,hybrid,base,mod}.rs` (~1600 LOC); `headroom/transforms/scoring.py` (459 LOC)
|
|
- **Evidence:** Sole consumer is `DropByScoreStrategy::try_fit`. Without ICM, no consumer.
|
|
- **Status:** Delete in Phase B (PR-B1). MessageScorer Rust port (PR #338, #343) becomes wasted work.
|
|
|
|
### P2-21. `crates/headroom-core/src/context/` — except `safety.rs`
|
|
- **Files:** `crates/headroom-core/src/context/{config,workspace,candidate,ccr_drop,manager,strategy/}.rs` (~1500 LOC)
|
|
- **Status:** Delete in Phase B (PR-B1). `safety.rs` (tool-pair atomicity) is moved to `crates/headroom-core/src/transforms/safety.rs` and kept.
|
|
|
|
### P2-22. `ToolCrusher` operates without `frozen_message_count`
|
|
- **File:** `headroom/transforms/tool_crusher.py:106`
|
|
- **Evidence:** Iterates all result_messages, no frozen check. Crushes any tool message above token threshold regardless of position.
|
|
- **Guide:** §10.1 (old tool results are cache-hot).
|
|
- **Status:** Delete in Phase B (PR-B1). ContentRouter covers the use case correctly.
|
|
|
|
### P2-23. `CacheAligner` rewrite path violates the very thing it claims to stabilize
|
|
- **File:** `headroom/transforms/cache_aligner.py:160-262`
|
|
- **Evidence:** Strips dynamic content from system prompt and re-inserts as a context block — mutates the cache hot zone. Currently `enabled=False` in `server.py:299`.
|
|
- **Guide:** §9.3.
|
|
- **Fix:** Delete the rewrite path (~400 LOC); keep detector + customer warning (~140 LOC).
|
|
- **Phase A → PR-A2 includes the deletion.**
|
|
|
|
### P2-24. Memory-handler injection at request lifecycle entry
|
|
- **File:** `headroom/proxy/memory_handler.py:498-510`; `headroom/proxy/handlers/openai.py:535-540`
|
|
- **Evidence:** Prepends a system message with retrieved memories on every turn. Retrieval is non-deterministic (vector store grows turn-to-turn).
|
|
- **Fix:** Move retrieval out of the request lifecycle; treat as an explicit customer-invoked tool.
|
|
- **Phase B → PR-B6 (memory refactor).**
|
|
|
|
### P2-25. CCR `ccr_retrieve` tool injected only when content was compressed
|
|
- **File:** `headroom/ccr/tool_injection.py:302-328`
|
|
- **Evidence:** `inject_tool_definition()` only adds the tool when `has_compressed_content` is true. Tool list size flips between requests.
|
|
- **Guide:** §6.3 #2 (tool list reordering).
|
|
- **Fix:** Inject `ccr_retrieve` on **every** request once a session has ever done CCR; or always inject for sessions that have CCR enabled.
|
|
- **Phase B → PR-B7.**
|
|
|
|
### P2-26. CCR markers computed but never injected into outgoing body in Rust path
|
|
- **File:** `crates/headroom-core/src/context/manager.rs:172-185`; `crates/headroom-proxy/src/proxy.rs:285`
|
|
- **Evidence:** `markers_inserted` is logged but never written into the body. The model is never told about dropped messages or about `ccr_retrieve`.
|
|
- **Guide:** §7.3 (reversibility).
|
|
- **Fix:** Once Phase B replaces ICM, CCR-on-live-zone-content writes the marker into the block content as a side-channel. Phase B PR-B7.
|
|
|
|
### P2-27. TOIN influences per-request decisions
|
|
- **File:** `headroom/telemetry/toin.py:853-927`
|
|
- **Evidence:** `get_recommendation()` consults pattern stats and returns hints that bias compression decisions; `pattern.observations += 1` mutates state during the call.
|
|
- **Guide:** §7.1, §11.17, §11.18.
|
|
- **Fix:** Strict observation-only. Recommendations published at deploy time, never altered request-time.
|
|
- **Phase B → PR-B5.**
|
|
|
|
---
|
|
|
|
## P3 — Missing infrastructure (Phase 3 cache stabilization)
|
|
|
|
### P3-28. No tool-array deterministic sort in Rust path
|
|
- **File:** Missing entirely in `crates/headroom-proxy/`
|
|
- **Evidence:** Python sorts at `handlers/anthropic.py:1198, 1217, 2041, 2118`; Rust does not.
|
|
- **Guide:** §8.5, §9.11.
|
|
- **Phase E → PR-E1.**
|
|
|
|
### P3-29. JSON Schema keys never sorted recursively
|
|
- **File:** None — `_sort_tools_deterministically` only sorts the tools array, not their `input_schema` contents.
|
|
- **Guide:** §8.5.
|
|
- **Phase E → PR-E2.**
|
|
|
|
### P3-30. No `prompt_cache_key` auto-injection
|
|
- **Evidence:** Zero references in the codebase.
|
|
- **Guide:** §4.17.
|
|
- **Phase E → PR-E4.**
|
|
|
|
### P3-31. No `cache_control` auto-placement (Anthropic)
|
|
- **Evidence:** `cache_control` only mentioned in stripping for hashing (`helpers.py:295-304`) and pass-through (`server.py:1053`).
|
|
- **Guide:** §2.19, §6.2.
|
|
- **Phase E → PR-E3.**
|
|
|
|
### P3-32. No volatile-content detector + warning
|
|
- **Evidence:** `cache_aligner` has detection but rewrites instead of warning.
|
|
- **Guide:** §9.3.
|
|
- **Phase E → PR-E5.**
|
|
|
|
### P3-33. No per-block token validation with fallback
|
|
- **Evidence:** Compression acceptance is `bytes_saved > 0` (`crates/headroom-core/src/transforms/pipeline/orchestrator.rs:158-165`); ICM aggregate-checks tokens (`anthropic.rs:162-168`) but per-block transforms don't.
|
|
- **Guide:** §7.5, §11.15, §11.20.
|
|
- **Phase B → PR-B4.**
|
|
|
|
### P3-34. No per-content-type byte thresholds
|
|
- **Evidence:** Threshold gating is by ratio (`bloat_threshold=0.5`) not by bytes (code>2KB, JSON>1KB, logs>500B, plain text>5KB per guide §7.6).
|
|
- **Phase B → PR-B4.**
|
|
|
|
### P3-35. No cache-bust drift detector telemetry
|
|
- **Evidence:** No prefix-hash drift detection across requests.
|
|
- **Phase E → PR-E6.**
|
|
|
|
### P3-36. No shared content-hash cache across customers (Phase 4)
|
|
- **Evidence:** `CompressionCache` is per-session, per-worker.
|
|
- **Status:** Out of scope for this realignment; queued for Phase 4 of the guide.
|
|
|
|
---
|
|
|
|
## P4 — OpenAI long-tail + Bedrock/Vertex
|
|
|
|
### P4-37. **Bedrock support is fake — lossy LiteLLM converter**
|
|
- **File:** `headroom/backends/litellm.py:486-628`
|
|
- **Evidence:** `_convert_messages_for_litellm` switch covers only `text` / `tool_use` / `tool_result`; drops `thinking`, `redacted_thinking`, `document`, `search_result`, `image`, `server_tool_use`, `mcp_tool_use`. Response converter hardcodes `"stop_sequence": None` (line 626) — §11.1 violation. Function-call arguments parsed and rewrapped (line 600) — string fidelity broken (§4.4).
|
|
- **Phase D → PR-D1, D2, D3 rebuild natively.**
|
|
|
|
### P4-38. Vertex same lossy converter
|
|
- **File:** Same — `headroom/backends/litellm.py`
|
|
- **Phase D → PR-D4 builds native Vertex.**
|
|
|
|
### P4-39. No native Bedrock/Vertex paths in Rust
|
|
- **File:** `crates/headroom-proxy/src/compression/mod.rs:50` only matches `/v1/messages`.
|
|
- **Phase D → PR-D1-D4.**
|
|
|
|
### P4-40. `/v1/conversations` blind spot (§4.14)
|
|
- **Evidence:** Zero references. Server-side prepended items invisible to Headroom; tokenizer count over-reports.
|
|
- **Phase C → PR-C4.**
|
|
|
|
### P4-41. `service_tier` never logged or surfaced
|
|
- **Guide:** §4.12.
|
|
- **Phase G → PR-G3 (observability).**
|
|
|
|
### P4-42. `incomplete`, `failed`, `cancelled` statuses never surfaced
|
|
- **Guide:** §4.10.
|
|
- **Phase C → PR-C3 / C4.**
|
|
|
|
### P4-43. `function_call.arguments` parsed-and-rewrapped in 2 places
|
|
- **File:** `headroom/backends/litellm.py:600`; `headroom/learn/plugins/codex.py:283`
|
|
- **Guide:** §4.4.
|
|
- **Phase D → PR-D1 deletes litellm.py; learn plugin moved to read-only.**
|
|
|
|
### P4-44. `phase` field "accidentally preserved" via `copy.copy(original)`
|
|
- **File:** `headroom/proxy/responses_converter.py:94, 235`
|
|
- **Status:** Already covered by P0-7. **Phase A → PR-A8** (hotfix), **Phase C → PR-C5** (full rebuild).
|
|
|
|
### P4-45. `image_generation_call` no log redaction
|
|
- **File:** `headroom/proxy/request_logger.py` — no base64/image redaction
|
|
- **Guide:** §11.6.
|
|
- **Phase G → PR-G3 includes a redaction step.**
|
|
|
|
### P4-46. `Cargo.toml` missing `arbitrary_precision` + `raw_value` features on `serde_json`
|
|
- **File:** `Cargo.toml:34`
|
|
- **Phase A → PR-A4 enables them.**
|
|
|
|
### P4-47. Apply patch V4A, local_shell_call argv, MCP items, compaction items only "accidentally" preserved
|
|
- **File:** `headroom/proxy/responses_converter.py:99` — "Unknown item type: preserve"
|
|
- **Evidence:** Survives only because the catch-all is conservative. No log line, no test. One refactor away from silent data loss.
|
|
- **Phase A → PR-A8 adds a warning log; Phase C → PR-C5 makes it explicit.**
|
|
|
|
### P4-48. No SSE parser in Rust at all
|
|
- **Status:** Phase 1 of the Rust proxy was passthrough; Phase C builds the parser.
|
|
- **Phase C → PR-C1.**
|
|
|
|
---
|
|
|
|
## P5 — Auth-mode + observability + fingerprinting
|
|
|
|
### P5-49. `X-Headroom-*` request headers leak upstream
|
|
- **File:** `headroom/proxy/handlers/anthropic.py:526` — `dict(request.headers.items())` captured unmodified, no strip step before `httpx.post(headers=headers)`.
|
|
- **Risk:** Subscription-revocation fingerprint.
|
|
- **Phase A → PR-A5.**
|
|
|
|
### P5-50. `anthropic-beta` mutated when memory enabled, not session-sticky
|
|
- **File:** `headroom/proxy/handlers/anthropic.py:1162-1168`
|
|
- **Status:** Already covered by P0-6. **Phase A → PR-A6, A7.**
|
|
|
|
### P5-51. `OpenAI-Beta` auto-injection on WS path
|
|
- **File:** `headroom/proxy/handlers/openai.py:1566-1567`
|
|
- **Risk:** OAuth scope rejection if scope doesn't grant the auto-injected beta.
|
|
- **Phase F → PR-F2 (gate by mode).**
|
|
|
|
### P5-52. `accept-encoding` stripped — fingerprint signal
|
|
- **File:** `handlers/anthropic.py:533`, `handlers/openai.py:264`
|
|
- **Risk:** Real Claude Code negotiates compression; stripping reveals the proxy.
|
|
- **Phase F → PR-F2 (preserve when subscription mode).**
|
|
|
|
### P5-53. `X-Forwarded-*` always added by Rust proxy
|
|
- **File:** `crates/headroom-proxy/src/headers.rs:103-117`
|
|
- **Phase F → PR-F4 (conditional on auth mode).**
|
|
|
|
### P5-54. Subscription tracker stores raw OAuth bearer token in process memory
|
|
- **File:** `headroom/subscription/tracker.py:166`
|
|
- **Risk:** Core dump or debugger attach exposes the token.
|
|
- **Phase F → PR-F3 hardens (hash + only the ID, not the token).**
|
|
|
|
### P5-55. Auth-mode never drives compression policy
|
|
- **Evidence:** Single policy applied to all three modes today.
|
|
- **Phase F → PR-F1 (`classify_auth_mode`), PR-F2 (gates).**
|
|
|
|
### P5-56. TOIN aggregates globally by `structure_hash` only
|
|
- **File:** `headroom/telemetry/toin.py:477, 496`
|
|
- **Risk:** Cross-tenant pattern leakage.
|
|
- **Phase F → PR-F3 changes key to `(auth_mode, model_family, structure_hash)`.**
|
|
|
|
### P5-57. Upstream `request-id` not captured in logs
|
|
- **File:** `crates/headroom-proxy/src/proxy.rs:355-358, 377-383`
|
|
- **Guide:** §11.10.
|
|
- **Phase A → PR-A8 (telemetry capture in Python); Phase C carries forward to Rust.**
|
|
|
|
### P5-58. Rate-limit headers forwarded but never observed
|
|
- **File:** `crates/headroom-proxy/src/headers.rs:126-139`
|
|
- **Guide:** §11.9.
|
|
- **Phase G → PR-G3 (Prometheus metric).**
|
|
|
|
### P5-59. Body size cap returns wrong status code (400 instead of 413)
|
|
- **File:** `crates/headroom-proxy/src/proxy.rs:243-263`
|
|
- **Phase A → PR-A8 fix, low priority.**
|
|
|
|
### P5-60. `tokens_saved_rtk` field is dead (allocated, never populated)
|
|
- **File:** `headroom/subscription/models.py:260`; `headroom/subscription/tracker.py:173`
|
|
- **Phase G → PR-G2.**
|
|
|
|
### P5-61. RTK never invoked from proxy (correct posture; document explicitly)
|
|
- **Status:** Per audit recommendation (Agent F): proxy-side invocation is wrong; cache hot zone risk + parallel impl with `log_compressor.rs`. Document explicitly so future contributors don't add it.
|
|
- **Phase G → PR-G1, G3.**
|
|
|
|
### P5-62. Wrap CLIs missing for cline, continue, goose, openhands, devin-style CLIs
|
|
- **Files:** `headroom/cli/wrap.py` — only Claude/Codex/Aider/Copilot/Cursor today.
|
|
- **Phase G → PR-G1.**
|
|
|
|
---
|
|
|
|
## P6 — Test-infra & parity
|
|
|
|
### P6-63. No SHA-256 byte-faithful round-trip test on recorded production payload
|
|
- **Phase A → PR-A8.**
|
|
|
|
### P6-64. `ccr`, `log_compressor`, `cache_aligner` parity comparators are `Skipped` stubs
|
|
- **File:** `crates/headroom-parity/src/lib.rs:172-174`
|
|
- **Phase I (parallel) — promote stubs to real comparators.**
|
|
|
|
### P6-65. `make test-parity` not a per-PR gate
|
|
- **File:** `.github/workflows/rust.yml:125-149` — nightly only, `continue-on-error: true`
|
|
- **Phase I — make per-PR; `Diff` fails build, `Skipped` allowed.**
|
|
|
|
### P6-66. No SSE corner-case fixtures (UTF-8 split, ping, all delta types, [DONE], mid-stream error)
|
|
- **Phase I — record fixtures during Phase C work.**
|
|
|
|
### P6-67. No real-traffic shadow test comparing Python vs Rust output byte-for-byte
|
|
- **Phase I — implement during Phase C.**
|
|
|
|
### P6-68. No per-session cache-hit-rate metric
|
|
- **File:** `headroom/proxy/prometheus_metrics.py` — only aggregate by provider
|
|
- **Phase G → PR-G3.**
|
|
|
|
### P6-69. No per-block compression-ratio histogram (only invocation count)
|
|
- **Phase G → PR-G3.**
|
|
|
|
### P6-70. No token-validation rejection counter
|
|
- **Phase B → PR-B4 emits the metric.**
|
|
|
|
### P6-71. WS-handshake `OpenAI-Beta` injection un-tested for OAuth-scope rejection paths
|
|
- **Phase I — record a fixture.**
|
|
|
|
### P6-72. Wrap E2E uses an `rtk` shim that just exits 0 (`e2e/wrap/run.py:250-267`) — doesn't exercise real RTK
|
|
- **Phase I — replace shim with a containerized real RTK or assert-on-shim-only-in-CI flag.**
|
|
|
|
---
|
|
|
|
## Summary table
|
|
|
|
| Priority | Count | Location |
|
|
|---|---:|---|
|
|
| P0 (cache-killer) | 7 | Phase A |
|
|
| P1 (wire-format) | 10 | Phase A + Phase C |
|
|
| P2 (over-build) | 10 | Phase B |
|
|
| P3 (missing Phase 3) | 9 | Phase E |
|
|
| P4 (long-tail + Bedrock) | 12 | Phase C + Phase D |
|
|
| P5 (auth + obs + fingerprint) | 14 | Phase F + Phase G |
|
|
| P6 (test infra) | 10 | Phase I (parallel) |
|
|
| **Total** | **72** | — |
|