mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
6744833afe
## Description
The Rust proxy's cache-bust drift detector
(`crates/headroom-proxy/src/cache_stabilization/drift_detector.rs`,
PR-E6) cannot currently tell drift from normal operation on interactive
agentic traffic, so it warns on nearly every turn and a real bust drowns
in the noise. Three compounding defects, all verified against live
Claude Code traffic:
1. `derive_session_key` stops at the credential hash — Claude Code sends
one OAuth bearer for every conversation, so all concurrent conversations
share one LRU slot and every conversation switch logs a false
`cache_drift_observed` (with `drift_dims` computed against the wrong
conversation's baseline).
2. The `early_messages` axis hashes the raw first-3-messages window, so
a lone conversation's normal growth (1 → 3 messages) and the client
relocating its `cache_control` breakpoint to the newest block both fire
a false `early_messages` drift at turn 2–3 of essentially every session.
3. `x-headroom-session-id` — the explicit session identity the Python
proxy honors everywhere session-sticky state exists — is ignored on the
Rust path.
This PR makes the detector's session identity conversation-scoped and
its comparison canonical, the same shape as the merged Python-side fix
for #2085 (`SessionTrackerStore.resolve_tracker` lineage resolution +
`_canonicalize_for_prefix_compare`):
- **`derive_session_key`**: honors `x-headroom-session-id` first
(hashed, like every other key input), then folds a conversation
discriminator into the credential/network arms: a 16-hex-char SHA-256
fingerprint of `(model, canonicalized first message)`. Provider prompt
caches are per-model, so a small-model sidecar call (title generation)
that reuses a conversation's opener stays a separate session instead of
false-drifting on `system`.
- **`canonicalize_for_hash`** on all axes and the discriminator: objects
rebuilt with sorted keys (this workspace enables serde_json
`preserve_order`, so a plain re-serialize would keep client wire order
and leave the hashes key-order sensitive) and `cache_control` stripped
outside opaque tool payloads (`input`/`arguments`/`json`/`input_schema`
— mirroring the Python canonicalizer's `_OPAQUE_PAYLOAD_KEYS`, so a user
field that happens to be *named* `cache_control` still counts as drift).
- **`early_messages`** becomes per-message hashes (`[Option<[u8; 32]>;
3]`) with a prefix-aware comparison: growing into the window is benign;
a settled message changing or disappearing under a stable session key is
still drift. `observe_drift` now gates the warning on drifted dimensions
rather than raw hash inequality.
True positives are preserved (`system`/`tools` changes, in-place history
rewrites under a pinned identity), and the detector remains a pure
observer — no forwarded byte changes, `does_not_mutate_input` still pins
that.
**Documented trade-off** (module doc + `conversation_discriminator`
doc): without the explicit header, a client that rewrites its first
message (history compaction, rolling-window truncation, Responses
chained mode) re-keys to a fresh session — the rewrite surfaces as
`cache_drift_first_request` rather than `cache_drift_observed` against
the old baseline. That is deliberate: the credential-keyed alternative
false-warned on every conversation switch, which buried those same
events anyway. `x-headroom-session-id` pins the identity and reports
rewrites as drift. Byte-identical openers on the same model under one
credential still conflate (rare; documented).
Closes #2300
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `derive_session_key`: `x-headroom-session-id` (hashed) wins;
credential/IP arms fold in `conversation_discriminator` — `(model,
canonicalized first message)`, 16 hex chars
- New `canonicalize_for_hash`: sorted-key object rebuild +
`cache_control` stripped outside `OPAQUE_PAYLOAD_KEYS`; applied to the
`system`/`tools`/`early_messages` axes and the discriminator
- `StructuralHash.early_messages`: `[u8; 32]` → `[Option<[u8; 32]>;
EARLY_MESSAGES_WINDOW]` per-message hashes;
`drift_dims`/`early_window_drifted` implement the prefix-aware rule;
`observe_drift` warns on non-empty dims instead of `!=`
- `conversation_messages` shape guard: bare-string message containers
only count for the Responses `input` sugar
- Docs: module header (canonicalization, trade-off, honest cost),
`conversation_discriminator` rationale + blind spots,
`DRIFT_DETECTOR_CAPACITY` cardinality note (per-conversation keys,
163-byte entry), `structural_hash_log_prefix` hex-length fix
- Tests: 13 new unit tests (conversation separation, turn-growth key
stability, explicit header priority, marker relocation + growth not
drift, rewrite/shrink still drift, per-model separation, key-order
neutrality, opaque-payload fields still count, Responses/Chat
discriminator shapes, string-container gating)
- `CHANGELOG.md`: Unreleased → Fixed entry
## Testing
- [x] Unit tests pass (`cargo test -p headroom-proxy` — full crate: lib
+ integration suites)
- [x] Linting passes (`cargo clippy -p headroom-proxy --all-targets` —
zero warnings; `cargo fmt --check` clean)
- [ ] Type checking passes (`mypy headroom`) — n/a, no Python files
touched
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ cargo test -p headroom-proxy --lib drift_detector
test result: ok. 27 passed; 0 failed; 0 ignored; 0 measured; 221 filtered out
$ cargo test -p headroom-proxy
(all suites) test result: ok. 248 passed (lib) + integration suites, 0 failed
$ cargo clippy -p headroom-proxy --all-targets
(no warnings)
```
## Real Behavior Proof
- Environment: macOS 15 (arm64), rustc 1.95.0, repo @
|
||
|---|---|---|
| .. | ||
| headroom-core | ||
| headroom-parity | ||
| headroom-proxy | ||
| headroom-py | ||
| headroom-simulators | ||