headroom/tests/test_cli/test_wrap_rtk_metrics.py
chopratejas 2a717a993e fix(observability): G3 remediation — bound cardinality + wire dead metrics
Phase G PR-G3 review identified 5 Critical + 4 High + 5 Medium
findings. This commit lands all 14 fixes plus the optional nits.

CRITICAL

* C1 (cardinality DoS): `service_tier` was read from inbound JSON
  and used verbatim as a metric label. A malicious client could
  blow up the metric vector unboundedly. Added bounded vocabulary
  in `metric_names.rs::service_tier` ({auto, default, flex,
  on_demand, priority, scale, other-sentinel}) + a `validate()`
  helper. Both request-side (`handlers/responses.rs`) and
  response-side (`proxy.rs` Responses arm) gate raw values through
  it.

* C2 (dead metric): `proxy_passthrough_bytes_modified_total` had
  no production emit site. Wired it in `proxy.rs` to fire when a
  dispatcher arm returning `NoCompression`/`Passthrough` produces
  a body of a different byte length (a true cache-poisoning
  regression detector). The check runs BEFORE the PR-E4
  prompt_cache_key injector so legitimate injector mutations do
  not trip the alarm.

* C3 (Python/Rust boundary): `proxy_image_generation_call_log_redacted_total`
  was a dead Rust counter — the redaction happens entirely in the
  Python proxy's request_logger. Removed the Rust counter; moved
  the metric to the Python proxy's `/metrics` exporter via the
  existing `redactions_total()` module-level counter.

* C4 (Python/Rust boundary): `wrap_rtk_invocations_total` was a
  dead Rust counter with no wrap-side bridge. Removed the Rust
  counter; added new `headroom/cli/wrap_rtk_metrics.py` with
  `record_rtk_invocation(tool, delta)` + `rtk_invocation_counts()`
  primitives and surfaced them via the Python proxy's `/metrics`
  exporter.

* C5 (dead metric): `proxy_compression_rejected_by_token_check_total`
  had no production caller. Wired it in
  `live_zone_anthropic.rs`, `live_zone_openai.rs`, and
  `live_zone_responses.rs` to increment on every
  `BlockAction::RejectedNotSmaller` block in the manifest. The
  metric now reflects real "compressor ran but kept original"
  cases.

HIGH

* H1 (per-strategy ratio garbage): `proxy_compression_ratio_by_strategy`
  emitted the same aggregate ratio for every strategy in
  `strategies_applied` when multiple strategies ran on one body.
  Added `per_strategy_tokens: Vec<PerStrategyTokens>` to
  `Outcome::Compressed`; per-strategy `(before, after)` is
  accumulated from the manifest at the wrapper sites and emitted
  one sample per strategy in `proxy.rs`. Empty vec → fallback to
  one aggregate-labelled sample with a debug log (Phase E
  normalization paths that don't track per-strategy tokens).

* H2 (aborted stream): cache_hit_rate observed on client
  disconnects mid-stream. Added a gate: Anthropic only fires when
  `state.status == MessageStop`, OpenAI Responses only when
  `terminal_status().is_some()`. Extracted the gate into the
  pure function `compute_anthropic_session_hit_rate(state)` so
  the H2 contract is unit-testable independent of the shared
  global registry.

* H3 (docs lie + alarm contract): docs claimed HELP/TYPE is
  reachable on fresh boot, then contradicted itself. Force-zero
  every counter / gauge MetricVec with an `__init__` sentinel
  label on each scrape so HELP/TYPE + a zero row are visible from
  boot. Histograms are NOT force-zeroed (a synthetic observe(0.0)
  would pollute percentiles). PromQL queries in docs filter
  `{... != "__init__"}` so the sentinel rows are excluded from
  aggregations.

* H4 (crate-version dependency): pinned `prometheus = "=0.13.4"`
  exactly (no caret) so a future minor bump cannot silently break
  the H3 force-zero contract that relies on this crate's gather()
  semantics. Added a clear "retest the alarm contract on bump"
  paragraph in docs.

MEDIUM

* M1 (saturate on cached > input): OpenAI Chat + Responses cache-
  hit-rate computed `non_cached = input.saturating_sub(cached)`,
  silently clamping to 0 if `cached > input`. Per "no silent
  fallbacks", log + skip the emit on this wire-format pathology.

* M2 (over-fire on non-image base64): Python redactor's "density
  heuristic" over-fired on encrypted blobs / signed tokens /
  minified JSON / tool outputs. Tightened: only redact strings
  inside known image-bearing JSON paths (`data`, `url`,
  `image_url`, `image`) OR strings starting with `data:image/`.

* M3 (NaN clamp): cache_hit_rate::observe used `f64::clamp(0,1)`
  which returns NaN for NaN input; the `debug_assert!` was
  compiled out in release. Added `is_finite()` guard with a
  loud-log + skip before observe.

* M4 (PromQL median-only): added p95, p99, mean (sum/count), and
  Phase H canary-gate query section to docs. Canary fails if ANY
  of {p50, p95, p99, mean} regresses below the Python baseline.

* M5 (label byte vs char): the `<image:base64-redacted bytes=N>`
  placeholder reported character count, not UTF-8 byte count.
  Switched to `.encode('utf-8').__len__()` so the label is
  honest for non-ASCII payloads (ASCII base64 still has byte ==
  char so existing scrapes are unchanged).

OPTIONAL

* Removed dead `debug_assert_eq!(buffered.len(), buffered.len(),
  ...)` no-op in proxy.rs.
* Normalised `record_response_status` log level from `info` to
  `debug` to match peer metric helpers.

Tests:

* Rust: 11 integration_metrics tests (was 6) + 9 cache_hit_rate
  unit tests (was 4) + 2 compression_ratio (unchanged). New
  coverage: service_tier known/unknown bucketing, C2 alarm wire,
  H1 per-strategy ratio, H2 abort gate, M3 NaN/inf skip.
* Python: 27 tests (was 13). New coverage: M2 path-gated
  redaction, M5 byte vs char label, wrap_rtk_metrics primitive
  thread safety and validation.

`cargo fmt --check`, `cargo clippy --workspace -- -D warnings`,
`cargo test -p headroom-proxy --lib` (221 passed) and the
integration_metrics + integration_compression +
integration_volatile_detector + integration_cache_control +
integration_cache_drift + integration_responses +
integration_bedrock_metrics test files all green. Full
`cargo test --workspace` deferred — disk pressure during the
agent session left insufficient space for the linker to write
the full integration test artifacts; runs that did fit all
passed. `make ci-precheck` deferred for the same reason.

ruff check + ruff format + mypy headroom/proxy/request_logger.py
+ headroom/cli/wrap_rtk_metrics.py + headroom/proxy/prometheus_metrics.py
green.
2026-05-24 10:41:56 -07:00

113 lines
3.4 KiB
Python

"""Phase G PR-G3 remediation (C4) — wrap-CLI RTK metrics primitive.
The Rust proxy previously held a dead `wrap_rtk_invocations_total`
counter. C4 remediation moved it Python-side because the wrap CLI
(headroom.cli.wrap) is where RTK invocations are actually counted.
These tests cover the counter primitives in isolation.
"""
from __future__ import annotations
import threading
import pytest
from headroom.cli.wrap_rtk_metrics import (
record_rtk_invocation,
reset_rtk_invocations,
rtk_invocation_counts,
)
@pytest.fixture(autouse=True)
def _reset_between_tests():
"""Reset the module-level counter map between tests so each
test owns a clean slate."""
reset_rtk_invocations()
yield
reset_rtk_invocations()
def test_record_increments_default_delta_one():
record_rtk_invocation("git")
counts = rtk_invocation_counts()
assert counts == {"git": 1}
def test_record_accumulates_per_tool():
record_rtk_invocation("git")
record_rtk_invocation("git")
record_rtk_invocation("ls")
record_rtk_invocation("cargo")
record_rtk_invocation("cargo")
record_rtk_invocation("cargo")
counts = rtk_invocation_counts()
assert counts == {"git": 2, "ls": 1, "cargo": 3}
def test_record_with_explicit_delta():
record_rtk_invocation("git", delta=5)
record_rtk_invocation("git", delta=2)
counts = rtk_invocation_counts()
assert counts == {"git": 7}
def test_record_zero_delta_is_noop_record():
# delta=0 is legal — caller may want to "touch" the counter to
# ensure the key exists before later increments.
record_rtk_invocation("git", delta=0)
counts = rtk_invocation_counts()
assert counts == {"git": 0}
def test_record_rejects_negative_delta():
with pytest.raises(ValueError, match="must be non-negative"):
record_rtk_invocation("git", delta=-1)
def test_record_rejects_non_string_tool():
with pytest.raises(TypeError, match="tool must be a str"):
record_rtk_invocation(123, delta=1) # type: ignore[arg-type]
def test_record_rejects_non_int_delta():
with pytest.raises(TypeError, match="delta must be an int"):
record_rtk_invocation("git", delta="1") # type: ignore[arg-type]
def test_counts_returns_snapshot_not_view():
# The returned mapping must be a plain dict copy, not the
# internal defaultdict — otherwise callers could pollute the
# counter map by reading absent keys.
record_rtk_invocation("git")
counts = rtk_invocation_counts()
# Reading a key that's not present must not add it to the
# internal map.
_ = counts.get("nonexistent_tool", 0)
counts2 = rtk_invocation_counts()
assert "nonexistent_tool" not in counts2
def test_thread_safe_concurrent_increments():
# 10 threads each bumping `git` 100 times: final count must be
# exactly 1000. The threading.Lock guards the dict update so
# races are impossible.
def worker():
for _ in range(100):
record_rtk_invocation("git")
threads = [threading.Thread(target=worker) for _ in range(10)]
for t in threads:
t.start()
for t in threads:
t.join()
counts = rtk_invocation_counts()
assert counts == {"git": 1000}
def test_reset_clears_counts():
record_rtk_invocation("git", delta=42)
record_rtk_invocation("ls", delta=7)
assert rtk_invocation_counts() != {}
reset_rtk_invocations()
assert rtk_invocation_counts() == {}