mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
2a717a993e
Phase G PR-G3 review identified 5 Critical + 4 High + 5 Medium
findings. This commit lands all 14 fixes plus the optional nits.
CRITICAL
* C1 (cardinality DoS): `service_tier` was read from inbound JSON
and used verbatim as a metric label. A malicious client could
blow up the metric vector unboundedly. Added bounded vocabulary
in `metric_names.rs::service_tier` ({auto, default, flex,
on_demand, priority, scale, other-sentinel}) + a `validate()`
helper. Both request-side (`handlers/responses.rs`) and
response-side (`proxy.rs` Responses arm) gate raw values through
it.
* C2 (dead metric): `proxy_passthrough_bytes_modified_total` had
no production emit site. Wired it in `proxy.rs` to fire when a
dispatcher arm returning `NoCompression`/`Passthrough` produces
a body of a different byte length (a true cache-poisoning
regression detector). The check runs BEFORE the PR-E4
prompt_cache_key injector so legitimate injector mutations do
not trip the alarm.
* C3 (Python/Rust boundary): `proxy_image_generation_call_log_redacted_total`
was a dead Rust counter — the redaction happens entirely in the
Python proxy's request_logger. Removed the Rust counter; moved
the metric to the Python proxy's `/metrics` exporter via the
existing `redactions_total()` module-level counter.
* C4 (Python/Rust boundary): `wrap_rtk_invocations_total` was a
dead Rust counter with no wrap-side bridge. Removed the Rust
counter; added new `headroom/cli/wrap_rtk_metrics.py` with
`record_rtk_invocation(tool, delta)` + `rtk_invocation_counts()`
primitives and surfaced them via the Python proxy's `/metrics`
exporter.
* C5 (dead metric): `proxy_compression_rejected_by_token_check_total`
had no production caller. Wired it in
`live_zone_anthropic.rs`, `live_zone_openai.rs`, and
`live_zone_responses.rs` to increment on every
`BlockAction::RejectedNotSmaller` block in the manifest. The
metric now reflects real "compressor ran but kept original"
cases.
HIGH
* H1 (per-strategy ratio garbage): `proxy_compression_ratio_by_strategy`
emitted the same aggregate ratio for every strategy in
`strategies_applied` when multiple strategies ran on one body.
Added `per_strategy_tokens: Vec<PerStrategyTokens>` to
`Outcome::Compressed`; per-strategy `(before, after)` is
accumulated from the manifest at the wrapper sites and emitted
one sample per strategy in `proxy.rs`. Empty vec → fallback to
one aggregate-labelled sample with a debug log (Phase E
normalization paths that don't track per-strategy tokens).
* H2 (aborted stream): cache_hit_rate observed on client
disconnects mid-stream. Added a gate: Anthropic only fires when
`state.status == MessageStop`, OpenAI Responses only when
`terminal_status().is_some()`. Extracted the gate into the
pure function `compute_anthropic_session_hit_rate(state)` so
the H2 contract is unit-testable independent of the shared
global registry.
* H3 (docs lie + alarm contract): docs claimed HELP/TYPE is
reachable on fresh boot, then contradicted itself. Force-zero
every counter / gauge MetricVec with an `__init__` sentinel
label on each scrape so HELP/TYPE + a zero row are visible from
boot. Histograms are NOT force-zeroed (a synthetic observe(0.0)
would pollute percentiles). PromQL queries in docs filter
`{... != "__init__"}` so the sentinel rows are excluded from
aggregations.
* H4 (crate-version dependency): pinned `prometheus = "=0.13.4"`
exactly (no caret) so a future minor bump cannot silently break
the H3 force-zero contract that relies on this crate's gather()
semantics. Added a clear "retest the alarm contract on bump"
paragraph in docs.
MEDIUM
* M1 (saturate on cached > input): OpenAI Chat + Responses cache-
hit-rate computed `non_cached = input.saturating_sub(cached)`,
silently clamping to 0 if `cached > input`. Per "no silent
fallbacks", log + skip the emit on this wire-format pathology.
* M2 (over-fire on non-image base64): Python redactor's "density
heuristic" over-fired on encrypted blobs / signed tokens /
minified JSON / tool outputs. Tightened: only redact strings
inside known image-bearing JSON paths (`data`, `url`,
`image_url`, `image`) OR strings starting with `data:image/`.
* M3 (NaN clamp): cache_hit_rate::observe used `f64::clamp(0,1)`
which returns NaN for NaN input; the `debug_assert!` was
compiled out in release. Added `is_finite()` guard with a
loud-log + skip before observe.
* M4 (PromQL median-only): added p95, p99, mean (sum/count), and
Phase H canary-gate query section to docs. Canary fails if ANY
of {p50, p95, p99, mean} regresses below the Python baseline.
* M5 (label byte vs char): the `<image:base64-redacted bytes=N>`
placeholder reported character count, not UTF-8 byte count.
Switched to `.encode('utf-8').__len__()` so the label is
honest for non-ASCII payloads (ASCII base64 still has byte ==
char so existing scrapes are unchanged).
OPTIONAL
* Removed dead `debug_assert_eq!(buffered.len(), buffered.len(),
...)` no-op in proxy.rs.
* Normalised `record_response_status` log level from `info` to
`debug` to match peer metric helpers.
Tests:
* Rust: 11 integration_metrics tests (was 6) + 9 cache_hit_rate
unit tests (was 4) + 2 compression_ratio (unchanged). New
coverage: service_tier known/unknown bucketing, C2 alarm wire,
H1 per-strategy ratio, H2 abort gate, M3 NaN/inf skip.
* Python: 27 tests (was 13). New coverage: M2 path-gated
redaction, M5 byte vs char label, wrap_rtk_metrics primitive
thread safety and validation.
`cargo fmt --check`, `cargo clippy --workspace -- -D warnings`,
`cargo test -p headroom-proxy --lib` (221 passed) and the
integration_metrics + integration_compression +
integration_volatile_detector + integration_cache_control +
integration_cache_drift + integration_responses +
integration_bedrock_metrics test files all green. Full
`cargo test --workspace` deferred — disk pressure during the
agent session left insufficient space for the linker to write
the full integration test artifacts; runs that did fit all
passed. `make ci-precheck` deferred for the same reason.
ruff check + ruff format + mypy headroom/proxy/request_logger.py
+ headroom/cli/wrap_rtk_metrics.py + headroom/proxy/prometheus_metrics.py
green.
|
||
|---|---|---|
| .. | ||
| app | ||
| components | ||
| content/docs | ||
| overrides | ||
| screenshots | ||
| spec | ||
| .gitignore | ||
| auth-modes.md | ||
| bedrock.md | ||
| bun.lock | ||
| next.config.mjs | ||
| observability.md | ||
| package-lock.json | ||
| package.json | ||
| postcss.config.mjs | ||
| proxy.ts | ||
| README.md | ||
| rtk-architecture.md | ||
| source.config.ts | ||
| tsconfig.json | ||
docs
This is a Next.js application generated with Create Fumadocs.
Run development server:
npm run dev
# or
pnpm dev
# or
yarn dev
Open http://localhost:3000 with your browser to see the result.
Explore
In the project, you can see:
lib/source.ts: Code for content source adapter,loader()provides the interface to access your content.lib/layout.shared.tsx: Shared options for layouts, optional but preferred to keep.
| Route | Description |
|---|---|
app/(home) |
The route group for your landing page and other pages. |
app/docs |
The documentation layout and pages. |
app/api/search/route.ts |
The Route Handler for search. |
Fumadocs MDX
A source.config.ts config file has been included, you can customise different options like frontmatter schema.
Read the Introduction for further details.
Learn More
To learn more about Next.js and Fumadocs, take a look at the following resources:
- Next.js Documentation - learn about Next.js features and API.
- Learn Next.js - an interactive Next.js tutorial.
- Fumadocs - learn about Fumadocs