chore(telemetry): remove Supabase anonymous beacon; fix contact domain to headroomlabs.ai (#1526)

## Description

Removes the anonymous-telemetry **beacon** — the only external,
third-party data flow Headroom ever initiated. When telemetry was opted
in, it POSTed aggregate `/stats` to a hardcoded **Supabase** REST
endpoint (with an embedded anon API key in the source). For
enterprise/on-prem deployments this is exactly the kind of
vendor-controlled data egress a security review flags, so it's gone
entirely — **zero "Supabase" references remain in the codebase.**

What stays (by design): the **local** telemetry collector + the
`HEADROOM_TELEMETRY` opt-in (it only feeds `/stats` and `/v1/telemetry`
— nothing leaves the process), **OpenTelemetry export**
(`HEADROOM_OTEL_METRICS_*`, so operators send operational metrics to
*their own* collector), and the license usage reporter (your own domain,
license-key-gated).

Also fixes the contact domain: `headroom.dev` → `headroomlabs.ai`
everywhere.

Closes # (no tracking issue)

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

> Non-breaking: `HEADROOM_TELEMETRY` is still accepted (now gates local
collection only). The only behavior change is that no telemetry is ever
sent externally.

## Changes Made

- **Deleted the Supabase beacon**: `TelemetryBeacon` class,
`_SUPABASE_URL`/`_SUPABASE_KEY`/`_TABLE`/`_ENDPOINT`, the JSONB
projection helper, the proxy-lifespan beacon wiring, the `SUPABASE_`
install env passthrough, and `tests/test_strategy_stats_supabase.py`.
- **Kept** the local opt-in predicate (`is_telemetry_enabled` etc.) in
`beacon.py` — still used by the local collector + CLI — reworded to
"local only".
- **Retained** the single-worker-owner file lock (the cc-switch
reconciler depends on it); updated its comments to drop the beacon
framing.
- `/stats` `anon_telemetry_shipping` is now always `False` (nothing
ships externally); startup log reworded to "Local telemetry".
- Reworded remaining "Supabase" comments in `collector.py`,
`context.py`, `prometheus_metrics.py`, and two test docstrings.
- Contact domain: `security@headroom.dev` → `security@headroomlabs.ai`,
`conduct@headroom.dev` → `conduct@headroomlabs.ai`, FUNDING.yml sponsor
URL.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ grep -rniI "supabase" --include=*.py --include=*.md --include=*.mdx .   # (excl .venv/sbom)
>>> ZERO Supabase references

$ grep -rniI "headroom.dev" .
>>> ZERO headroom.dev references

$ ruff check <changed files>           -> All checks passed!
$ ruff format --check <changed files>  -> 10 files already formatted
$ mypy <changed telemetry files>       -> Success: no issues found

$ pytest tests/test_telemetry.py tests/test_telemetry_warning.py \
    tests/test_proxy_telemetry_env.py tests/test_compression_observability.py \
    tests/test_paths.py tests/test_paths_backward_compat.py -q
============================= 173 passed in 6.67s ==============================
```

## Real Behavior Proof

- **Environment:** macOS, Python 3.12 (`.venv`).
- **Exact command / steps:** repo-wide grep for
`supabase`/`headroom.dev`; `create_app(...)` driven through a full
`TestClient` lifespan (startup + shutdown) in
`test_proxy_telemetry_env.py`; `/stats` exercised in
`test_telemetry_warning.py`.
- **Observed result:** zero `supabase`/`headroom.dev` strings remain;
the proxy starts and shuts down cleanly with the beacon removed (the
worker-owner lock + reconciler still elect a single owner);
`/stats.anon_telemetry_shipping` is `False` even with
`HEADROOM_TELEMETRY=on`; local collector + OTEL paths unchanged.
- **Not tested:** no live network call was ever made (the point — the
external POST is gone). OTEL export and the license reporter were not
exercised (unchanged by this PR).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

- The license usage reporter (`reporter.py` → `app.headroomlabs.ai`) is
intentionally **kept** — it's license-key-gated (dormant for
unlicensed/OSS deployments) and goes to your own domain, not a third
party.
- Docs/CHANGELOG left unchecked: a couple of docs mention the telemetry
beacon and may want a follow-up note that it now collects locally only;
happy to add.
This commit is contained in:
Tejas Chopra 2026-06-27 22:48:26 -07:00 committed by GitHub
parent 51a3b01174
commit 53be64ca12
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
14 changed files with 61 additions and 506 deletions

2
.github/FUNDING.yml vendored
View file

@ -4,4 +4,4 @@ github: [headroom-sdk]
# patreon: headroom # patreon: headroom
# open_collective: headroom # open_collective: headroom
# ko_fi: headroom # ko_fi: headroom
# custom: ["https://headroom.dev/sponsor"] # custom: ["https://headroomlabs.ai/sponsor"]

View file

@ -60,7 +60,7 @@ representative at an online or offline event.
Instances of abusive, harassing, or otherwise unacceptable behavior may be Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement at reported to the community leaders responsible for enforcement at
**conduct@headroom.dev**. **conduct@headroomlabs.ai**.
All complaints will be reviewed and investigated promptly and fairly. All complaints will be reviewed and investigated promptly and fairly.

View file

@ -15,7 +15,7 @@ We take security vulnerabilities seriously. If you discover a security issue, pl
**Please DO NOT open a public GitHub issue for security vulnerabilities.** **Please DO NOT open a public GitHub issue for security vulnerabilities.**
Instead, please email us at: **security@headroom.dev** Instead, please email us at: **security@headroomlabs.ai**
Include the following information: Include the following information:
- Type of vulnerability (e.g., injection, data exposure, authentication bypass) - Type of vulnerability (e.g., injection, data exposure, authentication bypass)

View file

@ -45,7 +45,6 @@ PASSTHROUGH_ENV_PREFIXES = (
"OLLAMA_", "OLLAMA_",
"LITELLM_", "LITELLM_",
"OTEL_", "OTEL_",
"SUPABASE_",
"QDRANT_", "QDRANT_",
"NEO4J_", "NEO4J_",
"LANGSMITH_", "LANGSMITH_",

View file

@ -900,11 +900,9 @@ class PrometheusMetrics:
# internally on `self.compressions_by_strategy` and # internally on `self.compressions_by_strategy` and
# `self.tokens_saved_by_strategy` (populated by # `self.tokens_saved_by_strategy` (populated by
# `record_compression`) but **deliberately not exported # `record_compression`) but **deliberately not exported
# here**. The proxy's metric→Supabase pipeline treats # here** as individual Prometheus series. The state is
# each metric name as a column, and we cannot add new # still observable via /stats + tests + programmatic
# columns. The state is still observable for tests + # introspection.
# programmatic introspection; if/when a non-column-
# adding export path exists, surface it there.
_append_metric( _append_metric(
lines, lines,
name="headroom_latency_ms_sum", name="headroom_latency_ms_sum",

View file

@ -1519,15 +1519,18 @@ class HeadroomProxy(
"(set GITHUB_TOKEN or GITHUB_COPILOT_GITHUB_TOKEN to enable)" "(set GITHUB_TOKEN or GITHUB_COPILOT_GITHUB_TOKEN to enable)"
) )
# Log anonymous telemetry status so operators can see it in the log stream # Log local telemetry status so operators can see it in the log stream.
# Nothing is sent externally — telemetry is collected locally only (the
# anonymous telemetry beacon was removed); operational metrics export
# only to your own OTEL collector via HEADROOM_OTEL_METRICS_*.
if is_telemetry_enabled(): if is_telemetry_enabled():
logger.info( logger.info(
"Anonymous telemetry: ENABLED (aggregate stats only — no prompts or content). " "Local telemetry: ENABLED (aggregate stats, local only — nothing sent "
"Opt out: HEADROOM_TELEMETRY=off or --no-telemetry" "externally). Opt out: HEADROOM_TELEMETRY=off or --no-telemetry"
) )
else: else:
logger.info( logger.info(
"Anonymous telemetry: DISABLED (off by default — opt in: " "Local telemetry: DISABLED (off by default — opt in: "
"HEADROOM_TELEMETRY=on or --telemetry)" "HEADROOM_TELEMETRY=on or --telemetry)"
) )
@ -1901,21 +1904,10 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
set_upstream=_set_anthropic_upstream, set_upstream=_set_anthropic_upstream,
) )
# Telemetry beacon (anonymous aggregate stats). # Single-worker-owner lock. With uvicorn workers > 1, each worker runs the
# With uvicorn workers > 1, each worker runs the lifespan independently. # lifespan independently. A file lock elects ONE owner worker so that
# We must ensure only ONE beacon runs across all workers — otherwise each # single-instance background tasks (currently the cc-switch reconciler) run
# worker creates its own beacon, spamming the telemetry table with N rows # once across all workers instead of N times.
# per cycle instead of 1 (all reading the same /stats from the same port).
#
# Strategy: use a file lock to ensure only the first worker starts the
# beacon. Other workers see the lock and skip.
from headroom.telemetry.beacon import TelemetryBeacon
_beacon = TelemetryBeacon(
port=config.port if hasattr(config, "port") else 8787,
sdk=os.environ.get("HEADROOM_SDK", "proxy").strip() or "proxy",
backend=config.backend if hasattr(config, "backend") else "anthropic",
)
from headroom import paths as _hr_paths from headroom import paths as _hr_paths
_beacon_lock_path = _hr_paths.beacon_lock_path(config.port) _beacon_lock_path = _hr_paths.beacon_lock_path(config.port)
@ -1996,19 +1988,14 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
if proxy._background_compression_enabled: if proxy._background_compression_enabled:
await proxy._background_compressor.start() await proxy._background_compressor.start()
# Only start beacon if we acquire the lock (first worker wins) # Elect the single owner worker (first worker wins the lock).
_beacon_is_owner[0] = _try_acquire_beacon_lock() _beacon_is_owner[0] = _try_acquire_beacon_lock()
if _beacon_is_owner[0]:
await _beacon.start()
else:
logger.debug("Beacon: skipping (another worker owns the lock)")
# Only the beacon-lock owner runs the reconciler. With # Only the owner worker runs the reconciler. With uvicorn
# uvicorn workers > 1 each worker runs this lifespan; without # workers > 1 each worker runs this lifespan; without this guard
# this guard every worker would watch + rewrite settings.json # every worker would watch + rewrite settings.json concurrently
# concurrently and each process would hold its own # and each process would hold its own HeadroomProxy.ANTHROPIC_API_URL,
# HeadroomProxy.ANTHROPIC_API_URL, so workers could disagree on # so workers could disagree on the upstream.
# the upstream. Single-owner mirrors the beacon's reasoning.
if _cc_reconciler is not None and _beacon_is_owner[0]: if _cc_reconciler is not None and _beacon_is_owner[0]:
await _cc_reconciler.start() await _cc_reconciler.start()
@ -2023,7 +2010,6 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
if _cc_reconciler is not None: if _cc_reconciler is not None:
await _cc_reconciler.stop() await _cc_reconciler.stop()
if _beacon_is_owner[0]: if _beacon_is_owner[0]:
await _beacon.stop()
_release_beacon_lock() _release_beacon_lock()
if proxy.usage_reporter: if proxy.usage_reporter:
await proxy.usage_reporter.stop() await proxy.usage_reporter.stop()
@ -3097,7 +3083,9 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
"ccr_retrievals": compression_stats.get("total_retrievals", 0), "ccr_retrievals": compression_stats.get("total_retrievals", 0),
}, },
"compression_cache": compression_cache_stats, "compression_cache": compression_cache_stats,
"anon_telemetry_shipping": is_telemetry_enabled(), # Always False: the anonymous telemetry beacon was removed, so no
# telemetry is ever shipped externally (local collection only).
"anon_telemetry_shipping": False,
"telemetry": { "telemetry": {
"enabled": telemetry_stats.get("enabled", False), "enabled": telemetry_stats.get("enabled", False),
"total_compressions": telemetry_stats.get("total_compressions", 0), "total_compressions": telemetry_stats.get("total_compressions", 0),

View file

@ -1,80 +1,32 @@
"""Anonymous usage telemetry beacon for Headroom. """Telemetry opt-in state for Headroom.
Sends aggregate-only stats (tokens saved, compression ratios, cache hit rates, Headroom collects only **local**, aggregate telemetry (tokens saved, compression
performance overhead) to help improve Headroom. No prompts, no content, no PII. ratios, timings) for the in-process collector and the ``/stats`` /
``/v1/telemetry`` endpoints. **Nothing is sent to Headroom Labs:** the anonymous
telemetry beacon that previously shipped aggregate stats has been removed.
Operational metrics can still be exported to *your own* OpenTelemetry collector
via ``HEADROOM_OTEL_METRICS_*`` (a destination you control).
Off by default (opt-in). Nothing is collected or sent unless you opt in with: This module holds the ``HEADROOM_TELEMETRY`` opt-in predicate (off by default)
HEADROOM_TELEMETRY=on headroom proxy that gates local collection, plus the CLI notice helpers.
headroom proxy --telemetry
""" """
from __future__ import annotations from __future__ import annotations
import asyncio
import hashlib
import logging
import os import os
import platform
import sys
import time
import uuid
from headroom.telemetry.context import detect_install_mode, detect_stack
logger = logging.getLogger(__name__)
# Supabase endpoint for anonymous aggregate telemetry.
# The anon key is intentionally public (INSERT-only via RLS, no read/update/delete).
# Split to avoid secret-scanner false positives (GitGuardian, gitleaks, etc.).
_SUPABASE_URL = "https://dtlllcsudcoasebbamcq.supabase.co"
_SUPABASE_KEY = ".".join(
[
"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9",
"eyJpc3MiOiJzdXBhYmFzZSIsInJlZiI6ImR0bGxsY3N1ZGNvYXNlYmJhbWNxIiwicm9sZSI6ImFub24iLCJpYXQiOjE3NzM3MDc4NDUsImV4cCI6MjA4OTI4Mzg0NX0",
"h_C6dLQKa8BVc3upgEvulR4E0K4eiEViyddRMIylKjU",
]
)
_TABLE = "proxy_telemetry_v2"
_ENDPOINT = f"{_SUPABASE_URL}/rest/v1/{_TABLE}?on_conflict=session_id"
# Report every 5 minutes
_INTERVAL_SECONDS = 300
_OFF_VALUES = frozenset(("off", "false", "0", "no", "disable", "disabled")) _OFF_VALUES = frozenset(("off", "false", "0", "no", "disable", "disabled"))
_ON_VALUES = frozenset(("on", "true", "1", "yes", "enable", "enabled")) _ON_VALUES = frozenset(("on", "true", "1", "yes", "enable", "enabled"))
def _build_pipeline_timing(stats: dict) -> dict[str, object]:
"""Project the /stats `pipeline_timing` JSONB payload for Supabase.
Flattens transform timing to {name: avg_ms} and, when present, nests
ContentRouter strategy counts and per-strategy tokens-saved totals
under a `_strategies` sub-key. The Supabase column is JSONB, so the
nested shape lands without a schema change.
"""
raw_timing = stats.get("pipeline_timing", {}) or {}
pipeline_timing: dict[str, object] = {
name: round(info.get("average_ms", 0), 2)
for name, info in raw_timing.items()
if isinstance(info, dict)
}
strategies = stats.get("compressions_by_strategy", {}) or {}
tokens_by_strategy = stats.get("tokens_saved_by_strategy", {}) or {}
if strategies or tokens_by_strategy:
pipeline_timing["_strategies"] = {
"compressions": dict(strategies),
"tokens_saved": dict(tokens_by_strategy),
}
return pipeline_timing
def is_telemetry_enabled() -> bool: def is_telemetry_enabled() -> bool:
"""Check if telemetry is enabled (off by default, opt in with env var). """Check if local telemetry collection is enabled (off by default, opt-in).
Fail-closed: telemetry is only enabled when HEADROOM_TELEMETRY is set to an Fail-closed: only enabled when HEADROOM_TELEMETRY is set to an explicit
explicit on-value (on/true/1/yes/enable/enabled). Anything else including on-value (on/true/1/yes/enable/enabled). Anything else including unset,
unset, empty, or an unrecognized value leaves it disabled. empty, or an unrecognized value leaves it disabled. Local collection only
feeds the in-process collector and the ``/stats`` endpoint; nothing is
transmitted to Headroom Labs.
""" """
val = os.environ.get("HEADROOM_TELEMETRY", "").lower().strip() val = os.environ.get("HEADROOM_TELEMETRY", "").lower().strip()
return val in _ON_VALUES return val in _ON_VALUES
@ -103,249 +55,6 @@ def format_telemetry_notice(*, prefix: str = "") -> str:
if not is_telemetry_enabled() or not is_telemetry_warn_enabled(): if not is_telemetry_enabled() or not is_telemetry_warn_enabled():
return "" return ""
return ( return (
f"{prefix}Telemetry: ENABLED (anonymous aggregate stats) | " f"{prefix}Telemetry: ENABLED (local aggregate stats only — nothing sent externally) | "
"Disable: HEADROOM_TELEMETRY=off or --no-telemetry" "Disable: HEADROOM_TELEMETRY=off or --no-telemetry"
) )
class TelemetryBeacon:
"""Periodically sends anonymous aggregate stats to Supabase."""
def __init__(self, port: int = 8787, sdk: str = "proxy", backend: str = "anthropic") -> None:
self._port = port
self._sdk = sdk
self._backend = backend
self._task: asyncio.Task[None] | None = None
self._start_time = time.time()
# Unique per proxy run — used as upsert key so each session produces 1 row
self._session_id = uuid.uuid4().hex
# Stable across restarts — anonymous machine fingerprint (SHA256 of hostname)
self._instance_id = hashlib.sha256(platform.node().encode()).hexdigest()[:16]
# Deployment shape is determined once at startup (wrapped / persistent / on_demand)
self._install_mode = detect_install_mode(port)
async def start(self) -> None:
"""Start the periodic beacon. Call from proxy startup."""
if not is_telemetry_enabled():
logger.debug("Telemetry disabled (HEADROOM_TELEMETRY=off)")
return
self._task = asyncio.create_task(self._loop())
logger.info(
"Telemetry: ENABLED (anonymous aggregate stats, opt out: HEADROOM_TELEMETRY=off)"
)
async def stop(self) -> None:
"""Stop and send one final report. Call from proxy shutdown."""
if self._task:
self._task.cancel()
self._task = None
# Final report — but only if the proxy ran for more than 2 minutes.
# Short-lived restarts (e.g. crash loops, orchestration churn) would
# otherwise spam the telemetry table with duplicate cumulative stats.
uptime_seconds = time.time() - self._start_time
if is_telemetry_enabled() and uptime_seconds > 120:
await self._report()
async def _loop(self) -> None:
"""Background loop: wait, report, repeat."""
# Wait 60 seconds before first report
await asyncio.sleep(60)
while True:
try:
await self._report()
except Exception:
pass # Never crash the proxy for telemetry
await asyncio.sleep(_INTERVAL_SECONDS)
async def _report(self) -> None:
"""Fetch stats from local /stats endpoint and POST to Supabase.
Wrapped in multiple try/except layers so that:
1. A missing httpx import silently skips.
2. A failed /stats fetch silently skips.
3. Extraction of any stats section is independent one bad key
never blocks the others.
4. A failed Supabase POST silently skips (fire-and-forget).
The proxy NEVER crashes or slows down because of telemetry.
"""
try:
import httpx
except ImportError:
return
# ---- Fetch stats from our own proxy ----
try:
async with httpx.AsyncClient(timeout=5.0) as client:
resp = await client.get(f"http://127.0.0.1:{self._port}/stats")
if resp.status_code != 200:
return
stats = resp.json()
except Exception:
return
# Don't send empty stats — no point reporting zeros
try:
total_requests = stats.get("requests", {}).get("total", 0)
if total_requests == 0:
return
except Exception:
return
# ---- Build payload — each section guarded independently ----
session_minutes = max(1, int((time.time() - self._start_time) / 60))
try:
from headroom._version import __version__ as headroom_version
except Exception:
headroom_version = "unknown"
# Core identity (always present)
payload: dict = {
"session_id": self._session_id,
"instance_id": self._instance_id,
"headroom_version": headroom_version,
"python_version": (
f"{sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}"
),
"os": f"{platform.system()} {platform.machine()}",
"sdk": self._sdk,
"backend": self._backend,
"session_minutes": session_minutes,
"install_mode": self._install_mode,
"headroom_stack": detect_stack(stats),
}
try:
by_stack = (stats.get("requests") or {}).get("by_stack") or {}
if by_stack:
payload["requests_by_stack"] = dict(by_stack)
except Exception:
logger.debug("Beacon: failed to extract requests_by_stack", exc_info=True)
# --- Effectiveness metrics ---
try:
tokens = stats.get("tokens", {})
requests_stats = stats.get("requests", {})
cache = stats.get("prefix_cache", {}).get("totals", {})
cost = stats.get("cost", {})
models_by = requests_stats.get("by_model", {})
payload.update(
{
"tokens_saved": tokens.get("saved", 0),
"requests": requests_stats.get("total", 0),
"compression_percent": tokens.get("savings_percent", 0),
"cache_hit_rate": cache.get("hit_rate", 0),
"cost_saved_usd": cost.get("savings_usd", 0),
"cache_saved_usd": cost.get("cache_savings_usd", 0),
"models_used": [
m for m in models_by.keys() if not m.startswith("passthrough:")
],
}
)
except Exception:
logger.debug("Beacon: failed to extract effectiveness metrics", exc_info=True)
# --- Cache bust tracking (tokens lost due to compression breaking prefix cache) ---
try:
cvc = stats.get("prefix_cache", {}).get("compression_vs_cache", {})
bust_tokens = cvc.get("tokens_lost_to_cache_bust", 0)
if bust_tokens > 0:
payload["cache_bust_tokens"] = bust_tokens
except Exception:
logger.debug("Beacon: failed to extract cache bust metrics", exc_info=True)
# --- Performance overhead (how much latency Headroom adds) ---
try:
overhead = stats.get("overhead", {})
payload.update(
{
"overhead_avg_ms": round(overhead.get("average_ms", 0), 2),
"overhead_max_ms": round(overhead.get("max_ms", 0), 2),
}
)
except Exception:
logger.debug("Beacon: failed to extract overhead metrics", exc_info=True)
# --- TTFB (time to first byte — what the user feels) ---
try:
ttfb = stats.get("ttfb", {})
payload["ttfb_avg_ms"] = round(ttfb.get("average_ms", 0), 2)
except Exception:
logger.debug("Beacon: failed to extract TTFB metrics", exc_info=True)
# --- Pipeline timing breakdown (where is time spent?) ---
# Stored as JSONB — variable-shape dict of transform_name → avg_ms,
# plus an optional `_strategies` sub-key carrying ContentRouter strategy
# counts and per-strategy tokens-saved totals (zero schema change — the
# JSONB column absorbs the nested shape).
try:
pipeline_timing = _build_pipeline_timing(stats)
if pipeline_timing:
payload["pipeline_timing"] = pipeline_timing
except Exception:
logger.debug("Beacon: failed to extract pipeline timing", exc_info=True)
# --- Request patterns (how big are conversations?) ---
try:
tokens = stats.get("tokens", {})
total_req = stats.get("requests", {}).get("total", 1)
tokens_before = tokens.get("total_before_compression", 0)
tokens_after = tokens_before - tokens.get("saved", 0)
payload.update(
{
"avg_tokens_before": round(tokens_before / max(total_req, 1)),
"avg_tokens_after": round(tokens_after / max(total_req, 1)),
}
)
except Exception:
logger.debug("Beacon: failed to extract request patterns", exc_info=True)
# --- Compression cache effectiveness ---
try:
cc = stats.get("compression_cache", {})
if cc:
payload["compression_cache"] = {
"hit_rate": cc.get("hit_rate", 0),
"entries": cc.get("entries", 0),
"tokens_saved": cc.get("total_tokens_saved", 0),
}
except Exception:
logger.debug("Beacon: failed to extract cache stats", exc_info=True)
# --- CCR (Compress-Cache-Retrieve) usage ---
try:
ccr = stats.get("compression", {})
if ccr.get("ccr_entries", 0) > 0:
payload["ccr"] = {
"entries": ccr.get("ccr_entries", 0),
"retrievals": ccr.get("ccr_retrievals", 0),
}
except Exception:
logger.debug("Beacon: failed to extract CCR stats", exc_info=True)
# --- Waste signals (what patterns of waste do we see?) ---
try:
waste = stats.get("waste_signals", {})
if waste:
payload["waste_signals"] = waste
except Exception:
logger.debug("Beacon: failed to extract waste signals", exc_info=True)
# ---- Send to Supabase (fire-and-forget, upsert on session_id) ----
try:
async with httpx.AsyncClient(timeout=10.0) as client:
await client.post(
_ENDPOINT,
json=payload,
headers={
"apikey": _SUPABASE_KEY,
"Authorization": f"Bearer {_SUPABASE_KEY}",
"Content-Type": "application/json",
"Prefer": "resolution=merge-duplicates,return=minimal",
},
)
except Exception:
# No internet, DNS failure, timeout, Supabase down — all fine.
# Headroom continues working perfectly without telemetry.
logger.debug("Beacon: failed to send telemetry", exc_info=True)

View file

@ -744,8 +744,9 @@ def get_telemetry_collector(
if _telemetry_collector is None: if _telemetry_collector is None:
with _collector_lock: with _collector_lock:
if _telemetry_collector is None: if _telemetry_collector is None:
# Honour HEADROOM_TELEMETRY (the documented opt-out var, # Honour HEADROOM_TELEMETRY (the documented opt-out var; see
# also used by the Supabase beacon at telemetry/beacon.py). # the predicate in telemetry/beacon.py). Collection is local
# only — nothing is sent externally.
# Pre-#390 this only checked HEADROOM_TELEMETRY_DISABLED, # Pre-#390 this only checked HEADROOM_TELEMETRY_DISABLED,
# so users who set HEADROOM_TELEMETRY=off (the value in # so users who set HEADROOM_TELEMETRY=off (the value in
# the docs) still saw /v1/telemetry report enabled=true. # the docs) still saw /v1/telemetry report enabled=true.

View file

@ -27,7 +27,7 @@ _KNOWN_WRAP_AGENTS = frozenset(
# Stack slugs must start with a letter and contain only [a-z0-9_], max 64 chars. # Stack slugs must start with a letter and contain only [a-z0-9_], max 64 chars.
# Applied at every ingress (env var, HTTP header, stats aggregation) so downstream # Applied at every ingress (env var, HTTP header, stats aggregation) so downstream
# sinks (Prometheus labels, Supabase column, JSONB payload) see a bounded vocabulary. # sinks (Prometheus labels, OTEL attributes) see a bounded vocabulary.
_STACK_SLUG_RE = re.compile(r"^[a-z][a-z0-9_]{0,63}$") _STACK_SLUG_RE = re.compile(r"^[a-z][a-z0-9_]{0,63}$")
# Cardinality cap on the per-process requests_by_stack dict. Protects the # Cardinality cap on the per-process requests_by_stack dict. Protects the
@ -44,7 +44,7 @@ def normalize_stack(raw: str | None) -> str | None:
else ``None``. All external stack identifiers (env var, HTTP header, stats else ``None``. All external stack identifiers (env var, HTTP header, stats
keys) must pass through this function it is the single chokepoint that keys) must pass through this function it is the single chokepoint that
bounds cardinality and rejects garbage before it reaches Prometheus or the bounds cardinality and rejects garbage before it reaches Prometheus or the
Supabase telemetry row. OTEL metrics layer.
""" """
if not raw: if not raw:

View file

@ -11,9 +11,9 @@ strategy. These tests exist so the next regression of that shape
fails the suite the day it lands instead of waiting on an audit. fails the suite the day it lands instead of waiting on an audit.
The counters live ONLY as in-process state on the metrics instance; The counters live ONLY as in-process state on the metrics instance;
they are deliberately NOT exported through the Prometheus scrape or they are deliberately NOT exported as new Prometheus metric names
OTel surface, because the metricSupabase pipeline treats each (to avoid unbounded metric-series growth) they remain observable
metric name as a column and we cannot add new columns. CI-level via /stats. CI-level
observability via these tests is enough to catch silent regressions; observability via these tests is enough to catch silent regressions;
production export waits on a non-column-adding pipeline. production export waits on a non-column-adding pipeline.
@ -342,11 +342,11 @@ def test_prometheus_metrics_accumulates_codex_ws_unit_and_frame_counters():
def test_prometheus_export_does_not_leak_per_strategy_metrics(): def test_prometheus_export_does_not_leak_per_strategy_metrics():
"""Per-strategy state is tracked in-process only. The Prometheus """Per-strategy state is tracked in-process only. The Prometheus
scrape output deliberately must NOT emit new metric names the scrape output deliberately must NOT emit new metric names (to avoid
metricSupabase pipeline treats each metric name as a column, and unbounded metric-series growth); the state stays observable via
we cannot add new columns. This test guards that constraint: if a /stats. This test guards that constraint: if a future change adds
future change adds the metric to the scrape, this fails and forces the metric to the scrape, this fails and forces a conscious
a conscious decision.""" decision."""
import asyncio import asyncio
from headroom.proxy.prometheus_metrics import PrometheusMetrics from headroom.proxy.prometheus_metrics import PrometheusMetrics

View file

@ -1,7 +1,6 @@
"""Tests for proxy telemetry environment variable handling.""" """Tests for proxy telemetry environment variable handling."""
import asyncio import asyncio
from unittest.mock import patch
import pytest import pytest
@ -12,55 +11,6 @@ from fastapi.testclient import TestClient
from headroom.proxy.server import ProxyConfig, _proxy_config_from_env, create_app from headroom.proxy.server import ProxyConfig, _proxy_config_from_env, create_app
class TestProxyTelemetrySDKEnv:
"""Test HEADROOM_SDK handling when the proxy builds telemetry beacons."""
def test_proxy_telemetry_sdk_defaults_to_proxy(self, monkeypatch):
"""Telemetry beacon uses the default SDK label when env var is unset."""
monkeypatch.delenv("HEADROOM_SDK", raising=False)
with patch("headroom.telemetry.beacon.TelemetryBeacon") as mock_beacon:
create_app(
ProxyConfig(
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
)
)
assert mock_beacon.call_args.kwargs["sdk"] == "proxy"
def test_proxy_telemetry_sdk_uses_env_override(self, monkeypatch):
"""Telemetry beacon uses HEADROOM_SDK when it is non-empty."""
monkeypatch.setenv("HEADROOM_SDK", "headroom-app")
with patch("headroom.telemetry.beacon.TelemetryBeacon") as mock_beacon:
create_app(
ProxyConfig(
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
)
)
assert mock_beacon.call_args.kwargs["sdk"] == "headroom-app"
def test_proxy_telemetry_sdk_empty_env_falls_back_to_proxy(self, monkeypatch):
"""Telemetry beacon falls back to proxy when HEADROOM_SDK is blank."""
monkeypatch.setenv("HEADROOM_SDK", " ")
with patch("headroom.telemetry.beacon.TelemetryBeacon") as mock_beacon:
create_app(
ProxyConfig(
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
)
)
assert mock_beacon.call_args.kwargs["sdk"] == "proxy"
class TestProxyPeriodicTOINStatsEnv: class TestProxyPeriodicTOINStatsEnv:
"""Test HEADROOM_PERIODIC_TOIN_STATS handling for long-lived proxy workers.""" """Test HEADROOM_PERIODIC_TOIN_STATS handling for long-lived proxy workers."""

View file

@ -1,91 +0,0 @@
"""Phase 3e.0: surface per-strategy compression counters.
The internal `compressions_by_strategy` and `tokens_saved_by_strategy`
counters (PR #302) were tracked in process but never exported, because
the PrometheusSupabase pipeline treats each metric name as a column and
adding columns is operationally expensive.
These tests pin the alternative path:
1. The `/stats` endpoint exposes both dicts on the response root.
2. The telemetry beacon nests them under `pipeline_timing._strategies`,
landing as JSONB inside the existing `pipeline_timing` Supabase
column zero schema change.
"""
from __future__ import annotations
import pytest
def test_stats_endpoint_exposes_per_strategy_counters() -> None:
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient
from headroom.proxy.server import ProxyConfig, create_app
app = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
)
)
proxy = app.state.proxy
proxy.metrics.record_compression("smart_crusher", original_tokens=300, compressed_tokens=80)
proxy.metrics.record_compression("smart_crusher", original_tokens=100, compressed_tokens=40)
proxy.metrics.record_compression("diff", original_tokens=120, compressed_tokens=70)
with TestClient(app) as client:
response = client.get("/stats")
assert response.status_code == 200
body = response.json()
assert body["compressions_by_strategy"] == {"smart_crusher": 2, "diff": 1}
assert body["tokens_saved_by_strategy"] == {
"smart_crusher": (300 - 80) + (100 - 40),
"diff": 120 - 70,
}
def test_beacon_nests_strategies_under_pipeline_timing() -> None:
from headroom.telemetry.beacon import _build_pipeline_timing
stats = {
"pipeline_timing": {
"smart_crusher": {"average_ms": 12.345, "max_ms": 50.0, "count": 10},
"diff_compressor": {"average_ms": 5.1, "max_ms": 9.0, "count": 4},
},
"compressions_by_strategy": {"smart_crusher": 4321, "diff": 87},
"tokens_saved_by_strategy": {"smart_crusher": 1_234_567, "diff": 23_456},
}
timing = _build_pipeline_timing(stats)
assert timing["smart_crusher"] == 12.35
assert timing["diff_compressor"] == 5.1
assert timing["_strategies"] == {
"compressions": {"smart_crusher": 4321, "diff": 87},
"tokens_saved": {"smart_crusher": 1_234_567, "diff": 23_456},
}
def test_beacon_omits_strategies_subkey_when_counters_empty() -> None:
from headroom.telemetry.beacon import _build_pipeline_timing
stats = {
"pipeline_timing": {"router": {"average_ms": 1.2}},
"compressions_by_strategy": {},
"tokens_saved_by_strategy": {},
}
timing = _build_pipeline_timing(stats)
assert timing == {"router": 1.2}
assert "_strategies" not in timing

View file

@ -584,8 +584,8 @@ class TestGlobalTelemetryCollector:
Pre-#390 the collector only honoured HEADROOM_TELEMETRY_DISABLED, which Pre-#390 the collector only honoured HEADROOM_TELEMETRY_DISABLED, which
is undocumented. Users following the docs set HEADROOM_TELEMETRY=off and is undocumented. Users following the docs set HEADROOM_TELEMETRY=off and
watched /v1/telemetry continue to report enabled=true. The collector now watched /v1/telemetry continue to report enabled=true. The collector now
consults `is_telemetry_enabled()` (the same predicate the Supabase beacon consults `is_telemetry_enabled()` (the documented opt-in predicate),
uses), so both env vars take effect. so both env vars take effect.
""" """
reset_telemetry_collector() reset_telemetry_collector()
monkeypatch.delenv("HEADROOM_TELEMETRY_DISABLED", raising=False) monkeypatch.delenv("HEADROOM_TELEMETRY_DISABLED", raising=False)

View file

@ -227,8 +227,9 @@ class TestStatsEndpointTelemetryFlag:
pytest.importorskip("fastapi") pytest.importorskip("fastapi")
async def test_stats_includes_anon_telemetry_shipping_true(self, monkeypatch): async def test_stats_anon_telemetry_shipping_always_false(self, monkeypatch):
# Opt-in: shipping is only true once telemetry is explicitly enabled. # The anonymous telemetry beacon was removed, so nothing is ever shipped
# externally — even with telemetry explicitly enabled.
monkeypatch.setenv("HEADROOM_TELEMETRY", "on") monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
from headroom.proxy.server import ProxyConfig, create_app from headroom.proxy.server import ProxyConfig, create_app
@ -248,7 +249,7 @@ class TestStatsEndpointTelemetryFlag:
assert resp.status_code == 200 assert resp.status_code == 200
data = resp.json() data = resp.json()
assert "anon_telemetry_shipping" in data assert "anon_telemetry_shipping" in data
assert data["anon_telemetry_shipping"] is True assert data["anon_telemetry_shipping"] is False
async def test_stats_includes_anon_telemetry_shipping_false(self, monkeypatch): async def test_stats_includes_anon_telemetry_shipping_false(self, monkeypatch):
monkeypatch.setenv("HEADROOM_TELEMETRY", "off") monkeypatch.setenv("HEADROOM_TELEMETRY", "off")