mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
fix(telemetry): switch anonymous telemetry to opt-in (off by default) (#1223)
## Description
Anonymous usage telemetry was **on by default** (opt-out). This flips it
to **opt-in**: nothing is collected or shipped unless the user
explicitly turns it on. Small change, but it makes "no data leaves the
proxy by default" the actual default rather than something users have to
discover and disable.
Closes # <!-- N/A: no tracking issue -->
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality) — adds
`--telemetry` opt-in flag
- [x] Breaking change (fix or feature that would cause existing
functionality to change) — telemetry no longer runs unless opted in
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `is_telemetry_enabled()` is now **fail-closed**: only explicit
on-values (`on`/`true`/`1`/`yes`/`enable`/`enabled`) enable telemetry;
unset, empty, or unrecognized values stay disabled. This single
predicate gates both the Supabase beacon and the local `/v1/telemetry`
collector.
- Added `--telemetry` opt-in flag to `headroom proxy` and `headroom
install apply`; kept `--no-telemetry` and `HEADROOM_TELEMETRY=off` for
back-compat. If both are passed, opt-out wins.
- Install manifests now write `HEADROOM_TELEMETRY` explicitly
(`on`/`off`) plus the matching flag, so generated systemd/docker/launchd
deployments are unambiguous and don't rely on the runtime default.
- Startup banner and proxy log show `DISABLED` by default and surface
how to opt in.
- Updated tests for opt-in defaults; added coverage for the default-off
banner, the `--telemetry` flag, and the explicit-on manifest path.
- Updated docs
(proxy/configuration/installation/benchmarks/community-savings mdx, spec
011/015, wiki proxy/cli/benchmarks/metrics) and `CHANGELOG.md`.
## Testing
- [x] Unit tests pass (`pytest`) — targeted telemetry + install-planner
suites
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_telemetry_warning.py tests/test_telemetry.py tests/test_install/test_planner.py -q
81 passed
$ ruff check <changed source + test files>
All checks passed!
$ mypy headroom/telemetry/beacon.py headroom/cli/proxy.py headroom/cli/install.py \
headroom/install/planner.py headroom/proxy/server.py
Success: no issues found in 5 source files
```
## Real Behavior Proof
- **Environment:** macOS (darwin 25.4.0), project `.venv`, `headroom`
CLI.
- **Exact command / steps:**
```
$ python -c "import os; from headroom.telemetry.beacon import
is_telemetry_enabled; \
os.environ.pop('HEADROOM_TELEMETRY', None); print('unset ->',
is_telemetry_enabled()); \
[ (os.environ.__setitem__('HEADROOM_TELEMETRY', v), print(repr(v), '->',
is_telemetry_enabled())) \
for v in ['on','TRUE','1','yes','off','0','garbage',''] ]"
$ headroom proxy --help | grep -i telemetry
$ headroom install apply --help | grep -i telemetry
```
- **Observed result:**
```
unset -> False # off by default
'on' -> True 'TRUE' -> True '1' -> True 'yes' -> True
'off' -> False '0' -> False
'garbage' -> False '' -> False # fail-closed
proxy: --telemetry Opt in to anonymous usage telemetry — off by default
(env: HEADROOM_TELEMETRY=on)
--no-telemetry Force anonymous usage telemetry off (already the default;
env: HEADROOM_TELEMETRY=off)
install: --telemetry Opt in to anonymous telemetry in the runtime (off
by default).
--no-telemetry Force anonymous telemetry off in the runtime (already the
default).
```
- **Not tested:** live Supabase beacon network round-trip (no opt-in
network call was made); full `pytest` suite was not run — only the
telemetry + install-planner targeted suites.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- "Breaking change" is checked because the default behavior changes
(telemetry stops running unless opted in). It is **not** an API break —
`--no-telemetry` and `HEADROOM_TELEMETRY=off` still work, so existing
opt-out configs are unaffected.
- No tracking issue, so `Closes #` is left N/A.
This commit is contained in:
parent
f4bd2fe68f
commit
b99869778b
20 changed files with 114 additions and 39 deletions
|
|
@ -8,6 +8,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
|
||||
## Unreleased
|
||||
|
||||
### Changed
|
||||
|
||||
* **telemetry:** anonymous usage telemetry is now **opt-in** (off by default) instead of opt-out. Nothing is collected or sent unless you set `HEADROOM_TELEMETRY=on` or pass `--telemetry` to `headroom proxy` / `headroom install apply`. `is_telemetry_enabled()` is fail-closed — only explicit on-values (`on`/`true`/`1`/`yes`/`enable`/`enabled`) enable it; unset, empty, or unrecognized values stay disabled. The existing `--no-telemetry` flag and `HEADROOM_TELEMETRY=off` remain accepted for back-compat, and install manifests now write the `HEADROOM_TELEMETRY` value explicitly so generated deployments are unambiguous.
|
||||
|
||||
### Features
|
||||
|
||||
* **proxy:** measure and surface rolling and current token throughput metrics (active/wall-clock input, compression, effective forward, and streamed generation) in `headroom perf` CLI and the dashboard ([#959](https://github.com/chopratejas/headroom/issues/959)).
|
||||
|
|
|
|||
|
|
@ -107,7 +107,7 @@ ContentRouter accounts for 91--98% of pipeline cost on average. CacheAligner and
|
|||
|
||||
## Production Telemetry
|
||||
|
||||
Real-world data from **50,000+ proxy sessions** across 250+ unique instances (March--April 2026). Collected via anonymous telemetry (opt-out: `HEADROOM_TELEMETRY=off`).
|
||||
Real-world data from **50,000+ proxy sessions** across 250+ unique instances (March--April 2026). Collected via anonymous telemetry (opt-in: `HEADROOM_TELEMETRY=on`; telemetry is off by default).
|
||||
|
||||
### Proxy Overhead
|
||||
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ title: Community Savings
|
|||
description: Aggregate savings from Headroom instances across the community. Anonymous telemetry data — no prompts, no content, no PII.
|
||||
---
|
||||
|
||||
Real-time aggregate metrics from Headroom proxy instances worldwide. All data is anonymous — only token counts, compression ratios, and cost estimates are collected. [Opt out anytime](https://github.com/chopratejas/headroom/blob/main/headroom/telemetry/beacon.py) with `HEADROOM_TELEMETRY=off`.
|
||||
Real-time aggregate metrics from Headroom proxy instances worldwide. All data is anonymous — only token counts, compression ratios, and cost estimates are collected. Telemetry is off by default; [opt in](https://github.com/chopratejas/headroom/blob/main/headroom/telemetry/beacon.py) with `HEADROOM_TELEMETRY=on`.
|
||||
|
||||
## Overview
|
||||
|
||||
|
|
|
|||
|
|
@ -224,7 +224,7 @@ headroom proxy --learn --min-evidence 3
|
|||
| `HEADROOM_MAX_CONNECTIONS` | Maximum upstream HTTP connections | `500` |
|
||||
| `HEADROOM_MAX_KEEPALIVE` | Maximum upstream keep-alive connections | `100` |
|
||||
| `HEADROOM_BUDGET` | Daily budget limit in USD | -- |
|
||||
| `HEADROOM_TELEMETRY` | Set to `off` to disable anonymous telemetry | enabled |
|
||||
| `HEADROOM_TELEMETRY` | Set to `on` to opt in to anonymous telemetry | `off` (opt-in) |
|
||||
| `HEADROOM_STATELESS` | Set to `true` to disable filesystem writes | `false` |
|
||||
| `HEADROOM_MODEL_LIMITS` | Custom model config (JSON string or file path) | -- |
|
||||
| `HEADROOM_BASE_URL` | Base URL of the Headroom proxy (TypeScript SDK) | `http://localhost:8787` |
|
||||
|
|
@ -234,7 +234,7 @@ headroom proxy --learn --min-evidence 3
|
|||
| `HEADROOM_SAVINGS_PATH` | Override persistent savings file location. Always wins when set. | derived from `${HEADROOM_WORKSPACE_DIR}` |
|
||||
| `HEADROOM_TOIN_PATH` | Override TOIN telemetry file location. Always wins when set. | derived from `${HEADROOM_WORKSPACE_DIR}` |
|
||||
| `HEADROOM_SUBSCRIPTION_STATE_PATH` | Override subscription tracker state file. Always wins when set. | derived from `${HEADROOM_WORKSPACE_DIR}` |
|
||||
| `HEADROOM_TELEMETRY` | Set to `off` to disable anonymous telemetry | `on` |
|
||||
| `HEADROOM_TELEMETRY` | Set to `on` to opt in to anonymous telemetry | `off` |
|
||||
| `HEADROOM_MEMORY_INJECTION_MODE` | Memory-context routing mode: `live_zone_tail` (default) or `disabled`. The legacy `system_prompt` mode was retired by PR-A2; supplying it raises. | `live_zone_tail` |
|
||||
| `HEADROOM_PROXY_PYTHON_FORWARDER_MODE` | Python forwarder serialization mode. `byte_faithful` (default) forwards original request bytes verbatim when no transform mutated the body and re-serializes canonically only when needed — keeps Anthropic prompt-cache hit-rate intact. `legacy_json_kwarg` is an explicit operator opt-in for emergency rollback to the historical `httpx ... json=body` behavior. NOT a fallback — only flip on explicit operator decision. | `byte_faithful` |
|
||||
| `HEADROOM_STRIP_INTERNAL_HEADERS` | Python proxy: whether to strip internal `x-headroom-*` request headers (e.g. `x-headroom-bypass`, `x-headroom-mode`, `x-headroom-user-id`, `x-headroom-stack`, `x-headroom-base-url`) before every upstream forwarder call (PR-A5, fixes P5-49). `enabled` (default) stops fingerprinting / leakage. `disabled` is an explicit operator opt-in for diagnostic shadow tracing — NOT a fallback. Inbound reads of these headers (bypass gating, memory user-id resolution) are unaffected because they read `request.headers` directly. | `enabled` |
|
||||
|
|
|
|||
|
|
@ -227,7 +227,7 @@ These variables configure Headroom at runtime. Set them in your shell, `.env` fi
|
|||
| `HEADROOM_PORT` | `8787` | Port the proxy listens on |
|
||||
| `HEADROOM_HOST` | `127.0.0.1` | Host the proxy binds to |
|
||||
| `HEADROOM_MODE` | `token` | Default optimization mode: `token` or `cache` |
|
||||
| `HEADROOM_TELEMETRY` | enabled | Set to `off` to disable anonymous telemetry |
|
||||
| `HEADROOM_TELEMETRY` | `off` (opt-in) | Set to `on` to opt in to anonymous telemetry |
|
||||
|
||||
### TypeScript SDK
|
||||
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@ headroom proxy \
|
|||
--budget 100.0
|
||||
```
|
||||
|
||||
Telemetry is enabled by default. Opt out with `HEADROOM_TELEMETRY=off` or `--no-telemetry`.
|
||||
Telemetry is **off by default** (opt-in). Opt in with `HEADROOM_TELEMETRY=on` or `--telemetry`.
|
||||
|
||||
## CLI options
|
||||
|
||||
|
|
@ -46,7 +46,8 @@ Telemetry is enabled by default. Opt out with `HEADROOM_TELEMETRY=off` or `--no-
|
|||
| `--gemini-api-url` | Gemini default | Custom Gemini API URL |
|
||||
| `--backend` | `anthropic` | Backend: `anthropic`, `bedrock`, `openrouter`, `anyllm`, or `litellm-<provider>` |
|
||||
| `--bedrock-api-url` | None | Bedrock InvokeModel upstream for the `/model/{id}/invoke` passthrough routes (see [Bedrock via a local gateway](#bedrock-via-a-local-gateway)) |
|
||||
| `--no-telemetry` | `false` | Disable anonymous telemetry |
|
||||
| `--telemetry` | `false` | Opt in to anonymous telemetry (off by default) |
|
||||
| `--no-telemetry` | `false` | Force anonymous telemetry off (already the default) |
|
||||
| `--stateless` | `false` | Disable filesystem writes and keep runtime state in memory |
|
||||
|
||||
### Context management
|
||||
|
|
|
|||
|
|
@ -125,7 +125,7 @@ deployment:
|
|||
| `HEADROOM_HOST` | `127.0.0.1` | Proxy host |
|
||||
| `ANTHROPIC_API_KEY` | - | Anthropic API key |
|
||||
| `OPENAI_API_KEY` | - | OpenAI API key |
|
||||
| `HEADROOM_TELEMETRY` | enabled | Set to `off` to disable telemetry |
|
||||
| `HEADROOM_TELEMETRY` | `off` (opt-in) | Set to `on` to opt in to telemetry |
|
||||
|
||||
### Config File
|
||||
|
||||
|
|
|
|||
|
|
@ -26,7 +26,8 @@ headroom proxy [OPTIONS]
|
|||
| `--memory` | `false` | Enable persistent memory |
|
||||
| `--learn` | `false` | Enable live traffic learning |
|
||||
| `--backend` | `anthropic` | Backend: anthropic, bedrock, openrouter, anyllm, or litellm-* |
|
||||
| `--no-telemetry` | `false` | Disable anonymous telemetry |
|
||||
| `--telemetry` | `false` | Opt in to anonymous telemetry (off by default) |
|
||||
| `--no-telemetry` | `false` | Force anonymous telemetry off (already the default) |
|
||||
| `--stateless` | `false` | Disable filesystem writes |
|
||||
|
||||
---
|
||||
|
|
@ -231,7 +232,7 @@ X-Headroom-Compressed-Tokens: 5325
|
|||
| `HEADROOM_MAX_CONNECTIONS` | `500` | Maximum upstream HTTP connections |
|
||||
| `HEADROOM_MAX_KEEPALIVE` | `100` | Maximum upstream keep-alive connections |
|
||||
| `HEADROOM_BUDGET` | - | Daily budget limit in USD |
|
||||
| `HEADROOM_TELEMETRY` | enabled | Set to `off` to disable anonymous telemetry |
|
||||
| `HEADROOM_TELEMETRY` | `off` (opt-in) | Set to `on` to opt in to anonymous telemetry |
|
||||
| `HEADROOM_STATELESS` | `false` | Disable filesystem writes |
|
||||
|
||||
### Provider
|
||||
|
|
@ -247,7 +248,7 @@ X-Headroom-Compressed-Tokens: 5325
|
|||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `HEADROOM_TELEMETRY` | enabled | Set to `off` to disable telemetry |
|
||||
| `HEADROOM_TELEMETRY` | `off` (opt-in) | Set to `on` to opt in to telemetry |
|
||||
| `HEADROOM_MIN_EVIDENCE` | `5` | Minimum observations before live learning persists a pattern |
|
||||
| `HEADROOM_PROXY_EXTENSIONS` | - | Comma-separated proxy extensions to enable |
|
||||
| `HEADROOM_STATELESS` | `false` | Disable filesystem writes |
|
||||
|
|
|
|||
|
|
@ -157,7 +157,16 @@ def _reject_task_lifecycle(manifest: DeploymentManifest, action: str) -> None:
|
|||
"--mode", "proxy_mode", default="token", show_default=True, help="Proxy optimization mode."
|
||||
)
|
||||
@click.option("--memory", is_flag=True, help="Enable persistent memory in the proxy runtime.")
|
||||
@click.option("--no-telemetry", is_flag=True, help="Disable anonymous telemetry in the runtime.")
|
||||
@click.option(
|
||||
"--telemetry",
|
||||
is_flag=True,
|
||||
help="Opt in to anonymous telemetry in the runtime (off by default).",
|
||||
)
|
||||
@click.option(
|
||||
"--no-telemetry",
|
||||
is_flag=True,
|
||||
help="Force anonymous telemetry off in the runtime (already the default).",
|
||||
)
|
||||
@click.option(
|
||||
"--image",
|
||||
default="ghcr.io/chopratejas/headroom:latest",
|
||||
|
|
@ -177,6 +186,7 @@ def install_apply(
|
|||
region: str | None,
|
||||
proxy_mode: str,
|
||||
memory: bool,
|
||||
telemetry: bool,
|
||||
no_telemetry: bool,
|
||||
image: str,
|
||||
) -> None:
|
||||
|
|
@ -198,7 +208,7 @@ def install_apply(
|
|||
region=region,
|
||||
proxy_mode=proxy_mode,
|
||||
memory_enabled=memory,
|
||||
telemetry_enabled=not no_telemetry,
|
||||
telemetry_enabled=telemetry and not no_telemetry,
|
||||
image=image,
|
||||
)
|
||||
|
||||
|
|
|
|||
|
|
@ -598,10 +598,15 @@ def _selected_context_tool() -> str:
|
|||
"(env: BEDROCK_TARGET_API_URL)"
|
||||
),
|
||||
)
|
||||
@click.option(
|
||||
"--telemetry",
|
||||
is_flag=True,
|
||||
help="Opt in to anonymous usage telemetry — off by default (env: HEADROOM_TELEMETRY=on)",
|
||||
)
|
||||
@click.option(
|
||||
"--no-telemetry",
|
||||
is_flag=True,
|
||||
help="Disable anonymous usage telemetry (env: HEADROOM_TELEMETRY=off)",
|
||||
help="Force anonymous usage telemetry off (already the default; env: HEADROOM_TELEMETRY=off)",
|
||||
)
|
||||
@click.option(
|
||||
"--stateless",
|
||||
|
|
@ -686,6 +691,7 @@ def proxy(
|
|||
bedrock_region: str | None,
|
||||
bedrock_profile: str | None,
|
||||
bedrock_api_url: str | None,
|
||||
telemetry: bool,
|
||||
no_telemetry: bool,
|
||||
stateless: bool,
|
||||
embedding_server: bool,
|
||||
|
|
@ -783,7 +789,10 @@ def proxy(
|
|||
"on",
|
||||
)
|
||||
|
||||
# Telemetry opt-out: --no-telemetry flag sets the env var
|
||||
# Telemetry is opt-in (off by default). --telemetry opts in; --no-telemetry
|
||||
# forces it off. If both are passed, the explicit opt-out wins (fail-closed).
|
||||
if telemetry:
|
||||
os.environ["HEADROOM_TELEMETRY"] = "on"
|
||||
if no_telemetry:
|
||||
os.environ["HEADROOM_TELEMETRY"] = "off"
|
||||
|
||||
|
|
@ -993,14 +1002,17 @@ Memory (Multi-Provider):
|
|||
|
||||
from headroom.telemetry.beacon import is_telemetry_enabled
|
||||
|
||||
# Build telemetry section for the startup banner
|
||||
# Build telemetry section for the startup banner. Telemetry is opt-in
|
||||
# (off by default); the disabled line surfaces how to opt in.
|
||||
if is_telemetry_enabled():
|
||||
telemetry_line = (
|
||||
" Telemetry: ENABLED (anonymous aggregate stats)\n"
|
||||
" Telemetry: ENABLED (anonymous aggregate stats — you opted in)\n"
|
||||
" Disable: HEADROOM_TELEMETRY=off or headroom proxy --no-telemetry"
|
||||
)
|
||||
else:
|
||||
telemetry_line = " Telemetry: DISABLED"
|
||||
telemetry_line = (
|
||||
" Telemetry: DISABLED (opt in: HEADROOM_TELEMETRY=on or headroom proxy --telemetry)"
|
||||
)
|
||||
|
||||
# Discover proxy extensions (third-party packages registered via the
|
||||
# `headroom.proxy_extension` entry-point group). Surfaced in the banner
|
||||
|
|
|
|||
|
|
@ -141,8 +141,9 @@ def build_manifest(
|
|||
base_env["HEADROOM_ANYLLM_PROVIDER"] = anyllm_provider
|
||||
if region:
|
||||
base_env["HEADROOM_REGION"] = region
|
||||
if not telemetry_enabled:
|
||||
base_env["HEADROOM_TELEMETRY"] = "off"
|
||||
# Telemetry is opt-in (off by default). Write the value explicitly so the
|
||||
# generated manifest is unambiguous and doesn't depend on the runtime default.
|
||||
base_env["HEADROOM_TELEMETRY"] = "on" if telemetry_enabled else "off"
|
||||
if memory_enabled:
|
||||
base_env["HEADROOM_MEMORY_ENABLED"] = "1"
|
||||
|
||||
|
|
@ -156,8 +157,7 @@ def build_manifest(
|
|||
"--backend",
|
||||
backend,
|
||||
]
|
||||
if not telemetry_enabled:
|
||||
proxy_args.append("--no-telemetry")
|
||||
proxy_args.append("--telemetry" if telemetry_enabled else "--no-telemetry")
|
||||
if memory_enabled:
|
||||
proxy_args.extend(["--memory", "--memory-db-path", str(_paths.memory_db_path())])
|
||||
if anyllm_provider:
|
||||
|
|
|
|||
|
|
@ -1376,7 +1376,10 @@ class HeadroomProxy(
|
|||
"Opt out: HEADROOM_TELEMETRY=off or --no-telemetry"
|
||||
)
|
||||
else:
|
||||
logger.info("Anonymous telemetry: DISABLED")
|
||||
logger.info(
|
||||
"Anonymous telemetry: DISABLED (off by default — opt in: "
|
||||
"HEADROOM_TELEMETRY=on or --telemetry)"
|
||||
)
|
||||
|
||||
self.pipeline_extensions.emit(
|
||||
PipelineStage.POST_START,
|
||||
|
|
|
|||
|
|
@ -3,9 +3,9 @@
|
|||
Sends aggregate-only stats (tokens saved, compression ratios, cache hit rates,
|
||||
performance overhead) to help improve Headroom. No prompts, no content, no PII.
|
||||
|
||||
On by default. Opt out with:
|
||||
HEADROOM_TELEMETRY=off headroom proxy
|
||||
headroom proxy --no-telemetry
|
||||
Off by default (opt-in). Nothing is collected or sent unless you opt in with:
|
||||
HEADROOM_TELEMETRY=on headroom proxy
|
||||
headroom proxy --telemetry
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -42,6 +42,7 @@ _INTERVAL_SECONDS = 300
|
|||
|
||||
|
||||
_OFF_VALUES = frozenset(("off", "false", "0", "no", "disable", "disabled"))
|
||||
_ON_VALUES = frozenset(("on", "true", "1", "yes", "enable", "enabled"))
|
||||
|
||||
|
||||
def _build_pipeline_timing(stats: dict) -> dict[str, object]:
|
||||
|
|
@ -69,9 +70,14 @@ def _build_pipeline_timing(stats: dict) -> dict[str, object]:
|
|||
|
||||
|
||||
def is_telemetry_enabled() -> bool:
|
||||
"""Check if telemetry is enabled (on by default, opt out with env var)."""
|
||||
val = os.environ.get("HEADROOM_TELEMETRY", "on").lower().strip()
|
||||
return val not in _OFF_VALUES
|
||||
"""Check if telemetry is enabled (off by default, opt in with env var).
|
||||
|
||||
Fail-closed: telemetry is only enabled when HEADROOM_TELEMETRY is set to an
|
||||
explicit on-value (on/true/1/yes/enable/enabled). Anything else — including
|
||||
unset, empty, or an unrecognized value — leaves it disabled.
|
||||
"""
|
||||
val = os.environ.get("HEADROOM_TELEMETRY", "").lower().strip()
|
||||
return val in _ON_VALUES
|
||||
|
||||
|
||||
def is_telemetry_warn_enabled() -> bool:
|
||||
|
|
|
|||
|
|
@ -39,6 +39,7 @@ def test_build_manifest_for_persistent_docker_sets_expected_defaults() -> None:
|
|||
assert manifest.health_url == "http://127.0.0.1:8787/readyz"
|
||||
assert manifest.base_env["HEADROOM_PORT"] == "8787"
|
||||
assert manifest.base_env["HEADROOM_TELEMETRY"] == "off"
|
||||
assert "--no-telemetry" in manifest.proxy_args
|
||||
assert manifest.tool_envs["claude"]["ANTHROPIC_BASE_URL"] == "http://127.0.0.1:8787"
|
||||
assert manifest.tool_envs["copilot"]["COPILOT_PROVIDER_TYPE"] == "anthropic"
|
||||
assert "--memory" in manifest.proxy_args
|
||||
|
|
@ -62,6 +63,9 @@ def test_build_manifest_uses_provider_slice_env_builders_for_all_supported_targe
|
|||
image="ghcr.io/chopratejas/headroom:latest",
|
||||
)
|
||||
|
||||
# telemetry_enabled=True must write the explicit opt-in value + flag.
|
||||
assert manifest.base_env["HEADROOM_TELEMETRY"] == "on"
|
||||
assert "--telemetry" in manifest.proxy_args
|
||||
assert manifest.tool_envs["claude"]["ANTHROPIC_BASE_URL"] == "http://127.0.0.1:9999"
|
||||
assert manifest.tool_envs["codex"]["OPENAI_BASE_URL"] == "http://127.0.0.1:9999/v1"
|
||||
assert manifest.tool_envs["aider"] == {
|
||||
|
|
|
|||
|
|
@ -601,7 +601,9 @@ class TestGlobalTelemetryCollector:
|
|||
)
|
||||
|
||||
def test_headroom_telemetry_on_keeps_collector_enabled(self, monkeypatch):
|
||||
"""Sanity check: the explicit on/unset path leaves the collector enabled."""
|
||||
"""Sanity check: the explicit opt-in path (HEADROOM_TELEMETRY=on) leaves
|
||||
the collector enabled. Telemetry is off by default, so this requires the
|
||||
user to have turned it on."""
|
||||
reset_telemetry_collector()
|
||||
monkeypatch.delenv("HEADROOM_TELEMETRY_DISABLED", raising=False)
|
||||
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
|
||||
|
|
|
|||
|
|
@ -97,6 +97,20 @@ class TestProxyCLITelemetryBanner:
|
|||
return CliRunner()
|
||||
|
||||
def test_banner_shows_telemetry_enabled(self, runner, monkeypatch):
|
||||
# Telemetry is opt-in: it only shows ENABLED once explicitly turned on.
|
||||
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
|
||||
|
||||
from headroom.cli.main import main
|
||||
|
||||
with patch("headroom.proxy.server.run_server", side_effect=SystemExit(0)):
|
||||
result = runner.invoke(main, ["proxy"])
|
||||
|
||||
assert "Telemetry:" in result.output
|
||||
assert "ENABLED" in result.output
|
||||
|
||||
def test_banner_disabled_by_default(self, runner, monkeypatch):
|
||||
# The whole point of opt-in: unset env => telemetry off, banner says so
|
||||
# and surfaces how to opt in.
|
||||
monkeypatch.delenv("HEADROOM_TELEMETRY", raising=False)
|
||||
|
||||
from headroom.cli.main import main
|
||||
|
|
@ -104,6 +118,18 @@ class TestProxyCLITelemetryBanner:
|
|||
with patch("headroom.proxy.server.run_server", side_effect=SystemExit(0)):
|
||||
result = runner.invoke(main, ["proxy"])
|
||||
|
||||
assert "Telemetry:" in result.output
|
||||
assert "DISABLED" in result.output
|
||||
assert "HEADROOM_TELEMETRY=on" in result.output or "--telemetry" in result.output
|
||||
|
||||
def test_telemetry_flag_opts_in(self, runner, monkeypatch):
|
||||
monkeypatch.delenv("HEADROOM_TELEMETRY", raising=False)
|
||||
|
||||
from headroom.cli.main import main
|
||||
|
||||
with patch("headroom.proxy.server.run_server", side_effect=SystemExit(0)):
|
||||
result = runner.invoke(main, ["proxy", "--telemetry"])
|
||||
|
||||
assert "Telemetry:" in result.output
|
||||
assert "ENABLED" in result.output
|
||||
|
||||
|
|
@ -130,7 +156,7 @@ class TestProxyCLITelemetryBanner:
|
|||
assert "DISABLED" in result.output
|
||||
|
||||
def test_banner_shows_opt_out_instructions_when_enabled(self, runner, monkeypatch):
|
||||
monkeypatch.delenv("HEADROOM_TELEMETRY", raising=False)
|
||||
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
|
||||
|
||||
from headroom.cli.main import main
|
||||
|
||||
|
|
@ -202,7 +228,8 @@ class TestStatsEndpointTelemetryFlag:
|
|||
pytest.importorskip("fastapi")
|
||||
|
||||
async def test_stats_includes_anon_telemetry_shipping_true(self, monkeypatch):
|
||||
monkeypatch.delenv("HEADROOM_TELEMETRY", raising=False)
|
||||
# Opt-in: shipping is only true once telemetry is explicitly enabled.
|
||||
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
|
||||
from headroom.proxy.server import ProxyConfig, create_app
|
||||
|
||||
app = create_app(
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ Tested on Apple M-series (CPU), headroom v0.5.18. Each test runs `compress()` on
|
|||
|
||||
## Production Telemetry
|
||||
|
||||
Real-world data from **50,000+ proxy sessions** across 250+ unique instances (March 30 – April 2, 2026). Collected via anonymous telemetry beacon (opt-out: `HEADROOM_TELEMETRY=off`).
|
||||
Real-world data from **50,000+ proxy sessions** across 250+ unique instances (March 30 – April 2, 2026). Collected via anonymous telemetry beacon (opt-in: `HEADROOM_TELEMETRY=on`; telemetry is off by default).
|
||||
|
||||
### Proxy Overhead
|
||||
|
||||
|
|
@ -184,7 +184,7 @@ A 94.9% compression means the output is 5.1% of the original size.
|
|||
- Collected via anonymous beacon (no prompts, no content, no PII)
|
||||
- Image-inflated instances excluded (base64 counted as text tokens — fixed in v0.5.18)
|
||||
- Multi-worker beacon spam excluded (per-instance MAX, not SUM)
|
||||
- Opt-out: `HEADROOM_TELEMETRY=off`
|
||||
- Opt-in: `HEADROOM_TELEMETRY=on` (telemetry is off by default)
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
11
wiki/cli.md
11
wiki/cli.md
|
|
@ -272,7 +272,8 @@ headroom proxy --mode cache
|
|||
| `--region` | `us-west-2` | Cloud region for Bedrock / Vertex / related backends |
|
||||
| `--bedrock-region` | unset | Deprecated Bedrock region override |
|
||||
| `--bedrock-profile` | unset | AWS profile name for Bedrock |
|
||||
| `--no-telemetry` | off | Disable anonymous usage telemetry |
|
||||
| `--telemetry` | off | Opt in to anonymous usage telemetry (off by default) |
|
||||
| `--no-telemetry` | off | Force anonymous usage telemetry off (already the default) |
|
||||
|
||||
Notes:
|
||||
|
||||
|
|
@ -603,7 +604,10 @@ Options:
|
|||
backends.
|
||||
--mode TEXT Proxy optimization mode. [default: token]
|
||||
--memory Enable persistent memory in the proxy runtime.
|
||||
--no-telemetry Disable anonymous telemetry in the runtime.
|
||||
--telemetry Opt in to anonymous telemetry in the runtime
|
||||
(off by default).
|
||||
--no-telemetry Force anonymous telemetry off in the runtime
|
||||
(already the default).
|
||||
--image TEXT Docker image to use when runtime=docker or
|
||||
preset=persistent-docker. [default:
|
||||
ghcr.io/chopratejas/headroom:latest]
|
||||
|
|
@ -632,7 +636,8 @@ headroom install apply --preset persistent-docker --scope user
|
|||
| `--region` | unset | Cloud region override |
|
||||
| `--mode` | `token` | Proxy optimization mode |
|
||||
| `--memory` | off | Enable persistent memory in the managed runtime |
|
||||
| `--no-telemetry` | off | Disable anonymous telemetry |
|
||||
| `--telemetry` | off | Opt in to anonymous telemetry (off by default) |
|
||||
| `--no-telemetry` | off | Force anonymous telemetry off (already the default) |
|
||||
| `--image` | `ghcr.io/chopratejas/headroom:latest` | Docker image for Docker-backed installs |
|
||||
|
||||
`apply` stores a manifest under
|
||||
|
|
|
|||
|
|
@ -248,7 +248,7 @@ Headroom's managed OTEL exporters are intentionally scoped to Headroom's own ins
|
|||
|
||||
Headroom has two separate systems:
|
||||
|
||||
- `HEADROOM_TELEMETRY` / `--no-telemetry` controls the privacy-preserving anonymous data-flywheel beacon and TOIN-related aggregate reporting.
|
||||
- `HEADROOM_TELEMETRY` / `--telemetry` / `--no-telemetry` controls the privacy-preserving anonymous data-flywheel beacon and TOIN-related aggregate reporting. It is **off by default** (opt-in): set `HEADROOM_TELEMETRY=on` or pass `--telemetry` to enable it.
|
||||
- `HEADROOM_OTEL_*` controls operational OTEL metric export.
|
||||
|
||||
They are independent by design so you can disable the anonymous beacon while keeping OTEL metrics enabled, or vice versa.
|
||||
|
|
|
|||
|
|
@ -36,7 +36,7 @@ OPENAI_BASE_URL=http://localhost:8787/v1 your-app
|
|||
|
||||
`headroom wrap copilot` uses Copilot CLI's BYOK provider settings under the hood. In `provider-type=auto`, it chooses Headroom's Anthropic route for the default proxy backend and the OpenAI-compatible `/v1` route for translated backends such as `anyllm` and LiteLLM.
|
||||
|
||||
Anonymous aggregate telemetry is enabled by default. Opt out with `HEADROOM_TELEMETRY=off` or `headroom proxy --no-telemetry`. Downstream apps can set `HEADROOM_SDK=headroom-app` to override the anonymous telemetry `sdk` label; the default remains `proxy`.
|
||||
Anonymous aggregate telemetry is **off by default** (opt-in). Opt in with `HEADROOM_TELEMETRY=on` or `headroom proxy --telemetry`. Downstream apps can set `HEADROOM_SDK=headroom-app` to override the anonymous telemetry `sdk` label; the default remains `proxy`.
|
||||
|
||||
Operational OTEL metrics are configured separately and are **off by default**. Install `headroom-ai[proxy,otel]` and set:
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue