mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
b99869778b
## Description
Anonymous usage telemetry was **on by default** (opt-out). This flips it
to **opt-in**: nothing is collected or shipped unless the user
explicitly turns it on. Small change, but it makes "no data leaves the
proxy by default" the actual default rather than something users have to
discover and disable.
Closes # <!-- N/A: no tracking issue -->
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality) — adds
`--telemetry` opt-in flag
- [x] Breaking change (fix or feature that would cause existing
functionality to change) — telemetry no longer runs unless opted in
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `is_telemetry_enabled()` is now **fail-closed**: only explicit
on-values (`on`/`true`/`1`/`yes`/`enable`/`enabled`) enable telemetry;
unset, empty, or unrecognized values stay disabled. This single
predicate gates both the Supabase beacon and the local `/v1/telemetry`
collector.
- Added `--telemetry` opt-in flag to `headroom proxy` and `headroom
install apply`; kept `--no-telemetry` and `HEADROOM_TELEMETRY=off` for
back-compat. If both are passed, opt-out wins.
- Install manifests now write `HEADROOM_TELEMETRY` explicitly
(`on`/`off`) plus the matching flag, so generated systemd/docker/launchd
deployments are unambiguous and don't rely on the runtime default.
- Startup banner and proxy log show `DISABLED` by default and surface
how to opt in.
- Updated tests for opt-in defaults; added coverage for the default-off
banner, the `--telemetry` flag, and the explicit-on manifest path.
- Updated docs
(proxy/configuration/installation/benchmarks/community-savings mdx, spec
011/015, wiki proxy/cli/benchmarks/metrics) and `CHANGELOG.md`.
## Testing
- [x] Unit tests pass (`pytest`) — targeted telemetry + install-planner
suites
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_telemetry_warning.py tests/test_telemetry.py tests/test_install/test_planner.py -q
81 passed
$ ruff check <changed source + test files>
All checks passed!
$ mypy headroom/telemetry/beacon.py headroom/cli/proxy.py headroom/cli/install.py \
headroom/install/planner.py headroom/proxy/server.py
Success: no issues found in 5 source files
```
## Real Behavior Proof
- **Environment:** macOS (darwin 25.4.0), project `.venv`, `headroom`
CLI.
- **Exact command / steps:**
```
$ python -c "import os; from headroom.telemetry.beacon import
is_telemetry_enabled; \
os.environ.pop('HEADROOM_TELEMETRY', None); print('unset ->',
is_telemetry_enabled()); \
[ (os.environ.__setitem__('HEADROOM_TELEMETRY', v), print(repr(v), '->',
is_telemetry_enabled())) \
for v in ['on','TRUE','1','yes','off','0','garbage',''] ]"
$ headroom proxy --help | grep -i telemetry
$ headroom install apply --help | grep -i telemetry
```
- **Observed result:**
```
unset -> False # off by default
'on' -> True 'TRUE' -> True '1' -> True 'yes' -> True
'off' -> False '0' -> False
'garbage' -> False '' -> False # fail-closed
proxy: --telemetry Opt in to anonymous usage telemetry — off by default
(env: HEADROOM_TELEMETRY=on)
--no-telemetry Force anonymous usage telemetry off (already the default;
env: HEADROOM_TELEMETRY=off)
install: --telemetry Opt in to anonymous telemetry in the runtime (off
by default).
--no-telemetry Force anonymous telemetry off in the runtime (already the
default).
```
- **Not tested:** live Supabase beacon network round-trip (no opt-in
network call was made); full `pytest` suite was not run — only the
telemetry + install-planner targeted suites.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- "Breaking change" is checked because the default behavior changes
(telemetry stops running unless opted in). It is **not** an API break —
`--no-telemetry` and `HEADROOM_TELEMETRY=off` still work, so existing
opt-out configs are unaffected.
- No tracking issue, so `Closes #` is left N/A.
297 lines
11 KiB
Python
297 lines
11 KiB
Python
"""Tests for anonymous telemetry warning feature.
|
||
|
||
Covers:
|
||
- is_telemetry_warn_enabled() feature flag
|
||
- format_telemetry_notice() helper
|
||
- proxy CLI banner includes telemetry status
|
||
- wrap CLI prints telemetry notice
|
||
- /stats endpoint exposes anon_telemetry_shipping flag
|
||
"""
|
||
|
||
from unittest.mock import patch
|
||
|
||
import pytest
|
||
|
||
click = pytest.importorskip("click")
|
||
from click.testing import CliRunner # noqa: E402
|
||
|
||
from headroom.telemetry.beacon import ( # noqa: E402
|
||
format_telemetry_notice,
|
||
is_telemetry_warn_enabled,
|
||
)
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# is_telemetry_warn_enabled
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
class TestIsTelemetryWarnEnabled:
|
||
"""Tests for the HEADROOM_TELEMETRY_WARN feature flag."""
|
||
|
||
def test_enabled_by_default(self, monkeypatch):
|
||
monkeypatch.delenv("HEADROOM_TELEMETRY_WARN", raising=False)
|
||
assert is_telemetry_warn_enabled() is True
|
||
|
||
@pytest.mark.parametrize("value", ["off", "OFF", "false", "0", "no", "disable", "disabled"])
|
||
def test_disabled_by_env_var(self, monkeypatch, value):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY_WARN", value)
|
||
assert is_telemetry_warn_enabled() is False
|
||
|
||
@pytest.mark.parametrize("value", ["on", "ON", "1", "yes", "true"])
|
||
def test_enabled_by_truthy_env_var(self, monkeypatch, value):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY_WARN", value)
|
||
assert is_telemetry_warn_enabled() is True
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# format_telemetry_notice
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
class TestFormatTelemetryNotice:
|
||
"""Tests for format_telemetry_notice()."""
|
||
|
||
def test_returns_notice_when_telemetry_on(self, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
|
||
monkeypatch.delenv("HEADROOM_TELEMETRY_WARN", raising=False)
|
||
notice = format_telemetry_notice()
|
||
assert notice != ""
|
||
assert "ENABLED" in notice
|
||
assert "HEADROOM_TELEMETRY=off" in notice
|
||
assert "--no-telemetry" in notice
|
||
|
||
def test_empty_when_telemetry_off(self, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "off")
|
||
monkeypatch.delenv("HEADROOM_TELEMETRY_WARN", raising=False)
|
||
assert format_telemetry_notice() == ""
|
||
|
||
def test_empty_when_warn_flag_off(self, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY_WARN", "off")
|
||
assert format_telemetry_notice() == ""
|
||
|
||
def test_prefix_is_applied(self, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
|
||
monkeypatch.delenv("HEADROOM_TELEMETRY_WARN", raising=False)
|
||
notice = format_telemetry_notice(prefix=" ")
|
||
assert notice.startswith(" ")
|
||
|
||
def test_no_prefix_by_default(self, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
|
||
monkeypatch.delenv("HEADROOM_TELEMETRY_WARN", raising=False)
|
||
notice = format_telemetry_notice()
|
||
# Default prefix is "" so the string should start with "Telemetry"
|
||
assert notice.startswith("Telemetry:")
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# proxy CLI banner
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
class TestProxyCLITelemetryBanner:
|
||
"""Proxy CLI startup banner must include telemetry status."""
|
||
|
||
@pytest.fixture
|
||
def runner(self):
|
||
return CliRunner()
|
||
|
||
def test_banner_shows_telemetry_enabled(self, runner, monkeypatch):
|
||
# Telemetry is opt-in: it only shows ENABLED once explicitly turned on.
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
|
||
|
||
from headroom.cli.main import main
|
||
|
||
with patch("headroom.proxy.server.run_server", side_effect=SystemExit(0)):
|
||
result = runner.invoke(main, ["proxy"])
|
||
|
||
assert "Telemetry:" in result.output
|
||
assert "ENABLED" in result.output
|
||
|
||
def test_banner_disabled_by_default(self, runner, monkeypatch):
|
||
# The whole point of opt-in: unset env => telemetry off, banner says so
|
||
# and surfaces how to opt in.
|
||
monkeypatch.delenv("HEADROOM_TELEMETRY", raising=False)
|
||
|
||
from headroom.cli.main import main
|
||
|
||
with patch("headroom.proxy.server.run_server", side_effect=SystemExit(0)):
|
||
result = runner.invoke(main, ["proxy"])
|
||
|
||
assert "Telemetry:" in result.output
|
||
assert "DISABLED" in result.output
|
||
assert "HEADROOM_TELEMETRY=on" in result.output or "--telemetry" in result.output
|
||
|
||
def test_telemetry_flag_opts_in(self, runner, monkeypatch):
|
||
monkeypatch.delenv("HEADROOM_TELEMETRY", raising=False)
|
||
|
||
from headroom.cli.main import main
|
||
|
||
with patch("headroom.proxy.server.run_server", side_effect=SystemExit(0)):
|
||
result = runner.invoke(main, ["proxy", "--telemetry"])
|
||
|
||
assert "Telemetry:" in result.output
|
||
assert "ENABLED" in result.output
|
||
|
||
def test_banner_shows_telemetry_disabled(self, runner, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "off")
|
||
|
||
from headroom.cli.main import main
|
||
|
||
with patch("headroom.proxy.server.run_server", side_effect=SystemExit(0)):
|
||
result = runner.invoke(main, ["proxy"])
|
||
|
||
assert "Telemetry:" in result.output
|
||
assert "DISABLED" in result.output
|
||
|
||
def test_no_telemetry_flag_disables(self, runner, monkeypatch):
|
||
monkeypatch.delenv("HEADROOM_TELEMETRY", raising=False)
|
||
|
||
from headroom.cli.main import main
|
||
|
||
with patch("headroom.proxy.server.run_server", side_effect=SystemExit(0)):
|
||
result = runner.invoke(main, ["proxy", "--no-telemetry"])
|
||
|
||
assert "Telemetry:" in result.output
|
||
assert "DISABLED" in result.output
|
||
|
||
def test_banner_shows_opt_out_instructions_when_enabled(self, runner, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
|
||
|
||
from headroom.cli.main import main
|
||
|
||
with patch("headroom.proxy.server.run_server", side_effect=SystemExit(0)):
|
||
result = runner.invoke(main, ["proxy"])
|
||
|
||
assert "HEADROOM_TELEMETRY=off" in result.output or "--no-telemetry" in result.output
|
||
|
||
def test_banner_shows_context_tool(self, runner, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_CONTEXT_TOOL", "lean-ctx")
|
||
|
||
from headroom.cli.main import main
|
||
|
||
with patch("headroom.proxy.server.run_server", side_effect=SystemExit(0)):
|
||
result = runner.invoke(main, ["proxy"])
|
||
|
||
assert result.exit_code == 0
|
||
assert "Context Tool: lean-ctx" in result.output
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# wrap CLI telemetry notice
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
class TestWrapCLITelemetryNotice:
|
||
"""_print_telemetry_notice() is called from wrap commands."""
|
||
|
||
def test_print_notice_outputs_when_telemetry_on(self, monkeypatch, capsys):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
|
||
monkeypatch.delenv("HEADROOM_TELEMETRY_WARN", raising=False)
|
||
|
||
from headroom.cli.wrap import _print_telemetry_notice
|
||
|
||
_print_telemetry_notice()
|
||
captured = capsys.readouterr()
|
||
assert "Telemetry" in captured.out
|
||
assert "HEADROOM_TELEMETRY=off" in captured.out
|
||
|
||
def test_print_notice_silent_when_telemetry_off(self, monkeypatch, capsys):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "off")
|
||
|
||
from headroom.cli.wrap import _print_telemetry_notice
|
||
|
||
_print_telemetry_notice()
|
||
captured = capsys.readouterr()
|
||
assert captured.out == ""
|
||
|
||
def test_print_notice_silent_when_warn_flag_off(self, monkeypatch, capsys):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY_WARN", "off")
|
||
|
||
from headroom.cli.wrap import _print_telemetry_notice
|
||
|
||
_print_telemetry_notice()
|
||
captured = capsys.readouterr()
|
||
assert captured.out == ""
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# /stats endpoint – anon_telemetry_shipping flag
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
class TestStatsEndpointTelemetryFlag:
|
||
"""The /stats endpoint must expose anon_telemetry_shipping."""
|
||
|
||
pytest.importorskip("fastapi")
|
||
|
||
async def test_stats_includes_anon_telemetry_shipping_true(self, monkeypatch):
|
||
# Opt-in: shipping is only true once telemetry is explicitly enabled.
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
|
||
from headroom.proxy.server import ProxyConfig, create_app
|
||
|
||
app = create_app(
|
||
ProxyConfig(
|
||
cache_enabled=False,
|
||
rate_limit_enabled=False,
|
||
cost_tracking_enabled=False,
|
||
)
|
||
)
|
||
|
||
from httpx import ASGITransport, AsyncClient
|
||
|
||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
|
||
resp = await client.get("/stats")
|
||
|
||
assert resp.status_code == 200
|
||
data = resp.json()
|
||
assert "anon_telemetry_shipping" in data
|
||
assert data["anon_telemetry_shipping"] is True
|
||
|
||
async def test_stats_includes_anon_telemetry_shipping_false(self, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "off")
|
||
from headroom.proxy.server import ProxyConfig, create_app
|
||
|
||
app = create_app(
|
||
ProxyConfig(
|
||
cache_enabled=False,
|
||
rate_limit_enabled=False,
|
||
cost_tracking_enabled=False,
|
||
)
|
||
)
|
||
|
||
from httpx import ASGITransport, AsyncClient
|
||
|
||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
|
||
resp = await client.get("/stats")
|
||
|
||
assert resp.status_code == 200
|
||
data = resp.json()
|
||
assert "anon_telemetry_shipping" in data
|
||
assert data["anon_telemetry_shipping"] is False
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# telemetry __init__ exports
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
class TestTelemetryModuleExports:
|
||
"""New helpers must be exported from headroom.telemetry."""
|
||
|
||
def test_is_telemetry_warn_enabled_exported(self):
|
||
from headroom.telemetry import is_telemetry_warn_enabled as fn
|
||
|
||
assert callable(fn)
|
||
|
||
def test_is_telemetry_enabled_exported(self):
|
||
from headroom.telemetry import is_telemetry_enabled as fn
|
||
|
||
assert callable(fn)
|
||
|
||
def test_format_telemetry_notice_exported(self):
|
||
from headroom.telemetry import format_telemetry_notice as fn
|
||
|
||
assert callable(fn)
|