mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-10 14:27:00 -04:00
9cfb00838a
## In one line
Headroom starts reporting **how well compression is working** — counters
and percentages only. **No prompts. No code. No file paths. Nothing
about what you're building.**
## Why
Right now nobody knows whether compression actually helps real users.
You can see your own numbers in `/stats`, but that's it — there's no way
to tell whether a given workload compresses well, or why it sometimes
doesn't. This closes that loop so we can make compression better for
everyone.
## Exactly what gets sent
One message per session, and every 5 minutes while you're active:
```json
{
"session": { "id": "random", "turns": 47, "duration_s": 4210, "seq": 3 },
"tokens": { "original": 890000, "attempted": 410000, "saved": 320000,
"tool_saved": 48000, "cache_read": 210000 },
"rates": { "saved_pct": 35.96, "eligible_pct": 46.07, "yield_pct": 78.05,
"cache_read_pct": 23.60, "overhead_pct": 1.96 },
"compression": { "transforms": {"crush": 47}, "passthrough_turns": 0 },
"skips": {},
"sources": { "proxy": 47 },
"providers": ["anthropic"],
"models": ["claude-sonnet-4-5-20250929"],
"failures": 2
}
```
Plus a random install ID, the Headroom version, and OS/architecture
(`darwin`, `arm64`).
That's the whole thing. A full example lives at
`deploy/beacon/sample-event.json`.
## What is never sent
- Your prompts or the model's responses
- Your code
- File paths, project names, repo names
- Tool names or MCP server names
- Hostname, username, or IP address
- Custom or fine-tuned model names (an id like `ft:gpt-4o:acme-corp:…`
contains a company name, so only models in a public registry are
reported)
**This is structural, not a pinky-swear.** Every value in the payload is
a number, a fixed word, or a random ID — there is no free-text field
anywhere for content to hide in. The receiver
(`deploy/beacon/worker.js`, in this repo so you can read it) drops
anything not on an explicit allowlist before storing.
## Turning it off
Any one of these:
```bash
HEADROOM_BEACON=off # or
DO_NOT_TRACK=1 # or
# offline mode
```
It's on by default, and Headroom says so at startup:
```
Telemetry: anonymous compression stats — never prompts, code, or file paths.
Helps us improve compression | Off: HEADROOM_BEACON=off
```
`HEADROOM_TELEMETRY` is a **separate** switch that still only affects
local stats. If you had turned that on, this change does not start
uploading anything — you answered a different question, and upgrading
should not change the answer.
## Why the percentages, not just "tokens saved"
"We saved 36%" hides the interesting part. In the example above only
**46% of tokens were eligible** for compression at all — the rest is
frozen cache prefix and system prompts we deliberately do not touch. Of
what we *could* touch, we removed **78%**.
Those are two separate problems. Raising eligibility is proxy work;
raising yield is compressor work. A single number cannot tell us which
to fix.
## Coverage
`emit_request_outcome` is a single chokepoint —
`handler.metrics.record_request` is called from exactly one place,
inside the funnel — so all 30 `RequestOutcome` construction sites are
covered: Anthropic, OpenAI, Gemini, Bedrock, batch, streaming, and the
long-lived Codex Responses-WS path.
The `headroom_compress` MCP path bypassed that funnel and is now wired
in separately. It has a different shape (no provider, no upstream
latency, and everything handed to the tool is eligible by construction),
so `sources` counts turns by origin — MCP turns always read
`eligible_pct: 100` and must not drag the proxy's real eligibility
ceiling upward.
**Subagents.** All subagent traffic through the proxy merges into one
session, which is correct for savings and retention but means `turns`
conflates fan-out with depth. Fan-out is still derivable —
`compression.latency_ms_total / session.duration_s` gives the
concurrency ratio (~1x serial, ~4x for four parallel agents), so no
extra field is needed. Verified no lost updates under 6-way concurrency
(1,200 turns).
**Known gap:** `--workers N` gives each process its own aggregator, so
one user session becomes up to N. Token totals and fleet rates stay
correct; session counts inflate. This matches the existing documented
limitation that TOIN state, CostTracker, and the prefix tracker are all
per-process.
## Notes for reviewers
- **Cumulative snapshots, not deltas.** Every report restates running
totals under one session ID, so the highest `seq` per `(install,
session)` is the complete session. Dedupe is a window function, and a
lost report costs nothing.
- **Never breaks the proxy.** Every path swallows its own exceptions;
uploads go out on a daemon thread so nothing blocks the request loop.
- **Explicit User-Agent is load-bearing.** urllib's default is blocked
by Cloudflare (error 1010). Combined with fire-and-forget error
handling, that would have failed every upload while looking perfectly
healthy.
- **The exit flush was broken and is fixed.** `atexit` handed the POST
to a daemon thread, and daemon threads are killed before they finish
during interpreter shutdown — so nothing was sent. That silently dropped
*every session shorter than the 5-minute heartbeat*, plus all
short-lived subagent MCP processes. The exit path now posts
synchronously with a 2s timeout.
- Receiver and query tooling are in `deploy/beacon/`.
## Testing
- `python -m headroom.telemetry.session` self-check: dedupe, cumulative
totals, dropped-report recovery, payload contains no model id or
prompt-derived string, allowlist coverage
- 175 telemetry/outcome tests pass; 6 new ones cover the opt-out notice
- Verified end to end against a live deployment: client → receiver →
storage → query
## Still to do before release
The default endpoint currently points at a temporary `workers.dev` URL.
It needs to move to a Headroom-owned hostname before this ships in a
tagged release — noted inline at `DEFAULT_ENDPOINT`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
339 lines
13 KiB
Python
339 lines
13 KiB
Python
"""Tests for anonymous telemetry warning feature.
|
||
|
||
Covers:
|
||
- is_telemetry_warn_enabled() feature flag
|
||
- format_telemetry_notice() helper
|
||
- proxy CLI banner includes telemetry status
|
||
- wrap CLI prints telemetry notice
|
||
- /stats endpoint exposes anon_telemetry_shipping flag
|
||
"""
|
||
|
||
from unittest.mock import patch
|
||
|
||
import pytest
|
||
|
||
click = pytest.importorskip("click")
|
||
from click.testing import CliRunner # noqa: E402
|
||
|
||
from headroom.telemetry.beacon import ( # noqa: E402
|
||
format_telemetry_notice,
|
||
is_telemetry_warn_enabled,
|
||
)
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# is_telemetry_warn_enabled
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
class TestIsTelemetryWarnEnabled:
|
||
"""Tests for the HEADROOM_TELEMETRY_WARN feature flag."""
|
||
|
||
def test_enabled_by_default(self, monkeypatch):
|
||
monkeypatch.delenv("HEADROOM_TELEMETRY_WARN", raising=False)
|
||
assert is_telemetry_warn_enabled() is True
|
||
|
||
@pytest.mark.parametrize("value", ["off", "OFF", "false", "0", "no", "disable", "disabled"])
|
||
def test_disabled_by_env_var(self, monkeypatch, value):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY_WARN", value)
|
||
assert is_telemetry_warn_enabled() is False
|
||
|
||
@pytest.mark.parametrize("value", ["on", "ON", "1", "yes", "true"])
|
||
def test_enabled_by_truthy_env_var(self, monkeypatch, value):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY_WARN", value)
|
||
assert is_telemetry_warn_enabled() is True
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# format_telemetry_notice
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
class TestFormatTelemetryNotice:
|
||
"""Tests for format_telemetry_notice()."""
|
||
|
||
def test_returns_notice_when_telemetry_on(self, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
|
||
monkeypatch.setenv("HEADROOM_BEACON", "off")
|
||
monkeypatch.delenv("HEADROOM_TELEMETRY_WARN", raising=False)
|
||
notice = format_telemetry_notice()
|
||
assert notice != ""
|
||
assert "ENABLED" in notice
|
||
assert "HEADROOM_TELEMETRY=off" in notice
|
||
assert "--no-telemetry" in notice
|
||
|
||
def test_empty_when_telemetry_off(self, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "off")
|
||
monkeypatch.setenv("HEADROOM_BEACON", "off")
|
||
monkeypatch.delenv("HEADROOM_TELEMETRY_WARN", raising=False)
|
||
assert format_telemetry_notice() == ""
|
||
|
||
def test_beacon_is_announced_by_default(self, monkeypatch):
|
||
"""The beacon is opt-out, so the notice is the only place a user finds
|
||
out it is running. Silence here is how anonymous telemetry becomes a
|
||
trust incident."""
|
||
for var in ("HEADROOM_TELEMETRY", "HEADROOM_BEACON", "DO_NOT_TRACK"):
|
||
monkeypatch.delenv(var, raising=False)
|
||
monkeypatch.delenv("HEADROOM_TELEMETRY_WARN", raising=False)
|
||
notice = format_telemetry_notice()
|
||
assert "compression stats" in notice
|
||
assert "HEADROOM_BEACON=off" in notice
|
||
|
||
def test_beacon_notice_names_what_is_not_sent(self, monkeypatch):
|
||
"""Vague reassurance is worse than none. The notice has to name the
|
||
three things users actually worry about."""
|
||
for var in ("HEADROOM_TELEMETRY", "HEADROOM_BEACON", "DO_NOT_TRACK"):
|
||
monkeypatch.delenv(var, raising=False)
|
||
monkeypatch.delenv("HEADROOM_TELEMETRY_WARN", raising=False)
|
||
notice = format_telemetry_notice()
|
||
assert "never prompts" in notice
|
||
assert "code" in notice
|
||
assert "file paths" in notice
|
||
|
||
def test_silent_when_beacon_disabled_and_no_local(self, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_BEACON", "off")
|
||
for var in ("HEADROOM_TELEMETRY", "DO_NOT_TRACK"):
|
||
monkeypatch.delenv(var, raising=False)
|
||
monkeypatch.delenv("HEADROOM_TELEMETRY_WARN", raising=False)
|
||
assert format_telemetry_notice() == ""
|
||
|
||
def test_do_not_track_silences_the_beacon_notice(self, monkeypatch):
|
||
monkeypatch.setenv("DO_NOT_TRACK", "1")
|
||
for var in ("HEADROOM_TELEMETRY", "HEADROOM_BEACON"):
|
||
monkeypatch.delenv(var, raising=False)
|
||
monkeypatch.delenv("HEADROOM_TELEMETRY_WARN", raising=False)
|
||
assert format_telemetry_notice() == ""
|
||
|
||
def test_empty_when_warn_flag_off(self, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY_WARN", "off")
|
||
assert format_telemetry_notice() == ""
|
||
|
||
def test_prefix_is_applied(self, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
|
||
monkeypatch.delenv("HEADROOM_TELEMETRY_WARN", raising=False)
|
||
notice = format_telemetry_notice(prefix=" ")
|
||
assert notice.startswith(" ")
|
||
|
||
def test_no_prefix_by_default(self, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
|
||
monkeypatch.delenv("HEADROOM_TELEMETRY_WARN", raising=False)
|
||
notice = format_telemetry_notice()
|
||
# Default prefix is "" so the string should start with "Telemetry"
|
||
assert notice.startswith("Telemetry:")
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# proxy CLI banner
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
class TestProxyCLITelemetryBanner:
|
||
"""Proxy CLI startup banner must include telemetry status."""
|
||
|
||
@pytest.fixture
|
||
def runner(self):
|
||
return CliRunner()
|
||
|
||
def test_banner_shows_telemetry_enabled(self, runner, monkeypatch):
|
||
# Telemetry is opt-in: it only shows ENABLED once explicitly turned on.
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
|
||
|
||
from headroom.cli.main import main
|
||
|
||
with patch("headroom.proxy.server.run_server", side_effect=SystemExit(0)):
|
||
result = runner.invoke(main, ["proxy"])
|
||
|
||
assert "Telemetry:" in result.output
|
||
assert "ENABLED" in result.output
|
||
|
||
def test_banner_disabled_by_default(self, runner, monkeypatch):
|
||
# The whole point of opt-in: unset env => telemetry off, banner says so
|
||
# and surfaces how to opt in.
|
||
monkeypatch.delenv("HEADROOM_TELEMETRY", raising=False)
|
||
|
||
from headroom.cli.main import main
|
||
|
||
with patch("headroom.proxy.server.run_server", side_effect=SystemExit(0)):
|
||
result = runner.invoke(main, ["proxy"])
|
||
|
||
assert "Telemetry:" in result.output
|
||
assert "DISABLED" in result.output
|
||
assert "HEADROOM_TELEMETRY=on" in result.output or "--telemetry" in result.output
|
||
|
||
def test_telemetry_flag_opts_in(self, runner, monkeypatch):
|
||
monkeypatch.delenv("HEADROOM_TELEMETRY", raising=False)
|
||
|
||
from headroom.cli.main import main
|
||
|
||
with patch("headroom.proxy.server.run_server", side_effect=SystemExit(0)):
|
||
result = runner.invoke(main, ["proxy", "--telemetry"])
|
||
|
||
assert "Telemetry:" in result.output
|
||
assert "ENABLED" in result.output
|
||
|
||
def test_banner_shows_telemetry_disabled(self, runner, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "off")
|
||
|
||
from headroom.cli.main import main
|
||
|
||
with patch("headroom.proxy.server.run_server", side_effect=SystemExit(0)):
|
||
result = runner.invoke(main, ["proxy"])
|
||
|
||
assert "Telemetry:" in result.output
|
||
assert "DISABLED" in result.output
|
||
|
||
def test_no_telemetry_flag_disables(self, runner, monkeypatch):
|
||
monkeypatch.delenv("HEADROOM_TELEMETRY", raising=False)
|
||
|
||
from headroom.cli.main import main
|
||
|
||
with patch("headroom.proxy.server.run_server", side_effect=SystemExit(0)):
|
||
result = runner.invoke(main, ["proxy", "--no-telemetry"])
|
||
|
||
assert "Telemetry:" in result.output
|
||
assert "DISABLED" in result.output
|
||
|
||
def test_banner_shows_opt_out_instructions_when_enabled(self, runner, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
|
||
|
||
from headroom.cli.main import main
|
||
|
||
with patch("headroom.proxy.server.run_server", side_effect=SystemExit(0)):
|
||
result = runner.invoke(main, ["proxy"])
|
||
|
||
assert "HEADROOM_TELEMETRY=off" in result.output or "--no-telemetry" in result.output
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# wrap CLI telemetry notice
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
class TestWrapCLITelemetryNotice:
|
||
"""_print_telemetry_notice() is called from wrap commands."""
|
||
|
||
def test_print_notice_outputs_when_telemetry_on(self, monkeypatch, capsys):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
|
||
monkeypatch.setenv("HEADROOM_BEACON", "off")
|
||
monkeypatch.delenv("HEADROOM_TELEMETRY_WARN", raising=False)
|
||
|
||
from headroom.cli.wrap import _print_telemetry_notice
|
||
|
||
_print_telemetry_notice()
|
||
captured = capsys.readouterr()
|
||
assert "Telemetry" in captured.out
|
||
assert "HEADROOM_TELEMETRY=off" in captured.out
|
||
|
||
def test_print_notice_announces_beacon_by_default(self, monkeypatch, capsys):
|
||
for var in ("HEADROOM_TELEMETRY", "HEADROOM_BEACON", "DO_NOT_TRACK"):
|
||
monkeypatch.delenv(var, raising=False)
|
||
monkeypatch.delenv("HEADROOM_TELEMETRY_WARN", raising=False)
|
||
|
||
from headroom.cli.wrap import _print_telemetry_notice
|
||
|
||
_print_telemetry_notice()
|
||
captured = capsys.readouterr()
|
||
assert "compression stats" in captured.out
|
||
assert "HEADROOM_BEACON=off" in captured.out
|
||
|
||
def test_print_notice_silent_when_telemetry_off(self, monkeypatch, capsys):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "off")
|
||
monkeypatch.setenv("HEADROOM_BEACON", "off")
|
||
|
||
from headroom.cli.wrap import _print_telemetry_notice
|
||
|
||
_print_telemetry_notice()
|
||
captured = capsys.readouterr()
|
||
assert captured.out == ""
|
||
|
||
def test_print_notice_silent_when_warn_flag_off(self, monkeypatch, capsys):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY_WARN", "off")
|
||
|
||
from headroom.cli.wrap import _print_telemetry_notice
|
||
|
||
_print_telemetry_notice()
|
||
captured = capsys.readouterr()
|
||
assert captured.out == ""
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# /stats endpoint – anon_telemetry_shipping flag
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
class TestStatsEndpointTelemetryFlag:
|
||
"""The /stats endpoint must expose anon_telemetry_shipping."""
|
||
|
||
pytest.importorskip("fastapi")
|
||
|
||
async def test_stats_anon_telemetry_shipping_always_false(self, monkeypatch):
|
||
# The anonymous telemetry beacon was removed, so nothing is ever shipped
|
||
# externally — even with telemetry explicitly enabled.
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
|
||
from headroom.proxy.server import ProxyConfig, create_app
|
||
|
||
app = create_app(
|
||
ProxyConfig(
|
||
cache_enabled=False,
|
||
rate_limit_enabled=False,
|
||
cost_tracking_enabled=False,
|
||
)
|
||
)
|
||
|
||
from httpx import ASGITransport, AsyncClient
|
||
|
||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
|
||
resp = await client.get("/stats")
|
||
|
||
assert resp.status_code == 200
|
||
data = resp.json()
|
||
assert "anon_telemetry_shipping" in data
|
||
assert data["anon_telemetry_shipping"] is False
|
||
|
||
async def test_stats_includes_anon_telemetry_shipping_false(self, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_TELEMETRY", "off")
|
||
from headroom.proxy.server import ProxyConfig, create_app
|
||
|
||
app = create_app(
|
||
ProxyConfig(
|
||
cache_enabled=False,
|
||
rate_limit_enabled=False,
|
||
cost_tracking_enabled=False,
|
||
)
|
||
)
|
||
|
||
from httpx import ASGITransport, AsyncClient
|
||
|
||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
|
||
resp = await client.get("/stats")
|
||
|
||
assert resp.status_code == 200
|
||
data = resp.json()
|
||
assert "anon_telemetry_shipping" in data
|
||
assert data["anon_telemetry_shipping"] is False
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# telemetry __init__ exports
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
class TestTelemetryModuleExports:
|
||
"""New helpers must be exported from headroom.telemetry."""
|
||
|
||
def test_is_telemetry_warn_enabled_exported(self):
|
||
from headroom.telemetry import is_telemetry_warn_enabled as fn
|
||
|
||
assert callable(fn)
|
||
|
||
def test_is_telemetry_enabled_exported(self):
|
||
from headroom.telemetry import is_telemetry_enabled as fn
|
||
|
||
assert callable(fn)
|
||
|
||
def test_format_telemetry_notice_exported(self):
|
||
from headroom.telemetry import format_telemetry_notice as fn
|
||
|
||
assert callable(fn)
|