2 commits

Author	SHA1	Message	Date
Tejas Chopra	e0870ef931	feat(beacon): hourly R2 compaction, per-strategy savings, and a stack that reports (#2853) ... Three beacon changes bundled because they are one story: the corpus got too slow to query, and then too coarse to answer the question it was collected for. ## 1. Hourly compaction (`deploy/beacon`) The beacon writes one ~1 KB object per heartbeat — **64,987 on 2026-08-06** and climbing. A full analysis `pull` was ~100k HTTPS round trips for 95 MB: minutes of pure per-object latency. Listing the bucket alone took 88 seconds. Moving the query server-side does not help — R2 SQL reads only Iceberg tables, and a pile of tiny files is the pathological case for every query engine. Compaction is the fix, and it is Iceberg's own answer to the same problem. An hourly cron collapses each **complete** hour of `sessions/` into one `rollup/dt=…/hh=…/data.ndjson`, keeping the highest-`seq` heartbeat per `(install, session)`. | measured on `dt=2026-08-06/hh=14` | before | after | |---|---|---| | objects | 3,938 | **1** | | rows | 3,938 | **1,061** | | analysis `pull` | minutes | **seconds** | Hourly rather than daily because every R2 binding call is a subrequest: a day is ~65k, an hour is ~4k. Newest-first, so a backlog drains from the present backwards and live data never starves behind it; a failing hour is logged and skipped rather than blocking every older hour behind it. **Raw objects are never deleted**, so any rollup is rebuildable by deleting it. Backfill runs to the **oldest surviving raw day**, not a fixed window. A fixed lookback strands everything older than it the moment analysis stops reading `sessions/`: the raw objects are still there, but nothing would ever compact them, so they disappear from every report. `oldestRawDay()` finds that floor in one delimited LIST, and the rollup listing starts from it — so the work is bounded by retention rather than by total history. Three failure modes the tests pin down, because each one is silent: - A **failed `get`** is transient, so the hour throws and writes nothing. A rollup is built once and trusted forever, so a short read would quietly become the permanent record. - A **corrupt record** loses only itself. This Worker wrote that content with `JSON.stringify`; it will never become valid, so blocking on it strands the hour instead of the record. - An **empty hour** writes an `empty` marker. Without one the hour stays "missing" and is re-listed on every run forever. `test-rollup.mjs` asserts the Worker's dedup picks exactly the same rows as the analysis-side `QUALIFY`. If those two ever disagree the reports go quietly wrong rather than loudly broken, which is why that check exists. Its stub paginates at 3 keys so the list cursor loop — load-bearing at the real ~4,000 objects/hour — runs in every case. ## 2. Per-strategy savings (`compression.by_strategy`) `compression.transforms` counts *invocations*, which cannot distinguish a compressor that saved 60% from one that ran constantly and saved nothing. The fleet's top transform by count contributes an unknown share of `tokens.saved`. It is worse than that in practice. Transform labels are slugged with `split(":", 1)[0]`, so every `router:<strategy>:<detail>` label collapses into a single `router` bucket. On 2026-08-08 that bucket held **19.1 M of the day's transform counts, across 8,528 of 9,616 sessions** — the compressors that do most of the work are indistinguishable from each other, by name as well as by yield: | transform | n | sessions | |---|---|---| | `router` | 19,108,118 | 8,528 | | `anthropic` | 688,124 | 4,734 | | `output_shaper` | 548,017 | 1,120 | No question about which strategy is earning its keep can be answered from that, which is what this field is for. The measurement already existed. `PrometheusMetrics.record_compression` is the configured `CompressionObserver` and already accumulates `tokens_saved_by_strategy` on the hot path — the numbers just never left the process. This forwards from that one chokepoint rather than adding a second observer and a second measurement pass. The paths that have **no** observer configured (MCP server, LangGraph, Strands hooks, the transform pipeline) get `BeaconCompressionObserver` passed directly. Compression runs on the executor thread *before* that request's outcome reaches `record()`, so events are **staged** into module state and drained by the next outcome. Staging is what makes two things true at once: - The first turn of a session still reports its numbers — otherwise every session's opening turn, and any session short enough to be one turn, would report nothing. - A compression event **never opens a session**. An abandoned request would otherwise emit a phantom `turns=0` row with all-zero tokens, inflating fleet session and install counts. Staging takes a dedicated mutex the request path never touches, so the fan-out stays off the aggregator's lock and `record_compression` keeps its "synchronous + lock-free" contract. ```json "by_strategy": [ {"strategy": "code_aware", "n": 1, "tokens_in": 800, "tokens_out": 800}, {"strategy": "smart_crusher", "n": 2, "tokens_in": 1500, "tokens_out": 700} ] ``` A **list of records, sorted by strategy** — not an object keyed by strategy. Keyed shapes change type as keys accumulate: DuckDB infers a STRUCT under ~24 keys and a MAP over it, so the analysis query breaks on the day the fleet picks up a 25th strategy. Sorted so heartbeats are byte-comparable. **These do not sum to `tokens.saved`,** and the field comment says so: strategies compose (the router routes, a strategy runs inside it) so the same text is measured more than once. A row means "of what this strategy was handed, it removed this much" — a per-strategy yield, not a share of the total. A strategy that saved nothing still appears; dropping it would make every strategy look effective. ## 3. `headroom.stack` `resource_attributes()` was called with no arguments at its one call site, so `headroom.stack` was absent from **all 24,040 sessions** in the corpus while `detect_stack` sat unused — dead code on both ends of a wire nobody connected. The fleet was unsegmentable by agent, which is the question the corpus is asked most often. Environment detection alone is not enough. It answers `wrap_claude` only under `headroom wrap`; every install that points an agent at a persistent proxy — the common deployment — reports `proxy`, which segments nothing. The per-request `X-Headroom-Stack` slugs are the only signal that names the harness there, so `record_stack()` stages them the same way and feeds `detect_stack`'s `by_stack` branch: ``` 9x wrap_claude, 1x wrap_cursor -> wrap_claude (dominant harness wins) 5x wrap_claude, 5x wrap_cursor -> mixed no per-request signal -> proxy (environment fallback) junk slug -> dropped before staging ``` ## Privacy The strategy string is slugged through the same `_safe_slug` as skip reasons and capped at `MAX_STRATEGIES`, because the observer protocol takes a free string and an extension could otherwise invent keys per request. Stack slugs are normalized and capped the same way. **Deliberately not collected:** the tool names in `smart_crush:<count>:<names>`. Those are user-defined MCP identifiers and can name internal tooling (`acme_deploy_prod`). They stay stripped by the existing `split(":", 1)[0]`, and this PR does not widen it. No new key was needed in `worker.js` — `by_strategy` nests under the already-allowlisted `compression`, and `headroom.stack` was already in `ALLOWED_RESOURCE`. Nothing here deletes or rewrites existing data: the Worker only ever writes, `sessions/` is never pruned by it, and readers merge old and new shapes with `union_by_name`, so pre-change heartbeats keep reading with the new fields null. ## Verification - `python -m headroom.telemetry.session` — self-check covers staging, the no-phantom-session case, drain exhaustiveness, a 0%-yield strategy staying visible, the cardinality cap, slug safety, and dominant/mixed/junk stack resolution - `node test-rollup.mjs /tmp/hr` — 3,938 real corpus objects → 1,061 sessions in 1 object, plus the pagination, partial-read, corrupt-record, empty-hour and `oldestRawDay` cases - 51 passing in `test_compression_observability`, `test_prometheus_obs_counters`, `test_telemetry_context`, `test_compression_strategy_outcomes` - Consumer side exists and is checked: `beacon.sh by_strategy` in headroom-beacon-stats reads the field end to end (sessions, installs, invocations, tokens in/out, yield %), verified against a synthetic parquet for the cases that matter — aggregation across installs, a 0%-yield strategy staying visible, pre-field sessions dropping out rather than erroring. It is guarded on the column, so it prints an instruction instead of a binder error until a release carrying this PR reaches the fleet. - Deployed and running against the live corpus on the `5 * * * *` trigger 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-09 15:42:12 -07:00
Tejas Chopra	9cfb00838a	fix(telemetry): anonymous compression stats — no prompts, no data (#2728) ... ## In one line Headroom starts reporting **how well compression is working** — counters and percentages only. **No prompts. No code. No file paths. Nothing about what you're building.** ## Why Right now nobody knows whether compression actually helps real users. You can see your own numbers in `/stats`, but that's it — there's no way to tell whether a given workload compresses well, or why it sometimes doesn't. This closes that loop so we can make compression better for everyone. ## Exactly what gets sent One message per session, and every 5 minutes while you're active: ```json { "session": { "id": "random", "turns": 47, "duration_s": 4210, "seq": 3 }, "tokens": { "original": 890000, "attempted": 410000, "saved": 320000, "tool_saved": 48000, "cache_read": 210000 }, "rates": { "saved_pct": 35.96, "eligible_pct": 46.07, "yield_pct": 78.05, "cache_read_pct": 23.60, "overhead_pct": 1.96 }, "compression": { "transforms": {"crush": 47}, "passthrough_turns": 0 }, "skips": {}, "sources": { "proxy": 47 }, "providers": ["anthropic"], "models": ["claude-sonnet-4-5-20250929"], "failures": 2 } ``` Plus a random install ID, the Headroom version, and OS/architecture (`darwin`, `arm64`). That's the whole thing. A full example lives at `deploy/beacon/sample-event.json`. ## What is never sent - Your prompts or the model's responses - Your code - File paths, project names, repo names - Tool names or MCP server names - Hostname, username, or IP address - Custom or fine-tuned model names (an id like `ft:gpt-4o:acme-corp:…` contains a company name, so only models in a public registry are reported) **This is structural, not a pinky-swear.** Every value in the payload is a number, a fixed word, or a random ID — there is no free-text field anywhere for content to hide in. The receiver (`deploy/beacon/worker.js`, in this repo so you can read it) drops anything not on an explicit allowlist before storing. ## Turning it off Any one of these: ```bash HEADROOM_BEACON=off # or DO_NOT_TRACK=1 # or # offline mode ``` It's on by default, and Headroom says so at startup: ``` Telemetry: anonymous compression stats — never prompts, code, or file paths. Helps us improve compression | Off: HEADROOM_BEACON=off ``` `HEADROOM_TELEMETRY` is a **separate** switch that still only affects local stats. If you had turned that on, this change does not start uploading anything — you answered a different question, and upgrading should not change the answer. ## Why the percentages, not just "tokens saved" "We saved 36%" hides the interesting part. In the example above only **46% of tokens were eligible** for compression at all — the rest is frozen cache prefix and system prompts we deliberately do not touch. Of what we *could* touch, we removed **78%**. Those are two separate problems. Raising eligibility is proxy work; raising yield is compressor work. A single number cannot tell us which to fix. ## Coverage `emit_request_outcome` is a single chokepoint — `handler.metrics.record_request` is called from exactly one place, inside the funnel — so all 30 `RequestOutcome` construction sites are covered: Anthropic, OpenAI, Gemini, Bedrock, batch, streaming, and the long-lived Codex Responses-WS path. The `headroom_compress` MCP path bypassed that funnel and is now wired in separately. It has a different shape (no provider, no upstream latency, and everything handed to the tool is eligible by construction), so `sources` counts turns by origin — MCP turns always read `eligible_pct: 100` and must not drag the proxy's real eligibility ceiling upward. **Subagents.** All subagent traffic through the proxy merges into one session, which is correct for savings and retention but means `turns` conflates fan-out with depth. Fan-out is still derivable — `compression.latency_ms_total / session.duration_s` gives the concurrency ratio (~1x serial, ~4x for four parallel agents), so no extra field is needed. Verified no lost updates under 6-way concurrency (1,200 turns). **Known gap:** `--workers N` gives each process its own aggregator, so one user session becomes up to N. Token totals and fleet rates stay correct; session counts inflate. This matches the existing documented limitation that TOIN state, CostTracker, and the prefix tracker are all per-process. ## Notes for reviewers - **Cumulative snapshots, not deltas.** Every report restates running totals under one session ID, so the highest `seq` per `(install, session)` is the complete session. Dedupe is a window function, and a lost report costs nothing. - **Never breaks the proxy.** Every path swallows its own exceptions; uploads go out on a daemon thread so nothing blocks the request loop. - **Explicit User-Agent is load-bearing.** urllib's default is blocked by Cloudflare (error 1010). Combined with fire-and-forget error handling, that would have failed every upload while looking perfectly healthy. - **The exit flush was broken and is fixed.** `atexit` handed the POST to a daemon thread, and daemon threads are killed before they finish during interpreter shutdown — so nothing was sent. That silently dropped *every session shorter than the 5-minute heartbeat*, plus all short-lived subagent MCP processes. The exit path now posts synchronously with a 2s timeout. - Receiver and query tooling are in `deploy/beacon/`. ## Testing - `python -m headroom.telemetry.session` self-check: dedupe, cumulative totals, dropped-report recovery, payload contains no model id or prompt-derived string, allowlist coverage - 175 telemetry/outcome tests pass; 6 new ones cover the opt-out notice - Verified end to end against a live deployment: client → receiver → storage → query ## Still to do before release The default endpoint currently points at a temporary `workers.dev` URL. It needs to move to a Headroom-owned hostname before this ships in a tagged release — noted inline at `DEFAULT_ENDPOINT`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 05:43:25 -07:00