2 commits

Author	SHA1	Message	Date
Tejas Chopra	e9a24f3ec1	fix(beacon): report all-layers savings, not context-compression only (#2796) ... ## Description `rates.saved_pct` and `rates.yield_pct` in the beacon payload divide `tokens_saved` by `original` / `attempted`. Tool-schema deferral never lands in either denominator — `outcome.py` says so explicitly, and `tokens.tool_saved` exists precisely because of it — so every beacon rate silently reports context compression only. On a tool-heavy fleet that is not a rounding difference. Across the first 516 sessions in the corpus the beacon reads **2.80%** where the dashboard headline for the same traffic reads **12.82%**: 157.6M context tokens vs 803.9M all-layers, with 646.3M of tool-schema deferral missing from the ratio. `headroom/proxy/server.py` already resolved this for the dashboard in #2737 — `savings_percent` is `all_layers_saved / (input + all_layers_saved)` and `active_savings_percent` puts tool savings on both sides of the ratio. The beacon was never brought along. This does that. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/telemetry/session.py`: add `rates.all_layers_saved_pct` and `rates.all_layers_yield_pct`, computed the way `server.py` builds `savings_percent` / `active_savings_percent` — tool savings added to **both** sides, since deferred schemas were attempted work that succeeded whole. - `headroom/telemetry/session.py`: extend the `demo()` self-test to assert both new rates against the existing tool-heavy fixture. - `deploy/beacon/query.sh`: add an `all_layers_pct` column to the fleet summary, so the reader stops showing the understated number too. **Kept alongside `saved_pct` rather than folded into it.** Every row already in the corpus means context-only under that name; redefining it would make old and new rows non-comparable with no field to tell them apart. **`SCHEMA_VERSION` deliberately stays at 1.** The change is purely additive, nothing reads the field, and the query uses `union_by_name = true`, so old and new rows mix cleanly. Happy to bump it if maintainers want the marker. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m headroom.telemetry.session ok $ python -m pytest tests/test_savings_tool_search_aggregation.py tests/test_outcome_dual_ruler_funnel.py -q FAILED tests/test_outcome_dual_ruler_funnel.py::test_ledger_delta_stays_on_the_local_ruler FAILED tests/test_outcome_dual_ruler_funnel.py::test_ledger_falls_back_to_billed_when_local_omitted 2 failed, 5 passed, 3 warnings in 4.42s $ ruff check headroom/ All checks passed! $ mypy --python-version 3.12 headroom/telemetry/session.py Success: no issues found in 1 source file ``` The two `test_outcome_dual_ruler_funnel.py` failures are **pre-existing on `main`, not caused by this PR** — verified by `git stash`-ing the change and re-running: ```text $ git stash && python -m pytest tests/test_outcome_dual_ruler_funnel.py -q FAILED tests/test_outcome_dual_ruler_funnel.py::test_ledger_delta_stays_on_the_local_ruler FAILED tests/test_outcome_dual_ruler_funnel.py::test_ledger_falls_back_to_billed_when_local_omitted 2 failed, 3 passed, 3 warnings in 3.71s ``` ## Real Behavior Proof - **Environment:** macOS 25.4.0 arm64, Python 3.12.6, repo `.venv`, branch rebased on `upstream/main` @ `d0a86d40`. - **Exact command / steps:** drive a real `SessionAggregator` with a tool-heavy outcome (`original=1000`, `attempted=400`, `tokens_saved=300`, plus `tool_search_deferred_tokens=800` and `turn_hook_tools_saved_tokens=200`) and print the emitted payload's `rates` block: ```text tokens: {"original": 1000, "attempted": 400, "saved": 300, "tool_saved": 1000} rates: { "saved_pct": 30.0, "eligible_pct": 40.0, "yield_pct": 75.0, "all_layers_saved_pct": 65.0, "all_layers_yield_pct": 92.86, "cache_read_pct": 50.0, "overhead_pct": 5.0 } ``` - **Observed result:** the pre-existing rates are byte-identical (30.0 / 40.0 / 75.0 / 50.0 / 5.0 — no regression), and the two new fields report the all-layers view: 1300 saved of 2000 sent = **65.0%**, 1300 of 1400 attempted = **92.86%**. Both denominators grow with the numerator, matching `server.py`. Cross-checked against the live corpus with DuckDB over the R2 bucket — restating all 516 sessions both ways reproduces the gap this PR closes: ```text ┌───────────┬────────────┬──────────────────┬──────────────────┬─────────────────────┐ │ ctx_saved │ tool_saved │ all_layers_saved │ beacon_saved_pct │ dashboard_saved_pct │ ├───────────┼────────────┼──────────────────┼──────────────────┼─────────────────────┤ │ 157561051 │ 646293457 │ 803854508 │ 2.8 │ 12.82 │ └───────────┴────────────┴──────────────────┴──────────────────┴─────────────────────┘ ``` - **Not tested:** no live proxy run was made against a real provider — the payload above comes from `SessionAggregator` driven directly, which is the same code path the proxy feeds. The 516 sessions already in R2 are **not backfilled**: they carry `tool_saved`, so the corrected rate is computable from them today, but their own `rates` block stays context-only. Only sessions emitted from the next release carry the new fields. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes - **Documentation:** N/A — no doc references the beacon `rates` field names (`grep -rn "saved_pct\|yield_pct" docs/` returns nothing). The field semantics are documented inline in `session.py`, which this PR extends. - **Worth a maintainer opinion:** the dashboard formula adds `tool_saved` to the *denominator* as well. That is defensible — deferred schemas were attempted work that succeeded 100% — but it does mean a tool-heavy session's ratio is partly measuring a layer that is near-always at full yield. This PR matches the dashboard rather than inventing a third convention; if the convention should change, it should change in both places at once. - **Follow-up:** nothing here changes what the fleet actually saved, only what the beacon admits to. The 4.6x gap was reporting, not performance.	2026-08-05 08:33:04 -07:00
Tejas Chopra	9cfb00838a	fix(telemetry): anonymous compression stats — no prompts, no data (#2728) ... ## In one line Headroom starts reporting **how well compression is working** — counters and percentages only. **No prompts. No code. No file paths. Nothing about what you're building.** ## Why Right now nobody knows whether compression actually helps real users. You can see your own numbers in `/stats`, but that's it — there's no way to tell whether a given workload compresses well, or why it sometimes doesn't. This closes that loop so we can make compression better for everyone. ## Exactly what gets sent One message per session, and every 5 minutes while you're active: ```json { "session": { "id": "random", "turns": 47, "duration_s": 4210, "seq": 3 }, "tokens": { "original": 890000, "attempted": 410000, "saved": 320000, "tool_saved": 48000, "cache_read": 210000 }, "rates": { "saved_pct": 35.96, "eligible_pct": 46.07, "yield_pct": 78.05, "cache_read_pct": 23.60, "overhead_pct": 1.96 }, "compression": { "transforms": {"crush": 47}, "passthrough_turns": 0 }, "skips": {}, "sources": { "proxy": 47 }, "providers": ["anthropic"], "models": ["claude-sonnet-4-5-20250929"], "failures": 2 } ``` Plus a random install ID, the Headroom version, and OS/architecture (`darwin`, `arm64`). That's the whole thing. A full example lives at `deploy/beacon/sample-event.json`. ## What is never sent - Your prompts or the model's responses - Your code - File paths, project names, repo names - Tool names or MCP server names - Hostname, username, or IP address - Custom or fine-tuned model names (an id like `ft:gpt-4o:acme-corp:…` contains a company name, so only models in a public registry are reported) **This is structural, not a pinky-swear.** Every value in the payload is a number, a fixed word, or a random ID — there is no free-text field anywhere for content to hide in. The receiver (`deploy/beacon/worker.js`, in this repo so you can read it) drops anything not on an explicit allowlist before storing. ## Turning it off Any one of these: ```bash HEADROOM_BEACON=off # or DO_NOT_TRACK=1 # or # offline mode ``` It's on by default, and Headroom says so at startup: ``` Telemetry: anonymous compression stats — never prompts, code, or file paths. Helps us improve compression | Off: HEADROOM_BEACON=off ``` `HEADROOM_TELEMETRY` is a **separate** switch that still only affects local stats. If you had turned that on, this change does not start uploading anything — you answered a different question, and upgrading should not change the answer. ## Why the percentages, not just "tokens saved" "We saved 36%" hides the interesting part. In the example above only **46% of tokens were eligible** for compression at all — the rest is frozen cache prefix and system prompts we deliberately do not touch. Of what we *could* touch, we removed **78%**. Those are two separate problems. Raising eligibility is proxy work; raising yield is compressor work. A single number cannot tell us which to fix. ## Coverage `emit_request_outcome` is a single chokepoint — `handler.metrics.record_request` is called from exactly one place, inside the funnel — so all 30 `RequestOutcome` construction sites are covered: Anthropic, OpenAI, Gemini, Bedrock, batch, streaming, and the long-lived Codex Responses-WS path. The `headroom_compress` MCP path bypassed that funnel and is now wired in separately. It has a different shape (no provider, no upstream latency, and everything handed to the tool is eligible by construction), so `sources` counts turns by origin — MCP turns always read `eligible_pct: 100` and must not drag the proxy's real eligibility ceiling upward. **Subagents.** All subagent traffic through the proxy merges into one session, which is correct for savings and retention but means `turns` conflates fan-out with depth. Fan-out is still derivable — `compression.latency_ms_total / session.duration_s` gives the concurrency ratio (~1x serial, ~4x for four parallel agents), so no extra field is needed. Verified no lost updates under 6-way concurrency (1,200 turns). **Known gap:** `--workers N` gives each process its own aggregator, so one user session becomes up to N. Token totals and fleet rates stay correct; session counts inflate. This matches the existing documented limitation that TOIN state, CostTracker, and the prefix tracker are all per-process. ## Notes for reviewers - **Cumulative snapshots, not deltas.** Every report restates running totals under one session ID, so the highest `seq` per `(install, session)` is the complete session. Dedupe is a window function, and a lost report costs nothing. - **Never breaks the proxy.** Every path swallows its own exceptions; uploads go out on a daemon thread so nothing blocks the request loop. - **Explicit User-Agent is load-bearing.** urllib's default is blocked by Cloudflare (error 1010). Combined with fire-and-forget error handling, that would have failed every upload while looking perfectly healthy. - **The exit flush was broken and is fixed.** `atexit` handed the POST to a daemon thread, and daemon threads are killed before they finish during interpreter shutdown — so nothing was sent. That silently dropped *every session shorter than the 5-minute heartbeat*, plus all short-lived subagent MCP processes. The exit path now posts synchronously with a 2s timeout. - Receiver and query tooling are in `deploy/beacon/`. ## Testing - `python -m headroom.telemetry.session` self-check: dedupe, cumulative totals, dropped-report recovery, payload contains no model id or prompt-derived string, allowlist coverage - 175 telemetry/outcome tests pass; 6 new ones cover the opt-out notice - Verified end to end against a live deployment: client → receiver → storage → query ## Still to do before release The default endpoint currently points at a temporary `workers.dev` URL. It needs to move to a Headroom-owned hostname before this ships in a tagged release — noted inline at `DEFAULT_ENDPOINT`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 05:43:25 -07:00