mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
7 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Tejas Chopra
|
02eb90f243
|
feat(metrics): record per-extension token savings (#2371)
## What Adds `PrometheusMetrics.record_extension_savings(key, saved)` so proxy extensions can report the tokens they save, and surfaces the per-extension totals in the `/stats` payload. ## Why Proxy extensions that perform their own token reduction currently have no supported way to report their savings to the metrics object — there is no method for it, so that telemetry is silently dropped. This adds the recording method and exposes the aggregate alongside the existing per-strategy compression breakdown. ## How - New `extension_savings: dict[str, int]` counter on `PrometheusMetrics`, populated lazily per extension-supplied `key` (no hardcoded list of extensions). - `record_extension_savings(key, saved)` accumulates positive savings per key, mirroring how `record_compression` aggregates `tokens_saved_by_strategy` (lock-free `defaultdict(int)`, atomic under the GIL for these key types); non-positive values are ignored. - Cleared in `reset_runtime()` with the other in-memory counters. - Surfaced in `/stats` as `extension_savings`, next to `compressions_by_strategy` / `tokens_saved_by_strategy`. No new Prometheus series. ## Behavior change None to existing metrics. ## Testing - Two focused tests in `tests/test_compression_observability.py` (per-key accumulation incl. zero/negative ignored; surfaced in `/stats` via `create_app`) → 2 passed (13 in file). - `ruff check` / `ruff format` → clean; `mypy` → clean on changed source. |
||
|
Tejas Chopra
|
53be64ca12
|
chore(telemetry): remove Supabase anonymous beacon; fix contact domain to headroomlabs.ai (#1526)
## Description
Removes the anonymous-telemetry **beacon** — the only external,
third-party data flow Headroom ever initiated. When telemetry was opted
in, it POSTed aggregate `/stats` to a hardcoded **Supabase** REST
endpoint (with an embedded anon API key in the source). For
enterprise/on-prem deployments this is exactly the kind of
vendor-controlled data egress a security review flags, so it's gone
entirely — **zero "Supabase" references remain in the codebase.**
What stays (by design): the **local** telemetry collector + the
`HEADROOM_TELEMETRY` opt-in (it only feeds `/stats` and `/v1/telemetry`
— nothing leaves the process), **OpenTelemetry export**
(`HEADROOM_OTEL_METRICS_*`, so operators send operational metrics to
*their own* collector), and the license usage reporter (your own domain,
license-key-gated).
Also fixes the contact domain: `headroom.dev` → `headroomlabs.ai`
everywhere.
Closes # (no tracking issue)
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)
> Non-breaking: `HEADROOM_TELEMETRY` is still accepted (now gates local
collection only). The only behavior change is that no telemetry is ever
sent externally.
## Changes Made
- **Deleted the Supabase beacon**: `TelemetryBeacon` class,
`_SUPABASE_URL`/`_SUPABASE_KEY`/`_TABLE`/`_ENDPOINT`, the JSONB
projection helper, the proxy-lifespan beacon wiring, the `SUPABASE_`
install env passthrough, and `tests/test_strategy_stats_supabase.py`.
- **Kept** the local opt-in predicate (`is_telemetry_enabled` etc.) in
`beacon.py` — still used by the local collector + CLI — reworded to
"local only".
- **Retained** the single-worker-owner file lock (the cc-switch
reconciler depends on it); updated its comments to drop the beacon
framing.
- `/stats` `anon_telemetry_shipping` is now always `False` (nothing
ships externally); startup log reworded to "Local telemetry".
- Reworded remaining "Supabase" comments in `collector.py`,
`context.py`, `prometheus_metrics.py`, and two test docstrings.
- Contact domain: `security@headroom.dev` → `security@headroomlabs.ai`,
`conduct@headroom.dev` → `conduct@headroomlabs.ai`, FUNDING.yml sponsor
URL.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ grep -rniI "supabase" --include=*.py --include=*.md --include=*.mdx . # (excl .venv/sbom)
>>> ZERO Supabase references
$ grep -rniI "headroom.dev" .
>>> ZERO headroom.dev references
$ ruff check <changed files> -> All checks passed!
$ ruff format --check <changed files> -> 10 files already formatted
$ mypy <changed telemetry files> -> Success: no issues found
$ pytest tests/test_telemetry.py tests/test_telemetry_warning.py \
tests/test_proxy_telemetry_env.py tests/test_compression_observability.py \
tests/test_paths.py tests/test_paths_backward_compat.py -q
============================= 173 passed in 6.67s ==============================
```
## Real Behavior Proof
- **Environment:** macOS, Python 3.12 (`.venv`).
- **Exact command / steps:** repo-wide grep for
`supabase`/`headroom.dev`; `create_app(...)` driven through a full
`TestClient` lifespan (startup + shutdown) in
`test_proxy_telemetry_env.py`; `/stats` exercised in
`test_telemetry_warning.py`.
- **Observed result:** zero `supabase`/`headroom.dev` strings remain;
the proxy starts and shuts down cleanly with the beacon removed (the
worker-owner lock + reconciler still elect a single owner);
`/stats.anon_telemetry_shipping` is `False` even with
`HEADROOM_TELEMETRY=on`; local collector + OTEL paths unchanged.
- **Not tested:** no live network call was ever made (the point — the
external POST is gone). OTEL export and the license reporter were not
exercised (unchanged by this PR).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
- The license usage reporter (`reporter.py` → `app.headroomlabs.ai`) is
intentionally **kept** — it's license-key-gated (dormant for
unlicensed/OSS deployments) and goes to your own domain, not a third
party.
- Docs/CHANGELOG left unchecked: a couple of docs mention the telemetry
beacon and may want a follow-up note that it now collects locally only;
happy to add.
|
||
|
Tejas Chopra
|
e9cae0131b |
fix: expose compression latency bottlenecks
Add Codex WS unit-level timing and bounded parallel compression, clarify context-tool session savings, and avoid costly diff/log fallbacks to Kompress. |
||
|
chopratejas
|
967b0db439 |
fix: B1 — retire ICM, RollingWindow, scoring, relevance + dependents
Phase B step 1 of the live-zone-only realignment. Removes ~10K LOC of
"drop messages from history" machinery that became unreachable after
PR-A1 made `/v1/messages` a passthrough on the proxy. Live-zone-only
compression (PR-B2..B7) operates on content blocks within messages;
message-list mutation no longer happens in the pipeline.
Python deletes:
- headroom/transforms/intelligent_context.py (1077 LOC)
- headroom/transforms/rolling_window.py (395 LOC)
- headroom/transforms/progressive_summarizer.py (508 LOC)
- headroom/transforms/scoring.py (459 LOC)
- headroom/transforms/tool_crusher.py (338 LOC)
- 5 corresponding tests/test_transforms/* and tests/test_proxy_intelligent_context.py
Rust deletes:
- crates/headroom-core/src/context/* (manager, config, workspace,
candidate, ccr_drop, strategy/, mod) + safety.rs replaced
- crates/headroom-core/src/scoring/* (mod, score, scorer, traits, weights)
- MessageScorerComparator from crates/headroom-parity (PR #338/#343
becomes deletable; sunk cost stays sunk)
- 13 message_scorer fixtures + record_message_scorer.py
Rust adds (move + rewrite):
- crates/headroom-core/src/transforms/safety.rs — `tool_pair_indices`
preserves the OpenAI/Anthropic tool_use ↔ tool_result pairing rule
the live-zone dispatcher (PR-B2) needs. No IcmConfig dependency.
Surface refactors:
- HeadroomConfig: drop `tool_crusher`, `rolling_window`,
`intelligent_context` fields; hoist `output_buffer_tokens` to top
level (used by client.py).
- ProxyConfig: drop `intelligent_context*` fields.
- `headroom wrap` proxy server: retire IntelligentContextManager
and RollingWindow imports + branch; pipeline is CacheAligner →
ContentRouter (smart_routing) or CacheAligner → SmartCrusher
(legacy).
- CLI: drop `--no-intelligent-context`, `--no-intelligent-scoring`,
`--no-compress-first` flags.
- LangChain memory integration: rename `_apply_rolling_window` →
`_apply_compression`, drop RollingWindowConfig dep. Threshold is
now advisory — B6 will rework the contract.
- TransformPipeline.create_pipeline now takes only cache_aligner_config.
- headroom/__init__.py + headroom/transforms/__init__.py: strip
exports of deleted symbols.
Bug fixes uncovered by full pytest sweep:
- providers/copilot/wrap.py: `environ or os.environ` collapsed
empty-dict to falsy → callers passing `environ={}` accidentally
pulled from os.environ. Use `environ if environ is not None else
os.environ`.
Test correctness fixes:
- _DummyAnthropicHandler._retry_request gains **_kwargs to match
the real handler signature post-A8.
- test_ws_http_fallback extracts JSON from `content=` (post-A3
byte-faithful) rather than the obsolete `json=` kwarg.
- test_ccr_response_handler_extra fixture joins SSE events with
`\n\n` per spec (post-A8 byte-buffer parser requirement).
- test_proxy_responses_phase_preservation: capture via direct
handler attached to the named logger, so the assertion is
order-independent (proxy `_setup_file_logging` flips
`headroom.propagate=False` once any earlier test triggers it).
- conftest.py autouse fixture resets `headroom.propagate=True`
before each test as a defensive measure for the same pollution.
- test_wrap_copilot_translated_backend_still_requires_byok:
monkeypatch.delenv every provider key so the BYOK error
actually fires.
- test_native_installers: skip when system bash < 4.3 (macOS ships 3.2).
- TestGeminiEmbedContent / TestGeminiBatchEmbedContents:
pytest.mark.skip — proxy currently has no :embedContent route;
feature gap, not regression.
Acceptance:
- cargo build --workspace + cargo clippy + cargo fmt --check: green.
- cargo test --workspace --exclude headroom-py: 777 passed.
- pytest: 4892 passed, 240 skipped, 0 failed.
- git grep returns only intentional comments referencing the deletion.
Per-PR-B1 plan: REALIGNMENT/04-phase-B-live-zone.md.
|
||
|
chopratejas
|
44944fb3fe |
fix(proxy): restore Anthropic compression on token mode (issue #327)
Three bugs combined to drive end-to-end compression on the Anthropic backend to ~0% in token mode (the default). User report #327 saw a ~9× drop in dashboard savings from one day to the next on Claude Code traffic; the dashboard headline was technically correct but the underlying compression genuinely was not running. After this change the same Claude Code-shape multi-turn conversation goes from 14987 → 14371 tokens at the request boundary on turn 1 and only recompresses the freshest tool_result on subsequent turns, with the prior turns frozen byte-identical to preserve the upstream prefix cache. Bug 1 — IntelligentContextManager inner ContentRouter has no observer PR #302 (commit |
||
|
chopratejas
|
2a6ab38177 |
test(observability): isolate SmartCrusher.apply tests from global TOIN file
The new SmartCrusher.apply() tests added in this PR feed the global TOIN learning store (default path `~/.headroom/toin.json`). On Python 3.11, CI runs the suite twice (regular + coverage); a pattern written in the first pass changes which rows the lossy sampler keeps in the second pass and breaks `tests/test_integrations/langchain/test_evals.py::TestRelevancePreservation::test_first_last_items_always_preserved`. Fix: an `isolated_toin` fixture that points `HEADROOM_TOIN_PATH` at a tempdir for the test and resets the singleton on enter and exit. Same isolation pattern PR #300 already uses for `test_smart_crusher_toin_attachment.py`. |
||
|
chopratejas
|
cf97995834 |
chore(proxy): per-strategy compression observability
Adds a `CompressionObserver` Protocol (`headroom.transforms.observability`) and wires `ContentRouter` and `SmartCrusher` to call it once per real compression event. `PrometheusMetrics` implements the protocol and accumulates per-strategy counters (`compressions_by_strategy`, `tokens_saved_by_strategy`). Why: the TOIN→SmartCrusher silent disconnect was invisible for three weeks because no signal distinguished by strategy. With per-strategy counters in place, the next regression of that shape fails the test suite the day it lands instead of waiting on a manual audit. Counters live as in-process state only — deliberately NOT exported via the Prometheus scrape or OTel surface. The metric→Supabase pipeline treats each metric name as a column, and we cannot add new columns. CI-level observability via `tests/test_compression_observability.py` is sufficient to catch silent regressions; production export waits on a non-column-adding pipeline. |