From 6b63b623e026efdc4f384f5fe574b4168d0fb7bb Mon Sep 17 00:00:00 2001 From: Tejas Chopra Date: Tue, 4 Aug 2026 18:46:39 -0700 Subject: [PATCH] docs(metrics): document OTLP metric export and Dynatrace ingest (#2785) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Description The proxy can already push its counters to any OTLP/HTTP endpoint via `HEADROOM_OTEL_METRICS_*`, but the docs site only surfaced this as a single row in the proxy env table (`proxy.mdx:287`). The endpoint, header, service-name, and resource-attribute variables were documented only in `wiki/metrics.md` — so an operator reading the Vercel docs had no way to wire Headroom into their existing observability stack. This adds that section, plus a Dynatrace subsection, because Dynatrace has a silent failure mode that costs an afternoon to diagnose. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `docs/content/docs/metrics.mdx` — new `### OpenTelemetry (OTLP) Export` section after the Prometheus section: the `headroom-ai[proxy,otel]` install, all seven `HEADROOM_OTEL_*` variables in a table, the exported counter names (`headroom.proxy.tokens.saved` et al.), the `curl /stats | jq .otel` verification, and the note that an app-managed global meter provider is recorded into automatically. - `docs/content/docs/metrics.mdx` — new `### Dynatrace` subsection: copy-paste env block, `metrics.ingest` token scope, a `warn` Callout on the delta-temporality requirement, the ActiveGate URL variant, the Collector + `cumulativetodelta` alternative, and one paragraph explaining that trace export needs `opentelemetry-instrument` (Headroom's self-configured tracing targets Langfuse only). - `docs/content/docs/proxy.mdx` — the `HEADROOM_OTEL_METRICS_ENABLED` row now links to `/docs/metrics#opentelemetry-otlp-export`. No code, config, or nav changes — the Observability nav slot already points at `metrics.mdx`. ## Testing - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [x] Manual testing performed Docs-only change: no Python touched, so pytest/ruff/mypy have nothing to cover here. `next build` was **not** run — `docs/node_modules` is absent in this checkout, which would require a full `npm install`; Vercel's preview build is the real gate. In its place I verified the MDX cannot break the build by parsing for the two things that actually fail MDX v3 — unbalanced JSX and bare `<`/`{` in prose. ### Test Output ```text $ python - <<'PY' # strip fenced + inline code, then scan prose for MDX hazards ... PY hazards: [(80, ''), (125, '')] Callout balance: 1 open / 1 close ``` Both flagged lines are pre-existing `` JSX expressions, untouched by this PR. The added prose introduces no bare `<` or `{` (every `` / `` placeholder sits inside a code fence or inline backticks). `type="warn"` is already used on three other pages, and the anchor `#opentelemetry-otlp-export` matches the GitHub-slugger form of the new heading. ## Real Behavior Proof - **Environment:** macOS (darwin 25.4.0), repo `.venv`, `opentelemetry-sdk` 1.44.0, `opentelemetry-exporter-otlp-proto-http`, headroom @ 6422a80a. - **Exact command / steps:** verified the central claim of the new Callout — that the OTLP HTTP exporter defaults to cumulative (which Dynatrace rejects) and that the standard env var flips it to delta with no Headroom code change: ```text $ python -c '...' DELTA= 1 CUMULATIVE= 2 default counter: 2 $ OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=DELTA python -c '...' counter temporality with env=DELTA: 1 ``` - **Observed result:** `OTLPMetricExporter._preferred_temporality[Counter]` is `CUMULATIVE` (2) by default and `DELTA` (1) with the env var set. Since every Headroom OTEL instrument is a `Counter` (`headroom/observability/metrics.py:136-189`), without the env var Dynatrace drops all of them — matching its documented `UNSUPPORTED_METRIC_TYPE_MONOTONIC_CUMULATIVE_SUM` rejection. Also confirmed against the code that `HEADROOM_OTEL_METRICS_ENDPOINT` is passed verbatim to the exporter (`metrics.py:539-543`), hence the doc's warning that `/v1/metrics` must be included by hand, and that `HEADROOM_OTEL_METRICS_HEADERS` splits on the first `=` so `Authorization=Api-Token dt0c01...` parses correctly. - **Not tested:** no live export against a real Dynatrace tenant — the URL shapes and token scopes come from Dynatrace's docs, not from an observed 200. The `opentelemetry-instrument` trace path is described from the code's global-provider fallback (`tracing.py:95-99`), not run end-to-end. `next build` not run (see Testing). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - N/A on the pytest / ruff / mypy / new-tests items: this PR changes two `.mdx` files and no Python. - Follow-up worth considering: `wiki/metrics.md:244` carries the same OTEL variable table and still lacks the Dynatrace guidance — happy to mirror it there, kept out of this PR to hold the diff to the Vercel docs as asked. - Second follow-up: the delta-temporality fix currently depends on an upstream OTEL SDK env var that Headroom neither sets nor documents in code. A `HEADROOM_OTEL_METRICS_TEMPORALITY=delta` passthrough would make the Dynatrace case self-contained instead of relying on a variable one layer down. --- docs/content/docs/metrics.mdx | 53 +++++++++++++++++++++++++++++++++++ docs/content/docs/proxy.mdx | 2 +- 2 files changed, 54 insertions(+), 1 deletion(-) diff --git a/docs/content/docs/metrics.mdx b/docs/content/docs/metrics.mdx index 6c8b1412f..ea8c2577f 100644 --- a/docs/content/docs/metrics.mdx +++ b/docs/content/docs/metrics.mdx @@ -88,6 +88,59 @@ headroom_latency_seconds_bucket{le="0.1"} 1150 headroom_cache_hits_total 456 ``` +### OpenTelemetry (OTLP) Export + +The proxy can also push its counters to any OTLP/HTTP endpoint. Install the extra and set four variables: + +```bash +pip install "headroom-ai[proxy,otel]" +``` + +```bash +HEADROOM_OTEL_METRICS_ENABLED=1 +HEADROOM_OTEL_METRICS_ENDPOINT=http://127.0.0.1:4318/v1/metrics +HEADROOM_OTEL_SERVICE_NAME=headroom-proxy +HEADROOM_OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod +``` + +| Variable | Default | Purpose | +| --- | --- | --- | +| `HEADROOM_OTEL_METRICS_ENABLED` | `0` | Enable Headroom-managed OTLP metric export | +| `HEADROOM_OTEL_METRICS_EXPORTER` | `otlp_http` | `otlp_http` or `console` (local debugging) | +| `HEADROOM_OTEL_METRICS_ENDPOINT` | unset | Full OTLP metrics URL — Headroom does **not** append `/v1/metrics` for you | +| `HEADROOM_OTEL_METRICS_HEADERS` | unset | Comma-separated `key=value` auth headers | +| `HEADROOM_OTEL_METRICS_EXPORT_INTERVAL_MS` | `10000` | Export interval | +| `HEADROOM_OTEL_SERVICE_NAME` | `headroom-proxy` | OTEL `service.name` | +| `HEADROOM_OTEL_RESOURCE_ATTRIBUTES` | unset | Comma-separated resource attributes | + +Exported counters include `headroom.proxy.requests`, `headroom.proxy.tokens.input`, `headroom.proxy.tokens.output`, `headroom.proxy.tokens.saved`, and `headroom.proxy.cache.read_tokens` / `write_tokens`. + +Confirm the exporter is live with `curl -s http://localhost:8787/stats | jq .otel`. + +If your application already configures a global OTEL meter provider, leave `HEADROOM_OTEL_*` unset — Headroom records into the ambient provider automatically. + +### Dynatrace + +Point the exporter at your environment's OTLP API and add the API token as a header. The token needs the `metrics.ingest` scope. + +```bash +HEADROOM_OTEL_METRICS_ENABLED=1 +HEADROOM_OTEL_METRICS_ENDPOINT="https://.live.dynatrace.com/api/v2/otlp/v1/metrics" +HEADROOM_OTEL_METRICS_HEADERS="Authorization=Api-Token dt0c01.XXXX" +OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=DELTA +HEADROOM_OTEL_SERVICE_NAME=headroom-proxy +``` + + +`OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=DELTA` is required. Dynatrace [only ingests delta counters](https://docs.dynatrace.com/docs/ingest-from/opentelemetry/getting-started/metrics/limitations) and rejects cumulative ones with `UNSUPPORTED_METRIC_TYPE_MONOTONIC_CUMULATIVE_SUM`, while the OTEL SDK default is cumulative. Without this line, every Headroom metric is dropped at ingest and the proxy logs no error. + + +Restart the proxy, then search the Dynatrace metric explorer for `headroom.proxy.tokens.saved` — data appears within ~30s. + +For an ActiveGate deployment, swap the base URL for `https://:9999/e//api/v2/otlp/v1/metrics`. If you already run an OpenTelemetry Collector, send Headroom to it instead and add the `cumulativetodelta` processor — then the temporality variable is unnecessary and the collector holds the token. + +Trace export is separate: Headroom's self-configured tracing targets Langfuse only. To land its spans in Dynatrace, leave `HEADROOM_LANGFUSE_*` unset and run the proxy under `opentelemetry-instrument` with the standard `OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_EXPORTER_OTLP_HEADERS` / `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` variables; Headroom records into the ambient tracer provider. + ### Health Check ```bash diff --git a/docs/content/docs/proxy.mdx b/docs/content/docs/proxy.mdx index b443be7bd..984285146 100644 --- a/docs/content/docs/proxy.mdx +++ b/docs/content/docs/proxy.mdx @@ -284,7 +284,7 @@ Rewrite the upstream model per request — for example, send small, tool-free ca | `--telemetry` / `HEADROOM_TELEMETRY` | off | **Local-only** usage stats for your own `/stats`, `/metrics`, and dashboard. Nothing leaves the machine. | | `--log-file` / `HEADROOM_LOG_FILE` | none | JSONL request/response log. | | `--log-messages` | `false` | Include full message bodies in the log (may contain sensitive data). | -| `HEADROOM_OTEL_METRICS_ENABLED` | `false` | Export OpenTelemetry metrics (`HEADROOM_OTEL_METRICS_ENDPOINT`, …). | +| `HEADROOM_OTEL_METRICS_ENABLED` | `false` | Export OpenTelemetry metrics (`HEADROOM_OTEL_METRICS_ENDPOINT`, …). See [OTLP export](/docs/metrics#opentelemetry-otlp-export). | | `HEADROOM_LANGFUSE_ENABLED` | `false` | Emit Langfuse traces (`LANGFUSE_PUBLIC_KEY` / `LANGFUSE_SECRET_KEY`). | See [Metrics](/docs/metrics) for the Prometheus and Grafana setup.