From 8275e4eeded86dff9eedcacee467c48153f8731f Mon Sep 17 00:00:00 2001 From: Tejas Chopra Date: Tue, 4 Aug 2026 18:10:05 -0700 Subject: [PATCH] docs(metrics): document OTLP metric export and Dynatrace ingest MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The proxy has been able to push its counters to any OTLP/HTTP endpoint since HEADROOM_OTEL_METRICS_* landed, but the docs site only mentioned it as a single row in the proxy env table — the endpoint, header, and resource-attribute variables were undocumented outside the wiki. Add an "OpenTelemetry (OTLP) Export" section to the observability page covering the full variable set, the exported counter names, and the ambient-meter-provider fallback, plus a Dynatrace subsection. The Dynatrace path has one non-obvious failure mode worth calling out: Dynatrace rejects cumulative counters (UNSUPPORTED_METRIC_TYPE_MONOTONIC_ CUMULATIVE_SUM) while the OTEL SDK defaults to cumulative, so every Headroom metric is dropped at ingest with no error logged locally. The fix is OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=DELTA, which the OTLP HTTP exporter honours without any Headroom code change. --- docs/content/docs/metrics.mdx | 53 +++++++++++++++++++++++++++++++++++ docs/content/docs/proxy.mdx | 2 +- 2 files changed, 54 insertions(+), 1 deletion(-) diff --git a/docs/content/docs/metrics.mdx b/docs/content/docs/metrics.mdx index 6c8b1412f..ea8c2577f 100644 --- a/docs/content/docs/metrics.mdx +++ b/docs/content/docs/metrics.mdx @@ -88,6 +88,59 @@ headroom_latency_seconds_bucket{le="0.1"} 1150 headroom_cache_hits_total 456 ``` +### OpenTelemetry (OTLP) Export + +The proxy can also push its counters to any OTLP/HTTP endpoint. Install the extra and set four variables: + +```bash +pip install "headroom-ai[proxy,otel]" +``` + +```bash +HEADROOM_OTEL_METRICS_ENABLED=1 +HEADROOM_OTEL_METRICS_ENDPOINT=http://127.0.0.1:4318/v1/metrics +HEADROOM_OTEL_SERVICE_NAME=headroom-proxy +HEADROOM_OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod +``` + +| Variable | Default | Purpose | +| --- | --- | --- | +| `HEADROOM_OTEL_METRICS_ENABLED` | `0` | Enable Headroom-managed OTLP metric export | +| `HEADROOM_OTEL_METRICS_EXPORTER` | `otlp_http` | `otlp_http` or `console` (local debugging) | +| `HEADROOM_OTEL_METRICS_ENDPOINT` | unset | Full OTLP metrics URL — Headroom does **not** append `/v1/metrics` for you | +| `HEADROOM_OTEL_METRICS_HEADERS` | unset | Comma-separated `key=value` auth headers | +| `HEADROOM_OTEL_METRICS_EXPORT_INTERVAL_MS` | `10000` | Export interval | +| `HEADROOM_OTEL_SERVICE_NAME` | `headroom-proxy` | OTEL `service.name` | +| `HEADROOM_OTEL_RESOURCE_ATTRIBUTES` | unset | Comma-separated resource attributes | + +Exported counters include `headroom.proxy.requests`, `headroom.proxy.tokens.input`, `headroom.proxy.tokens.output`, `headroom.proxy.tokens.saved`, and `headroom.proxy.cache.read_tokens` / `write_tokens`. + +Confirm the exporter is live with `curl -s http://localhost:8787/stats | jq .otel`. + +If your application already configures a global OTEL meter provider, leave `HEADROOM_OTEL_*` unset — Headroom records into the ambient provider automatically. + +### Dynatrace + +Point the exporter at your environment's OTLP API and add the API token as a header. The token needs the `metrics.ingest` scope. + +```bash +HEADROOM_OTEL_METRICS_ENABLED=1 +HEADROOM_OTEL_METRICS_ENDPOINT="https://.live.dynatrace.com/api/v2/otlp/v1/metrics" +HEADROOM_OTEL_METRICS_HEADERS="Authorization=Api-Token dt0c01.XXXX" +OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=DELTA +HEADROOM_OTEL_SERVICE_NAME=headroom-proxy +``` + + +`OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=DELTA` is required. Dynatrace [only ingests delta counters](https://docs.dynatrace.com/docs/ingest-from/opentelemetry/getting-started/metrics/limitations) and rejects cumulative ones with `UNSUPPORTED_METRIC_TYPE_MONOTONIC_CUMULATIVE_SUM`, while the OTEL SDK default is cumulative. Without this line, every Headroom metric is dropped at ingest and the proxy logs no error. + + +Restart the proxy, then search the Dynatrace metric explorer for `headroom.proxy.tokens.saved` — data appears within ~30s. + +For an ActiveGate deployment, swap the base URL for `https://:9999/e//api/v2/otlp/v1/metrics`. If you already run an OpenTelemetry Collector, send Headroom to it instead and add the `cumulativetodelta` processor — then the temporality variable is unnecessary and the collector holds the token. + +Trace export is separate: Headroom's self-configured tracing targets Langfuse only. To land its spans in Dynatrace, leave `HEADROOM_LANGFUSE_*` unset and run the proxy under `opentelemetry-instrument` with the standard `OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_EXPORTER_OTLP_HEADERS` / `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` variables; Headroom records into the ambient tracer provider. + ### Health Check ```bash diff --git a/docs/content/docs/proxy.mdx b/docs/content/docs/proxy.mdx index aa9c1dc61..e03aaffa2 100644 --- a/docs/content/docs/proxy.mdx +++ b/docs/content/docs/proxy.mdx @@ -284,7 +284,7 @@ Rewrite the upstream model per request — for example, send small, tool-free ca | `--telemetry` / `HEADROOM_TELEMETRY` | off | **Local-only** usage stats for your own `/stats`, `/metrics`, and dashboard. Nothing leaves the machine. | | `--log-file` / `HEADROOM_LOG_FILE` | none | JSONL request/response log. | | `--log-messages` | `false` | Include full message bodies in the log (may contain sensitive data). | -| `HEADROOM_OTEL_METRICS_ENABLED` | `false` | Export OpenTelemetry metrics (`HEADROOM_OTEL_METRICS_ENDPOINT`, …). | +| `HEADROOM_OTEL_METRICS_ENABLED` | `false` | Export OpenTelemetry metrics (`HEADROOM_OTEL_METRICS_ENDPOINT`, …). See [OTLP export](/docs/metrics#opentelemetry-otlp-export). | | `HEADROOM_LANGFUSE_ENABLED` | `false` | Emit Langfuse traces (`LANGFUSE_PUBLIC_KEY` / `LANGFUSE_SECRET_KEY`). | See [Metrics](/docs/metrics) for the Prometheus and Grafana setup.