akadmin/headroom
1
0
Fork 0
mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions

docs(metrics): document OTLP metric export and Dynatrace ingest

Browse source 
The proxy has been able to push its counters to any OTLP/HTTP endpoint
since HEADROOM_OTEL_METRICS_* landed, but the docs site only mentioned it
as a single row in the proxy env table — the endpoint, header, and
resource-attribute variables were undocumented outside the wiki.

Add an "OpenTelemetry (OTLP) Export" section to the observability page
covering the full variable set, the exported counter names, and the
ambient-meter-provider fallback, plus a Dynatrace subsection.

The Dynatrace path has one non-obvious failure mode worth calling out:
Dynatrace rejects cumulative counters (UNSUPPORTED_METRIC_TYPE_MONOTONIC_
CUMULATIVE_SUM) while the OTEL SDK defaults to cumulative, so every
Headroom metric is dropped at ingest with no error logged locally. The
fix is OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=DELTA, which the
OTLP HTTP exporter honours without any Headroom code change.
This commit is contained in:
Tejas Chopra 2026-08-04 18:10:05 -07:00
parent 6422a80a58
commit 8275e4eede
2 changed files with 54 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

53
docs/content/docs/metrics.mdx
View file

@ -88,6 +88,59 @@ headroom_latency_seconds_bucket{le="0.1"} 1150
headroom_cache_hits_total 456
```
### OpenTelemetry (OTLP) Export
The proxy can also push its counters to any OTLP/HTTP endpoint. Install the extra and set four variables:
```bash
pip install "headroom-ai[proxy,otel]"
```
```bash
HEADROOM_OTEL_METRICS_ENABLED=1
HEADROOM_OTEL_METRICS_ENDPOINT=http://127.0.0.1:4318/v1/metrics
HEADROOM_OTEL_SERVICE_NAME=headroom-proxy
HEADROOM_OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod
```
| Variable | Default | Purpose |
| --- | --- | --- |
| `HEADROOM_OTEL_METRICS_ENABLED` | `0` | Enable Headroom-managed OTLP metric export |
| `HEADROOM_OTEL_METRICS_EXPORTER` | `otlp_http` | `otlp_http` or `console` (local debugging) |
| `HEADROOM_OTEL_METRICS_ENDPOINT` | unset | Full OTLP metrics URL — Headroom does **not** append `/v1/metrics` for you |
| `HEADROOM_OTEL_METRICS_HEADERS` | unset | Comma-separated `key=value` auth headers |
| `HEADROOM_OTEL_METRICS_EXPORT_INTERVAL_MS` | `10000` | Export interval |
| `HEADROOM_OTEL_SERVICE_NAME` | `headroom-proxy` | OTEL `service.name` |
| `HEADROOM_OTEL_RESOURCE_ATTRIBUTES` | unset | Comma-separated resource attributes |
Exported counters include `headroom.proxy.requests`, `headroom.proxy.tokens.input`, `headroom.proxy.tokens.output`, `headroom.proxy.tokens.saved`, and `headroom.proxy.cache.read_tokens` / `write_tokens`.
Confirm the exporter is live with `curl -s http://localhost:8787/stats | jq .otel`.
If your application already configures a global OTEL meter provider, leave `HEADROOM_OTEL_*` unset — Headroom records into the ambient provider automatically.
### Dynatrace
Point the exporter at your environment's OTLP API and add the API token as a header. The token needs the `metrics.ingest` scope.
```bash
HEADROOM_OTEL_METRICS_ENABLED=1
HEADROOM_OTEL_METRICS_ENDPOINT="https://<env-id>.live.dynatrace.com/api/v2/otlp/v1/metrics"
HEADROOM_OTEL_METRICS_HEADERS="Authorization=Api-Token dt0c01.XXXX"
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=DELTA
HEADROOM_OTEL_SERVICE_NAME=headroom-proxy
```
<Callout type="warn" title="Delta temporality is not optional">
`OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=DELTA` is required. Dynatrace [only ingests delta counters](https://docs.dynatrace.com/docs/ingest-from/opentelemetry/getting-started/metrics/limitations) and rejects cumulative ones with `UNSUPPORTED_METRIC_TYPE_MONOTONIC_CUMULATIVE_SUM`, while the OTEL SDK default is cumulative. Without this line, every Headroom metric is dropped at ingest and the proxy logs no error.
</Callout>
Restart the proxy, then search the Dynatrace metric explorer for `headroom.proxy.tokens.saved` — data appears within ~30s.
For an ActiveGate deployment, swap the base URL for `https://<activegate>:9999/e/<env-id>/api/v2/otlp/v1/metrics`. If you already run an OpenTelemetry Collector, send Headroom to it instead and add the `cumulativetodelta` processor — then the temporality variable is unnecessary and the collector holds the token.
Trace export is separate: Headroom's self-configured tracing targets Langfuse only. To land its spans in Dynatrace, leave `HEADROOM_LANGFUSE_*` unset and run the proxy under `opentelemetry-instrument` with the standard `OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_EXPORTER_OTLP_HEADERS` / `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` variables; Headroom records into the ambient tracer provider.
### Health Check
```bash

2
docs/content/docs/proxy.mdx
View file

@ -284,7 +284,7 @@ Rewrite the upstream model per request — for example, send small, tool-free ca
| `--telemetry` / `HEADROOM_TELEMETRY` | off | **Local-only** usage stats for your own `/stats`, `/metrics`, and dashboard. Nothing leaves the machine. |
| `--log-file` / `HEADROOM_LOG_FILE` | none | JSONL request/response log. |
| `--log-messages` | `false` | Include full message bodies in the log (may contain sensitive data). |
| `HEADROOM_OTEL_METRICS_ENABLED` | `false` | Export OpenTelemetry metrics (`HEADROOM_OTEL_METRICS_ENDPOINT`, …). |
| `HEADROOM_OTEL_METRICS_ENABLED` | `false` | Export OpenTelemetry metrics (`HEADROOM_OTEL_METRICS_ENDPOINT`, …). See [OTLP export](/docs/metrics#opentelemetry-otlp-export). |
| `HEADROOM_LANGFUSE_ENABLED` | `false` | Emit Langfuse traces (`LANGFUSE_PUBLIC_KEY` / `LANGFUSE_SECRET_KEY`). |
See [Metrics](/docs/metrics) for the Prometheus and Grafana setup.