akadmin/headroom
1
0
Fork 0
mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions

docs(metrics): document OTLP metric export and Dynatrace ingest (#2785)

Browse source 
## Description

The proxy can already push its counters to any OTLP/HTTP endpoint via
`HEADROOM_OTEL_METRICS_*`, but the docs site only surfaced this as a
single row in the proxy env table (`proxy.mdx:287`). The endpoint,
header, service-name, and resource-attribute variables were documented
only in `wiki/metrics.md` — so an operator reading the Vercel docs had
no way to wire Headroom into their existing observability stack.

This adds that section, plus a Dynatrace subsection, because Dynatrace
has a silent failure mode that costs an afternoon to diagnose.

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `docs/content/docs/metrics.mdx` — new `### OpenTelemetry (OTLP)
Export` section after the Prometheus section: the
`headroom-ai[proxy,otel]` install, all seven `HEADROOM_OTEL_*` variables
in a table, the exported counter names (`headroom.proxy.tokens.saved` et
al.), the `curl /stats | jq .otel` verification, and the note that an
app-managed global meter provider is recorded into automatically.
- `docs/content/docs/metrics.mdx` — new `### Dynatrace` subsection:
copy-paste env block, `metrics.ingest` token scope, a `warn` Callout on
the delta-temporality requirement, the ActiveGate URL variant, the
Collector + `cumulativetodelta` alternative, and one paragraph
explaining that trace export needs `opentelemetry-instrument`
(Headroom's self-configured tracing targets Langfuse only).
- `docs/content/docs/proxy.mdx` — the `HEADROOM_OTEL_METRICS_ENABLED`
row now links to `/docs/metrics#opentelemetry-otlp-export`.

No code, config, or nav changes — the Observability nav slot already
points at `metrics.mdx`.

## Testing

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed

Docs-only change: no Python touched, so pytest/ruff/mypy have nothing to
cover here. `next build` was **not** run — `docs/node_modules` is absent
in this checkout, which would require a full `npm install`; Vercel's
preview build is the real gate. In its place I verified the MDX cannot
break the build by parsing for the two things that actually fail MDX v3
— unbalanced JSX and bare `<`/`{` in prose.

### Test Output

```text
$ python - <<'PY'   # strip fenced + inline code, then scan prose for MDX hazards
...
PY
hazards: [(80, '<Tabs groupId="lang" items={[\'TypeScript\', \'Python\']}>'),
          (125, '<Tabs groupId="lang" items={[\'Python\', \'Proxy\']}>')]
Callout balance: 1 open / 1 close
```

Both flagged lines are pre-existing `<Tabs>` JSX expressions, untouched
by this PR. The added prose introduces no bare `<` or `{` (every
`<env-id>` / `<activegate>` placeholder sits inside a code fence or
inline backticks). `type="warn"` is already used on three other pages,
and the anchor `#opentelemetry-otlp-export` matches the GitHub-slugger
form of the new heading.

## Real Behavior Proof

- **Environment:** macOS (darwin 25.4.0), repo `.venv`,
`opentelemetry-sdk` 1.44.0, `opentelemetry-exporter-otlp-proto-http`,
headroom @ 6422a80a.
- **Exact command / steps:** verified the central claim of the new
Callout — that the OTLP HTTP exporter defaults to cumulative (which
Dynatrace rejects) and that the standard env var flips it to delta with
no Headroom code change:

```text
$ python -c '...'
DELTA= 1 CUMULATIVE= 2
default counter: 2

$ OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=DELTA python -c '...'
counter temporality with env=DELTA: 1
```

- **Observed result:**
`OTLPMetricExporter._preferred_temporality[Counter]` is `CUMULATIVE` (2)
by default and `DELTA` (1) with the env var set. Since every Headroom
OTEL instrument is a `Counter`
(`headroom/observability/metrics.py:136-189`), without the env var
Dynatrace drops all of them — matching its documented
`UNSUPPORTED_METRIC_TYPE_MONOTONIC_CUMULATIVE_SUM` rejection. Also
confirmed against the code that `HEADROOM_OTEL_METRICS_ENDPOINT` is
passed verbatim to the exporter (`metrics.py:539-543`), hence the doc's
warning that `/v1/metrics` must be included by hand, and that
`HEADROOM_OTEL_METRICS_HEADERS` splits on the first `=` so
`Authorization=Api-Token dt0c01...` parses correctly.
- **Not tested:** no live export against a real Dynatrace tenant — the
URL shapes and token scopes come from Dynatrace's docs, not from an
observed 200. The `opentelemetry-instrument` trace path is described
from the code's global-provider fallback (`tracing.py:95-99`), not run
end-to-end. `next build` not run (see Testing).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- N/A on the pytest / ruff / mypy / new-tests items: this PR changes two
`.mdx` files and no Python.
- Follow-up worth considering: `wiki/metrics.md:244` carries the same
OTEL variable table and still lacks the Dynatrace guidance — happy to
mirror it there, kept out of this PR to hold the diff to the Vercel docs
as asked.
- Second follow-up: the delta-temporality fix currently depends on an
upstream OTEL SDK env var that Headroom neither sets nor documents in
code. A `HEADROOM_OTEL_METRICS_TEMPORALITY=delta` passthrough would make
the Dynatrace case self-contained instead of relying on a variable one
layer down.
This commit is contained in:
Tejas Chopra 2026-08-04 18:46:39 -07:00 • committed by GitHub
parent 3c10e8ff00
commit 6b63b623e0
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 54 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

53
docs/content/docs/metrics.mdx
View file

@ -88,6 +88,59 @@ headroom_latency_seconds_bucket{le="0.1"} 1150
headroom_cache_hits_total 456
```
### OpenTelemetry (OTLP) Export
The proxy can also push its counters to any OTLP/HTTP endpoint. Install the extra and set four variables:
```bash
pip install "headroom-ai[proxy,otel]"
```
```bash
HEADROOM_OTEL_METRICS_ENABLED=1
HEADROOM_OTEL_METRICS_ENDPOINT=http://127.0.0.1:4318/v1/metrics
HEADROOM_OTEL_SERVICE_NAME=headroom-proxy
HEADROOM_OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod
```
| Variable | Default | Purpose |
| --- | --- | --- |
| `HEADROOM_OTEL_METRICS_ENABLED` | `0` | Enable Headroom-managed OTLP metric export |
| `HEADROOM_OTEL_METRICS_EXPORTER` | `otlp_http` | `otlp_http` or `console` (local debugging) |
| `HEADROOM_OTEL_METRICS_ENDPOINT` | unset | Full OTLP metrics URL — Headroom does **not** append `/v1/metrics` for you |
| `HEADROOM_OTEL_METRICS_HEADERS` | unset | Comma-separated `key=value` auth headers |
| `HEADROOM_OTEL_METRICS_EXPORT_INTERVAL_MS` | `10000` | Export interval |
| `HEADROOM_OTEL_SERVICE_NAME` | `headroom-proxy` | OTEL `service.name` |
| `HEADROOM_OTEL_RESOURCE_ATTRIBUTES` | unset | Comma-separated resource attributes |
Exported counters include `headroom.proxy.requests`, `headroom.proxy.tokens.input`, `headroom.proxy.tokens.output`, `headroom.proxy.tokens.saved`, and `headroom.proxy.cache.read_tokens` / `write_tokens`.
Confirm the exporter is live with `curl -s http://localhost:8787/stats | jq .otel`.
If your application already configures a global OTEL meter provider, leave `HEADROOM_OTEL_*` unset — Headroom records into the ambient provider automatically.
### Dynatrace
Point the exporter at your environment's OTLP API and add the API token as a header. The token needs the `metrics.ingest` scope.
```bash
HEADROOM_OTEL_METRICS_ENABLED=1
HEADROOM_OTEL_METRICS_ENDPOINT="https://<env-id>.live.dynatrace.com/api/v2/otlp/v1/metrics"
HEADROOM_OTEL_METRICS_HEADERS="Authorization=Api-Token dt0c01.XXXX"
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=DELTA
HEADROOM_OTEL_SERVICE_NAME=headroom-proxy
```
<Callout type="warn" title="Delta temporality is not optional">
`OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=DELTA` is required. Dynatrace [only ingests delta counters](https://docs.dynatrace.com/docs/ingest-from/opentelemetry/getting-started/metrics/limitations) and rejects cumulative ones with `UNSUPPORTED_METRIC_TYPE_MONOTONIC_CUMULATIVE_SUM`, while the OTEL SDK default is cumulative. Without this line, every Headroom metric is dropped at ingest and the proxy logs no error.
</Callout>
Restart the proxy, then search the Dynatrace metric explorer for `headroom.proxy.tokens.saved` — data appears within ~30s.
For an ActiveGate deployment, swap the base URL for `https://<activegate>:9999/e/<env-id>/api/v2/otlp/v1/metrics`. If you already run an OpenTelemetry Collector, send Headroom to it instead and add the `cumulativetodelta` processor — then the temporality variable is unnecessary and the collector holds the token.
Trace export is separate: Headroom's self-configured tracing targets Langfuse only. To land its spans in Dynatrace, leave `HEADROOM_LANGFUSE_*` unset and run the proxy under `opentelemetry-instrument` with the standard `OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_EXPORTER_OTLP_HEADERS` / `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` variables; Headroom records into the ambient tracer provider.
### Health Check
```bash

2
docs/content/docs/proxy.mdx
View file

@ -284,7 +284,7 @@ Rewrite the upstream model per request — for example, send small, tool-free ca
| `--telemetry` / `HEADROOM_TELEMETRY` | off | **Local-only** usage stats for your own `/stats`, `/metrics`, and dashboard. Nothing leaves the machine. |
| `--log-file` / `HEADROOM_LOG_FILE` | none | JSONL request/response log. |
| `--log-messages` | `false` | Include full message bodies in the log (may contain sensitive data). |
| `HEADROOM_OTEL_METRICS_ENABLED` | `false` | Export OpenTelemetry metrics (`HEADROOM_OTEL_METRICS_ENDPOINT`, …). |
| `HEADROOM_OTEL_METRICS_ENABLED` | `false` | Export OpenTelemetry metrics (`HEADROOM_OTEL_METRICS_ENDPOINT`, …). See [OTLP export](/docs/metrics#opentelemetry-otlp-export). |
| `HEADROOM_LANGFUSE_ENABLED` | `false` | Emit Langfuse traces (`LANGFUSE_PUBLIC_KEY` / `LANGFUSE_SECRET_KEY`). |
See [Metrics](/docs/metrics) for the Prometheus and Grafana setup.