akadmin/headroom
1
0
Fork 0
mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
headroom/examples/grafana/headroom-dashboard.json
Matthew Jackson 6bdc8c44a3
docs(metrics): ship an importable Grafana dashboard (#2168) 
## Description

<!-- Briefly explain the change and why it is needed. -->

The metrics docs describe the `headroom_*` Prometheus metric family and
suggest example Grafana panels, but ship no importable dashboard — users
have to build one by hand. This adds a ready-to-import Grafana dashboard
built **only** on documented metric names (`headroom_requests_total`,
`headroom_tokens_saved_total`, `headroom_tokens_input_total`, and the
`headroom_overhead_ms_*` millisecond summary), and links it from the
**Grafana Dashboard** section of `docs/content/docs/metrics.mdx`.

This is a docs/examples-only addition — no source code changes.

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added `examples/grafana/headroom-dashboard.json` — a ready-to-import
Grafana dashboard (7 panels, uid `headroom-compression`) built entirely
on Headroom's documented `/metrics` names. Panels cover tokens saved,
input tokens, request rate, average processing overhead
(`headroom_overhead_ms_sum` / `headroom_overhead_ms_count` with
min/max), tokens-saved/sec, and request rate by pool. It uses **no
histograms** (the proxy emits none). The `pool`/`source` template
variables use regex matchers (`=~`) so they are optional and match
series without those labels.
- Updated `docs/content/docs/metrics.mdx` — linked the new dashboard
from the **Grafana Dashboard** section with import instructions, keeping
the existing ad-hoc PromQL query table alongside it.

## Testing

<!-- Check what you actually ran, then paste the real command output
below. -->

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed

Docs/examples-only change, manually verified: the dashboard JSON is
well-formed and every PromQL query references only the documented
`headroom_*` metric names from `docs/content/docs/metrics.mdx`.

### Test Output

```text
$ python3 -c "import json; d=json.load(open('examples/grafana/headroom-dashboard.json')); print('valid JSON,', len(d['panels']), 'panels, uid', d['uid'])"
valid JSON, 7 panels, uid headroom-compression
```

PromQL queries used by the panels (all against documented `headroom_*`
metrics):

```text
sum(headroom_tokens_saved_total{pool=~"$pool", hook=~"$hook"})
sum(headroom_tokens_input_total{pool=~"$pool", hook=~"$hook"})
sum(rate(headroom_requests_total{pool=~"$pool", hook=~"$hook"}[$__rate_interval]))
sum(rate(headroom_overhead_ms_sum{pool=~"$pool", hook=~"$hook"}[$__rate_interval])) / clamp_min(sum(rate(headroom_overhead_ms_count{pool=~"$pool", hook=~"$hook"}[$__rate_interval])), 1)
sum(rate(headroom_tokens_saved_total{pool=~"$pool", hook=~"$hook"}[$__rate_interval])) by (pool)
max(headroom_overhead_ms_max{pool=~"$pool", hook=~"$hook"})
min(headroom_overhead_ms_min{pool=~"$pool", hook=~"$hook"})
sum(rate(headroom_requests_total{pool=~"$pool", hook=~"$hook"}[$__rate_interval])) by (pool)
```

## Real Behavior Proof

- Environment: local checkout of the PR branch; Python 3 for JSON
validation.
- Exact command / steps: ran the JSON-validation command above (see Test
Output) — parses cleanly, reports 7 panels and uid
`headroom-compression`; then read every panel target and confirmed each
PromQL query references only metric names documented in
`docs/content/docs/metrics.mdx` (`headroom_requests_total`,
`headroom_tokens_saved_total`, `headroom_tokens_input_total`,
`headroom_overhead_ms_{sum,count,min,max}`). No histogram metrics are
referenced.
- Observed result: JSON is valid and importable via Grafana's
**Dashboards → New → Import → Upload**; no datasource UID is hard-coded,
so the importer prompts for a Prometheus datasource. Queries match the
documented metric family.
- Not tested: a full live Grafana import against a running proxy
scraping real `/metrics` was not performed in CI. Verification was
limited to JSON validity and query/metric-name correctness against the
documented metrics.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

Additive docs/examples only — no source code, tests, or runtime behavior
changed.

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A — dashboard is imported from JSON; see the PromQL and panel list
above.

## Additional Notes

<!-- Mention any N/A checklist items, tradeoffs, follow-ups, or
maintainer context. -->

Test-related checklist items are N/A: this is an additive docs/examples
change with no application code, so `pytest`/`mypy`/`ruff` and new unit
tests do not apply. The dashboard JSON was validated and its queries
checked against the documented metric names instead.

---------

Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
2026-07-14 16:07:25 -04:00

368 lines
13 KiB
JSON
Raw Permalink Blame History

{
"__note": "Headroom compression dashboard. Built on the headroom_* Prometheus metrics exposed at GET /metrics (verified against the running proxy): counters for requests/tokens and the headroom_overhead_ms_* millisecond summary. No histograms are used because the proxy emits none. The pool/source template variables use regex matchers (=~) so they are optional: they match series with no such label as well, and let multi-tenant scrapers that add a per-source label filter by it. Import: Grafana -> Dashboards -> New -> Import -> Upload this file, then pick your Prometheus datasource.",
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": { "type": "grafana", "uid": "-- Grafana --" },
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"description": "Tokens saved, request throughput, and processing overhead for the Headroom compression proxy, read from the headroom_* Prometheus metrics.",
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"id": null,
"links": [],
"liveNow": false,
"refresh": "30s",
"schemaVersion": 39,
"tags": ["headroom", "compression", "llm"],
"templating": {
"list": [
{
"current": {},
"hide": 0,
"includeAll": false,
"label": "Datasource",
"multi": false,
"name": "datasource",
"options": [],
"query": "prometheus",
"refresh": 1,
"regex": "",
"type": "datasource"
},
{
"allValue": ".*",
"current": { "text": "All", "value": "$__all" },
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"definition": "label_values(headroom_requests_total, pool)",
"hide": 0,
"includeAll": true,
"label": "Pool",
"multi": true,
"name": "pool",
"options": [],
"query": { "query": "label_values(headroom_requests_total, pool)", "refId": "PoolQuery" },
"refresh": 2,
"regex": "",
"sort": 1,
"type": "query"
},
{
"allValue": ".*",
"current": { "text": "All", "value": "$__all" },
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"definition": "label_values(headroom_requests_total, hook)",
"description": "Optional. Present when a multi-tenant scraper re-exposes headroom_* with a per-source 'hook' label; empty against Headroom's own /metrics.",
"hide": 0,
"includeAll": true,
"label": "Source",
"multi": true,
"name": "hook",
"options": [],
"query": { "query": "label_values(headroom_requests_total, hook)", "refId": "HookQuery" },
"refresh": 2,
"regex": "",
"sort": 1,
"type": "query"
}
]
},
"time": { "from": "now-6h", "to": "now" },
"timepicker": {},
"timezone": "",
"title": "Headroom — Compression",
"uid": "headroom-compression",
"version": 1,
"weekStart": "",
"panels": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"description": "Input tokens saved by compression since the process started.",
"fieldConfig": {
"defaults": {
"color": { "mode": "thresholds" },
"mappings": [],
"thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] },
"unit": "short"
},
"overrides": []
},
"gridPos": { "h": 4, "w": 6, "x": 0, "y": 0 },
"id": 1,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"textMode": "auto"
},
"pluginVersion": "10.4.0",
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "sum(headroom_tokens_saved_total{pool=~\"$pool\", hook=~\"$hook\"})",
"legendFormat": "Tokens saved",
"refId": "A"
}
],
"title": "Tokens Saved",
"type": "stat"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"description": "Total input tokens seen.",
"fieldConfig": {
"defaults": {
"color": { "mode": "thresholds" },
"mappings": [],
"thresholds": { "mode": "absolute", "steps": [{ "color": "blue", "value": null }] },
"unit": "short"
},
"overrides": []
},
"gridPos": { "h": 4, "w": 6, "x": 6, "y": 0 },
"id": 2,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"textMode": "auto"
},
"pluginVersion": "10.4.0",
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "sum(headroom_tokens_input_total{pool=~\"$pool\", hook=~\"$hook\"})",
"legendFormat": "Input tokens",
"refId": "A"
}
],
"title": "Input Tokens",
"type": "stat"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"description": "Requests processed per second.",
"fieldConfig": {
"defaults": {
"color": { "mode": "thresholds" },
"mappings": [],
"thresholds": { "mode": "absolute", "steps": [{ "color": "purple", "value": null }] },
"unit": "reqps"
},
"overrides": []
},
"gridPos": { "h": 4, "w": 6, "x": 12, "y": 0 },
"id": 3,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"textMode": "auto"
},
"pluginVersion": "10.4.0",
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "sum(rate(headroom_requests_total{pool=~\"$pool\", hook=~\"$hook\"}[$__rate_interval]))",
"legendFormat": "req/s",
"refId": "A"
}
],
"title": "Request Rate",
"type": "stat"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"description": "Mean Headroom processing overhead — overhead_ms_sum / overhead_ms_count over the window.",
"fieldConfig": {
"defaults": {
"color": { "mode": "thresholds" },
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 5 },
{ "color": "red", "value": 25 }
]
},
"unit": "ms"
},
"overrides": []
},
"gridPos": { "h": 4, "w": 6, "x": 18, "y": 0 },
"id": 4,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"textMode": "auto"
},
"pluginVersion": "10.4.0",
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "sum(rate(headroom_overhead_ms_sum{pool=~\"$pool\", hook=~\"$hook\"}[$__rate_interval])) / clamp_min(sum(rate(headroom_overhead_ms_count{pool=~\"$pool\", hook=~\"$hook\"}[$__rate_interval])), 1)",
"legendFormat": "mean overhead",
"refId": "A"
}
],
"title": "Avg Overhead",
"type": "stat"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"description": "Tokens saved per second, split by pool when a per-pool label is present.",
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"drawStyle": "line",
"fillOpacity": 15,
"gradientMode": "opacity",
"lineInterpolation": "smooth",
"lineWidth": 2,
"pointSize": 5,
"showPoints": "never",
"spanNulls": true,
"stacking": { "group": "A", "mode": "normal" }
},
"unit": "short"
},
"overrides": []
},
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 4 },
"id": 5,
"options": {
"legend": { "calcs": ["sum"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "sum(rate(headroom_tokens_saved_total{pool=~\"$pool\", hook=~\"$hook\"}[$__rate_interval])) by (pool)",
"legendFormat": "{{pool}}",
"refId": "A"
}
],
"title": "Tokens Saved / sec",
"type": "timeseries"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"description": "Headroom processing overhead in milliseconds: mean (overhead_ms_sum/overhead_ms_count) with the reported min and max.",
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"drawStyle": "line",
"fillOpacity": 8,
"gradientMode": "none",
"lineInterpolation": "smooth",
"lineWidth": 2,
"pointSize": 5,
"showPoints": "never",
"spanNulls": true,
"stacking": { "group": "A", "mode": "none" }
},
"unit": "ms"
},
"overrides": []
},
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 4 },
"id": 6,
"options": {
"legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "sum(rate(headroom_overhead_ms_sum{pool=~\"$pool\", hook=~\"$hook\"}[$__rate_interval])) / clamp_min(sum(rate(headroom_overhead_ms_count{pool=~\"$pool\", hook=~\"$hook\"}[$__rate_interval])), 1)",
"legendFormat": "mean",
"refId": "A"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "max(headroom_overhead_ms_max{pool=~\"$pool\", hook=~\"$hook\"})",
"legendFormat": "max",
"refId": "B"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "min(headroom_overhead_ms_min{pool=~\"$pool\", hook=~\"$hook\"})",
"legendFormat": "min",
"refId": "C"
}
],
"title": "Processing Overhead",
"type": "timeseries"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"description": "Request throughput split by pool (when a per-pool label is present).",
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"drawStyle": "line",
"fillOpacity": 15,
"gradientMode": "opacity",
"lineInterpolation": "smooth",
"lineWidth": 2,
"pointSize": 5,
"showPoints": "never",
"spanNulls": true,
"stacking": { "group": "A", "mode": "normal" }
},
"unit": "reqps"
},
"overrides": []
},
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 12 },
"id": 7,
"options": {
"legend": { "calcs": ["mean"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "sum(rate(headroom_requests_total{pool=~\"$pool\", hook=~\"$hook\"}[$__rate_interval])) by (pool)",
"legendFormat": "{{pool}}",
"refId": "A"
}
],
"title": "Request Rate by Pool",
"type": "timeseries"
}
]
}
Reference in a new issue View git blame Copy permalink