From 7f7af667ed3db718adad34f34d83517ac880760d Mon Sep 17 00:00:00 2001 From: Krishna Chaitanya Date: Sat, 11 Jul 2026 12:04:54 -0400 Subject: [PATCH] feat(observability): add gen_ai.request.model to the compression span (#1667) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Description Emit the OpenTelemetry GenAI semantic-convention attribute `gen_ai.request.model` on the existing `headroom.compression.pipeline` span, alongside the current `headroom.*` attributes. Today Headroom's OTel spans use only proprietary `headroom.*` names, so a team pointing an OTel-native backend at Headroom can't join its telemetry to their existing `gen_ai.*` LLM dashboards. This makes the compression span groupable/filterable by the standard schema. Proposed and scoped in #1671. Per CONTRIBUTING (new features want a maintainer 👍 + spec first), this is opened as a **draft** to get sign-off on the approach and v1 scope before finalizing. ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Short spec - API surface: one additive span attribute, `gen_ai.request.model`, on the existing `headroom.compression.pipeline` span. No new endpoints, headers, or config; nothing renamed. - Scope (v1, deliberately minimal): only `gen_ai.request.model` — the one gen_ai attribute this pre-flight compression span can set correctly and unconditionally (the model is always known here). - Deferred to v2 (each needs work this span cannot do correctly, and I'd value your steer on all three): - `gen_ai.operation.name`: `apply()` is shared by many callers (chat, `/v1/compress`, batch, Gemini `countTokens`), so no single hardcoded value is right — it has to be threaded from each caller. - `gen_ai.provider.name`: Headroom's provider label can't distinguish Bedrock/Gemini from Anthropic/OpenAI at this layer (Bedrock routes through the Anthropic provider). - `gen_ai.usage.*`: provider-authoritative usage lives on the response path, not this span; the compressed-input estimate stays under `headroom.tokens.after`. - Failure modes: model missing → attribute omitted (never a blank string); span not recording / `record_metrics=False` → no attribute, no crash. - Security: no new input surface; derived from data already on the span. ## Changes Made - `headroom/transforms/pipeline.py`: emit `gen_ai.request.model` on the pipeline span (guarded on model present), with a comment documenting why the other gen_ai.* attributes are deferred. - `tests/test_observability_tracing.py`: assert the attribute is emitted, the deferred attrs are omitted, the model-missing guard, and the non-recording path. - `CHANGELOG.md`: Unreleased → Features entry. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output ```text $ uv run pytest tests/test_observability_tracing.py -q 7 passed $ uv run pytest tests/test_observability_tracing.py tests/test_compression_observability.py \ tests/test_observability_metrics.py tests/test_pipeline.py tests/test_canonical_pipeline.py tests/test_telemetry.py -q 76 passed $ uv run ruff check . && uv run mypy headroom/transforms/pipeline.py --ignore-missing-imports All checks passed! / Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: local, Python 3.12, `opentelemetry-sdk` 1.39.1, real `ConsoleSpanExporter` (not a mock). Ran the actual `TransformPipeline.apply()` emission path. - Exact command / steps: configured a real `TracerProvider` + `ConsoleSpanExporter`, set it as Headroom's tracer, ran `TransformPipeline([]).apply([{user msg}], model="claude-3-5-sonnet-20241022", model_limit=8192)`, then `force_flush()` and inspected the exported span. - Observed result: the exported `headroom.compression.pipeline` span carries `gen_ai.request.model` alongside the existing `headroom.*` attributes: ```json "attributes": { "headroom.model": "claude-3-5-sonnet-20241022", "headroom.provider": "unknown", "headroom.message_count": 1, "headroom.tokens.before": 83, "gen_ai.request.model": "claude-3-5-sonnet-20241022", "headroom.tokens.after": 83, "headroom.tokens.saved": 0 } ``` - Not tested: no live OTLP collector / Grafana backend (used the console exporter, which is the same span pipeline); the deferred v2 attributes (`operation.name`/`provider.name`/`usage.*`) are intentionally not emitted. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review (Draft: awaiting a maintainer 👍 on the approach and the v1 scope before marking ready.) ## Additional Notes Purely additive and back-compatible — no `headroom.*` attribute changed or removed. The gen_ai attribute name is a string literal because the `gen_ai.*` conventions are stability=development in the semconv registry (no stable constants published). No new dependencies. Signed-off-by: Krishnachaitanyakc Co-authored-by: JerrettDavis --- CHANGELOG.md | 1 + headroom/transforms/pipeline.py | 28 +++++++++++- tests/test_observability_tracing.py | 70 +++++++++++++++++++++++++++++ 3 files changed, 98 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b6a7591f0..82fe80f67 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 * **proxy:** add provider-only HTTP proxy routing via `--http-proxy` and `HEADROOM_HTTP_PROXY`. Upstream LLM provider calls can now use an HTTP proxy without setting process-wide `HTTP_PROXY`/`HTTPS_PROXY` variables that are inherited by tool executions; proxied provider clients use HTTP/1.1 so HTTPS provider APIs can tunnel through CONNECT. * **proxy:** add output shaping for OpenAI Responses traffic on `/v1/responses` HTTP requests and Codex WebSocket `response.create` frames, with stable output-savings holdout keys and counted WS token strata for the experiment. +* **observability:** the `headroom.compression.pipeline` span now also carries the OpenTelemetry GenAI semantic-convention attribute `gen_ai.request.model` alongside the existing `headroom.*` attributes, so Headroom's traces group and filter by the standard `gen_ai.*` schema in any OTel-native backend (Grafana, Datadog, etc.). Purely additive; no existing attribute changed. `gen_ai.operation.name`, `gen_ai.provider.name`, and `gen_ai.usage.*` are deliberately deferred (they need per-caller operation threading, reliable upstream-provider resolution, and response-path usage respectively). * **wrap:** `headroom wrap claude --1m` preserves the 1M context window. Behind a custom `ANTHROPIC_BASE_URL` (the proxy) Claude Code drops the `context-1m` beta header and caps the window at 200k for entitled subscription users; the opt-in flag sets `ANTHROPIC_MODEL=[1m]` on the launched process so the 1M window activates through Headroom. A model already selected via `ANTHROPIC_MODEL` is preserved (only the `[1m]` suffix is appended) ([#1158](https://github.com/chopratejas/headroom/issues/1158)). * **learn:** weight loops in `headroom learn`. A new loop detector (`headroom/learn/loops.py`) recognizes repeated tool-call patterns — including RTK re-fetch loops, where RTK's output truncation makes the agent re-run larger-limit variants of a *successful* command — collapses output-limit variants to one signature, measures the wasted tokens, surfaces loops as a highest-priority digest section, and weights loop guardrails above one-off rules by their measured waste. Previously loops had no special weight and a no-failure re-fetch loop was skipped entirely. Adds an RTK-loop eval (`benchmarks/rtk_loop_learn_eval.py`) that reproduces a loop, runs it through Learn, and asserts the generated guardrail ranks first and prevents re-triggering. * **learn:** write per-project learnings to the personal, gitignored `CLAUDE.local.md` by default instead of the team-shared `CLAUDE.md`, matching Claude Code's memory convention so machine-specific paths and tool-discovery byproducts no longer pollute the shared file. Adds a `--target` flag to override the destination (e.g. `--target CLAUDE.md` to opt back into the shared file, or any custom path), and auto-migrates a stale learned-patterns block out of an existing `CLAUDE.md` into `CLAUDE.local.md` with a warning ([#1072](https://github.com/chopratejas/headroom/issues/1072)). diff --git a/headroom/transforms/pipeline.py b/headroom/transforms/pipeline.py index fec4701a2..c7061dee1 100644 --- a/headroom/transforms/pipeline.py +++ b/headroom/transforms/pipeline.py @@ -42,6 +42,28 @@ MAX_WASTE_SIGNAL_DETECTION_TOKENS = 100_000 # runs when compression saved more than this many tokens. _MIN_TOKENS_SAVED_FOR_WASTE_SIGNALS = 100 +# OTel GenAI semantic conventions (open-telemetry/semantic-conventions-genai). +# The compression-pipeline span carries this gen_ai.* attribute alongside the +# proprietary headroom.* ones, so Headroom's telemetry groups/filters by the +# standard schema in any OTel-native backend (Grafana, Datadog, etc.). It is a +# string literal rather than an opentelemetry.semconv constant because the gen_ai +# attributes are stability=development (no stable constants are published). +# +# v1 emits only gen_ai.request.model — the one attribute this span can set +# correctly and unconditionally (the model is always known here). The rest are +# deliberately deferred to v2 because this span cannot set them correctly: +# - gen_ai.operation.name: apply() is shared by many callers (chat, /v1/compress, +# batch, Gemini countTokens), so no single value is right — it must be threaded +# from each caller, not hardcoded. +# - gen_ai.provider.name: Headroom's provider label can't distinguish Bedrock / +# Gemini from Anthropic / OpenAI at this layer (Bedrock routes via the +# Anthropic provider). +# - gen_ai.usage.*: provider-authoritative usage lives on the response path, not +# this pre-flight compression span (the compressed-input estimate stays under +# headroom.tokens.after). +GEN_AI_REQUEST_MODEL = "gen_ai.request.model" + + _N = TypeVar("_N", int, float) @@ -269,12 +291,16 @@ class TransformPipeline: ) tracer = get_headroom_tracer() - span_attributes = { + span_attributes: dict[str, Any] = { "headroom.model": model, "headroom.provider": provider_name or "unknown", "headroom.message_count": len(messages), "headroom.tokens.before": tokens_before, } + # OTel GenAI semconv request descriptor — emitted alongside headroom.* so + # the span is groupable by the standard schema (v1: model only). + if model: + span_attributes[GEN_AI_REQUEST_MODEL] = model pipeline_span_context = ( tracer.start_as_current_span( "headroom.compression.pipeline", diff --git a/tests/test_observability_tracing.py b/tests/test_observability_tracing.py index d10054631..5a072fa34 100644 --- a/tests/test_observability_tracing.py +++ b/tests/test_observability_tracing.py @@ -55,6 +55,76 @@ def test_transform_pipeline_emits_trace_spans() -> None: reset_headroom_tracing() +def test_pipeline_span_emits_gen_ai_request_model() -> None: + """The compression-pipeline span carries the v1 OTel GenAI semconv descriptor + (gen_ai.request.model) alongside headroom.*, so it groups by the standard + schema. operation.name / provider.name / usage.* are intentionally v2.""" + exporter = InMemorySpanExporter() + provider = TracerProvider(resource=Resource.create({"service.name": "headroom-test"})) + provider.add_span_processor(SimpleSpanProcessor(exporter)) + set_headroom_tracer(HeadroomTracer(tracer_provider=provider)) + + try: + pipeline = TransformPipeline(transforms=[]) + pipeline.apply( + [{"role": "user", "content": "hello world"}], + model="claude-3-5-sonnet-20241022", + model_limit=1024, + ) + + span = exporter.get_finished_spans()[0] + assert span.name == "headroom.compression.pipeline" + assert span.attributes["gen_ai.request.model"] == "claude-3-5-sonnet-20241022" + # v1 deliberately emits ONLY request.model — no operation/provider/usage + # (each is inaccurate at this span; see pipeline.py). + assert "gen_ai.operation.name" not in span.attributes + assert "gen_ai.provider.name" not in span.attributes + assert "gen_ai.usage.input_tokens" not in span.attributes + finally: + reset_headroom_tracing() + + +def test_pipeline_span_omits_request_model_when_model_missing() -> None: + """gen_ai.request.model is omitted (not set to an empty string) when no model + is provided — never emit a blank standard attribute.""" + exporter = InMemorySpanExporter() + provider = TracerProvider(resource=Resource.create({"service.name": "headroom-test"})) + provider.add_span_processor(SimpleSpanProcessor(exporter)) + set_headroom_tracer(HeadroomTracer(tracer_provider=provider)) + + try: + pipeline = TransformPipeline(transforms=[]) + pipeline.apply([{"role": "user", "content": "hi"}], model="", model_limit=1024) + span = exporter.get_finished_spans()[0] + assert span.name == "headroom.compression.pipeline" + assert "gen_ai.request.model" not in span.attributes + finally: + reset_headroom_tracing() + + +def test_pipeline_runs_with_metrics_disabled() -> None: + """record_metrics=False takes the nullcontext (no-span) path: the run still + returns a valid result and emits no spans (guards that building span_attributes + with the gen_ai key never breaks the non-recording path).""" + exporter = InMemorySpanExporter() + provider = TracerProvider(resource=Resource.create({"service.name": "headroom-test"})) + provider.add_span_processor(SimpleSpanProcessor(exporter)) + set_headroom_tracer(HeadroomTracer(tracer_provider=provider)) + + try: + pipeline = TransformPipeline(transforms=[]) + result = pipeline.apply( + [{"role": "user", "content": "hi"}], + model="gpt-4o", + model_limit=1024, + record_metrics=False, + ) + assert result.messages # pipeline produced output + assert exporter.get_finished_spans() == () + finally: + reset_headroom_tracing() + + def test_langfuse_tracing_status_defaults_to_unconfigured() -> None: reset_headroom_tracing() status = get_langfuse_tracing_status()