mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
9cba64d89e
## Description Users upgrading 0.27.0 → 0.31.0 report that the dashboard's compression / "Tokens Saved" figures drop to ~0 and conclude Headroom stopped working. The #2248 reporter ran the same prompt on both versions and captured the telltale detail: **0.31.0 actually spent fewer total tokens than 0.27.0, despite showing 0 saved.** This is a default-mode change, not a regression. 0.31.0 ships the `coding` savings profile as the out-of-box default (`headroom/agent_savings.py`: `DEFAULT_PROFILE = "coding"`), and `coding` sets `proxy_mode="cache"`. Cache mode freezes the provider prefix and compresses only the newest turn *delta* — deliberately, to avoid busting the prompt cache — so the **compression** number is small while savings shift to **cheaper prefix-cache reads**. On a short prompt there's little delta to compress, so the compression tile reads ~0 even as real cost drops. The reference behavior is already documented in the proxy docs' [Savings profiles](/docs/proxy#savings-profiles) section, but there was no discoverable troubleshooting entry connecting the alarming "0 saved after upgrade" symptom to this cause — so it gets filed as a bug. Closes #2248 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made `docs/content/docs/troubleshooting.mdx` only — a new `### Dashboard shows 0 compressed/saved tokens after upgrading to 0.31.0` subsection appended to the existing `## No Token Savings` section: - **Symptom** — compression figures ~0 after upgrade, while total spend is flat or lower (so users can match it by search). - **Cause** — the `coding`/cache-mode default and why delta-only compression makes the compression tile small. - **Where the savings show up** — the **Prefix Cache Impact** panel and **Compression vs Cache** tile, which reflect cache-read savings; the headline "Tokens Saved" tile counts compression only and understates the benefit in cache mode. - **How to get 0.27.0-style numbers back** — `--mode token`, or `HEADROOM_SAVINGS_PROFILE=balanced` / `agent-90`, with the explicit trade-off that token mode raises visible compression but can reduce prefix-cache hits. Placed under the existing `## No Token Savings` heading (which covers the separate SDK/library case: audit mode, sub-threshold tool outputs) rather than rewriting it. Cross-links to the existing Savings-profiles reference instead of restating the profile table, keeping one source of truth. No code change. ## Testing - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [x] Manual testing performed ### Test Output Docs-only; verification is fact cross-check against source plus MDX sanity: ```text $ grep -n 'DEFAULT_PROFILE = \|proxy_mode="cache"' headroom/agent_savings.py 18:DEFAULT_PROFILE = "coding" 173: proxy_mode="cache", # delta-only compression at ~0 prefix-cache busts $ grep -n "_estimate_cache_savings_usd" headroom/proxy/savings_tracker.py 248:def _estimate_cache_savings_usd(model: str, cache_read_tokens: int) -> float: $ grep -c "Prefix Cache Impact" headroom/dashboard/templates/dashboard.html # 2 $ grep -c "Compression vs Cache" headroom/dashboard/templates/dashboard.html # 1 $ grep -n "### Savings profiles" docs/content/docs/proxy.mdx 94:### Savings profiles # cross-link target for /docs/proxy#savings-profiles # placement: new "### Dashboard shows 0 compressed..." (line 145) sits between # "## No Token Savings" (89) and "## Claude Code context window..." (166) # MDX sanity: code fences balance (even count) ``` ## Real Behavior Proof - **Environment:** Docs source verified against the current `main` base (`718c8dc5`). - **Exact command / steps:** Issue #2248 contains a complete reproduction — the same prompt run under 0.27.0 and 0.31.0 via `headroom wrap claude --dangerously-skip-permissions` (Sonnet 5, same files, same Claude Code version, reproduced on macOS and Debian 12), with dashboard screenshots showing savings on 0.27.0 and ~0 on 0.31.0. Every claim in the new section is verified against the tree with the greps above: the `coding` default and its `proxy_mode="cache"`, the cache-read savings estimator, and both dashboard panel/tile labels users are pointed to. - **Observed result:** The documented cause matches the code — the compression tile legitimately reads ~0 in cache mode while cache-read savings accrue in the Prefix Cache Impact panel, which explains the reporter's own observation that 0.31.0 spent *fewer* tokens while showing 0 saved. - **Not tested:** I did not re-run a live 0.27.0-vs-0.31.0 dashboard comparison (that requires installing an old release and generating real provider traffic); the reporter's reproduction with screenshots already establishes the symptom, and the cause is verified in source. No local Fumadocs site build was run, so the section is validated by MDX syntax checks rather than a rendered preview. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A (troubleshooting prose addition). ## Additional Notes - Test/tests-added and CHANGELOG checklist items are N/A — documentation-only change, kept to a single file (matching the merged #2031 and #2237 precedent). - If maintainers would rather resolve this in the UI than the docs, an alternative is a dashboard hint shown when mode is `cache` and compression savings are ~0 (pointing at the Prefix Cache Impact panel). That touches `dashboard.html` and has UX implications, so it's intentionally not attempted here. - This is the second report rooted in the cache-mode default (following the confusion behind #2031), which is why it's framed as a searchable troubleshooting entry rather than another reference-section edit. |
||
|---|---|---|
| .. | ||
| app | ||
| components | ||
| content/docs | ||
| lib | ||
| overrides | ||
| screenshots | ||
| .gitignore | ||
| bun.lock | ||
| claude-code-bedrock-headroom.md | ||
| next.config.mjs | ||
| observability.md | ||
| package-lock.json | ||
| package.json | ||
| platform-feature-matrix.json | ||
| platform-stabilization.md | ||
| postcss.config.mjs | ||
| proxy.ts | ||
| README.md | ||
| rtk-architecture.md | ||
| source.config.ts | ||
| tsconfig.json | ||
| vercel.json | ||
docs
This is a Next.js application generated with Create Fumadocs.
Run development server:
npm run dev
# or
pnpm dev
# or
yarn dev
Open http://localhost:3000 with your browser to see the result.
Explore
In the project, you can see:
lib/source.ts: Code for content source adapter,loader()provides the interface to access your content.lib/layout.shared.tsx: Shared options for layouts, optional but preferred to keep.
| Route | Description |
|---|---|
app/(home) |
The route group for your landing page and other pages. |
app/docs |
The documentation layout and pages. |
app/api/search/route.ts |
The Route Handler for search. |
Fumadocs MDX
A source.config.ts config file has been included, you can customise different options like frontmatter schema.
Read the Introduction for further details.
Learn More
To learn more about Next.js and Fumadocs, take a look at the following resources:
- Next.js Documentation - learn about Next.js features and API.
- Learn Next.js - an interactive Next.js tutorial.
- Fumadocs - learn about Fumadocs