akadmin/headroom
1
0
Fork 0
mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions

docs(configuration): document cold-prefix hook flags + bound the TTL observation log (#2557)

Browse source 
## Description

Follow-up to #2555. Documents the cold-prefix hook /
reasoning-compaction /
cache-TTL-learner flags (what to set for what, and whether each can be
on by
default), and makes two small safety fixes so the learning seam is
production-ready and free when off.

## Type of Change

- [x] Documentation update
- [x] Performance improvement (learning seam is now free when disabled)

## Changes Made

- **docs/content/docs/configuration.mdx** — env-var table rows for
`HEADROOM_THINKING_COMPACT` (+`_KEEP_LAST`), `HEADROOM_COLD_RECOMPACT`,
`HEADROOM_DEDUPE`, `HEADROOM_CACHE_TTL_LEARN`,
`HEADROOM_KOMPRESS_ENDPOINT`, plus a
**Cold-prefix hook & reasoning compaction** section: what to set for
what, how
cold detection reads the real TTL (CC config vs learned), and a per-flag
  "can this be on by default?" analysis.
- **docs/content/docs/cache-optimization.mdx** — a cold-prefix
recompaction
  section linking to the flags.
- **headroom/cache/ttl_observations.py** — the observation log is now
size-bounded (single-backup rotation) and respects `HEADROOM_STATELESS`.
- **headroom/proxy/handlers/openai.py** — the extra
`classify_cache_miss`
attribution is gated behind `observations_enabled()` so it costs nothing
when
  learning is off.

Everything remains **off by default**.

## Testing

- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] Manual testing performed (module self-check)

### Test Output

```text
$ ruff check headroom/cache/ttl_observations.py headroom/proxy/handlers/openai.py
All checks passed!

$ mypy headroom/cache/ttl_observations.py headroom/proxy/handlers/openai.py
Success: no issues found in 2 source files

$ python headroom/cache/ttl_observations.py
ttl_observations self-check OK
```

## Real Behavior Proof

- Environment: local repo, Python 3.12 venv.
- Exact command / steps: ran the module self-check (covers gated-off
no-write,
gated-on write, learned-table read with model→provider fallback) and
ruff+mypy.
- Observed result: self-check passes; when `HEADROOM_CACHE_TTL_LEARN` is
unset no
file is written; when `HEADROOM_STATELESS` is truthy no file is written;
the
  observation log rotates to `.1` past the size cap.
- Not tested: live multi-turn provider run (unchanged from #2555, which
carried
  the live Kimi/CC proofs); docs render is Markdown/MDX only.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] New and existing checks pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- Default-on stance (in the docs): `THINKING_COMPACT` stays opt-in
(rewrites model
inputs); `COLD_RECOMPACT` is a candidate to default for Claude Code once
TTL
detection is field-validated; `CACHE_TTL_LEARN` is the safest to default
on
  (observation-only, bounded, stateless-aware) — kept opt-in for now.
This commit is contained in:
Tejas Chopra 2026-07-25 11:07:57 -07:00 • committed by GitHub
parent cb8f4b6436
commit 58555c5be0
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
4 changed files with 96 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

18
docs/content/docs/cache-optimization.mdx
View file

@ -71,3 +71,21 @@ Keep the prefix stable in your integration layer; Headroom reports when the cach
Keep the stable prefix first, keep volatile content out of it, and treat CacheAligner warnings as a signal that the caller needs to move assembly logic.
CacheAligner surfaces prefix instability, provider caches reward byte-identical prefixes, and the caller owns the actual message layout.
## Cold-prefix recompaction (when the cache lapses)
Byte-identical forwarding only pays off while the prompt cache is warm. When a
session goes idle past the provider's cache TTL, that cache is dead — so
re-sending the prefix verbatim buys nothing, and it's the one safe moment to
rewrite it. The **cold-prefix hook** detects a lapsed cache and recompacts the
whole prefix (cross-turn dedupe + superseded-read drop + lossless folds) instead,
then re-caches the smaller result. It only fires on a *confirmed*-cold turn — a
wrong call would bust a warm cache — so it reads the real TTL (exact for Claude
Code via its cache-control env vars; learned over time for other providers).
For models that re-send **reasoning** as plain text every turn (Kimi/GLM/DeepSeek),
a companion hook Kompresses that reasoning on warm turns and drops it on cold ones.
Both are off by default. See
[Cold-prefix hook & reasoning compaction](/docs/configuration#cold-prefix-hook--reasoning-compaction)
for the exact flags and when each is safe to enable.

46
docs/content/docs/configuration.mdx
View file

@ -315,9 +315,55 @@ headroom proxy --learn --min-evidence 3
| `HEADROOM_BETA_TRACKER_MAX_SESSIONS` | LRU capacity of the in-memory session beta tracker. Once full, the oldest session entry is evicted. | `1000` |
| `HEADROOM_MODEL_ROUTER_ENABLED` | Enable cost-aware model routing. `1`/`true`/`yes`/`on`/`enabled` turns it on and requires `HEADROOM_MODEL_ROUTES`. See [Cost-aware model routing](/docs/configuration#cost-aware-model-routing). | `off` |
| `HEADROOM_MODEL_ROUTES` | JSON array of ordered routing rules for cost-aware model routing (schema below). | -- |
| `HEADROOM_THINKING_COMPACT` | Compact plain-text reasoning that models re-send every turn (Kimi/GLM/DeepSeek `reasoning_content` / inline `<think>`): Kompress it on warm turns, drop it on cold turns. No-op for Claude/Codex/OpenAI (encrypted reasoning). See [Cold-prefix hook](#cold-prefix-hook--reasoning-compaction). | `off` |
| `HEADROOM_THINKING_COMPACT_KEEP_LAST` | Most-recent assistant turns whose reasoning is left intact (the active reasoning the model still uses). Only applies with `HEADROOM_THINKING_COMPACT`. | `1` |
| `HEADROOM_COLD_RECOMPACT` | On a confirmed-cold turn (idle past the real cache TTL), recompact the whole prefix — cross-turn dedupe + superseded-read drop + lossless folds — instead of forwarding a byte-identical prefix to a dead cache. Warm turns unchanged. Pair with `HEADROOM_DEDUPE`. See [Cold-prefix hook](#cold-prefix-hook--reasoning-compaction). | `off` |
| `HEADROOM_DEDUPE` | Whole-conversation verbatim cross-turn dedup in the router (cache-safe, information-preserving via retrieval markers). Superseded-read drop + lossless folds run without it; this adds verbatim dedup. | `off` |
| `HEADROOM_CACHE_TTL_LEARN` | Append per-turn cache-outcome observations (provider, model, idle, hit/miss) to `cache_ttl_observations.jsonl` for the offline `headroom-cache-ttl` learner. Observation-only (no request-behavior change); respects `HEADROOM_STATELESS`; the log is size-bounded. | `off` |
| `HEADROOM_KOMPRESS_ENDPOINT` / `HEADROOM_KOMPRESS_ENDPOINT_TOKEN` | Offload ML compression (Kompress) to a remote endpoint instead of the local ONNX model — used by reasoning compaction and the router when set. | -- |
For provider-only proxying, prefer `HEADROOM_HTTP_PROXY` over process-wide variables such as `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, or `NO_PROXY`. HTTPX reads those global variables, but Headroom also passes them through to tool executions.
### Cold-prefix hook & reasoning compaction
Two related optimizations target tokens that agent harnesses re-send every turn:
model **reasoning** (re-billed as input on Kimi/GLM/DeepSeek and on Claude 4.6+),
and the **prefix** itself once its prompt cache has lapsed. All flags below are
**off by default** — the base proxy is unchanged until you opt in.
**What to set for what:**
| Goal | Set | Notes |
|------|-----|-------|
| Shrink Kimi/GLM/DeepSeek reasoning re-sent each turn | `HEADROOM_THINKING_COMPACT=1` (+ `HEADROOM_KOMPRESS_ENDPOINT` for the best ratio) | Warm turns Kompress the reasoning; cold turns drop it. No-op for Claude/Codex/OpenAI (their reasoning is an encrypted handle, already cheap on resend). |
| Recompact a dead-cache prefix instead of re-sending it whole | `HEADROOM_COLD_RECOMPACT=1` (+ `HEADROOM_DEDUPE=1`) | Fires only when the cache has genuinely lapsed. Works in both `cache` and `token` mode. |
| Learn each provider's real cache TTL | `HEADROOM_CACHE_TTL_LEARN=1`, then run the `headroom-cache-ttl` estimator periodically | Sharpens cold detection for providers that don't expose a TTL (Kimi/OpenAI). |
**How cold detection knows the TTL.** A wrong "cold" call would recompact a
*warm* prefix and bust the cache, so the TTL must be right:
- **Claude Code:** read exactly, from the request's `cache_control.ttl` plus CC's
own controls — `ENABLE_PROMPT_CACHING_1H` (1h), `FORCE_PROMPT_CACHING_5M` (5m),
and `DISABLE_PROMPT_CACHING` (+ per-model `DISABLE_PROMPT_CACHING_<FAMILY>`),
which turns caching off and makes every turn a free recompaction candidate.
- **Kimi / OpenAI / Codex:** they don't expose a TTL, so detection uses a
conservative default until the learner (`HEADROOM_CACHE_TTL_LEARN` + the
`headroom-cache-ttl` plugin) fills in the real value from observed hits/misses.
**Can these be on by default?**
- `HEADROOM_THINKING_COMPACT` — **stays opt-in.** It rewrites model inputs
(reasoning the model actively uses) and depends on Kompress, so its quality /
latency trade should be a deliberate choice.
- `HEADROOM_COLD_RECOMPACT` — **opt-in today; a candidate to default for Claude
Code** once TTL detection is field-validated. It only fires on confirmed-cold
turns and recompacts losslessly, but a mis-read TTL would bust a warm cache
(expensive), so it waits for confidence. For Kimi/OpenAI it should stay opt-in
until the learner has data.
- `HEADROOM_CACHE_TTL_LEARN` — **the safest to default on:** observation-only, a
size-bounded local log, and it respects `HEADROOM_STATELESS`. Kept opt-in for
now so nothing is written to disk unasked.
### Cost-aware model routing
Complementary to content compression, Headroom can rewrite the upstream model per request to stretch quota and control spend, for example by sending small, tool-free requests to a cheaper model. Routing is opt-in and disabled by default, so behavior is unchanged unless you configure it.

18
headroom/cache/ttl_observations.py vendored
View file

@ -32,6 +32,10 @@ _LEARN_ENV = "HEADROOM_CACHE_TTL_LEARN"
_HEADROOM_DIR = os.path.expanduser("~/.headroom")
_OBS_DEFAULT = os.path.join(_HEADROOM_DIR, "cache_ttl_observations.jsonl")
_LEARNED_DEFAULT = os.path.join(_HEADROOM_DIR, "cache_ttl_learned.json")
# Bound the observation log so it can never grow unbounded (a prerequisite for ever
# defaulting learning on). When exceeded, rotate to ".1" (single backup); the
# estimator reads the current window, which is plenty for a TTL boundary.
_OBS_MAX_BYTES = 50 * 1024 * 1024
def observations_enabled() -> bool:
@ -57,7 +61,18 @@ def record_cache_observation(*, provider: str, model: str, attribution: Any) ->
"""
if not observations_enabled():
return
# Respect the global no-filesystem-writes switch (safe to default learning on
# without forcing writes in stateless/CI deployments).
if os.environ.get("HEADROOM_STATELESS", "").strip().lower() in _TRUTHY:
return
try:
path = _obs_path()
os.makedirs(os.path.dirname(path), exist_ok=True)
try:
if os.path.getsize(path) > _OBS_MAX_BYTES:
os.replace(path, path + ".1") # bounded: single-backup rotation
except OSError:
pass
row = {
"ts": round(time.time(), 3),
"provider": provider,
@ -69,8 +84,7 @@ def record_cache_observation(*, provider: str, model: str, attribution: Any) ->
"cache_read": int(getattr(attribution, "cache_read_tokens", 0) or 0),
"expected_cached": int(getattr(attribution, "expected_cached_tokens", 0) or 0),
}
os.makedirs(os.path.dirname(_obs_path()), exist_ok=True)
with open(_obs_path(), "a", encoding="utf-8") as f:
with open(path, "a", encoding="utf-8") as f:
f.write(json.dumps(row) + "\n")
except Exception:
pass # observability must never break a request

24
headroom/proxy/handlers/openai.py
View file

@ -3915,11 +3915,16 @@ class OpenAIHandlerMixin:
# update_from_response overwrites the prior-turn state. Feeds the
# offline TTL learner (HEADROOM_CACHE_TTL_LEARN); best-effort.
try:
if hasattr(openai_prefix_tracker, "classify_cache_miss"):
from headroom.cache.ttl_observations import (
record_cache_observation,
)
from headroom.cache.ttl_observations import (
observations_enabled,
record_cache_observation,
)
# Only run the extra attribution when learning is enabled —
# otherwise this path adds nothing over the base proxy.
if observations_enabled() and hasattr(
openai_prefix_tracker, "classify_cache_miss"
):
record_cache_observation(
provider="openai",
model=model,
@ -4236,11 +4241,14 @@ class OpenAIHandlerMixin:
# Cache-TTL learning seam (see the /v1/chat path above): record the
# cache-outcome attribution before update_from_response. Best-effort.
try:
if hasattr(openai_prefix_tracker, "classify_cache_miss"):
from headroom.cache.ttl_observations import (
record_cache_observation,
)
from headroom.cache.ttl_observations import (
observations_enabled,
record_cache_observation,
)
if observations_enabled() and hasattr(
openai_prefix_tracker, "classify_cache_miss"
):
record_cache_observation(
provider="openai",
model=model,