mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
295 lines
10 KiB
Markdown
295 lines
10 KiB
Markdown
# I Was Wasting 85% of My LLM Tokens on JSON Boilerplate
|
||
|
||
I recently built an agent to handle some SRE tasks—fetching logs, querying databases, searching code. It worked, but when I looked at the traces, I was annoyed.
|
||
|
||
It wasn't just that it was expensive (though the bill was climbing). It was the sheer **inefficiency**.
|
||
|
||
I looked at a single tool output—a search for Python files. It was 40,000 tokens.
|
||
About 35,000 of those tokens were just `"type": "file"` and `"language": "python"` repeated 2,000 times.
|
||
|
||
We are paying premium compute prices to force state-of-the-art models to read standard JSON boilerplate.
|
||
|
||
I couldn't find a tool that solved this without breaking the agent, so I wrote one. It's called **Headroom**. It's a context optimization layer that sits between your app and your LLM. It compresses context by ~85% without losing semantic meaning.
|
||
|
||
It's open source (Apache-2.0). If you just want the code:
|
||
**[github.com/chopratejas/headroom](https://github.com/chopratejas/headroom)**
|
||
|
||
---
|
||
|
||
## Why Truncation and Summarization Don't Work
|
||
|
||
When your context window fills up, the standard industry solution is **truncation** (chopping off the oldest messages or the middle of the document).
|
||
|
||
But for an agent, truncation is dangerous.
|
||
|
||
* If you chop the middle of a log file, you might lose the one error line that explains the crash.
|
||
* If you chop a file list, you might lose the exact config file the user asked for.
|
||
|
||
I tried **summarization** (using a cheaper model to summarize the data first), but that introduced hallucination. I had a summarizer tell me a deployment "looked fine" because it ignored specific error codes in the raw log.
|
||
|
||
I needed a third option: **Lossless compression.** Or at least, "intent-lossless."
|
||
|
||
---
|
||
|
||
## The Core Idea: Statistical Analysis, Not Blind Truncation
|
||
|
||
I realized that 90% of the data in a tool output is just schema scaffolding. The LLM doesn't need to see `status: active` repeated a thousand times. It needs the **anomalies**.
|
||
|
||
Headroom's SmartCrusher runs statistical analysis before touching your data:
|
||
|
||
**1. Constant Factoring**
|
||
If every item in an array has `"type": "file"`, it doesn't repeat that 2,000 times. It extracts constants once.
|
||
|
||
**2. Outlier Detection**
|
||
It calculates standard deviation of numerical fields. It preserves the spikes—the values that are >2σ from the mean. Those are usually what matters.
|
||
|
||
**3. Error Preservation**
|
||
Hard rule: never discard strings that look like stack traces, error messages, or failures. Errors are sacred.
|
||
|
||
**4. Relevance Scoring**
|
||
If you searched for "auth", items containing "auth" get preserved. Uses BM25 + semantic embeddings (hybrid scoring) to match items against the user's query context.
|
||
|
||
**5. First/Last Retention**
|
||
Always keeps first few and last few items. The LLM expects to see some examples, and recency matters.
|
||
|
||
The result: 40,000 tokens → 4,000 tokens. Same information density. No hallucination risk.
|
||
|
||
---
|
||
|
||
## CCR: Making Compression Reversible
|
||
|
||
Here's the insight that changed everything: **compression should be reversible**.
|
||
|
||
I call the architecture **CCR** (Compress-Cache-Retrieve):
|
||
|
||
### 1. Compress
|
||
SmartCrusher compresses the tool output from 2,000 items to 20.
|
||
|
||
### 2. Cache
|
||
The original 2,000 items are cached locally (5-minute TTL, LRU eviction).
|
||
|
||
### 3. Retrieve
|
||
Headroom injects a tool called `headroom_retrieve()` into the LLM's context. If the model looks at the compressed summary and decides it needs more data—maybe the user asked a follow-up question—it can call that tool. Headroom fetches from the cache and returns the relevant items.
|
||
|
||
This changes the risk calculus. You can compress aggressively (90%+) because **nothing is ever truly lost**. The model can always "unzip" what it needs.
|
||
|
||
I've had conversations like this:
|
||
|
||
```
|
||
Turn 1: "Search for all Python files"
|
||
→ 1000 files returned, compressed to 15
|
||
|
||
Turn 5: "Actually, what was that file handling JWT tokens?"
|
||
→ LLM calls headroom_retrieve("jwt")
|
||
→ Returns jwt_handler.py from cached data
|
||
```
|
||
|
||
No extra API calls. No "sorry, I don't have that information anymore."
|
||
|
||
---
|
||
|
||
## TOIN: The Network Effect
|
||
|
||
Here's where it gets interesting. Headroom learns from compression patterns.
|
||
|
||
**TOIN** (Tool Output Intelligence Network) tracks—anonymously—what happens after compression:
|
||
- Which fields get retrieved most often?
|
||
- Which tool types have high retrieval rates?
|
||
- What query patterns trigger retrievals?
|
||
|
||
This data feeds back into compression recommendations. If TOIN learns that users frequently retrieve `error_code` fields after compression, it tells SmartCrusher to preserve `error_code` more aggressively next time.
|
||
|
||
Privacy is built in:
|
||
- No actual data values stored
|
||
- Tool names are structure hashes
|
||
- Field names are SHA256[:8] hashes
|
||
- No user identifiers
|
||
|
||
The network effect: more users → more compression events → better recommendations for everyone.
|
||
|
||
---
|
||
|
||
## Memory: Cross-Conversation Learning
|
||
|
||
Agents often need to remember things across conversations. "I prefer dark mode." "My timezone is PST." "I'm working on the auth refactor."
|
||
|
||
Headroom has a memory system that extracts and stores these facts automatically.
|
||
|
||
Two approaches:
|
||
|
||
**Fast Memory (Recommended)**
|
||
Zero extra latency. The LLM outputs a `<memory>` block inline with its response. Headroom parses it out and stores the memory.
|
||
|
||
```python
|
||
from headroom.memory import with_fast_memory
|
||
client = with_fast_memory(OpenAI(), user_id="alice")
|
||
|
||
# Memories extracted automatically from responses
|
||
# Injected automatically into future requests
|
||
```
|
||
|
||
**Background Memory**
|
||
Separate LLM call extracts memories asynchronously. More accurate but adds latency.
|
||
|
||
```python
|
||
from headroom import with_memory
|
||
client = with_memory(OpenAI(), user_id="alice")
|
||
```
|
||
|
||
Memories are stored locally (SQLite) and injected into future conversations. The model remembers that Alice prefers dark mode without you managing state.
|
||
|
||
---
|
||
|
||
## The Transform Pipeline
|
||
|
||
Headroom runs four transforms on each request:
|
||
|
||
### 1. CacheAligner
|
||
LLM providers offer cached token pricing (Anthropic: 90% off, OpenAI: 50% off). But caching only works if your prompt prefix is stable.
|
||
|
||
Problem: your system prompt probably has a timestamp. `Current time: 2024-01-15 10:32:45`. That breaks caching.
|
||
|
||
CacheAligner extracts dynamic content and moves it to the end, stabilizing the prefix. Same information, better cache hits.
|
||
|
||
### 2. SmartCrusher
|
||
The statistical compression engine. Analyzes arrays, detects patterns, preserves anomalies, factors constants.
|
||
|
||
### 3. ContentRouter
|
||
Different content needs different compression. Code isn't JSON isn't logs isn't prose.
|
||
|
||
ContentRouter uses ML-based content detection to route data to specialized compressors:
|
||
- **Code** → AST-aware compression (tree-sitter)
|
||
- **JSON** → SmartCrusher
|
||
- **Logs** → LogCompressor (clusters similar messages)
|
||
- **Text** → Optional LLMLingua integration (20x compression, adds latency)
|
||
|
||
### 4. RollingWindow
|
||
When context exceeds the model limit, something has to go. RollingWindow drops oldest tool calls + responses together (never orphans data), preserves system prompt and recent turns.
|
||
|
||
---
|
||
|
||
## Three Ways to Use It
|
||
|
||
### Option 1: Proxy Server (Zero Code Changes)
|
||
|
||
```bash
|
||
pip install headroom-ai
|
||
headroom proxy --port 8787
|
||
```
|
||
|
||
Point your OpenAI client to `http://localhost:8787/v1`. Done.
|
||
|
||
```python
|
||
from openai import OpenAI
|
||
client = OpenAI(base_url="http://localhost:8787/v1")
|
||
# No other changes
|
||
```
|
||
|
||
Works with Claude Code, Cursor, any OpenAI-compatible client.
|
||
|
||
### Option 2: SDK Wrapper
|
||
|
||
```python
|
||
from headroom import HeadroomClient
|
||
from openai import OpenAI
|
||
|
||
client = HeadroomClient(OpenAI())
|
||
|
||
response = client.chat.completions.create(
|
||
model="gpt-4o",
|
||
messages=[...],
|
||
headroom_mode="optimize" # or "audit" or "simulate"
|
||
)
|
||
```
|
||
|
||
Three modes:
|
||
- **audit**: Observe only. Logs what would be optimized, doesn't change anything.
|
||
- **optimize**: Apply compression. This is what saves tokens.
|
||
- **simulate**: Dry run. Returns the optimized messages without calling the API.
|
||
|
||
Start with `audit` to see potential savings, then flip to `optimize` when you're confident.
|
||
|
||
### Option 3: Framework Integrations
|
||
|
||
**LangChain:**
|
||
```python
|
||
from langchain_openai import ChatOpenAI
|
||
from headroom.integrations.langchain import HeadroomChatModel
|
||
|
||
base_model = ChatOpenAI(model="gpt-4o")
|
||
model = HeadroomChatModel(base_model, mode="optimize")
|
||
|
||
# Use in any chain or agent
|
||
chain = prompt | model | parser
|
||
```
|
||
|
||
**Agno:**
|
||
```python
|
||
from agno.agent import Agent
|
||
from headroom.integrations.agno import HeadroomAgnoModel
|
||
|
||
model = HeadroomAgnoModel(original_model, mode="optimize")
|
||
agent = Agent(model=model, tools=[...])
|
||
```
|
||
|
||
**MCP (Model Context Protocol):**
|
||
```python
|
||
from headroom.integrations.mcp import compress_tool_result
|
||
|
||
# Compress any tool result before returning to LLM
|
||
compressed = compress_tool_result(tool_name, result_data)
|
||
```
|
||
|
||
---
|
||
|
||
## Real Numbers
|
||
|
||
I've been running this in production for months. Here's what the token reduction looks like:
|
||
|
||
| Workload | Before | After | Savings |
|
||
|----------|--------|-------|---------|
|
||
| Log Analysis | 22,000 | 3,300 | 85% |
|
||
| Code Search | 45,000 | 4,500 | 90% |
|
||
| Database Queries | 18,000 | 2,700 | 85% |
|
||
| Long Conversations | 80,000 | 32,000 | 60% |
|
||
|
||
Latency overhead: 3-5ms per request. No extra LLM calls.
|
||
|
||
---
|
||
|
||
## What's Coming Next
|
||
|
||
This is actively maintained. On the roadmap:
|
||
|
||
**More Frameworks**
|
||
- CrewAI integration
|
||
- AutoGen integration
|
||
- Semantic Kernel integration
|
||
|
||
**Managed Storage**
|
||
- Cloud-hosted TOIN backend (opt-in)
|
||
- Cross-device memory sync
|
||
- Team-shared compression patterns
|
||
|
||
**Better Compression**
|
||
- Domain-specific profiles (SRE, coding, data analysis)
|
||
- Custom compressor plugins
|
||
- Streaming compression for real-time tools
|
||
|
||
---
|
||
|
||
## Why I Built This
|
||
|
||
I'm a believer that we're in the "optimization phase" of the AI hype cycle. Getting things to work is table stakes; getting them to work cheaply and reliably is the actual engineering work.
|
||
|
||
Headroom is my attempt to fix the "context bloat" problem properly. Not with heuristics or truncation, but with statistical analysis and reversible compression.
|
||
|
||
It runs entirely locally. No data leaves your machine (except to OpenAI/Anthropic as usual). Apache-2.0 licensed.
|
||
|
||
**Repo:** [github.com/chopratejas/headroom](https://github.com/chopratejas/headroom)
|
||
|
||
If you find bugs or have ideas, open an issue. I'm actively maintaining this.
|
||
|
||
---
|
||
|
||
*Tags: #llm #ai #python #openai #anthropic #agents #optimization*
|