headroom/docs/memory.md

Memory

Persistent memory for LLM applications. Enable your AI to remember across conversations without carrying full history.

Why Memory?

LLMs have two fundamental limitations:

1. Context windows overflow - Too much history, need to truncate
2. No persistence - Every conversation starts from zero

Memory solves both: extract key facts, persist them, inject when relevant.

This is temporal compression - instead of carrying 10,000 tokens of conversation history, carry 100 tokens of extracted memories.

---

Quick Start

Zero-Latency Memory (Recommended)

from openai import OpenAI
from headroom.memory import with_fast_memory

# One line - that's it
client = with_fast_memory(OpenAI(), user_id="alice")

# Use exactly like normal
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "I prefer Python for backend work"}]
)
# Memory extracted INLINE - zero extra latency

# Later, in a new conversation...
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What language should I use?"}]
)
# → Response uses the Python preference from memory

How It Works

┌─────────────────────────────────────────────────────────────┐
│                    with_fast_memory()                        │
│                                                              │
│   1. INJECT: Search memories → prepend to user message      │
│   2. INSTRUCT: Add memory extraction instruction            │
│   3. CALL: Forward to LLM                                   │
│   4. PARSE: Extract <memory> block from response            │
│   5. STORE: Save memories with embeddings                   │
│   6. RETURN: Clean response (without memory block)          │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Key insight: Memory extraction happens inline as part of the LLM response. No extra API calls, no extra latency.

---

Two Approaches

1. Fast Memory (Inline Extraction)

from headroom.memory import with_fast_memory

client = with_fast_memory(
    OpenAI(),
    user_id="alice",
    db_path="memory.db",           # SQLite storage
    top_k=5,                       # Memories to inject
    use_local_embeddings=True,    # Local model (fast) vs OpenAI API
)

Characteristics:

• Zero extra latency (extraction is part of response)
• ~100 extra output tokens per response
• Smart extraction (LLM decides what's important)
• Semantic retrieval (vector similarity)

2. Background Memory (Separate Extraction)

from headroom.memory import with_memory

client = with_memory(
    OpenAI(),
    user_id="alice",
    db_path="memory.db",
)

Characteristics:

• Non-blocking (extraction happens in background worker)
• Separate LLM call for extraction
• Good when you don't want to modify responses

---

Memory API

Both wrappers provide a .memory API for direct access:

client = with_fast_memory(OpenAI(), user_id="alice")

# Search memories
results = client.memory.search("python preferences", top_k=5)
for memory, score in results:
    print(f"{score:.2f}: {memory.text}")

# Add manual memory
client.memory.add("User is a senior engineer", category="fact")

# Get all memories
all_memories = client.memory.get_all()

# Clear memories
client.memory.clear()

# Get stats
stats = client.memory.stats()
print(f"Total memories: {stats['total_chunks']}")

---

Memory Categories

Memories are categorized for better organization:

Category	Description	Examples
preference	Likes, dislikes, preferred approaches	"Prefers Python", "Likes async/await"
fact	Identity, role, constraints	"Works at fintech startup", "Senior engineer"
context	Current goals, ongoing tasks	"Migrating to microservices", "Working on auth"

---

Configuration

Storage

# SQLite (default, local)
client = with_fast_memory(OpenAI(), user_id="alice", db_path="memory.db")

# Custom path
client = with_fast_memory(OpenAI(), user_id="alice", db_path="/data/memories.db")

Embeddings

# Local embeddings (recommended - fast, free)
client = with_fast_memory(
    OpenAI(),
    user_id="alice",
    use_local_embeddings=True,
    embedding_model="all-MiniLM-L6-v2",  # 384 dimensions
)

# OpenAI embeddings (higher quality, costs money)
client = with_fast_memory(
    OpenAI(),
    user_id="alice",
    use_local_embeddings=False,  # Uses text-embedding-3-small
)

Retrieval

# Number of memories to inject
client = with_fast_memory(
    OpenAI(),
    user_id="alice",
    top_k=10,  # Inject up to 10 relevant memories
)

---

Multi-User Isolation

Memories are isolated by user_id:

# Alice's memories
alice_client = with_fast_memory(OpenAI(), user_id="alice")

# Bob's memories (completely separate)
bob_client = with_fast_memory(OpenAI(), user_id="bob")

# Agent memories
agent_client = with_fast_memory(OpenAI(), user_id="agent-researcher")

---

How Memory Enables Compression

Memory is temporal compression. Instead of carrying full conversation history:

WITHOUT MEMORY:
Context = Turn 1 + Turn 2 + ... + Turn 50 = 10,000 tokens

WITH MEMORY:
Context = 5 relevant memories = 100 tokens
Compression ratio: 100x

This lets you use aggressive rolling window truncation while preserving important facts.

from headroom.memory import with_fast_memory
from headroom.transforms import RollingWindowTransform

# Memory + aggressive truncation = best of both worlds
client = with_fast_memory(OpenAI(), user_id="alice")
transform = RollingWindowTransform(max_tokens=4000)

# Old messages get truncated, but key facts live in memory
messages = transform.apply(very_long_conversation)
response = client.chat.completions.create(model="gpt-4o", messages=messages)

---

Performance

Operation	Latency	Notes
Memory injection	<50ms	Local embeddings + vector search
Memory extraction	+50-100ms	Part of LLM response (inline)
Memory storage	<10ms	SQLite write + cache update

Overhead: ~100 extra output tokens per response for the <memory> block.

---

Providers

Memory works with any OpenAI-compatible client:

from openai import OpenAI
from anthropic import Anthropic
from groq import Groq

# OpenAI
client = with_fast_memory(OpenAI(), user_id="alice")

# Anthropic (via OpenAI-compatible wrapper)
client = with_fast_memory(OpenAI(base_url="..."), user_id="alice")

# Groq
client = with_fast_memory(Groq(), user_id="alice")

# Any OpenAI-compatible client
client = with_fast_memory(YourClient(), user_id="alice")

---

Example: Multi-Turn Conversation

from openai import OpenAI
from headroom.memory import with_fast_memory

client = with_fast_memory(OpenAI(), user_id="developer_jane")

# Conversation 1: User shares context
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{
        "role": "user",
        "content": "I'm a Python developer at a fintech startup. We use PostgreSQL."
    }]
)
# Memories extracted: "Python developer", "fintech startup", "uses PostgreSQL"

# Conversation 2 (new session): User asks question
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{
        "role": "user",
        "content": "What database should I use for my new project?"
    }]
)
# Response references PostgreSQL preference from memory
print(response.choices[0].message.content)
# → "Given your experience with PostgreSQL at your fintech company..."

---

Troubleshooting

Memories not being extracted

1. Check if the conversation has memory-worthy content (not just greetings)
2. Verify the LLM is following the memory instruction
3. Check logs for parsing errors

Memories not being retrieved

1. Verify user_id matches between sessions
2. Check if memories exist: client.memory.get_all()
3. Try a more specific search query

High latency

1. Switch to local embeddings: use_local_embeddings=True
2. Reduce top_k for fewer memories to retrieve
3. Check database size and consider pruning old memories

---

Best Practices

1. Use consistent user_id - Same ID across sessions for continuity
2. Start with local embeddings - Faster, free, good enough for most cases
3. Combine with rolling window - Memory + truncation = aggressive compression
4. Monitor memory growth - Periodically review and prune if needed
5. Use categories - Helps with debugging and selective retrieval