## Description
`headroom learn` with the claude-cli backend used `subprocess.run` with
a hard 120s wall-clock cap and no liveness signal. A successful long
analysis and a hung connection looked identical — exit 0 with "0
recommendations" was the only user-visible signal when the LLM call
timed out, which silently hides genuine learnings.
This PR makes the CLI backend timeout-aware, with progress detection for
claude-cli and configurable wall-clock caps for every backend.
Fixes #(issue number)
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **Streaming claude-cli with idle timeout**: invoke `claude -p
--output-format stream-json --verbose` and run a watchdog loop that
drains stdout/stderr via reader threads. Each stream-json event resets
an idle deadline. Kill the process if no output for
`HEADROOM_LEARN_CLI_IDLE_TIMEOUT_SECS` (default 60s) or if total elapsed
exceeds `HEADROOM_LEARN_CLI_TIMEOUT_SECS` (default 300s, was 120s). The
final `type:"result"` event carries the assistant response, which is
then parsed as JSON. Reader threads (rather than `select`) are used so
the watchdog works on Windows where `select` does not support pipe
handles.
- **Bumped default `_CLI_TIMEOUT` from 120s to 300s** as the hard cap
for all CLI backends. The previous 120s was too tight for large digests
on slower networks.
- **Env-var overrides** via new helper `_resolve_timeout_secs(env_var,
default)`:
- `HEADROOM_LEARN_CLI_TIMEOUT_SECS` — hard wall-clock cap (all CLI
backends)
- `HEADROOM_LEARN_CLI_IDLE_TIMEOUT_SECS` — idle cap (streaming
claude-cli only)
- Non-positive or non-integer values log a warning and fall back to
defaults, so a typo can't disable the timeout.
- **gemini-cli and codex-cli** keep `subprocess.run(timeout=hard_cap)`
since they do not emit progress events. They benefit from the bumped
default and the env-var override.
- **CHANGELOG.md** updated under `[Unreleased]` → `### Fixed`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed (existing repro: 16k-call digest that
previously timed out at 120s)
New test coverage in `tests/test_learn/test_analyzer.py`:
- `test_claude_cli_streams_and_parses_result_event` — happy path, fake
Popen yields system/assistant/result events
- `test_claude_cli_parses_fenced_result` — markdown fences in the result
event still parse
- `test_claude_cli_idle_timeout_kills_hang` —
`HEADROOM_LEARN_CLI_IDLE_TIMEOUT_SECS=1` + a hanging stdout iterator
triggers the idle watchdog
- `test_claude_cli_hard_cap_kills_continuous_chatter` — continuous
events with a low hard cap fire the wall-clock kill (proves idle reset
alone can't keep a runaway alive)
- `test_claude_cli_missing_result_event_raises` — graceful failure when
no `result` event is emitted
- `test_claude_cli_nonzero_exit_raises` /
`test_claude_cli_unparseable_result_raises_with_context` /
`test_claude_cli_not_installed_raises` — error paths
- Parallel codex-cli error coverage (timeout-honors-env-override
included) so the wall-clock path is exercised
- `TestResolveTimeoutSecs` — unset / empty / non-integer / non-positive
/ valid override
## Test Output
```
$ uv run pytest tests/test_learn/test_analyzer.py
============================== 67 passed in 2.14s ==============================
$ uv run ruff check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
All checks passed!
$ uv run ruff format --check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
2 files already formatted
$ uv run mypy headroom/learn/analyzer.py
Success: no issues found in 1 source file
```
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- The contract assumed for claude-cli stream-json output is: each line
is a JSON object with a `type` field; the final event has
`type:"result"` with a string `result` field carrying the assistant
text. This matches the documented Anthropic CLI behavior. If the
contract changes upstream, `_call_claude_cli_streaming` raises a clear
"did not emit a final \`result\` event" error rather than silently
succeeding.
- Backwards-compatible for users without env-var configuration: behavior
just becomes "longer hard cap, plus idle watchdog for claude-cli",
neither of which can falsely succeed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Google deprecated gemini/gemini-2.0-flash; headroom learn silently fails
when GEMINI_API_KEY is set. PR #532 updated the default in analyzer.py
but left stale references in the CLI help text and unit test assertion.
When `headroom learn` re-surfaced a section heading that already existed
in CLAUDE.md / MEMORY.md, the writer replaced that section wholesale —
but the LLM never saw the prior block, so it emitted condensed bullets
like "X is *also* large — same rule as Y, Z" assuming Y and Z would
remain siblings. After replacement, Y and Z were gone and the "also"
dangled.
This threads the project's current `<!-- headroom:learn -->` block (from
both CLAUDE.md and MEMORY.md) into the digest as a "Prior Learned
Patterns" section, and extends the system prompt to make the re-emission
contract explicit: re-stating a section replaces it wholesale, so the
LLM must copy forward prior bullets it still agrees with. Prior sections
the LLM omits entirely are still carried forward by the writer (#231
behavior preserved as a safety net).
Changes:
- New `extract_marker_block(file_content)` helper in `learn.writer` that
returns the raw marker block (delimiters included) or None.
- New `_build_prior_patterns_section(project)` in `learn.analyzer` reads
`project.context_file` and `project.memory_file` via the new helper
and formats a labeled section ahead of the per-session event stream.
- `_build_digest` emits the prior-patterns section when present; char
budget accounting unchanged (prior blocks are small).
- `_SYSTEM_PROMPT` gains a "Prior Learned Patterns" rule block telling
the LLM how to integrate prior bullets (preserve / revise / drop-only-
if-contradicted) and warning against unresolved cross-references.
- Tests: 6 new `TestPriorPatternsInjection` cases (present/absent files,
no-marker-block, both-files, end-to-end via mocked `_call_llm`); 4 new
`TestExtractMarkerBlock` cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When --model codex-cli is used but codex is not in PATH,
subprocess.run raises FileNotFoundError. Catch it and raise
a clear RuntimeError with guidance.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Allow `headroom learn` to use locally installed coding agent CLIs
(claude, gemini, codex) as LLM backends, so subscription users
without raw API keys can run failure analysis.
Priority: --model flag > API key > HEADROOM_LEARN_CLI env var > auto-detect
- Pass prompts via stdin to avoid ARG_MAX limits
- Handle TimeoutExpired, truncate stderr, enrich JSONDecodeError
- Add 31 new tests (48 total), all passing
- Update docs/learn.md with CLI backend documentation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
SessionAnalyzer() without a model calls _detect_default_model() which
raises when no API keys are set (e.g., in CI). Pass model="test-model"
in the three tests that mock _call_llm.
Proxy performance logging (`headroom perf`):
- Add always-on RotatingFileHandler to ~/.headroom/logs/proxy.log (10MB x 5 backups)
- Replace scattered log lines with structured PERF lines containing model, msgs,
tok_before/after/saved, cache_read/write/hit_pct, opt_ms, and transforms
- Emit PERF lines from all three response paths (streaming Anthropic, non-streaming
Anthropic, Bedrock streaming)
- Add `headroom perf` CLI that parses proxy logs and reports token savings, cache
hit rates, prefix stability, transform effectiveness, routing breakdown, TOIN
status, and actionable recommendations
- Support --hours and --raw flags for time filtering and raw record output
Learn module rewrite (LLM-based analysis):
- Replace all regex/heuristic analyzers with a single LLM call via LiteLLM
- New SessionAnalyzer builds compact digests and sends to any of 100+ models
- Auto-detect best model from API keys (Anthropic → OpenAI → Gemini)
- Add --model flag for explicit model selection
- Enrich scanner with SessionEvent (user messages, interruptions, subagent summaries),
token usage tracking, and timestamps
- Simplify models: remove EnvironmentFact, StructureNote, Correction, CommandPattern,
RetryPattern, AnalysisReport; add SessionEvent, AnalysisResult
- Simplify writer: remove Recommender class (LLM now produces recommendations directly)
- Update tests for new analyzer and models
Analyzes past conversation history to find tool call failure patterns,
correlates each failure with what eventually succeeded, and writes
specific project-level learnings to CLAUDE.md and MEMORY.md.
Key design:
- Success correlation: extracts the diff between failed and successful
inputs as the learning (not generic advice)
- Generic architecture: tool-agnostic ToolCall model with pluggable
Scanner/Writer adapters (Claude Code first, extensible to Cursor/Codex)
- 5 analyzers: Environment, Structure, Commands, Retries, Cross-Session
- Dry-run by default, --apply to write, --all for all projects
Also fixes mypy errors in litellm_callback, asgi, langchain chat_model,
and anthropic provider (AsyncClient typing, ToolCall arg-type, int cast).
