mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
14e8dc4c84
## Description `headroom learn` ranked recommendations by a single LLM-guessed `estimated_tokens_saved` with a flat hardcoded `confidence`, and had **no notion of a loop**. So (1) RTK re-fetch loops were invisible - RTK truncates a command's output, the agent re-runs larger-limit variants, those calls *succeed* (`is_error=False`), and `analyze()` even early-returned when a session had no failures and no events - and (2) even when surfaced, a loop ranked no higher than a one-off mistake. This adds loop-aware weighting plus the eval that reproduces an RTK loop, runs it through Learn, and checks the guardrail prevents re-triggering. Closes #1159 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - New `headroom/learn/loops.py`: `detect_loops()` (canonical signature collapses RTK pagination/limit variants; classifies error vs rtk-refetch loops; **measured** wasted tokens), `format_loops_for_digest()`, `apply_loop_weighting()`. - `analyzer.py`: detect loops up front (fixes the no-failure early-return), lead the digest with them, prioritize loops in the system prompt, re-sort after weighting. - `models.py`: `Recommendation.is_loop_guardrail` / `loop_occurrences`. - `benchmarks/rtk_loop_learn_eval.py` + `headroom/learn/fixtures.py`: the two-phase RTK-loop eval and its session fixtures. - Tests, `docs/rtk-loop-weighting.md`, CHANGELOG entry. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - not run (mypy not in my minimal env; see Not tested) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_learn/ -q 190 passed, 3 skipped, 1 warning in 5.85s $ ruff check <changed files> All checks passed! ``` ## Real Behavior Proof - Environment: macOS (Darwin 25.0), Python 3.10.18, fresh venv (`pip install -e` minus the optional `hnswlib`/proxy extras, which are unrelated to `learn`); real LLM via the analyzer's claude CLI backend (`HEADROOM_LEARN_CLI=claude`, claude-cli 2.1.158) — no API key used. - Exact command / steps: `HEADROOM_LEARN_CLI=claude python -c "from benchmarks.rtk_loop_learn_eval import run_eval; c=run_eval(use_real_llm=True); print(c.render())"` - Observed result: the analyzer shelled out to a real model and produced the "Commands" guardrail quoted below, naming the looping command. The digest reports the measured 5,005-token waste and asks the model to rank loops first, so the model emitted that figure; in this run the guardrail ranked **#1** and the scorecard was all-PASS (below). Caveat — real-mode is run-dependent: the rule's wording, and whether the post-hoc `apply_loop_weighting` fuzzy match fires, vary across runs (in one run it did not tag the rule). The **deterministic CI eval** (stub LLM) is the stable, reproducible artifact; this real run corroborates it. - Not tested: the analyzer's API-key path (ANTHROPIC/OPENAI/GEMINI) — exercised the equivalent claude CLI backend instead; `mypy`; a live agent *obeying* the written rule end-to-end (Phase 2 is a non-recurrence check, not a live agent — called out in the doc). Real model output from this run, ranked #1 at the measured 5,005-token weight: > **Commands** — When grepping logs (or any large file), never loop with increasing `| head -N` limits — tool output is capped at ~4 KB regardless of N, so repeated attempts return identical bytes. Instead: redirect to a temp file (`grep ... > /tmp/out.txt`) then read it, or use `grep -c` first… ```text [PASS] loop_detected (1 loop(s), ~5,005 tok wasted) [PASS] guardrail_produced [PASS] ranked_first [PASS] names_command [PASS] prescribes_fix [PASS] weight_reflects_waste [PASS] guardrail_holds RESULT: PASS ``` (One real-mode run via the claude CLI backend. The deterministic `pytest` eval above is the stable artifact; see the run-dependence caveat under Observed result.) The real run also caught an over-brittle check: an earlier `names_command` required the literal "TimeoutError"; the real model wrote a *more general* rule (grep + `head -N`) without it, so I fixed the check to verify the looping **command** is named, not an incidental literal. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - No new dependencies. No network, no user/assistant content dropped — operates on already-captured session digests. - Kept as one logical change. mypy not run locally (minimal env); happy to address anything CI's mypy flags. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
27 lines
794 B
Python
27 lines
794 B
Python
"""CI wrapper for the RTK-loop eval (benchmarks/rtk_loop_learn_eval.py).
|
|
|
|
The deterministic path runs everywhere and gates the loop-weighting behavior
|
|
end-to-end. The real-LLM path is opt-in via the repo's ``real_llm`` marker and
|
|
only runs when an API key is present.
|
|
"""
|
|
|
|
import os
|
|
|
|
import pytest
|
|
|
|
from benchmarks.rtk_loop_learn_eval import run_eval
|
|
|
|
|
|
def test_rtk_loop_eval_deterministic():
|
|
card = run_eval(use_real_llm=False)
|
|
assert card.passed, "RTK-loop eval failed:\n" + card.render()
|
|
|
|
|
|
@pytest.mark.real_llm
|
|
@pytest.mark.skipif(
|
|
not os.environ.get("ANTHROPIC_API_KEY"),
|
|
reason="real_llm eval needs ANTHROPIC_API_KEY",
|
|
)
|
|
def test_rtk_loop_eval_real_llm():
|
|
card = run_eval(use_real_llm=True)
|
|
assert card.passed, "RTK-loop eval (real LLM) failed:\n" + card.render()
|