fix(learn): parse fenced JSON even with a prose preamble (#1988)

## Description

`_strip_fenced_json` only stripped a markdown fence when the string
*started with* ```` ``` ````. When the model prefixed prose before the
fence (e.g. `Here is the JSON:\n\n```json ...`) despite being told to
return JSON only, the guard was skipped and `json.loads` ran on the
prose, raising `JSONDecodeError`. The claude-cli streaming path surfaced
this as `returned unparseable output`, and `headroom learn` silently
discarded the LLM analysis, degrading to "No actionable patterns found".
This is the parsing-side cousin of the silent-degradation issue fixed in
#373.

Closes #1989. Related: #373.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/learn/analyzer.py`: rewrote `_strip_fenced_json` to locate
the fenced block wherever it appears, then fall back to the whole text,
then to a first-`{` / last-`}` slice, only re-raising `JSONDecodeError`
if nothing parses as a JSON object. Preserves the prior "first opening /
last closing fence" behaviour and triple-backtick content inside the
payload. Fixes all three call sites (non-streaming CLI, claude-cli
streaming, litellm).
- `tests/test_learn/test_analyzer.py`: added regression cases to
`TestStripFencedJson` for preamble-before-fence, prose around a bare
object, and triple-backticks inside the payload.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`) — scoped to the changed
module (see Additional Notes)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_learn/test_analyzer.py -q
........................................................................ [ 86%]
...........                                                              [100%]
83 passed, 1 warning in 2.18s

$ ruff check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
All checks passed!
$ ruff format --check headroom/learn/analyzer.py tests/test_learn/test_analyzer.py
2 files already formatted

$ mypy --ignore-missing-imports --follow-imports=silent headroom/learn/analyzer.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS, Python 3.13.12, headroom-ai at this branch
(runtime deps from an installed 0.30.0 env).
- Exact command / steps: ran the old vs new `_strip_fenced_json` on the
exact failing model output (a prose preamble followed by a ```json
fence), then applied the fix over an installed 0.30.0 and re-ran the
previously failing `headroom learn --apply`. Input sample: `'The JSON is
my deliverable for this analysis task. Here it
is:\n\n```json\n{"context_file_rules": [], "memory_file_rules":
[]}\n```'`
- Observed result: OLD raised `JSONDecodeError: Expecting value: line 1
column 1 (char 0)`; NEW returned `{'context_file_rules': [],
'memory_file_rules': []}`. The real `headroom learn --apply` run that
had been failing with `returned unparseable output` then completed and
consumed the LLM analysis instead of dropping it. Full transcript:
  ```text
  OLD: JSONDecodeError -> Expecting value: line 1 column 1 (char 0)
  NEW: {'context_file_rules': [], 'memory_file_rules': []}
  ```
- Not tested: full end-to-end `headroom learn --apply` was not re-run
inside CI here (it shells out to a live `claude` CLI); the parser is
exercised deterministically by the added unit tests and the before/after
repro above.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A —
updated the function docstring only; no external docs affected)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable (N/A — no CHANGELOG
entry convention observed for this fix; happy to add if maintainers
prefer)

## Additional Notes

- `mypy` was run against the changed module in isolation
(`--ignore-missing-imports --follow-imports=silent`) rather than the
full project, because I validated in an ad-hoc environment; the change
keeps the existing `-> dict` signature and annotations, so it is
type-neutral.
- Not addressed here (possible follow-up): the failure is swallowed as a
warning in `analyze()`, so users only see "No actionable patterns found"
with no signal the LLM pass produced nothing — the same
silent-degradation class as #373, on the parsing side.

headroom/learn/analyzer.py

@@ -470,28 +470,50 @@ Return ONLY valid JSON matching this schema — no other text:
 def _strip_fenced_json(raw: str) -> dict:
     """Strip optional markdown fences and parse JSON.
 
-    Handles both raw JSON and fenced code blocks (e.g. ``​`json ... ``​`).
-    Only the first opening fence and last closing fence are removed, preserving
-    any triple-backtick content that may appear inside the JSON payload.
+    Handles raw JSON and fenced code blocks (e.g. ``​`json ... ``​`), including
+    the case where the model prefixes prose before the fence (e.g. "Here is the
+    JSON:") despite being told to return JSON only. Between the first opening
+    fence and last closing fence is preferred, preserving any triple-backtick
+    content inside the JSON payload; a first-``{`` / last-``}`` slice is the
+    final fallback.
 
     Args:
-        raw: Raw text output from an LLM, possibly wrapped in markdown fences.
+        raw: Raw text output from an LLM, possibly wrapped in markdown fences
+            and/or preceded by explanatory prose.
 
     Returns:
         Parsed JSON as a dictionary.
 
     Raises:
-        json.JSONDecodeError: If the text is not valid JSON after stripping.
+        json.JSONDecodeError: If no candidate parses as a JSON object.
     """
     text = raw.strip()
-    if text.startswith("```"):
-        lines = text.split("\n")
-        # Remove the first line (opening fence, e.g. ```json)
-        lines = lines[1:]
-        # Remove the last line if it is a closing fence
-        if lines and lines[-1].strip().startswith("```"):
-            lines = lines[:-1]
-        text = "\n".join(lines)
+
+    candidates: list[str] = []
+    # 1. Fenced block located anywhere (tolerates a prose preamble before it).
+    lines = text.split("\n")
+    fence_idxs = [i for i, ln in enumerate(lines) if ln.strip().startswith("```")]
+    if len(fence_idxs) >= 2:
+        candidates.append("\n".join(lines[fence_idxs[0] + 1 : fence_idxs[-1]]))
+    elif len(fence_idxs) == 1:
+        candidates.append("\n".join(lines[fence_idxs[0] + 1 :]))
+    # 2. The whole text as-is (the common raw-JSON case).
+    candidates.append(text)
+    # 3. First-``{`` .. last-``}`` slice (prose on both sides, no fence).
+    start, end = text.find("{"), text.rfind("}")
+    if start != -1 and end > start:
+        candidates.append(text[start : end + 1])
+
+    for candidate in candidates:
+        try:
+            parsed = json.loads(candidate)
+        except json.JSONDecodeError:
+            continue
+        if isinstance(parsed, dict):
+            return parsed
+
+    # Nothing parsed as an object: re-raise the natural error on the raw text
+    # so callers see a JSONDecodeError, preserving the documented contract.
     result: dict = json.loads(text)
     return result
 

tests/test_learn/test_analyzer.py

@@ -592,6 +592,23 @@ class TestStripFencedJson:
         with pytest.raises(json.JSONDecodeError):
             _strip_fenced_json("not json at all")
 
+    def test_prose_preamble_before_fence(self):
+        # Models sometimes add a preamble before the fence despite being told
+        # to return JSON only (e.g. "Here it is:\n\n```json ...").
+        raw = 'The JSON is my deliverable. Here it is:\n\n```json\n{"key": "value"}\n```'
+        result = _strip_fenced_json(raw)
+        assert result == {"key": "value"}
+
+    def test_prose_around_bare_object(self):
+        raw = 'Sure, here you go: {"key": "value"} hope that helps!'
+        result = _strip_fenced_json(raw)
+        assert result == {"key": "value"}
+
+    def test_triple_backtick_inside_payload(self):
+        raw = '```json\n{"note": "run ```code``` here", "n": 1}\n```'
+        result = _strip_fenced_json(raw)
+        assert result == {"note": "run ```code``` here", "n": 1}
+
 
 def _fake_claude_popen(
     *,