mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
6129808462
## Description Fixes #1202. On a Windows (cp1252) locale, `headroom learn` cannot complete a run: the whole pipeline opens transcript files and pipes analyzer prompts without `encoding="utf-8"`, so any non-ASCII byte (em-dashes, arrows — ubiquitous in code and prose) breaks it. Same bug class already fixed for `headroom wrap` (#65, #1126) and the dashboard (#533), never swept through `learn`. Three independent failure points, each hidden behind the previous: 1. **Reading transcripts** — six bare `open()` calls in the learn plugins. The **Codex** JSONL scanner caught only `OSError`, so a `UnicodeDecodeError` propagated and **aborted the whole cross-agent run**; the **Claude** scanner caught it and **silently dropped the session**. `analyzer.py` also read the user's own CLAUDE.md/MEMORY.md with no encoding. 2. **Analyzer subprocess** — `subprocess.run`/`Popen(..., text=True)` with no encoding raised `UnicodeEncodeError` on the piped prompt; it was swallowed, so the run produced **0 recommendations** with no obvious failure. 3. **`--apply` merge** — `writer.py` read the existing context file with strict `encoding="utf-8"`, which aborts on a single stray legacy byte. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `learn/plugins/{claude,codex,gemini}.py`: add `encoding="utf-8", errors="replace"` to the six transcript `open()` calls. - `learn/analyzer.py`: same on the `read_text` of the user's context files and on both analyzer subprocess calls (`subprocess.run` and `Popen`). - `learn/writer.py`: add `_read_text_tolerant` — decode the to-be-rewritten context file as UTF-8, falling back to UTF-8-with-replacement on a stray byte (a whole-file cp1252 fallback is wrong: it mojibakes genuine UTF-8 em-dashes); the subsequent `write_text(encoding="utf-8")` self-heals the file. - `cli/learn.py`: wrap `plugin.scan_project` so one unreadable agent/project is skipped with a warning instead of aborting the whole run. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_learn/test_writer.py tests/test_learn/test_plugin_encoding.py -q 22 passed $ ruff check headroom/learn/plugins/*.py headroom/learn/analyzer.py \ headroom/learn/writer.py headroom/cli/learn.py tests/test_learn/test_*.py All checks passed! ``` New tests are **red on the old code, green with the fix**: - `test_plugin_encoding.py` — a transcript with a stray `0x9d` byte (undefined in cp1252 *and* an invalid UTF-8 start byte, so a bare `open()` fails on any locale): the Codex scanner no longer raises, the Claude scanner now recovers the session instead of dropping it. - `test_writer.py::TestEncodingResilience` — `_read_text_tolerant` preserves valid UTF-8 (no mojibake) and `--apply` merges over a file with a stray byte. ## Real Behavior Proof - Environment: Windows 11, Python 3.10, against the real learn plugins/writer (no live LLM backend; the decode failures occur before any backend call). - Exact command / steps: write a Claude transcript and a Codex rollout JSONL containing a valid em-dash/arrow line plus a stray `0x9d` byte, then call `ClaudeCodePlugin._scan_session` / `CodexPlugin._scan_jsonl_session`; for the writer, `write_bytes` an `AGENTS.md` with a stray `0x97` and run `_merge_into_file`. - Observed result: **before** the fix → `CodexPlugin._scan_jsonl_session` raises `UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9d` (aborts the run) and `ClaudeCodePlugin._scan_session` returns `None` (session dropped); **after** → Codex completes, Claude returns the `SessionData` (`total_input_tokens == 5`), and `_merge_into_file` keeps `Notes — existing` with no mojibake. - Not tested: a full end-to-end `headroom learn --apply` against live agent histories + a real LLM backend (verified at the plugin/writer level, which is where the decode failures live). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review
51 lines
2 KiB
Python
51 lines
2 KiB
Python
"""Regression tests for #1202 — the ``learn`` session scanners must read agent
|
|
transcripts as UTF-8 with replacement, so a stray non-UTF-8 byte cannot abort
|
|
(or silently drop) a scan.
|
|
|
|
``0x9d`` is undefined in cp1252 *and* an invalid UTF-8 start byte, so a bare
|
|
``open()`` fails on it regardless of the host locale. Before the fix this made
|
|
the Codex JSONL scanner raise ``UnicodeDecodeError`` (the scan caught only
|
|
``OSError``), aborting the whole cross-agent run, while the Claude scanner
|
|
caught it and silently dropped the session.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
from pathlib import Path
|
|
|
|
from headroom.learn.models import SessionData
|
|
from headroom.learn.plugins.claude import ClaudeCodePlugin
|
|
from headroom.learn.plugins.codex import CodexPlugin
|
|
|
|
|
|
def _stray_byte_line() -> bytes:
|
|
# A line that is neither valid UTF-8 nor decodable in cp1252.
|
|
return b"\x9d arrow \xe2\x86\x92 junk\n"
|
|
|
|
|
|
def test_claude_scan_recovers_session_with_stray_byte(tmp_path: Path) -> None:
|
|
jsonl = tmp_path / "session.jsonl"
|
|
valid = json.dumps(
|
|
{"type": "assistant", "message": {"usage": {"input_tokens": 5}}, "text": "em — arrow →"}
|
|
)
|
|
jsonl.write_bytes(valid.encode() + b"\n" + _stray_byte_line())
|
|
|
|
result = ClaudeCodePlugin(claude_dir=tmp_path)._scan_session(jsonl)
|
|
|
|
# Before the fix this returned None (session silently dropped); now the
|
|
# valid line is read and the stray-byte line is skipped, not fatal.
|
|
assert result is not None
|
|
assert result.session_id == "session"
|
|
assert result.total_input_tokens == 5
|
|
|
|
|
|
def test_codex_jsonl_scan_does_not_crash_on_stray_byte(tmp_path: Path) -> None:
|
|
jsonl = tmp_path / "rollout.jsonl"
|
|
meta = json.dumps({"type": "session_meta", "payload": {"id": "abc"}})
|
|
jsonl.write_bytes(meta.encode() + b"\n" + _stray_byte_line())
|
|
|
|
# Before the fix this raised UnicodeDecodeError and aborted the run.
|
|
result = CodexPlugin()._scan_jsonl_session(jsonl)
|
|
|
|
assert result is None or isinstance(result, SessionData)
|