fix(wrap): read/write instruction files as UTF-8 on Windows (#1245)

## Description

Fixes #1126. On a Windows (cp1252) locale, `headroom wrap` crashes with
`UnicodeDecodeError` the first time it injects guidance into a user
instruction
file that contains non-ASCII prose (e.g. typographic quotes `“happy
places”` or
an em-dash). `_inject_rtk_instructions` and `_inject_memory_agents_md`
both read
the existing file and append/create it with a bare `read_text()` /
`open()` /
`write_text()`, so the default codec (cp1252, not UTF-8) chokes on the
multi-byte characters.

This is the same bug class already fixed for the `learn` pipeline
(#1202) and
earlier for other wrap paths — here it's the instruction-file injectors.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cli/wrap.py`: in `_inject_rtk_instructions` and
  `_inject_memory_agents_md`, read the existing instruction file as
  `encoding="utf-8", errors="replace"` and append/create with
`encoding="utf-8"`. The read only feeds the marker-existence check and
the
append doesn't rewrite existing bytes, so replacement can't corrupt the
file.
- `tests/test_cli/test_wrap_encoding.py`: new regression tests.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_cli/test_wrap_encoding.py tests/test_cli/test_wrap_hintfile_agents.py -q
16 passed

$ ruff check headroom/cli/wrap.py tests/test_cli/test_wrap_encoding.py
All checks passed!
```

The new tests are **red on the old code, green with the fix**: injecting
into a
file with a typographic quote plus a stray `0x9d` byte (undefined in
cp1252 and
invalid UTF-8, so a bare `open()` fails on any locale) — the append and
idempotent paths fail before the fix (4 failed) and pass after (6
passed).

## Real Behavior Proof

- Environment: Windows 11, Python 3.10, against the real
`headroom.cli.wrap`
injectors (no live agent launch; the decode failure is at file read
time).
- Exact command / steps: `write_bytes` an `AGENTS.md` containing
`"Be in “happy places” — really.\n"` plus a stray `0x9d` byte, then call
  `_inject_rtk_instructions(path)` / `_inject_memory_agents_md(path)`.
- Observed result: **before** the fix → `UnicodeDecodeError: 'utf-8'
codec
  can't decode byte 0x9d` (and on a real cp1252 locale, the same on the
typographic quotes alone); **after** → both return `True`, the marker is
present, the pre-existing prose is preserved, and re-running is
idempotent.
- Not tested: a full end-to-end `headroom wrap copilot` against a live
Copilot
CLI (verified at the injector level, which is where the decode crash
lives).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
jichaowang02-lang 2026-07-15 15:26:34 +01:00 committed by GitHub
parent 9e376afabe
commit 6413cc75a2
No known key found for this signature in database
GPG key ID: B5690EEEBB952194

View file

@ -0,0 +1,62 @@
"""Regression tests for #1126 — `headroom wrap` instruction injection must read
and write user instruction files as UTF-8, so non-ASCII prose (typographic
quotes, em-dashes) does not crash on a cp1252 (Windows) locale.
The reads use `errors="replace"`, so a stray non-UTF-8 byte (e.g. `0x9d`, which
fails a bare `open()` on *any* locale) cannot abort the wrap either.
"""
from __future__ import annotations
from pathlib import Path
import pytest
from headroom.cli.wrap import (
_MEMORY_AGENTS_MARKER,
_RTK_MARKER,
_inject_memory_agents_md,
_inject_rtk_instructions,
)
# (inject_fn, marker) for the two prose injectors that share the bug.
INJECTORS = [
pytest.param(_inject_rtk_instructions, _RTK_MARKER, id="rtk"),
pytest.param(_inject_memory_agents_md, _MEMORY_AGENTS_MARKER, id="memory_agents"),
]
# A typographic quote / em-dash (valid UTF-8) plus a stray byte that is
# undefined in cp1252 and an invalid UTF-8 start byte.
_EXISTING = "Be in “happy places” — really.\n".encode() + b"legacy \x9d byte\n"
@pytest.mark.parametrize("inject, marker", INJECTORS)
def test_inject_appends_into_file_with_non_ascii_and_stray_byte(inject, marker, tmp_path: Path):
target = tmp_path / "AGENTS.md"
target.write_bytes(_EXISTING)
# Before the fix this raised UnicodeDecodeError reading the existing file.
assert inject(target) is True
text = target.read_text(encoding="utf-8", errors="replace")
assert marker in text
# The pre-existing prose is preserved (append, not rewrite).
assert "happy places" in text
@pytest.mark.parametrize("inject, marker", INJECTORS)
def test_inject_creates_file_when_absent(inject, marker, tmp_path: Path):
target = tmp_path / "nested" / "AGENTS.md"
assert inject(target) is True
assert marker in target.read_text(encoding="utf-8")
@pytest.mark.parametrize("inject, marker", INJECTORS)
def test_inject_is_idempotent(inject, marker, tmp_path: Path):
target = tmp_path / "AGENTS.md"
target.write_bytes(_EXISTING)
assert inject(target) is True
assert inject(target) is True # marker already present -> no duplicate / no crash
assert target.read_text(encoding="utf-8", errors="replace").count(marker) == 1