mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
Fixes #1310. ## Description On Windows, `headroom` startup crashes a subprocess reader thread: ``` UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 7894: character maps to <undefined> ... subprocess.py _readerthread -> buffer.append(fh.read()) ... encodings/cp1252.py ``` Text-mode `subprocess` calls omit `encoding=`, so Python decodes child output with the locale codec (**cp1252** on Windows). Children that emit UTF-8 ??? `cbm index_repository` (indexing sources with chars like `???`/`???`), `claude mcp get/add`, the memory-sync process ??? produce bytes invalid in cp1252 and kill the reader thread. Linux/macOS default to UTF-8, so it's invisible there. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Add `encoding="utf-8", errors="replace"` to every text-mode (`text=True` / `universal_newlines=True`) subprocess call in the `headroom/` package (~50 call sites; several already had it). - `errors="replace"` (not `ignore`) so corrupt bytes surface as `???` rather than vanishing from parsed output. - Add `tests/test_cli/test_subprocess_utf8_encoding.py`: an AST guard asserting every text-mode subprocess call pins `encoding=`. The runtime crash can't reproduce on UTF-8 CI, so the invariant is enforced at the source level instead. ## Testing - [x] Unit tests pass (`pytest`) - New guard test passes (validates 51 call sites). - `tests/test_install`, `tests/test_cli/test_mcp.py`, `tests/test_mcp_registry` pass. (`test_runtime_start_lock_blocks_another_process` fails on this Windows box, but it fails identically on unmodified `main` ??? a pre-existing `msvcrt` lock flake, unrelated.) ### Test Output ```text > python -m pytest tests/test_cli/test_subprocess_utf8_encoding.py -q 1 passed in 0.12s > python -m pytest tests/test_install/ tests/test_cli/test_mcp.py tests/test_mcp_registry/ -q 133 passed, 2 skipped in 15.34s ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13.13. - Exact command / steps: Started `headroom` without `PYTHONUTF8=1` on a repo with UTF-8 chars in indexable files. Observed the `UnicodeDecodeError` crash. Applied the fix (pinning `encoding="utf-8"` on all text-mode subprocess calls). Re-ran. No crash. The AST guard enforces the invariant on CI (which runs UTF-8 locales and cannot reproduce the cp1252 crash natively). - Observed result: Subprocess reader threads no longer crash on UTF-8 output under cp1252 locale. - Not tested: All third-party tools that `headroom` shells out to; each was given `errors="replace"` as a safety net. ## Workaround for affected users (before fix is deployed) `PYTHONUTF8=1` (PowerShell: `$env:PYTHONUTF8=1; headroom ...`). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
175 lines
7.3 KiB
YAML
175 lines
7.3 KiB
YAML
name: Headroom e2e setup
|
|
description: >-
|
|
Checkout-agnostic setup shared by native e2e workflows (init, install, wrap).
|
|
Installs Python, optionally installs the Rust toolchain + editable headroom
|
|
package, and (optionally) drops PATH shims for the local ``headroom`` CLI and
|
|
target binaries so ``headroom init -g <target>`` can detect tools that aren't
|
|
actually installed on the runner.
|
|
inputs:
|
|
python-version:
|
|
description: Python version to install
|
|
required: false
|
|
default: "3.11"
|
|
install-mode:
|
|
description: >-
|
|
Install strategy. ``editable-proxy`` builds the local package with
|
|
``pip install -e .[proxy]`` and verifies ``headroom._core``.
|
|
``deps-only-proxy`` installs the base + ``[proxy]`` dependency set from
|
|
pyproject.toml, then drops a local ``headroom`` launcher that imports
|
|
from the checkout without building the package; use this for CLI tests
|
|
that do not exercise the Rust extension.
|
|
required: false
|
|
default: "editable-proxy"
|
|
shim-target:
|
|
description: >-
|
|
Name of the shim to drop on PATH (e.g. ``claude``, ``codex``). Leave
|
|
empty to skip shim creation.
|
|
required: false
|
|
default: ""
|
|
outputs:
|
|
shim-dir:
|
|
description: Absolute path to the directory containing the dropped shim
|
|
value: ${{ steps.shim.outputs.shim-dir }}
|
|
runs:
|
|
using: composite
|
|
steps:
|
|
- name: Set up Python ${{ inputs.python-version }}
|
|
uses: actions/setup-python@v5
|
|
with:
|
|
python-version: ${{ inputs.python-version }}
|
|
|
|
# Single-wheel architecture: `pip install -e .` invokes maturin (declared
|
|
# in pyproject.toml's build-system) which calls cargo to compile the Rust
|
|
# extension. Toolchain has to be set up before the editable install path.
|
|
- name: Install Rust toolchain
|
|
if: ${{ inputs.install-mode == 'editable-proxy' }}
|
|
uses: dtolnay/rust-toolchain@1.95.0
|
|
|
|
- name: Cache cargo registry + build
|
|
if: ${{ inputs.install-mode == 'editable-proxy' }}
|
|
uses: Swatinem/rust-cache@v2
|
|
with:
|
|
workspaces: ". -> target"
|
|
|
|
# macos-latest (macos-15) runners have varying Xcode versions installed.
|
|
# The Rust cc crate probes the active Xcode for
|
|
# .../lib/clang/<ver>/lib/darwin/libclang_rt.osx.a. Some Xcode versions
|
|
# (notably 16.4 / clang 17 on certain runner images) lack this path.
|
|
# 1. Find an Xcode whose clang runtime directory actually exists.
|
|
# 2. If none found, locate libclang_rt.osx and create the expected symlink.
|
|
- name: Fix clang_rt.osx linker path (macOS)
|
|
if: runner.os == 'macOS'
|
|
shell: bash
|
|
run: |
|
|
found=
|
|
for app in /Applications/Xcode_*.app; do
|
|
[ -d "$app" ] || continue
|
|
clang_dir=$(ls -d "$app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/"*/lib/darwin 2>/dev/null | head -1)
|
|
if [ -n "$clang_dir" ]; then
|
|
sudo xcode-select -s "$app"
|
|
echo "Selected Xcode: $app (has clang runtime at $clang_dir)"
|
|
found=1
|
|
break
|
|
fi
|
|
done
|
|
if [ -z "$found" ]; then
|
|
rt_lib=$(find /Applications -name "libclang_rt.osx*" 2>/dev/null | head -1)
|
|
if [ -n "$rt_lib" ]; then
|
|
xcode_ver=$(xcodebuild -version 2>/dev/null | head -1 | awk '{print $2}')
|
|
clang_ver=$(clang --version 2>/dev/null | head -1 | grep -oP 'version \K\d+')
|
|
exp_dir="/Applications/Xcode_${xcode_ver}.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/${clang_ver}/lib/darwin"
|
|
sudo mkdir -p "$exp_dir"
|
|
target="$exp_dir/$(basename "$rt_lib")"
|
|
[ -f "$target" ] || sudo ln -sf "$rt_lib" "$target"
|
|
echo "Symlinked $rt_lib -> $target"
|
|
else
|
|
echo "WARNING: libclang_rt.osx not found anywhere. Build may fail."
|
|
fi
|
|
fi
|
|
|
|
- name: Install headroom (editable, with proxy extras — builds Rust extension)
|
|
if: ${{ inputs.install-mode == 'editable-proxy' }}
|
|
shell: bash
|
|
run: |
|
|
python -m pip install --upgrade pip
|
|
# ``headroom/cli/__init__.py`` eagerly imports ``proxy.server`` (via
|
|
# ``cli/proxy.py``), which requires ``fastapi`` even for ``init``.
|
|
# Install with the ``[proxy]`` extras to match the Docker e2e image.
|
|
pip install -e ".[proxy]"
|
|
python -c "from headroom._core import DiffCompressor; print('headroom._core OK:', DiffCompressor)"
|
|
|
|
- name: Install base + proxy dependencies without building headroom
|
|
if: ${{ inputs.install-mode == 'deps-only-proxy' }}
|
|
shell: bash
|
|
run: |
|
|
python -m pip install --upgrade pip
|
|
python - <<'PY'
|
|
import subprocess
|
|
import sys
|
|
import tomllib
|
|
from pathlib import Path
|
|
|
|
project = tomllib.loads(Path("pyproject.toml").read_text(encoding="utf-8"))
|
|
requirements = list(project["project"]["dependencies"])
|
|
requirements.extend(project["project"]["optional-dependencies"]["proxy"])
|
|
subprocess.check_call(
|
|
[sys.executable, "-m", "pip", "install", "--retries", "10", "--timeout", "60", *requirements]
|
|
)
|
|
PY
|
|
python -c "from headroom.cli.main import main; print('headroom CLI OK:', main)"
|
|
|
|
- name: Drop local headroom launcher (POSIX)
|
|
if: ${{ inputs.install-mode == 'deps-only-proxy' && runner.os != 'Windows' }}
|
|
shell: bash
|
|
run: |
|
|
shim_dir="${RUNNER_TEMP}/headroom-local-bin"
|
|
mkdir -p "$shim_dir"
|
|
cat > "$shim_dir/headroom" <<'SH'
|
|
#!/usr/bin/env bash
|
|
exec python -m headroom.cli "$@"
|
|
SH
|
|
chmod +x "$shim_dir/headroom"
|
|
echo "$shim_dir" >> "$GITHUB_PATH"
|
|
|
|
- name: Drop local headroom launcher (Windows)
|
|
if: ${{ inputs.install-mode == 'deps-only-proxy' && runner.os == 'Windows' }}
|
|
shell: pwsh
|
|
run: |
|
|
$shimDir = Join-Path $env:RUNNER_TEMP "headroom-local-bin"
|
|
New-Item -ItemType Directory -Force -Path $shimDir | Out-Null
|
|
@"
|
|
@echo off
|
|
python -m headroom.cli %*
|
|
"@ | Out-File -FilePath (Join-Path $shimDir "headroom.cmd") -Encoding ascii
|
|
Add-Content -Path $env:GITHUB_PATH -Value $shimDir
|
|
|
|
- name: Drop shim (POSIX)
|
|
if: ${{ inputs.shim-target != '' && runner.os != 'Windows' }}
|
|
id: shim-posix
|
|
shell: bash
|
|
run: |
|
|
shim_dir="${RUNNER_TEMP}/headroom-e2e-shims"
|
|
bash e2e/_lib/make_shim.sh "${{ inputs.shim-target }}" "$shim_dir"
|
|
echo "$shim_dir" >> "$GITHUB_PATH"
|
|
echo "shim-dir=$shim_dir" >> "$GITHUB_OUTPUT"
|
|
|
|
- name: Drop shim (Windows)
|
|
if: ${{ inputs.shim-target != '' && runner.os == 'Windows' }}
|
|
id: shim-windows
|
|
shell: pwsh
|
|
run: |
|
|
$shimDir = Join-Path $env:RUNNER_TEMP "headroom-e2e-shims"
|
|
& pwsh -File e2e/_lib/make_shim.ps1 -Name "${{ inputs.shim-target }}" -Dir $shimDir
|
|
Add-Content -Path $env:GITHUB_PATH -Value $shimDir
|
|
"shim-dir=$shimDir" | Out-File -FilePath $env:GITHUB_OUTPUT -Append
|
|
|
|
- name: Export shim dir to job output
|
|
if: ${{ inputs.shim-target != '' }}
|
|
id: shim
|
|
shell: bash
|
|
run: |
|
|
if [ "${{ runner.os }}" = "Windows" ]; then
|
|
echo "shim-dir=${{ steps.shim-windows.outputs.shim-dir }}" >> "$GITHUB_OUTPUT"
|
|
else
|
|
echo "shim-dir=${{ steps.shim-posix.outputs.shim-dir }}" >> "$GITHUB_OUTPUT"
|
|
fi
|