akadmin/headroom
1
0
Fork 0
mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
2696 commits 410 branches 169 tags 142 MiB
Commit graph

5 commits

Author SHA1 Message Date
Parideboy
d633e8172c
fix(windows): pin UTF-8 encoding on text-mode subprocess calls (#1311) 
Fixes #1310.

## Description

On Windows, `headroom` startup crashes a subprocess reader thread:

```
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 7894: character maps to <undefined>
  ... subprocess.py _readerthread -> buffer.append(fh.read())
  ... encodings/cp1252.py
```

Text-mode `subprocess` calls omit `encoding=`, so Python decodes child
output with the locale codec (**cp1252** on Windows). Children that emit
UTF-8 ??? `cbm index_repository` (indexing sources with chars like
`???`/`???`), `claude mcp get/add`, the memory-sync process ??? produce
bytes invalid in cp1252 and kill the reader thread. Linux/macOS default
to UTF-8, so it's invisible there.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Add `encoding="utf-8", errors="replace"` to every text-mode
(`text=True` / `universal_newlines=True`) subprocess call in the
`headroom/` package (~50 call sites; several already had it).
- `errors="replace"` (not `ignore`) so corrupt bytes surface as `???`
rather than vanishing from parsed output.
- Add `tests/test_cli/test_subprocess_utf8_encoding.py`: an AST guard
asserting every text-mode subprocess call pins `encoding=`. The runtime
crash can't reproduce on UTF-8 CI, so the invariant is enforced at the
source level instead.

## Testing

- [x] Unit tests pass (`pytest`)
- New guard test passes (validates 51 call sites).
- `tests/test_install`, `tests/test_cli/test_mcp.py`,
`tests/test_mcp_registry` pass.
(`test_runtime_start_lock_blocks_another_process` fails on this Windows
box, but it fails identically on unmodified `main` ??? a pre-existing
`msvcrt` lock flake, unrelated.)

### Test Output

```text
> python -m pytest tests/test_cli/test_subprocess_utf8_encoding.py -q
1 passed in 0.12s

> python -m pytest tests/test_install/ tests/test_cli/test_mcp.py tests/test_mcp_registry/ -q
133 passed, 2 skipped in 15.34s
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13.13.
- Exact command / steps: Started `headroom` without `PYTHONUTF8=1` on a
repo with UTF-8 chars in indexable files. Observed the
`UnicodeDecodeError` crash. Applied the fix (pinning `encoding="utf-8"`
on all text-mode subprocess calls). Re-ran. No crash. The AST guard
enforces the invariant on CI (which runs UTF-8 locales and cannot
reproduce the cp1252 crash natively).
- Observed result: Subprocess reader threads no longer crash on UTF-8
output under cp1252 locale.
- Not tested: All third-party tools that `headroom` shells out to; each
was given `errors="replace"` as a safety net.

## Workaround for affected users (before fix is deployed)

`PYTHONUTF8=1` (PowerShell: `$env:PYTHONUTF8=1; headroom ...`).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-23 12:52:49 -05:00
Rod Boev
ad7993bf15
fix(codex): stop pinning Codex memory MCP to one project db (#1269) 
## Description

Stop `headroom wrap codex --memory` from pinning the global
`headroom_memory` MCP server to one absolute SQLite path. Today the
wrapper writes `--db <wrap-cwd>/.headroom/memory.db` into
`~/.codex/config.toml`, which makes later Codex sessions either reopen a
stale project-local DB or fail with `unable to open database file` when
that original path disappears. This change lets the MCP server use its
existing per-cwd default again, so each Codex session resolves
`.headroom/memory.db` from the active project instead of a serialized
past cwd. Closes #1147

The current Codex-memory config surface was shaped by
https://github.com/chopratejas/headroom/issues/462 and
https://github.com/chopratejas/headroom/issues/730; this PR keeps that
surface project-scoped again instead of globally pinning one DB.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- remove the injected `--db` argument from the global `headroom_memory`
Codex MCP block while keeping `--user` intact
- preserve the wrap-time local `.headroom/memory.db` setup and
Claude-memory import path for the current project
- treat only wrap-owned Codex markers as snapshot-suppression and
unwrap-cleanup signals, so pre-existing named MCP blocks still back up
and restore
- log a startup diagnostic from `headroom.memory.mcp_server` that
records the configured DB path, config source, cwd/project root,
resolved storage scope, path existence/readability, and whether the path
was static or cwd-derived
- add a shared MCP SDK test stub so both the memory MCP and CCR MCP test
surfaces still run in CI when `mcp` is absent
- make the shared MCP stub re-import target modules under the stubbed
dependency set and restore any pre-existing target module object plus
dotted parent-package attribute state after cleanup
- add focused regressions and guard coverage for the persisted Codex
config shape, named-MCP marker backup and restore, the no-backup
memory-only unwrap path, the wrap-memory-then-unwrap cleanup path, the
failed-wrap memory-only cleanup path, the startup-diagnostic path
classification, the shared-store CCR retrieval path, and the shared MCP
stub import lifecycle
- add a `CHANGELOG.md` entry for the user-visible Codex memory scoping
fix

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
uv run pytest tests/test_ccr_mcp_server.py tests/test_memory/test_mcp_server.py tests/test_cli/test_wrap_codex.py tests/test_mcp_stub.py
======================== 78 passed, 1 warning in 5.96s ========================
Pytest warning:
PytestConfigWarning: Unknown config option: asyncio_mode
Pytest post-success atexit noise:
PermissionError: [WinError 5] Access is denied: 'C:\Users\Rod\AppData\Local\Temp\pytest-of-Rod\pytest-current'

uv run ruff check headroom/cli/wrap.py headroom/memory/mcp_server.py tests/_mcp_stub.py tests/test_ccr_mcp_server.py tests/test_cli/test_wrap_codex.py tests/test_mcp_stub.py tests/test_memory/test_mcp_server.py
All checks passed!

uv run ruff format headroom/cli/wrap.py headroom/memory/mcp_server.py tests/_mcp_stub.py tests/test_ccr_mcp_server.py tests/test_cli/test_wrap_codex.py tests/test_mcp_stub.py tests/test_memory/test_mcp_server.py --check
7 files already formatted
```

## Real Behavior Proof

- Environment: isolated temp project directories, a temp Codex home, the
real `wrap codex` and `unwrap codex` CLI commands under pytest, a mocked
missing-`codex` launch path for the failed-wrap cleanup case, and shared
MCP-SDK stubs for the memory MCP and CCR MCP test modules so CI still
exercises those paths without a real `mcp` install.
- Exact command / steps: run `uv run pytest tests/test_ccr_mcp_server.py
tests/test_memory/test_mcp_server.py tests/test_cli/test_wrap_codex.py
tests/test_mcp_stub.py`; prove the persisted config shape with
`TestCodexMemoryMcpConfig::test_inject_omits_db_and_replaces_existing_memory_block`;
prove prepare-only wrap cleanup with
`test_wrap_codex_memory_prepare_only_unwrap_removes_memory_mcp_without_prior_config`;
prove failed-wrap cleanup with
`test_wrap_codex_memory_launch_failure_unwrap_cleans_memory_only_config`;
guard pre-existing named Codex MCP preservation with
`test_memory_only_wrap_restores_preexisting_named_mcp_block` and
`test_memory_only_wrap_without_backup_preserves_named_mcp_block`; prove
the startup diagnostic classifications with
`test_memory_mcp_startup_context_reports_dynamic_project_db` and
`test_memory_mcp_startup_context_reports_static_external_db`; prove the
shared-store CCR retrieval path with
`test_mcp_uses_shared_singleton_store` and
`test_mcp_retrieves_proxy_stored_content`; prove stub import cleanup
with `test_import_module_with_mcp_stub_imports_target_and_cleans_up`,
`test_import_module_with_mcp_stub_reimports_target_and_restores_originals`,
and
`test_import_module_with_mcp_stub_cleans_up_dotted_target_attribute`.
- Observed result: the persisted global `headroom_memory` block now
keeps `--user` but omits `--db`; prepare-only memory setup still
bootstraps the current project's `.headroom/memory.db`; `headroom unwrap
codex --no-stop-proxy` now removes both the prepare-only generated
config and the failed-wrap memory-only config instead of leaving
`[mcp_servers.headroom_memory]` behind; pre-existing named Codex MCP
blocks remain restorable across both normal and no-backup memory-only
unwrap paths because only wrap-owned markers suppress backups or trigger
named-block cleanup; the memory MCP server now logs whether its DB path
came from the cwd default or an explicit static path, along with the
resolved path and scope it will open; CI can exercise both MCP test
modules even when the `mcp` package is absent from the shard
environment, and the shared stub now re-imports target modules under the
stubbed SDK while restoring both dependency and dotted parent-package
target-module import state after cleanup.
- Not tested: full end-to-end interactive Codex CLI launch.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

The code change stays narrowly scoped to Codex memory config
persistence, cleanup, and startup observability. It does not widen into
larger memory-routing redesign or startup-failure recovery logic.
 2026-06-23 07:49:07 -05:00
chopratejas
2a91cbb4b4 refactor: single-wheel maturin build backend (fixes #355) 
Eliminates the dual-package architecture that was the root cause of #355.
`pip install headroom-ai` now produces ONE wheel containing both the Python
source (headroom/*.py) and the compiled Rust extension (headroom/_core.so).
No more separate `headroom-core-py` package, no more chicken-and-egg with
PyPI publication, no more wheelhouse / PIP_FIND_LINKS / composite-action
plumbing in CI.

This is the canonical pattern used by cryptography, polars, ruff,
pydantic-core, and other Rust-as-core Python packages. Honors the
"Rust as core engine" direction.

## What changed

- pyproject.toml: `[build-system]` swapped from hatchling to maturin.
  `[tool.hatch.*]` deleted; `[tool.maturin]` added pointing at
  `crates/headroom-py/Cargo.toml` for the cdylib. `python-source = "."`
  picks up the root `headroom/` package directly (dashboard HTML
  templates and other non-Python files included automatically).
- crates/headroom-py/pyproject.toml: deleted. The crate is no longer a
  separate published package; its Cargo.toml stays as the cdylib build
  target invoked via `[tool.maturin] manifest-path`.
- crates/headroom-py/python/: deleted (placeholder layout for the old
  separate package).

## CI updates

- ci.yml: `test` / `test-extras` / `test-agno` jobs simplified — Rust
  toolchain set up before `pip install -e .` (which now invokes maturin
  via build-system). Removed the "build wheel + symlink .so" dance.
  `build` job swapped from `python -m build` (hatch) to
  `maturin build` + `maturin sdist`.
- release.yml: collapsed dual-package matrix into one. New `build-wheels`
  matrix produces cross-platform wheels for cp310/11/12/13 ×
  {linux x86_64, linux aarch64, macos x86_64, macos aarch64}. New
  `collect-dist` aggregator merges artifacts. publish-pypi consumes the
  merged dist.
- init-native-e2e.yml: dropped windows-latest from the matrix —
  upstream `esaxx-rs` (/MT) and `ort-sys` (/MD) link with conflicting
  MSVC C runtime libraries, so the Rust extension cannot build for
  win_amd64 today. Tracked as a follow-up; not a blocker for Linux+macOS.
- headroom-e2e-setup: composite action now sets up Rust toolchain +
  Swatinem/rust-cache before `pip install -e .[proxy]`.
- eval.yml, publish.yml, rust.yml: same pattern — rust toolchain before
  install. rust.yml's wheels job builds from root pyproject.toml (no
  more `-m crates/headroom-py/Cargo.toml`).
- e2e/init/Dockerfile, e2e/wrap/Dockerfile: install rust + maturin in
  the build stage; copy `crates/` + workspace `Cargo.toml/lock` so the
  install can build the extension. Dropped `HEADROOM_REQUIRE_RUST_CORE=false`
  from wrap-e2e — the image now ships the full Rust core.
- Dockerfile (main): simplified — no more Layer 2/3 dance with
  `headroom-core-py` install + symlink. Single `uv pip install` builds
  + installs everything.
- .devcontainer/Dockerfile: rust toolchain + libssl-dev + maturin
  added so `uv sync` builds the extension inside the devcontainer.

## Lockfile + script

- uv.lock: regenerated. No `headroom-core-py` entries remain.
- scripts/build_rust_extension.sh: simplified from a symlink-into-tree
  workaround to a thin wrapper around `pip install -e .`. The maturin
  build-backend handles placement automatically.

## Local validation (all green on macOS aarch64)

1. Clean venv `pip install -e .` → `from headroom._core import …` works.
2. `maturin build --release` → 13.8 MB wheel, 336 files including
   `headroom/_core.cpython-311-darwin.so` (32 MB cdylib) and
   `headroom/dashboard/templates/dashboard.html`.
3. `pip install <wheel>` in fresh venv → import works.
4. Wheel contents verified via `unzip -l`.
5. `pytest tests/test_transforms/test_diff_compressor.py` — 29 passed.
6. `pytest tests/test_relevance.py` — 30 passed.
7. `cargo build --workspace` + `cargo test --workspace` — all green.
8. `make ci-precheck` — 176 Python tests + Rust + commitlint green.

## Migration notes

Users on `pip install headroom-ai` get the Rust core automatically
(linux + macos wheels). sdist installs require rust toolchain available
locally — pip will build via maturin.

Closes #355
Supersedes #357 (workarounds-based fix abandoned in favor of
architectural fix)
 2026-05-03 13:16:41 -07:00
JerrettDavis
bc7a95a7c7 ci(init-native): install [proxy] extras and use pwsh for Windows shim check 
Two fixes for the init-native-e2e matrix surfaced on PR #256:

1. Composite action installed `headroom` without extras, but
   `headroom/cli/__init__.py` eagerly imports `proxy.server` (via
   `cli/proxy.py`), which requires `fastapi`. All 6 POSIX jobs hit
   `ModuleNotFoundError: No module named 'fastapi'` before `init` ran.
   Fix: install `-e .[proxy]` to match the Docker e2e image.

2. On Windows, shims are `.cmd` files and Git Bash's `which` cannot
   resolve them (exact-match only). Python's `shutil.which` (used by
   `headroom init`) honors PATHEXT and finds the shim fine, but the
   pre-flight `which` step failed first. Fix: use `Get-Command` via
   `pwsh` for the Windows verification step.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 16:30:30 -05:00
JerrettDavis
0cfbc3f436 ci: add init-native-e2e workflow across linux / macos / windows 
Existing Docker init-e2e runs on ubuntu only. Platform-specific bugs
(Windows path separators in written hook commands, PowerShell-vs-bash
matcher strings, macOS keychain prompts, shutil.which PATHEXT quirks)
slip past it. Add a matrix workflow that drops a noop shim for each
target agent and runs ``headroom init -g <target>`` on each of the
three supported OSes, then asserts the settings file was written to
the platform-correct location.

Matrix: [ubuntu-latest, macos-latest, windows-latest] x [claude,
codex, copilot]. ``openclaw`` is excluded because it delegates to
``headroom wrap openclaw`` which needs a real OpenClaw CLI and can't
be stubbed with a noop shim; the Docker suite already covers its
negative path.

Common setup (Python install, editable headroom install, shim drop,
PATH wiring) is factored into a composite action at
.github/actions/headroom-e2e-setup so follow-up per-command workflows
(install-native-e2e, wrap-native-e2e) can be near-copies that only
supply their matrix and assertion blocks. The composite action uses
the cross-platform shim scripts from e2e/_lib/make_shim.{sh,ps1} that
landed with the harness refactor.

Scoped trigger: pull_request touching init code OR the harness, plus
pushes to main and manual dispatch. This avoids burning CI minutes on
every push to unrelated feature branches while still gating every PR
that could regress init behavior.

Not verified locally: Windows runner behavior. Reviewer should watch
the first matrix run on PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 16:14:32 -05:00