2 commits

Author	SHA1	Message	Date
Tejas Chopra	5d23a0aec2	refactor(wrap): retire tokensave; Serena is the code-memory MCP (#2499) ... ## Description `tokensave` was a **downloaded third-party Rust binary** (`aovestdipaperino/tokensave`) that `headroom wrap` registered as a code-graph MCP server. This removes it entirely and standardises on **Serena** as the code-memory MCP — which was already the default in `wrap`. Serena runs on demand via `uvx`, so Headroom no longer downloads or executes a binary of its own for code memory. The change is a *removal + safe transition*, not a behaviour flip: Serena was already the default, so existing users move over automatically. This PR also folds in a small README repositioning (Headroom = the proxy; Serena is the recommended companion; RTK/lean-ctx are third-party tools we don't control), since it's the same tooling-stack story. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [x] Code refactoring (no functional changes) ## Changes Made - **Removed the external tool:** `headroom/graph/tokensave_installer.py` (the download-and-execute path), `build_tokensave_spec`, and the `_ensure_tokensave_binary` / `_index_tokensave_project` / `_setup_tokensave_mcp` helpers; dropped the dead `_setup_code_graph`. - **`--code-memory`** now offers `serena` (default) or `none` — the `tokensave` choice is gone. The strands `HeadroomBundle` uses Serena as its (default-on) code-memory MCP. - **Tombstone / transition (ledger-verified):** `headroom wrap` **and** `headroom unwrap` remove a previously Headroom-installed `tokensave` MCP entry so upgrading users stop launching it, and print that the leftover `~/.local/bin/tokensave` binary and `.tokensave/` folders are safe to delete. A user-managed `tokensave` entry is left untouched. Mirrors the existing `codebase-memory-mcp` retirement. - **Graceful for existing users:** `HEADROOM_CODE_MEMORY=tokensave` and `--no-tokensave` resolve to Serena instead of erroring; `--no-serena` now means "no code memory". **No state migration needed** — both tools' indexes are regenerable caches of the source, so Serena simply re-indexes. - **Docs/README:** replaced the "tokensave binary trust model" section with a Serena note + an "Upgrading from tokensave?" callout; retired the RTK "first-class part of our stack" framing. - **Tests:** deleted the tokensave-only test files (`test_graph_tokensave.py`, `test_cli/test_tokensave_helpers.py`, `test_cli/test_tokensave_setup.py`), rewrote `test_wrap_code_memory.py` for the new resolver/dispatch, fixed a codex test that patched a removed symbol. Net: **+102 / −1187 lines.** ## Testing - [x] Unit tests pass (`pytest`) — targeted to the affected areas - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check headroom/cli/wrap.py headroom/mcp_registry/ headroom/integrations/strands/ tests/test_wrap_code_memory.py tests/test_cli/conftest.py tests/test_cli/test_wrap_codex.py All checks passed! $ mypy headroom/cli/wrap.py headroom/mcp_registry headroom/integrations/strands/bundle.py Success: no issues found in 12 source files $ pytest tests/test_wrap_code_memory.py tests/test_cli/test_wrap_codex.py \ tests/test_cli/test_wrap_claude_vertex_proxy_env.py \ tests/test_cli/test_wrap_claude_finally_unbound.py -q ======================== 120 passed in 97.86s (0:01:37) ======================== $ pytest tests/test_cli --collect-only -q ========================= 713 tests collected in 1.82s ========================= # no import errors after symbol removal ``` ## Real Behavior Proof - **Environment:** macOS (darwin), Python 3.12.6, local `.venv`, on branch `tejas/remove-tokensave`. - **Exact command / steps:** - `python -c "from headroom.integrations.strands.bundle import HeadroomBundle; b=HeadroomBundle(enable_headroom_mcp=False, enable_serena_mcp=False); print(len(b.tools))"` → confirms the module imports after `build_tokensave_spec` removal (the import that my change would otherwise break). - CliRunner-driven `wrap codex --prepare-only` (in `test_wrap_codex.py`) writes `[mcp_servers.serena]` (with `command = "uvx"`, `"--context", "codex"`) to the codex config and **no** tokensave entry. - `_resolve_code_memory` unit tests confirm: default → `serena`; `HEADROOM_CODE_MEMORY=tokensave` → `serena`; `--no-serena` → `none`; `--code-memory bogus` → `ClickException`. - **Observed result:** import OK (`tools: 0`); Serena registered, tokensave absent; resolver behaves as above; 120/120 tests pass. - **Not tested:** a live `headroom wrap` against a real agent on a machine with a *previously-installed* tokensave MCP entry — the tombstone-removal path is covered by unit tests with a fake registrar/ledger, not an end-to-end run. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title ## Additional Notes - `--no-tokensave` / `--serena` / `--no-serena` are retained as hidden, deprecated flags (no-ops or mapped) so existing scripts don't break. - `--code-graph` is unchanged — it's the proxy's live file-watcher flag and was never the tokensave MCP; only the dead tokensave hook behind it was removed. - `CHANGELOG.md` intentionally left untouched. 🤖 Generated with [Claude Code](https://claude.com/claude-code)	2026-07-22 20:59:24 -07:00
Paperinik	dca9853ed9	feat(wrap): make tokensave the primary coding-task compressor, Serena the backup (#1230) ... ## Description Makes **tokensave** ([github.com/aovestdipaperino/tokensave](https://github.com/aovestdipaperino/tokensave)) the **primary coding-task compressor** that `headroom wrap` installs, and demotes **Serena** to a **backup**. tokensave is a local semantic code-graph MCP server (`tokensave serve`): the agent queries it for symbols, call chains, and impact analysis instead of grepping/reading whole files — the same role Serena filled, but as a pre-indexed graph. Serena now only registers when tokensave is unavailable (or when forced with `--serena`). Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/graph/tokensave_installer.py` (new): fetch the prebuilt tokensave release binary for the platform (release-binary only — no `cargo` compile at wrap time); honors `HEADROOM_BINARIES_OFFLINE`; returns `None` (→ Serena) when no asset exists (e.g. x86_64 macOS) or the download fails. - `mcp_registry`: `build_tokensave_spec()`; registration/disable/migrate go through the existing `ServerSpec` + ownership-ledger flow, identical to Serena. - `cli/wrap.py`: new `_setup_coding_compressor` primary/backup policy; tokensave setup/disable/migrate/index helpers. New flags `--no-tokensave` (skip primary) and `--serena` (force backup on); `--no-serena` now means "never register the backup". Default wrap removes a previously Headroom-installed Serena entry once tokensave is primary (user-managed entries preserved). `--code-graph` repointed to tokensave; the legacy `codebase-memory-mcp` install path is dropped (unwrap still cleans up legacy entries). `unwrap claude|codex` remove a ledger-owned tokensave entry. - Strands `HeadroomBundle`: `enable_tokensave_mcp=True` (primary); `enable_serena_mcp` now defaults `False` (backup). - `docs/content/docs/proxy.mdx`: `--code-graph` description updated from codebase-memory-mcp to tokensave. - Tests: tokensave installer (incl. error paths), register/disable/migrate, primary/backup policy, and the binary-resolution/indexing helpers. A scoped `tests/test_cli/conftest.py` offline guard keeps the CLI suite hermetic. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_graph_tokensave.py tests/test_cli/test_tokensave_setup.py tests/test_cli/test_tokensave_helpers.py 41 passed $ uv run pytest -q tests/test_cli/ tests/test_graph.py tests/test_graph_tokensave.py 421 passed # full CLI + graph suites, incl. all pre-existing Serena/unwrap/registry tests $ uv run pytest -q tests/test_mcp_registry/ tests/test_proxy_healthchecks.py passed $ uv run ruff format --check headroom/ tests/ # 822 files already formatted $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/graph/tokensave_installer.py headroom/mcp_registry/install.py Success: no issues found in 2 source files # Coverage on new module headroom/graph/tokensave_installer.py 99% ``` ## Real Behavior Proof - Environment: macOS (darwin arm64), Python 3.14, `uv` dev env; tokensave 7.0.2 binary present on PATH and exercised against this repo's `.tokensave/` graph during development. The installer pins release **v7.0.2** (SHA-256-verified) across macOS arm64, Linux aarch64/x86_64, and Windows x86_64/aarch64. - Exact command / steps: `headroom wrap claude` registers `tokensave serve` as the primary MCP code-graph server and indexes the project; with the binary removed from PATH and `HEADROOM_BINARIES_OFFLINE=1`, the same command falls back to registering Serena. Behavior is pinned by the unit tests (binary-present → tokensave registered + Serena entry removed; binary-absent → Serena fallback; `--serena` forces backup on; `--no-serena` suppresses it; `--no-tokensave` disables primary). - Observed result: tokensave registered as primary on the binary-present path; Serena registered on the unavailable path; unwrap removes only ledger-owned entries. - Not tested: live end-to-end agent session inside Claude Code / Codex against a real provider API; Windows/Linux release-asset download (covered by unit tests with mocked archives, not a live fetch). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG is left untouched: this repo generates it via release-please from Conventional Commits, so a manual edit is N/A. - `strands/bundle.py` shows 0% patch coverage because that module hard-imports the optional `strands` SDK, which CI does not install (the pre-existing `_make_serena_client` was likewise uncovered) — not a regression. - A `test (3)` shard failure on `headroom.memory.bridge` is a pre-existing offline-CI flake (cannot reach huggingface.co); it touches no file in this PR and the scoped offline guard only applies under `tests/test_cli/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-25 16:55:37 -05:00