diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 45d5243e8..83cf8f78f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -70,7 +70,6 @@ jobs: - 'headroom/cli/**' - 'headroom/install/**' - 'headroom/providers/**' - - 'headroom/rtk/**' - 'crates/**' - '**/*.rs' - 'Cargo.toml' diff --git a/.github/workflows/wrap-e2e.yml b/.github/workflows/wrap-e2e.yml index cc041e092..dae4a147a 100644 --- a/.github/workflows/wrap-e2e.yml +++ b/.github/workflows/wrap-e2e.yml @@ -9,7 +9,6 @@ on: paths: - 'headroom/cli/**' - 'headroom/providers/**' - - 'headroom/rtk/**' - 'crates/**' - 'docker/**' - 'Dockerfile' diff --git a/.github/workflows/wrap-native-e2e.yml b/.github/workflows/wrap-native-e2e.yml index 0e98d780a..3684368a4 100644 --- a/.github/workflows/wrap-native-e2e.yml +++ b/.github/workflows/wrap-native-e2e.yml @@ -16,7 +16,6 @@ on: paths: - "headroom/cli/**" - "headroom/providers/**" - - "headroom/rtk/**" - "tests/test_cli/test_wrap_bridge.py" - ".github/actions/headroom-e2e-setup/**" - ".github/workflows/wrap-native-e2e.yml" diff --git a/README.md b/README.md index da2536151..266c9914f 100644 --- a/README.md +++ b/README.md @@ -533,12 +533,10 @@ Headroom runs **locally**, covers **every** content type, works with every major | | Scope | Deploy | Local | Reversible | |------------------------------------------------------------------------------|------------------------------------------------|------------------------------------|:-----:|:----------:| | **Headroom** | All context — tools, RAG, logs, files, history | Proxy · library · middleware · MCP | Yes | Yes | -| [RTK](https://github.com/rtk-ai/rtk) | CLI command outputs | CLI wrapper | Yes | No | -| [lean-ctx](https://github.com/yvgude/lean-ctx) | Tool output, files, shell, history | Proxy · library · middleware · MCP · CLI | Yes | Yes | | [Compresr](https://compresr.ai), [Token Co.](https://thetokencompany.ai) | Text sent to their API | Hosted API call | No | No | | OpenAI Compaction | Conversation history | Provider-native | No | No | -> **Stack & integrations.** Headroom is the **proxy** — that's what we build and offer, and it compresses everything flowing through it no matter what sits upstream. Our recommended companion is **[Serena](https://github.com/oraios/serena)** (installed by default when you wrap an agent) for semantic code navigation — plus **Ponytail** if you want leaner model output. Everything else is your call: Headroom vendors the third-party [RTK](https://github.com/rtk-ai/rtk) and [lean-ctx](https://github.com/yvgude/lean-ctx) binaries for shell-output rewriting, but we don't own or control either project — swap between them with `HEADROOM_CONTEXT_TOOL`, or turn them off. You're free to attach your own tooling too — code-memory MCP, Graphify, Caveman, or any MCP server — and Headroom compresses downstream of all of it. +> **Stack & integrations.** Headroom is the **proxy** — that's what we build and offer, and it compresses everything flowing through it no matter what sits upstream. Our recommended companion is **[Serena](https://github.com/oraios/serena)** (installed by default when you wrap an agent) for semantic code navigation — plus **Ponytail** if you want leaner model output. Everything else is your call: you're free to attach your own tooling — code-memory MCP, Graphify, Caveman, or any MCP server — and Headroom compresses downstream of all of it. ## Contributing diff --git a/REALIGNMENT/09-phase-G-rtk-observability.md b/REALIGNMENT/09-phase-G-rtk-observability.md index 4b89b7ef7..3c634fb36 100644 --- a/REALIGNMENT/09-phase-G-rtk-observability.md +++ b/REALIGNMENT/09-phase-G-rtk-observability.md @@ -1,5 +1,14 @@ # Phase G — RTK Breadth + Observability +> **SUPERSEDED.** RTK and lean-ctx were removed from Headroom entirely: the +> `headroom/rtk/` and `headroom/lean_ctx/` packages, all `--rtk` / `--context-tool` +> flags, the wrap-side hooks and hint-file injection, and the proxy-side `rtk gain` +> polling are all gone, and `headroom/context_tool_cleanup.py` uninstalls what +> earlier versions left on disk. The RTK-specific plan below is historical; the +> non-RTK observability items (cache-hit rate, compression ratio, token +> validation) were kept. `docs/rtk-architecture.md`, referenced throughout this +> document, was deleted with the feature. + **Goal:** Extend RTK coverage to more wrap-CLI agents; close the dead `tokens_saved_rtk` data plane; add per-invocation RTK metrics; add the cache-hit-rate, compression-ratio, token-validation observability surface that's missing today. **Calendar:** 1 week. diff --git a/REALIGNMENT/12-decisions-needed.md b/REALIGNMENT/12-decisions-needed.md index 757ad3520..7e9dc9d64 100644 --- a/REALIGNMENT/12-decisions-needed.md +++ b/REALIGNMENT/12-decisions-needed.md @@ -92,7 +92,11 @@ No re-scoping needed; revisit after Phase D lands. ## Q9. RTK proxy-side invocation — ever revisit? -**Recommendation:** **No, document the decision in `docs/rtk-architecture.md`** (Phase G PR-G3). The argument: +**Resolved — moot.** RTK was removed from Headroom outright (see +`09-phase-G-rtk-observability.md`), so there is no proxy-side invocation to +revisit. The original recommendation was "no, document the decision in +`docs/rtk-architecture.md`" (that doc was deleted with the feature). The argument +is kept because reasons 1–3 apply to any future shell-output rewriter: 1. Cache hot zone risk: shell-out + buffer per tool result is correctness-fragile. 2. Parallel implementation: `crates/headroom-core/src/transforms/log_compressor.rs` covers post-hoc log/output compression; RTK rewrites *commands* (different value). 3. RTK itself is a third-party binary the team doesn't control; an upstream version change silently busts cache. diff --git a/benchmarks/rtk_loop_learn_eval.py b/benchmarks/rtk_loop_learn_eval.py deleted file mode 100644 index cd36fbf98..000000000 --- a/benchmarks/rtk_loop_learn_eval.py +++ /dev/null @@ -1,287 +0,0 @@ -"""RTK-loop eval — does Headroom Learn catch a loop and write a guardrail that -would prevent it recurring? - -This is the agentic eval for the loop-weighting work. It runs in two phases: - - Phase 1 — TRIGGER + LEARN - Reproduce an RTK re-fetch loop (a grep whose RTK-truncated output forces the - agent to re-run larger-limit variants), run it through ``SessionAnalyzer``, - and SCORE the resulting guardrail: - • produced — a loop guardrail was emitted at all - • ranked_first — it outranks the one-off rules (the weighting works) - • names_command — the rule identifies the command that looped - • prescribes_fix — the rule says how to avoid it (fetch full output once) - • weight_reflects — its savings estimate >= the MEASURED wasted tokens - - Phase 2 — GUARDRAIL HOLDS - Inject that guardrail as a prior learned pattern, then feed a session where - the agent FOLLOWED it (one full-output fetch, no loop). Re-run the analyzer - and assert NO new loop guardrail is produced for that command — i.e. once - the rule exists and is honored, the loop does not re-trigger and Learn does - not need to relearn it. - -Runs deterministically by default (a stubbed analyzer LLM so CI is hermetic). -With ``--real`` it drives the real analyzer LLM and scores the actually-generated -rule, using an API key (ANTHROPIC/OPENAI/GEMINI) or an installed CLI backend. - -Usage: - python benchmarks/rtk_loop_learn_eval.py # deterministic - python benchmarks/rtk_loop_learn_eval.py --real # real LLM (API key) - HEADROOM_LEARN_CLI=claude python benchmarks/rtk_loop_learn_eval.py --real # via CLI -""" - -from __future__ import annotations - -import argparse -import os -import sys -from contextlib import nullcontext -from dataclasses import dataclass, field -from pathlib import Path -from unittest.mock import patch - -# Allow running as a plain script from the repo root. -sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) - -from headroom.learn.analyzer import SessionAnalyzer # noqa: E402 -from headroom.learn.fixtures import rtk_refetch_loop_session # noqa: E402 -from headroom.learn.loops import detect_loops # noqa: E402 -from headroom.learn.models import ( # noqa: E402 - ProjectInfo, - SessionData, - ToolCall, -) - -REPETITIONS = 6 - - -# ============================================================================= -# Deterministic LLM stub — stands in for the analyzer's _call_llm in CI. -# It mimics a competent model: emits the loop guardrail (under-estimating its -# savings, so the weighting layer has real work to do) plus a one-off rule the -# model would naively rank higher. In Phase 2 it emits NO loop rule, because a -# non-looping guarded session gives it nothing to relearn. -# ============================================================================= - - -def _stub_llm_phase1(digest: str, model: str) -> dict: - return { - "context_file_rules": [ - { - "section": "Use uv for Python", - "content": "Use `uv run python` instead of `python3`.", - "estimated_tokens_saved": 900, # model rates the one-off high - "evidence_count": 2, - }, - { - "section": "Avoid grep TimeoutError re-fetch loop", - "content": ( - "When searching logs for TimeoutError, capture the full " - "result once (grep into a file and read it) instead of " - "re-running grep with larger `head` limits." - ), - "estimated_tokens_saved": 150, # simulated low estimate (stub value, not a real-model figure) - "evidence_count": 1, - }, - ], - "memory_file_rules": [], - } - - -def _stub_llm_phase2(digest: str, model: str) -> dict: - # Guarded, non-looping session → nothing new to learn about the grep. - return {"context_file_rules": [], "memory_file_rules": []} - - -# ============================================================================= -# Scoring -# ============================================================================= - - -@dataclass -class Scorecard: - checks: dict[str, bool] = field(default_factory=dict) - notes: dict[str, str] = field(default_factory=dict) - - def add(self, name: str, passed: bool, note: str = "") -> None: - self.checks[name] = passed - if note: - self.notes[name] = note - - @property - def passed(self) -> bool: - return all(self.checks.values()) - - def render(self) -> str: - width = max(len(k) for k in self.checks) - lines = [] - for name, ok in self.checks.items(): - mark = "PASS" if ok else "FAIL" - note = f" ({self.notes[name]})" if name in self.notes else "" - lines.append(f" [{mark}] {name.ljust(width)}{note}") - return "\n".join(lines) - - -def _guarded_session() -> SessionData: - """A session where the agent followed the guardrail: one full-output fetch, - no re-fetch loop.""" - return SessionData( - session_id="guarded", - tool_calls=[ - ToolCall( - name="Bash", - tool_call_id="tc_0", - input_data={"command": "grep -rn 'TimeoutError' logs/ > /tmp/hits.txt"}, - output="(wrote 1240 matches to /tmp/hits.txt)", - is_error=False, - msg_index=0, - output_bytes=40, - ), - ToolCall( - name="Read", - tool_call_id="tc_1", - input_data={"file_path": "/tmp/hits.txt"}, - output="logs/app.log:42: TimeoutError ...", - is_error=False, - msg_index=1, - output_bytes=8000, - ), - ], - ) - - -def run_eval(*, use_real_llm: bool) -> Scorecard: - project = ProjectInfo( - name="rtk-loop-eval", - project_path=Path("/tmp/rtk-loop-eval"), - data_path=Path("/tmp/rtk-loop-eval-data"), - ) - card = Scorecard() - - # ---- Phase 1: trigger + learn ----------------------------------------- - loop_session = rtk_refetch_loop_session(repetitions=REPETITIONS) - loops = detect_loops([loop_session]) - measured_waste = loops[0].wasted_tokens if loops else 0 - card.add("loop_detected", bool(loops), f"{len(loops)} loop(s), ~{measured_waste:,} tok wasted") - - analyzer = SessionAnalyzer(model=None if use_real_llm else "stub") - phase1_ctx = ( - nullcontext() - if use_real_llm - else patch("headroom.learn.analyzer._call_llm", _stub_llm_phase1) - ) - with phase1_ctx: - result = analyzer.analyze(project, [loop_session]) - - recs = result.recommendations - loop_recs = [r for r in recs if r.is_loop_guardrail] - card.add("guardrail_produced", bool(loop_recs)) - - top = recs[0] if recs else None - card.add( - "ranked_first", - bool(top and top.is_loop_guardrail), - "" if (top and top.is_loop_guardrail) else "loop rule did not rank #1", - ) - - guardrail = loop_recs[0] if loop_recs else None - text = (guardrail.section + " " + guardrail.content).lower() if guardrail else "" - # The rule must identify the LOOPING COMMAND (grep + its output-limit shape), - # not the incidental search string — a good fix generalizes beyond it. (The - # real-LLM run surfaced this: the model wrote a general "grepping logs / `head - # -N` limits" rule and never echoed "TimeoutError", which an earlier - # literal-match check wrongly failed.) - card.add( - "names_command", - "grep" in text and any(k in text for k in ("head", "log", "limit")), - ) - card.add( - "prescribes_fix", - any(k in text for k in ("full", "once", "into a file", "instead", "limit")), - ) - card.add( - "weight_reflects_waste", - bool(guardrail and guardrail.estimated_tokens_saved >= measured_waste), - "" - if (guardrail and guardrail.estimated_tokens_saved >= measured_waste) - else f"savings {getattr(guardrail, 'estimated_tokens_saved', 0)} < waste {measured_waste}", - ) - - # ---- Phase 2: guardrail holds ----------------------------------------- - # Inject the produced guardrail as a prior pattern via the project's - # context file, then analyze a guarded (non-looping) session. - held = True - note = "" - if guardrail: - ctx_path = Path("/tmp/rtk-loop-eval-CLAUDE.md") - ctx_path.write_text( - "\n" - f"### {guardrail.section}\n{guardrail.content}\n" - "\n", - encoding="utf-8", - ) - project.context_file = ctx_path - phase2_ctx = ( - nullcontext() - if use_real_llm - else patch("headroom.learn.analyzer._call_llm", _stub_llm_phase2) - ) - with phase2_ctx: - held_result = analyzer.analyze(project, [_guarded_session()]) - # No NEW loop guardrail should be needed for the (now-guarded) grep. - new_loop_rules = [ - r - for r in held_result.recommendations - if r.is_loop_guardrail and "grep" in (r.section + r.content).lower() - ] - held = not new_loop_rules - note = "" if held else f"{len(new_loop_rules)} new grep loop rule(s) re-emitted" - else: - held = False - note = "no guardrail from phase 1 to test" - card.add("guardrail_holds", held, note) - - return card - - -def _real_backend_available() -> bool: - """True when the analyzer can reach a real LLM — API key or installed CLI.""" - import shutil - - if any(os.environ.get(k) for k in ("ANTHROPIC_API_KEY", "OPENAI_API_KEY", "GEMINI_API_KEY")): - return True - return any(shutil.which(cli) for cli in ("claude", "gemini", "codex")) - - -def main() -> int: - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument( - "--real", - action="store_true", - help="Drive the real analyzer LLM — needs an API key (ANTHROPIC_API_KEY / " - "OPENAI_API_KEY / GEMINI_API_KEY) or an installed CLI backend " - "(claude / gemini / codex; force one with HEADROOM_LEARN_CLI=claude).", - ) - args = parser.parse_args() - - if args.real and not _real_backend_available(): - print( - "--real needs an LLM backend (API key or claude/gemini/codex CLI); " - "falling back to deterministic mode.\n" - ) - args.real = False - - mode = "REAL LLM" if args.real else "deterministic stub" - print(f"RTK-loop eval — mode: {mode}\n") - card = run_eval(use_real_llm=args.real) - print(card.render()) - print() - if card.passed: - print("RESULT: PASS — loop caught, guardrail ranked first, and it holds.") - return 0 - print("RESULT: FAIL — see failed checks above.") - return 1 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/crates/headroom-proxy/src/observability/metric_names.rs b/crates/headroom-proxy/src/observability/metric_names.rs index 7708f92f5..07f5772d5 100644 --- a/crates/headroom-proxy/src/observability/metric_names.rs +++ b/crates/headroom-proxy/src/observability/metric_names.rs @@ -98,11 +98,11 @@ pub const METRIC_PROXY_RESPONSE_STATUS_COUNT_TOTAL_HELP: &str = // for `proxy_image_generation_call_log_redacted_total`, // `wrap_rtk_invocations_total`, and `wrap_rtk_tokens_saved_per_session` // were removed because the underlying counters had no production -// emit site on the Rust side. The same metrics are exported by the -// Python proxy (`headroom/proxy/prometheus_metrics.py`) which is the -// natural owner: image redaction is a Python-proxy operation and RTK -// invocation tracking lives in the wrap CLI, both Python-side -// surfaces. See `docs/observability.md`. +// emit site on the Rust side. Image redaction is exported by the +// Python proxy (`headroom/proxy/prometheus_metrics.py`), its natural +// owner. The two `wrap_rtk_*` names are gone for good: the rtk +// integration they measured has been removed from Headroom. +// See `docs/observability.md`. // ---------- shared label keys ---------- diff --git a/crates/headroom-proxy/src/observability/proxy_metrics.rs b/crates/headroom-proxy/src/observability/proxy_metrics.rs index 371b2887f..a93eee396 100644 --- a/crates/headroom-proxy/src/observability/proxy_metrics.rs +++ b/crates/headroom-proxy/src/observability/proxy_metrics.rs @@ -283,10 +283,10 @@ pub fn record_response_status(status: &str, reason: Option<&str>, request_id: &s // Phase G PR-G3 remediation (C3 + C4): the image-redacted counter // and the wrap_rtk_invocations counter were originally registered // here but neither had a production emit site that crossed the -// Python/Rust boundary. Both have moved Python-side -// (`headroom.proxy.request_logger::redactions_total` and -// `headroom.cli.wrap_rtk_metrics::rtk_invocation_counts`) and the -// Python proxy's `/metrics` exporter surfaces them — see +// Python/Rust boundary. The image-redacted counter moved Python-side +// (`headroom.proxy.request_logger::redactions_total`) and the Python +// proxy's `/metrics` exporter surfaces it; the RTK counter is gone +// entirely along with the rtk integration itself — see // `docs/observability.md` for the placement decision. Keeping a // dead Rust counter would (a) violate the "no dead metrics // registered" review finding and (b) mislead Phase H canary diff --git a/docker-compose.yml b/docker-compose.yml index f081de838..47b707b2d 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -38,8 +38,6 @@ services: # if you want to use a custom OpenAI-compatible API endpoint, # uncomment and set the following line with the desired URL # - OPENAI_TARGET_API_URL=https://api.x.ai - # CLI-filtering dashboard figures require the `rtk` binary inside this - # container; it is not installed by this image. See docs/content/docs/docker-install.mdx. ports: - "8787:8787" volumes: diff --git a/docker/docker-compose.native.yml b/docker/docker-compose.native.yml index 0c1fd6c1d..c999023f1 100644 --- a/docker/docker-compose.native.yml +++ b/docker/docker-compose.native.yml @@ -13,8 +13,6 @@ services: # the Docker bind-mount source and is intentionally different. HEADROOM_WORKSPACE_DIR: /tmp/headroom-home/.headroom HEADROOM_CONFIG_DIR: /tmp/headroom-home/.headroom/config - # CLI-filtering dashboard figures require the `rtk` binary inside this - # container; it is not installed by this image. See docs/content/docs/docker-install.mdx. volumes: - ${HEADROOM_WORKSPACE:-.}:/workspace - ${HEADROOM_HOST_HOME:?set HEADROOM_HOST_HOME}/.headroom:/tmp/headroom-home/.headroom @@ -35,8 +33,6 @@ services: # above for rationale. HEADROOM_WORKSPACE_DIR: /tmp/headroom-home/.headroom HEADROOM_CONFIG_DIR: /tmp/headroom-home/.headroom/config - # CLI-filtering dashboard figures require the `rtk` binary inside this - # container; it is not installed by this image. See docs/content/docs/docker-install.mdx. ports: - "${HEADROOM_PORT:-8787}:${HEADROOM_PORT:-8787}" volumes: diff --git a/docs/content/docs/configuration.mdx b/docs/content/docs/configuration.mdx index 2e301a051..c4bf1bc8f 100644 --- a/docs/content/docs/configuration.mdx +++ b/docs/content/docs/configuration.mdx @@ -5,27 +5,8 @@ description: All configuration options for the Headroom Python and TypeScript SD Headroom can be configured via the SDK constructor, proxy command line, environment variables, or per-request overrides. -## CLI Context Tool - -`headroom wrap ...` uses RTK for local shell-output filtering by default. -Set `HEADROOM_CONTEXT_TOOL=lean-ctx` to have wrap commands install or reuse -`lean-ctx` and run `lean-ctx init --agent ` instead of RTK setup. - -```bash -export HEADROOM_CONTEXT_TOOL=lean-ctx -headroom wrap claude -headroom wrap codex --prepare-only -``` - -Supported values are `rtk` and `lean-ctx`; unset defaults to `rtk`. - If Codex history disappeared after using an older wrapper, see [Recover Codex State](/docs/codex-recovery) before wrapping Codex again. -The proxy reads RTK lifetime savings with global scope by default so a shared -daemon reports savings across the operator's projects. Set -`HEADROOM_RTK_GAIN_SCOPE=project` to query `rtk gain --project` from the -proxy process working directory. - ## SDK Modes (`default_mode` / `headroom_mode`) These modes apply to SDK usage via `HeadroomClient(default_mode=...)` or per-request `headroom_mode=...`. They are **not** the same as the proxy `--mode` flag. diff --git a/docs/content/docs/docker-install.mdx b/docs/content/docs/docker-install.mdx index 747330cd9..06e90fe38 100644 --- a/docs/content/docs/docker-install.mdx +++ b/docs/content/docs/docker-install.mdx @@ -75,7 +75,7 @@ docker run --rm -it \ `wrap` is host-oriented in Docker-native mode: - the wrapper starts the Headroom proxy in Docker -- container-side prep writes Headroom config, memory, and selected CLI context-tool setup into mounted host files +- container-side prep writes Headroom config and memory into mounted host files - the target CLI itself is launched on the host by the wrapper Supported host wrap flows: @@ -191,9 +191,6 @@ That keeps provider auth and runtime config working without maintaining a separa - The install scripts are idempotent: rerunning them refreshes the wrapper and image without duplicating shell profile blocks. - For persistent service and task installs, use the Python-native `headroom install ...` workflow — see [Persistent Installs](/docs/persistent-installs). - For Docker-native `headroom install ...`, the wrapper persists its profile manifest under `~/.headroom/deploy//`. -- The `rtk` binary is not bundled in the Docker image. Dashboard CLI-filtering - savings figures show as "not installed" (not `0`) until `rtk` is installed - inside the container. ## Next steps diff --git a/docs/content/docs/filesystem-contract.mdx b/docs/content/docs/filesystem-contract.mdx index 6aeda5451..d54b75347 100644 --- a/docs/content/docs/filesystem-contract.mdx +++ b/docs/content/docs/filesystem-contract.mdx @@ -54,8 +54,6 @@ Examples: | Memory bridge state | `${WORKSPACE_DIR}/bridge_state.json` | — | | Proxy log directory | `${WORKSPACE_DIR}/logs/` | — | | HTTP 400 debug dumps | `${WORKSPACE_DIR}/logs/debug_400/` | — | -| Vendored `rtk` binary | `${WORKSPACE_DIR}/bin/rtk[.exe]` | — | -| Vendored `lean-ctx` binary | `${WORKSPACE_DIR}/bin/lean-ctx[.exe]` | — | | Deployment profiles | `${WORKSPACE_DIR}/deploy/` | — | | Beacon lock file | `${WORKSPACE_DIR}/.beacon_lock_` | — | diff --git a/docs/content/docs/grok-build.mdx b/docs/content/docs/grok-build.mdx index e11b658c0..583f92f81 100644 --- a/docs/content/docs/grok-build.mdx +++ b/docs/content/docs/grok-build.mdx @@ -3,7 +3,7 @@ title: Grok Build Integration description: Route Grok Build traffic through Headroom for token compression and per-project savings attribution. --- -Use `headroom wrap grok-build` to route Grok Build LLM traffic through the local Headroom proxy. The wrapper starts or reuses the proxy, injects a reversible `[model.grok-build]` override into `~/.grok/config.toml` (or `$GROK_HOME/config.toml`), optionally sets up RTK or `lean-ctx`, and prints next steps for launching `grok`. +Use `headroom wrap grok-build` to route Grok Build LLM traffic through the local Headroom proxy. The wrapper starts or reuses the proxy, injects a reversible `[model.grok-build]` override into `~/.grok/config.toml` (or `$GROK_HOME/config.toml`), and prints next steps for launching `grok`. ## Quick Start @@ -30,7 +30,6 @@ headroom unwrap grok-build | Proxy | Starts the Headroom proxy unless `--no-proxy` is set | | Model config | Writes or updates `[model.grok-build] base_url` in Grok's `config.toml`, pointing at `http://127.0.0.1:/v1` (with optional `/p/` prefix for savings attribution) | | Existing config | If you already have a `[model.grok-build]` table, Headroom rewrites `base_url` in place instead of appending a duplicate table (invalid TOML) | -| Context tool | Injects RTK or `lean-ctx` guidance into project `AGENTS.md` unless `--no-context-tool` is set | | MCP install | `headroom mcp install` can register Headroom MCP via `GrokRegistrar` | | Backup | Snapshots `config.toml` to `config.toml.headroom-backup` before the first injection | @@ -39,7 +38,6 @@ headroom unwrap grok-build ```bash headroom wrap grok-build \ --port 8787 \ # Proxy port (default: 8787) - --no-context-tool \ # Skip RTK / lean-ctx setup --no-proxy \ # Use an existing proxy instead of starting one --learn \ # Enable live traffic learning --memory # Enable persistent memory @@ -50,7 +48,6 @@ headroom wrap grok-build \ | Variable | Description | |---|---| | `GROK_HOME` | Override Grok config directory (default: `~/.grok`) | -| `HEADROOM_CONTEXT_TOOL` | Set to `lean-ctx` to use lean-ctx instead of RTK | | `XAI_API_KEY` | Grok API key (also accepts `GROK_CODE_XAI_API_KEY`) | ## Persistent Install diff --git a/docs/content/docs/opencode.mdx b/docs/content/docs/opencode.mdx index 837fa5e56..6c71d27ff 100644 --- a/docs/content/docs/opencode.mdx +++ b/docs/content/docs/opencode.mdx @@ -3,7 +3,7 @@ title: OpenCode Integration description: Route OpenCode traffic through Headroom for token compression, MCP tools, and cached model access. One command to wrap, one to unwrap. --- -Use `headroom wrap opencode` to route OpenCode LLM traffic through the Headroom proxy with a single command. The wrapper starts or reuses the proxy, writes OpenCode config, injects Headroom MCP tools, adds RTK context filtering, and launches OpenCode with the generated config. +Use `headroom wrap opencode` to route OpenCode LLM traffic through the Headroom proxy with a single command. The wrapper starts or reuses the proxy, writes OpenCode config, injects Headroom MCP tools, and launches OpenCode with the generated config. The `headroom-opencode` npm package also exports a native OpenCode plugin. The plugin can be used directly from OpenCode config when you want in-process transport interception plus the Headroom retrieve tool. @@ -27,7 +27,6 @@ headroom unwrap opencode | Provider injection | Writes a `headroom` provider using `@ai-sdk/openai-compatible` into `opencode.json`, pointing at `http://127.0.0.1:/v1` | | Runtime env | Sets `OPENCODE_CONFIG_CONTENT` with provider, plugin, and optional local MCP config so OpenCode picks up Headroom at launch | | Provider compatibility | Leaves `OPENAI_BASE_URL` and `ANTHROPIC_BASE_URL` untouched so OpenCode `/connect` providers keep their own routing | -| Context tool | Injects RTK (or `lean-ctx`) instructions into `~/.config/opencode/AGENTS.md` and project `AGENTS.md` | | MCP setup | Registers the Headroom MCP server (`headroom_compress`, `headroom_retrieve`, `headroom_stats`) | | Serena MCP | Optionally registers Serena code graph tools (`--no-serena` to skip) | | Backup | Snapshots `opencode.json` to `opencode.json.headroom-backup` before making any changes | @@ -38,7 +37,6 @@ headroom unwrap opencode ```bash headroom wrap opencode \ --port 8787 \ - --no-rtk \ --no-mcp \ --no-serena \ --code-graph \ @@ -71,7 +69,6 @@ The default model is `headroom/claude-sonnet-4-6`. Change it in `opencode.json` |---|---| | `OPENCODE_CONFIG_CONTENT` | JSON payload with provider, plugin, and optional local MCP config injected by `wrap` | | `HEADROOM_PROXY_URL` | Proxy URL passed to Headroom MCP when a non-default port is used, and to the native plugin when configured | -| `HEADROOM_CONTEXT_TOOL` | Set to `lean-ctx` to use lean-ctx instead of RTK | ## Failure Learning diff --git a/docs/observability.md b/docs/observability.md index 33b2f20f6..b6de50dc6 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -3,7 +3,7 @@ The Headroom Rust proxy exposes Prometheus-format metrics on the `/metrics` endpoint of every running proxy instance. The metric catalogue below covers Phase D (Bedrock route instrumentation) and -Phase G PR-G3 (per-invocation RTK + proxy-wide observability). +Phase G PR-G3 (proxy-wide observability). All metric names + label keys are constants in `crates/headroom-proxy/src/observability/metric_names.rs`, so any @@ -59,17 +59,6 @@ intentional byte mutations do not trip the alarm. | `proxy_service_tier_count_total` | Counter | `tier` | Service-tier distribution observed at the proxy. | | `proxy_response_status_count_total` | Counter | `status` | Terminal status distribution (`completed`, `incomplete`, `failed`, `cancelled`, `in_progress`). | -#### Wrap CLI / RTK (Python-side) - -| Name | Type | Labels | Purpose | -|------|------|--------|---------| -| `wrap_rtk_invocations_total` | Counter | `tool` | RTK invocations observed via the wrap-CLI tail. Surfaced via the Python proxy's `/metrics` exporter; the wrap CLI bumps `headroom.cli.wrap_rtk_metrics.record_rtk_invocation(...)`. | - -> **C4 remediation:** This counter is Python-side because RTK is -> wrapped by `headroom wrap` (Python CLI) and the wrap-side tail -> is the natural emit site. The Rust proxy previously held a dead -> counter for this metric; that has been removed. - #### Image log redaction (Python-side) | Name | Type | Labels | Purpose | @@ -141,9 +130,6 @@ sum by (strategy) (rate(proxy_compression_rejected_by_token_check_total{strategy # Upstream rate-limit headroom (smaller = closer to throttle). proxy_rate_limit_remaining_tokens{provider="anthropic"} -# RTK invocation rate (Python-side). -sum by (tool) (rate(wrap_rtk_invocations_total{tool!="__init__"}[5m])) - # Image-redaction rate (Python-side). rate(proxy_image_generation_call_log_redacted_total[5m]) ``` @@ -259,16 +245,12 @@ Every label vocabulary is bounded by code, not customer input: `"other"` and a `tracing::warn!` is emitted so wire-format drift surfaces loudly in logs. - `status`: 5-variant enum. -- `tool` (Python-side `wrap_rtk_invocations_total`): bounded by the - set of tools the wrap CLI rewrites, captured by - `headroom.cli.wrap_rtk_metrics`. There is no code path where a malicious client can drive label cardinality unbounded. ## See also -- `docs/rtk-architecture.md` — why RTK lives wrap-side, not proxy-side. - `crates/headroom-proxy/src/observability/` — implementation. - `REALIGNMENT/09-phase-G-rtk-observability.md` — spec. - `REALIGNMENT/10-phase-H-python-retirement.md` — H1 acceptance gate. diff --git a/docs/rtk-architecture.md b/docs/rtk-architecture.md deleted file mode 100644 index 35ff9aadf..000000000 --- a/docs/rtk-architecture.md +++ /dev/null @@ -1,122 +0,0 @@ -# RTK architecture — why wrap-CLI only - -**Status:** decided. Locked at Phase G PR-G3 (2026-05). -**Owner:** Headroom realignment. - -## TL;DR - -**RTK is a wrap-CLI hook, not a proxy-side compressor.** The Headroom -proxy does NOT invoke RTK on tool-result content. Future contributors -who consider moving RTK into the proxy hot path: read this doc first. - -## Background - -RTK (Realtime Token Kompress) rewrites shell **commands** at exec -time so that a `git diff` or `grep` invocation emits a more -compressed output before the agent ever ingests it. RTK runs in the -wrap-CLI tail — `headroom wrap claude`, `headroom wrap codex`, etc. -— where it installs a `~/.rtk/bin/rtk` shim ahead of the agent CLI -and intercepts shelled-out subprocesses. - -It surfaces value in two places: -1. **Tokens saved per invocation** — measured by `rtk gain --format json`. -2. **Tokens saved per session** — aggregated at wrap-session end. - -Both signals feed `wrap_rtk_invocations_total` and -`wrap_rtk_tokens_saved_per_session` (registered by the Rust proxy's -observability surface so a single `/metrics` scrape exposes the full -picture). - -## Proxy-side RTK was considered and rejected - -At Phase G scoping, three reviewers floated the idea of invoking -RTK on the **proxy** side: when a `tool_result` block flows -upstream, dispatch it through RTK to shrink the content before it -hits the model. - -**Decision: rejected.** Three load-bearing reasons. - -### 1. Cache hot zone risk - -The proxy's Phase B cache-safety contract pins `tool_result` -content as part of the cache hot zone. Compression there bursts -the prompt cache because the rewritten bytes diverge from the -canonical wire bytes the upstream cached. Phase B PR-B2 → PR-B7 -spent ~3000 LOC carving the live-zone-only surface specifically -to prevent this class of cache-invalidation. Inserting RTK -proxy-side would re-introduce it. - -### 2. Parallel implementation with `log_compressor.rs` - -The Rust proxy already has a `crates/headroom-core/src/transforms/log_compressor.rs` -that compresses **tool output text** in the live zone. It uses the -same heuristics RTK uses (whitespace de-dup, line de-dup, -file-listing collapse) but invoked at the proxy's per-block -dispatcher rather than at the shell exec boundary. Adding RTK -proxy-side would mean two implementations of the same compression -in the same hot path; "no silent fallbacks, no parallel impls" is -explicit project policy. - -### 3. Command-rewrite vs output-rewrite — different value propositions - -RTK rewrites **commands** before they execute. The -`git log --oneline` you typed becomes `git log --oneline -n 50` -because RTK has learned that the first 50 commits are usually -enough context. That's a fundamentally different mechanism from -compressing the **output** of an unmodified command. A proxy-side -invocation would skip the command-rewrite half — the half that -generates the largest savings on heavy shell workloads — and only -catch the output side, which is already covered by -`log_compressor` and `code_compressor`. - -## What the proxy does provide - -Per Phase G PR-G3, the proxy exposes RTK-derived metrics via its -registry: - -- `wrap_rtk_invocations_total{tool}` — driven by the wrap-CLI - polling `rtk gain --format json` and incrementing the registered - counter by the delta since last poll. -- `wrap_rtk_tokens_saved_per_session` — emitted at wrap-session - close. - -This keeps the operator dashboard single-pane-of-glass without -re-implementing RTK inside the proxy. - -## What the wrap CLI does - -Every `headroom wrap ` subcommand: - -1. Ensures the RTK binary is installed via `_ensure_rtk_binary()`. -2. Injects the `` block into the - agent's instruction file (e.g. `AGENTS.md`, `.cursorrules`). -3. Spawns the proxy and the agent CLI side-by-side. -4. Polls `rtk gain --format json` on a 5-second memoization window - and feeds the delta into the proxy's metric registry. - -See `headroom/cli/wrap/` for the per-agent shims. - -## Re-litigation policy - -A change to this architecture should: - -1. Quote the live-zone-only contract from - `REALIGNMENT/04-phase-B-live-zone.md` and explain why the - cache-burst risk is acceptable. -2. Show measurements (not estimates) that proxy-side RTK adds value - beyond `log_compressor.rs` on real production traffic. -3. Have an exit ramp: a CLI flag to disable proxy-side RTK without - reverting the wrap-CLI integration. - -Without all three, treat the proposal as a regression and link this -doc. - -## References - -- `REALIGNMENT/09-phase-G-rtk-observability.md` — Phase G plan. -- `REALIGNMENT/04-phase-B-live-zone.md` — cache hot-zone contract. -- `headroom/cli/wrap/` — wrap-CLI implementation. -- `crates/headroom-core/src/transforms/log_compressor.rs` — the - proxy-side log compressor RTK would parallel. -- 2026-05-01 user direction message archived in - `project_compression_realignment_2026_05` memory note. diff --git a/e2e/wrap/run.py b/e2e/wrap/run.py index c779b7042..4546ffa8d 100644 --- a/e2e/wrap/run.py +++ b/e2e/wrap/run.py @@ -21,7 +21,6 @@ import httpx REPO_ROOT = Path("/workspace") PLUGIN_DIR = REPO_ROOT / "plugins" / "openclaw" SDK_DIR = REPO_ROOT / "sdk" / "typescript" -RTK_MARKER = "" PROXY_PORT = 28887 CODEX_PORT = 28888 AIDER_PORT = 28889 @@ -368,25 +367,10 @@ def create_shims(shim_dir: Path) -> None: raise SystemExit(0) """ ) - rtk_shim = textwrap.dedent( - """\ - #!/usr/bin/env python3 - from __future__ import annotations - - import sys - - if "--version" in sys.argv: - print("rtk e2e-shim") - else: - print("rtk shim") - raise SystemExit(0) - """ - ) write_executable(shim_dir / "claude", generic_shim) write_executable(shim_dir / "codex", codex_shim) write_executable(shim_dir / "aider", generic_shim) write_executable(shim_dir / "opencode", generic_shim) - write_executable(shim_dir / "rtk", rtk_shim) def start_mock_server(port: int) -> tuple[MockOpenAIServer, threading.Thread]: @@ -569,16 +553,6 @@ def verify_codex_wrap( cwd=project_dir, timeout=120, ) - # RTK guidance for Codex is global-only (#1240): it is injected into - # ~/.codex/AGENTS.md, never a project-level AGENTS.md. A project AGENTS.md is - # written only when `wrap codex --memory` is used (for memory guidance), which - # this scenario does not exercise. - global_agents = Path(base_env["HOME"]) / ".codex" / "AGENTS.md" - assert_true(global_agents.exists(), "Codex wrap should create ~/.codex/AGENTS.md") - assert_true( - RTK_MARKER in global_agents.read_text(encoding="utf-8"), "Missing global RTK marker" - ) - config_path = Path(base_env["HOME"]) / ".codex" / "config.toml" assert_true( config_path.exists(), @@ -688,13 +662,6 @@ def verify_aider_wrap(base_env: dict[str, str], project_dir: Path, log_dir: Path cwd=project_dir, timeout=120, ) - conventions = project_dir / "CONVENTIONS.md" - assert_true(conventions.exists(), "Aider wrap should create CONVENTIONS.md") - assert_true( - RTK_MARKER in conventions.read_text(encoding="utf-8"), - "Aider wrap should inject RTK instructions", - ) - entries = read_jsonl(log_dir / "aider.jsonl") assert_true(len(entries) > 0, "Aider shim should have been invoked") env_vars = entries[-1]["env"] @@ -745,84 +712,52 @@ def verify_cursor_wrap(base_env: dict[str, str], project_dir: Path) -> None: "Cursor wrap should print the Anthropic base URL override", ) wait_for_http(f"http://127.0.0.1:{port}/health", timeout=15) - # rtk registers a native Cursor hook (rtk init --agent cursor) when it - # can (~/.cursor exists); headroom only falls back to injecting - # .cursorrules text if that registration fails (GH #756). Accept - # either outcome rather than assuming the fallback path. - cursorrules = project_dir / ".cursorrules" - cursor_hooks_json = Path(base_env["HOME"]) / ".cursor" / "hooks.json" - native_hook_registered = ( - cursor_hooks_json.exists() and "rtk" in cursor_hooks_json.read_text(encoding="utf-8") - ) - if not native_hook_registered: - assert_true( - cursorrules.exists(), - "Cursor wrap should create .cursorrules when the native rtk hook is unavailable", - ) - assert_true( - RTK_MARKER in cursorrules.read_text(encoding="utf-8"), - "Cursor wrap should inject RTK instructions", - ) finally: stop_process(proc) def verify_cline_wrap(base_env: dict[str, str], project_dir: Path) -> None: - """Smoke test: `wrap cline --prepare-only` writes RTK guidance to .clinerules.""" + """Smoke test: `wrap cline --prepare-only` exits clean. + + These three wraps used to be verified by the hint-file guidance they wrote. + With the CLI context tools removed they produce no on-disk artifact, so the + remaining assertion is that the prepare path still runs without crashing — + ``run`` raises on a non-zero exit. + """ run( ["headroom", "wrap", "cline", "--prepare-only", "--port", str(CLINE_PORT)], env=base_env, cwd=project_dir, timeout=60, ) - clinerules = project_dir / ".clinerules" - assert_true(clinerules.exists(), "Cline wrap should create .clinerules") - assert_true( - RTK_MARKER in clinerules.read_text(encoding="utf-8"), - "Cline wrap should inject RTK instructions", - ) def verify_continue_wrap(base_env: dict[str, str], project_dir: Path) -> None: - """Smoke test: `wrap continue --prepare-only` injects RTK into .continue/config.json.""" + """Smoke test: `wrap continue --prepare-only` exits clean (see verify_cline_wrap).""" run( ["headroom", "wrap", "continue", "--prepare-only", "--port", str(CONTINUE_PORT)], env=base_env, cwd=project_dir, timeout=60, ) - config_file = project_dir / ".continue" / "config.json" - assert_true(config_file.exists(), "Continue wrap should create .continue/config.json") - data = json.loads(config_file.read_text(encoding="utf-8")) - system_message = data.get("systemMessage", "") - assert_true( - RTK_MARKER in system_message, - "Continue wrap should inject RTK instructions into systemMessage", - ) def verify_goose_wrap(base_env: dict[str, str], project_dir: Path) -> None: - """Smoke test: `wrap goose --prepare-only` writes RTK guidance to .goosehints.""" + """Smoke test: `wrap goose --prepare-only` exits clean (see verify_cline_wrap).""" run( ["headroom", "wrap", "goose", "--prepare-only", "--port", str(GOOSE_PORT)], env=base_env, cwd=project_dir, timeout=60, ) - goosehints = project_dir / ".goosehints" - assert_true(goosehints.exists(), "Goose wrap should create .goosehints") - assert_true( - RTK_MARKER in goosehints.read_text(encoding="utf-8"), - "Goose wrap should inject RTK instructions", - ) def verify_openhands_wrap(base_env: dict[str, str], project_dir: Path) -> None: - """Smoke test: `wrap openhands --prepare-only` exits clean and ensures rtk is present. + """Smoke test: `wrap openhands --prepare-only` exits clean. - OpenHands wires instructions via the OPENHANDS_INSTRUCTIONS env var at launch - time (no on-disk artifact), so --prepare-only just exercises the rtk-binary - setup path. The env-var wiring is covered by the unit tests. + This one is a real regression guard: openhands used to *require* the rtk + binary, so once rtk became opt-in the default path exited 1. Nothing is + stubbed here, so a reintroduced hard dependency fails the run. """ run( ["headroom", "wrap", "openhands", "--prepare-only", "--port", str(OPENHANDS_PORT)], @@ -953,9 +888,6 @@ def main() -> None: "PATH": f"{shim_dir}{os.pathsep}{base_env['PATH']}", "HEADROOM_E2E_LOG_DIR": str(log_dir), "OPENAI_TARGET_API_URL": "http://127.0.0.1:19001/v1", - # RTK is opt-in (off by default). These wrap smoke tests assert - # RTK-instruction injection, so exercise the RTK-on path. - "HEADROOM_RTK": "1", } ) @@ -987,19 +919,6 @@ def verify_opencode_wrap(base_env: dict[str, str], project_dir: Path, log_dir: P cwd=project_dir, timeout=120, ) - global_agents = Path(base_env["HOME"]) / ".config" / "opencode" / "AGENTS.md" - project_agents = project_dir / "AGENTS.md" - assert_true(global_agents.exists(), "Opencode wrap should create ~/.config/opencode/AGENTS.md") - assert_true(project_agents.exists(), "Opencode wrap should create project AGENTS.md") - assert_true( - RTK_MARKER in global_agents.read_text(encoding="utf-8"), - "Missing RTK marker in global AGENTS.md", - ) - assert_true( - RTK_MARKER in project_agents.read_text(encoding="utf-8"), - "Missing RTK marker in project AGENTS.md", - ) - entries = read_jsonl(log_dir / "opencode.jsonl") assert_true(len(entries) > 0, "Opencode shim should have been invoked") env_vars = entries[-1]["env"] diff --git a/headroom/audit/codex.py b/headroom/audit/codex.py index 934b085b3..d528519d9 100644 --- a/headroom/audit/codex.py +++ b/headroom/audit/codex.py @@ -1,8 +1,7 @@ """Codex transcript audit — read-pattern analysis for shell-based clients. Codex has no structured Read tool: it reads files through shell commands -(``cat``, ``sed -n 'a,bp'``, ``head``/``tail``, ``nl``) — frequently -wrapped by rtk (``rtk read ``, ``rtk proxy ``). This module +(``cat``, ``sed -n 'a,bp'``, ``head``/``tail``, ``nl``). This module classifies ``exec_command`` calls in Codex session transcripts (``~/.codex/sessions/**/*.jsonl``) and measures the read pattern so the read-maturation mechanism can be sized for Codex workloads. @@ -23,8 +22,8 @@ from collections import Counter from dataclasses import asdict, dataclass, field from pathlib import Path -# Programs whose output is file content. "read" is rtk's read command. -_READ_PROGS = frozenset({"cat", "sed", "head", "tail", "nl", "bat", "more", "read"}) +# Programs whose output is file content. +_READ_PROGS = frozenset({"cat", "sed", "head", "tail", "nl", "bat", "more"}) _SEARCH_PROGS = frozenset({"rg", "grep", "ugrep", "ag", "fd", "find"}) _BUILD_PROGS = frozenset({"python", "python3", "pytest", "cargo", "npm", "make", "uv", "ruff"}) _RANGE_RE = re.compile(r"^\d+([,:-]\d+)?p?$") @@ -57,19 +56,6 @@ class CodexAuditReport: return asdict(self) -def strip_wrappers(cmd: str) -> str: - """Peel rtk wrappers: ``rtk `` and ``rtk proxy ``.""" - c = cmd.strip() - while True: - if c.startswith("rtk "): - c = c[4:].strip() - continue - if c.startswith("proxy "): - c = c[6:].strip() - continue - return c - - def _resolve_path(path: str | None, workdir: str = "") -> str | None: if not path: return None @@ -129,7 +115,7 @@ def classify_command(cmd: str, workdir: str = "") -> tuple[str, str | None, bool Categories: read, search, git, edit, build/test, compound, other. For reads and edits, the path is resolved against ``workdir`` when relative. """ - c = strip_wrappers(cmd) + c = cmd.strip() try: toks = shlex.split(c) except ValueError: diff --git a/headroom/audit/maturation.py b/headroom/audit/maturation.py index e21b4cad3..18cab19ce 100644 --- a/headroom/audit/maturation.py +++ b/headroom/audit/maturation.py @@ -177,7 +177,7 @@ def simulate_codex_maturation(root: Path) -> MaturationSimReport: """Run the maturation simulation over Codex shell-based transcripts. Codex has no structured ``Read`` tool. It reads files through - ``exec_command`` calls such as ``cat``, ``sed -n``, and ``rtk read``. + ``exec_command`` calls such as ``cat``, ``sed -n``, and ``head``. This mirrors ``simulate_maturation`` with the Codex command classifier so ``headroom audit-reads --codex --simulate-maturation`` sizes the same read-maturation policy from Codex traffic instead of returning an diff --git a/headroom/cli/audit.py b/headroom/cli/audit.py index 26d2b7329..389801ce4 100644 --- a/headroom/cli/audit.py +++ b/headroom/cli/audit.py @@ -32,7 +32,7 @@ from .main import main "--codex", "codex_mode", is_flag=True, - help="Audit Codex transcripts instead (shell-based reads: cat/sed/rtk read). " + help="Audit Codex transcripts instead (shell-based reads: cat/sed/head). " "Default path becomes ~/.codex/sessions.", ) def audit_reads_cmd( diff --git a/headroom/cli/proxy.py b/headroom/cli/proxy.py index 513b14dd7..1a4dbe7ee 100644 --- a/headroom/cli/proxy.py +++ b/headroom/cli/proxy.py @@ -66,11 +66,6 @@ warnings.filterwarnings("ignore", category=UserWarning, module="huggingface_hub" # --------------------------------------------------------------------------- -_CONTEXT_TOOL_ENV = "HEADROOM_CONTEXT_TOOL" -_CONTEXT_TOOL_RTK = "rtk" -_CONTEXT_TOOL_LEAN_CTX = "lean-ctx" -_VALID_CONTEXT_TOOLS = {_CONTEXT_TOOL_RTK, _CONTEXT_TOOL_LEAN_CTX} - def _get_env_bool(name: str, default: bool) -> bool: val = os.environ.get(name) @@ -117,19 +112,6 @@ def _get_env_float_optional(name: str) -> float | None: raise click.ClickException(f"{name} must be a number, got {val!r}") from None -def _selected_context_tool() -> str: - raw = os.environ.get(_CONTEXT_TOOL_ENV, "").strip().lower().replace("_", "-") - if not raw: - return _CONTEXT_TOOL_RTK - if raw == "leanctx": - raw = _CONTEXT_TOOL_LEAN_CTX - if raw not in _VALID_CONTEXT_TOOLS: - raise click.ClickException( - f"{_CONTEXT_TOOL_ENV} must be one of: {', '.join(sorted(_VALID_CONTEXT_TOOLS))}" - ) - return raw - - @main.command() @click.option( "--port", @@ -1460,7 +1442,6 @@ Memory (Multi-Provider): from headroom.proxy.server import _get_code_aware_banner_status code_aware_line = f" Code-Aware: {_get_code_aware_banner_status(config)}" - context_tool_line = f" Context Tool: {_selected_context_tool()}" # Performance tuning section — only shown when at least one tuning var is active. _embed_socket = os.environ.get("HEADROOM_EMBEDDING_SERVER_SOCKET") or ( @@ -1490,7 +1471,6 @@ Starting proxy server... Memory: {memory_status} License: {license_status} {code_aware_line} -{context_tool_line} {extensions_line} {security_line} {stateless_line}{telemetry_line} diff --git a/headroom/cli/wrap.py b/headroom/cli/wrap.py index 0afd8ec65..45dfa623e 100644 --- a/headroom/cli/wrap.py +++ b/headroom/cli/wrap.py @@ -1,7 +1,7 @@ """Wrap CLI commands to run through Headroom proxy. Usage: - headroom wrap claude # Start proxy + context tool + claude + headroom wrap claude # Start proxy + claude headroom wrap copilot -- --model ... # Start proxy + launch GitHub Copilot CLI headroom wrap codex # Start proxy + OpenAI Codex CLI headroom wrap aider # Start proxy + aider @@ -11,7 +11,6 @@ Usage: headroom wrap cursor # Start proxy + print Cursor config instructions headroom wrap grok-build # Start proxy + configure Grok Build headroom wrap openclaw # Install + configure OpenClaw plugin - headroom wrap claude --no-context-tool # Without CLI context-tool setup headroom wrap claude --port 9999 # Custom proxy port headroom wrap claude -- --model opus # Pass args to claude """ @@ -29,7 +28,6 @@ import signal import socket import subprocess import sys -import tempfile import time import urllib.parse from collections.abc import Callable @@ -205,15 +203,52 @@ def _write_text(path: Path, content: str) -> None: fsutil.write_text(path, content) +def _read_settings_for_write(path: Path) -> dict[str, Any]: + """Read a Claude settings file that is about to be mutated, or refuse to write. + + Callers previously fell back to ``{}`` when the file existed but would not + parse, then wrote that back — turning a hand-edited typo or a transient read + error into total loss of the user's ``permissions``/``env``/``hooks``. Abort + instead, mirroring ``mcp_registry.claude._read_json_for_write``: a malformed + config is the user's to fix, and no Headroom feature is worth erasing it. + + An **empty** file is the one safe exception and is treated as ``{}``: there + are no settings in it to lose, and refusing would strand the user behind a + file they cannot see anything wrong with. A zero-byte settings.json is also + the classic residue of an interrupted non-atomic write (the failure mode + :func:`headroom.fsutil.write_text` now prevents), so recovering from it is + exactly right. Anything non-empty that will not parse is treated as data. + """ + if not path.exists(): + return {} + try: + raw = _read_text(path) + except OSError as exc: + raise click.ClickException( + f"could not read {path} ({exc}). Fix or move it, then re-run — " + "refusing to overwrite it and lose your settings." + ) from exc + if not raw.strip(): + return {} + try: + payload = json.loads(raw) + except json.JSONDecodeError as exc: + raise click.ClickException( + f"{path} is not valid JSON ({exc}). Fix or move it, then re-run — " + "refusing to overwrite it and lose your settings." + ) from exc + if not isinstance(payload, dict): + raise click.ClickException( + f"{path} does not contain a JSON object. Fix or move it, then re-run." + ) + return cast("dict[str, Any]", payload) + + def _append_text(path: Path, content: str) -> None: """Append to a text file as UTF-8 without translating line endings.""" fsutil.append_text(path, content) -_CONTEXT_TOOL_ENV = "HEADROOM_CONTEXT_TOOL" -_CONTEXT_TOOL_RTK = "rtk" -_CONTEXT_TOOL_LEAN_CTX = "lean-ctx" -_VALID_CONTEXT_TOOLS = {_CONTEXT_TOOL_RTK, _CONTEXT_TOOL_LEAN_CTX} _AGENT_SAVINGS_TARGET_AGENTS = {"claude", "codex", "cursor", "grok", "grok_build", "opencode"} _WRAP_PROXY_TIMEOUT_ENV = "HEADROOM_WRAP_PROXY_TIMEOUT" _WRAP_PROXY_TIMEOUT_DEFAULT_SECONDS = 45 @@ -382,26 +417,6 @@ def _live_wrap_module() -> Any: return cast(Any, sys.modules[__name__]) -def _selected_context_tool() -> str: - """Return the configured CLI context tool. - - RTK remains the default for backward compatibility. Set - ``HEADROOM_CONTEXT_TOOL=lean-ctx`` to let lean-ctx configure the supported - coding agent instead. - """ - - raw = os.environ.get(_CONTEXT_TOOL_ENV, "").strip().lower().replace("_", "-") - if not raw: - return _CONTEXT_TOOL_RTK - if raw == "leanctx": - raw = _CONTEXT_TOOL_LEAN_CTX - if raw not in _VALID_CONTEXT_TOOLS: - raise click.ClickException( - f"{_CONTEXT_TOOL_ENV} must be one of: {', '.join(sorted(_VALID_CONTEXT_TOOLS))}" - ) - return raw - - def _module_available(module_name: str) -> bool: """Return whether an optional module is installed without importing it.""" @@ -713,38 +728,95 @@ def _start_proxy( stdio_log_file.close() -def _rtk_opt_in() -> bool: - """Whether RTK CLI-command filtering was explicitly enabled. +# CLI context tools (rtk, lean-ctx) were removed from Headroom. The selector is +# kept only long enough to fail loudly: it lives in shell profiles, scripts and +# CI jobs, and silently ignoring it would look like Headroom had stopped working. +# See :mod:`headroom.context_tool_cleanup`, which uninstalls what they left behind. +_RETIRED_CONTEXT_TOOL_ENV = "HEADROOM_CONTEXT_TOOL" +_RETIRED_CONTEXT_TOOL_MESSAGE = ( + "CLI context tools (rtk, lean-ctx) have been removed from Headroom: they " + "rewrote shell commands through a third-party binary Headroom no longer " + "manages. Drop --context-tool / --no-context-tool and unset " + f"{_RETIRED_CONTEXT_TOOL_ENV}; `headroom wrap` uninstalls what they left " + "behind on first run." +) - RTK is opt-in (off by default): turn it on with ``--rtk`` (which sets - ``HEADROOM_RTK=1``) or by exporting ``HEADROOM_RTK=1``. ``--no-rtk`` remains - accepted as a deprecated no-op. + +def _retired_context_tool_callback(ctx: Any, param: Any, value: str | None) -> str | None: + """Click eager callback: reject any surviving context-tool selection. + + Also checks the env var (the callback runs on every wrap subcommand, flag + passed or not), so an exported ``HEADROOM_CONTEXT_TOOL`` fails with the same + message instead of silently doing nothing. """ - return os.environ.get("HEADROOM_RTK", "").strip().lower() in ("1", "true", "yes", "on") - - -def _rtk_flag_callback(ctx: Any, param: Any, value: bool) -> bool: - """Click eager callback: ``--rtk`` sets HEADROOM_RTK so the central RTK gate - (:func:`_rtk_opt_in`) sees the opt-in without threading a param through every - wrap subcommand.""" - if value: - os.environ["HEADROOM_RTK"] = "1" + if value is not None or os.environ.get(_RETIRED_CONTEXT_TOOL_ENV, "").strip(): + raise click.ClickException(_RETIRED_CONTEXT_TOOL_MESSAGE) return value -# Shared opt-in flag applied to every ``wrap`` subcommand. ``expose_value=False`` -# so no subcommand signature changes; it works purely through HEADROOM_RTK. -_rtk_option = click.option( - "--rtk", - is_flag=True, - default=False, +# Applied to every ``wrap`` subcommand. ``expose_value=False`` so no subcommand +# signature carries it; both spellings the flag ever had are accepted and +# rejected with one message. +_retired_context_tool_option = click.option( + "--context-tool", + "--no-context-tool", + default=None, + is_flag=False, + flag_value="", + metavar="TOOL", expose_value=False, is_eager=True, - callback=_rtk_flag_callback, - help="Enable RTK CLI-command filtering (opt-in; off by default). Also enabled by HEADROOM_RTK=1.", + hidden=True, + callback=_retired_context_tool_callback, + help="Removed: CLI context tools (rtk, lean-ctx) are no longer supported.", ) +def _should_purge_context_tools(ctx: click.Context) -> bool: + """Whether this invocation should run the retired-context-tool cleanup. + + Two exemptions, both about not doing filesystem surgery from a command the + caller expects to be inert: + + * ``wrap selfheal`` — runs from a SessionStart hook on every new + conversation, where rewriting ``~/.claude.json`` would race Claude Code's + own writer for no benefit. + * any ``--help`` invocation — help must stay read-only. Click resolves a + subcommand's help *after* this group callback, so it cannot be detected + from ``ctx``; scanning argv is blunt but correct, and a false positive only + defers the cleanup to the next real run. + """ + if ctx.invoked_subcommand == "selfheal": + return False + return not any(arg in ("--help", "-h") for arg in sys.argv[1:]) + + +def _report_context_tool_purge() -> None: + """Uninstall leftover rtk / lean-ctx state, reporting anything removed. + + Removing the integration code cannot help a machine that already ran the old + default: the Claude ``PreToolUse`` hook, the vendored binaries and the + injected hint-file guidance are all durable on disk. Running this once per + ``wrap`` / ``unwrap`` invocation is what actually makes the tools go away. + Silent when there is nothing to do, which is the steady state after the first + run, and never fatal — a cleanup failure must not block launching the tool. + + Reports on **stderr**: some subcommands (``wrap/unwrap openclaw + --prepare-only``) emit machine-readable JSON on stdout as their entire + contract, and a human cleanup line prepended to it breaks every + ``json.loads(stdout)`` consumer on the one run that has something to remove. + """ + from headroom.context_tool_cleanup import purge_context_tool_artifacts + + try: + removed = purge_context_tool_artifacts() + except Exception as exc: # pragma: no cover - defensive, cleanup is best-effort + click.echo(f"Warning: could not finish removing retired CLI context tools: {exc}", err=True) + return + for line in removed: + click.echo(f"Retired CLI context tool cleanup: {line}", err=True) + + def _serena_instructions_opt_in() -> bool: """Whether Serena instruction injection into the agent's hint file is enabled. @@ -775,7 +847,7 @@ def _serena_instructions_flag_callback(ctx: Any, param: Any, value: bool) -> boo # Shared opt-in flag for Serena instruction injection, applied to the wrap # subcommands that set up Serena. ``expose_value=False`` so no subcommand # signature changes; it works purely through HEADROOM_SERENA_INSTRUCTIONS. Same -# approach as _rtk_option above — set via the callback with NO ``envvar=`` so the +# approach as _code_memory_option below — set via the callback with NO ``envvar=`` so the # settings_store drift guard doesn't flag it. _serena_instructions_option = click.option( "--serena-instructions", @@ -792,7 +864,7 @@ _serena_instructions_option = click.option( # The code-memory MCP is Serena by default; turn it off with --code-memory none. # Selection flows through HEADROOM_CODE_MEMORY (set by the eager --code-memory # callback) so it works the same on every agent without threading a param -# through each subcommand — the same approach as _rtk_option above. +# through each subcommand — the same approach as _serena_instructions_option above. _CODE_MEMORY_ENV = "HEADROOM_CODE_MEMORY" _CODE_MEMORY_SERENA = "serena" _CODE_MEMORY_NONE = "none" @@ -850,165 +922,11 @@ _code_memory_option = click.option( ) -def _setup_rtk(verbose: bool = False) -> Path | None: - """Ensure rtk is installed and hooks are registered.""" - if not _rtk_opt_in(): - return None - from headroom.rtk import get_rtk_path - from headroom.rtk.installer import ensure_rtk, register_claude_hooks - - rtk_path = get_rtk_path() - - if rtk_path: - if verbose: - click.echo(f" rtk found at {rtk_path}") - else: - click.echo(" Downloading rtk (Rust Token Killer)...") - rtk_path = ensure_rtk() - if rtk_path: - click.echo(f" rtk installed at {rtk_path}") - else: - click.echo(" rtk download failed — continuing without it") - return None - - # Register hooks (idempotent) - if register_claude_hooks(rtk_path): - if verbose: - click.echo(" rtk hooks registered in Claude Code") - try: - linked = _ensure_rtk_on_path(rtk_path) - if linked and verbose: - click.echo(f" rtk linked onto PATH at {linked}") - except Exception as e: - if verbose: - click.echo(f" rtk PATH link skipped: {e}") - else: - click.echo(" rtk hook registration failed — continuing without it") - - return rtk_path - - -def _ensure_rtk_on_path(rtk_path: Path, path_dirs: list[str] | None = None) -> Path | None: - """Make the Headroom-managed rtk resolvable as a bare ``rtk`` on PATH. - - ``rtk init --global --auto-patch`` writes ``~/.claude/hooks/rtk-rewrite.sh``, - and ``rtk rewrite`` emits a bare ``rtk`` token at runtime that the hook feeds - back to the shell — so bare ``rtk`` has to resolve on PATH regardless of the - hook's contents. Since ``~/.headroom/bin`` (where Headroom installs rtk) is - not on PATH by default, that lookup fails and compression silently never - runs (issue #487). - - An earlier fix rewrote the generated hook to hard-code rtk's absolute path. - That mutates the hook *after* ``rtk init`` bakes in its expected SHA-256, so - rtk's integrity guard rejects it (``hook integrity check FAILED … RTK will - not execute``) and only absolutizes the hook's own ``rtk`` call — not the - bare ``rtk`` that ``rtk rewrite`` emits at runtime (issue #1631). Instead, - leave the canonical hook untouched and link the managed binary into a PATH - directory so bare ``rtk`` resolves. - - Idempotent and conservative: - * no-op if a ``rtk`` already resolves on PATH (managed or system); - * no-op on Windows (symlinks need privilege; hooks resolve differently); - * only creates/refreshes a symlink Headroom owns — never clobbers an - existing real file or foreign binary. - - Returns the link path that was created or already correct, else ``None``. - """ - if sys.platform == "win32": - return None - - # A bare `rtk` already resolves — the hook will find it, nothing to do. - if shutil.which("rtk"): - return None - - if path_dirs is None: - path_dirs = os.environ.get("PATH", "").split(os.pathsep) - - preferred = Path.home() / ".local" / "bin" - - # Prefer ~/.local/bin (conventionally on PATH), then any other PATH dir. - ordered: list[Path] = [] - if str(preferred) in path_dirs: - ordered.append(preferred) - for entry in path_dirs: - if not entry: - continue - candidate = Path(entry) - if candidate not in ordered: - ordered.append(candidate) - - target = rtk_path.resolve() - - for target_dir in ordered: - link = target_dir / "rtk" - try: - # Existing correct link — done. - if link.is_symlink() and link.resolve() == target: - return link - # Never clobber a real file or a link pointing elsewhere. - if link.exists() or link.is_symlink(): - continue - # Create ~/.local/bin on demand; other PATH dirs must already exist. - if target_dir == preferred: - target_dir.mkdir(parents=True, exist_ok=True) - if not target_dir.is_dir() or not os.access(target_dir, os.W_OK): - continue - link.symlink_to(target) - return link - except OSError: - continue - - return None - - -def _setup_lean_ctx_agent(agent: str, verbose: bool = False) -> Path | None: - """Run lean-ctx agent setup for the requested coding tool.""" - - from headroom.lean_ctx import get_lean_ctx_path - from headroom.lean_ctx.installer import ensure_lean_ctx - - lean_ctx = get_lean_ctx_path() - if not lean_ctx: - click.echo(" Downloading lean-ctx...") - lean_ctx = ensure_lean_ctx() - if not lean_ctx: - click.echo(" lean-ctx download failed — continuing without it") - return None - - try: - with tempfile.TemporaryDirectory(prefix="headroom-lean-ctx-") as setup_cwd: - # lean-ctx writes project-local files when initialized from a git - # checkout. Run from a non-project directory so setup is limited to - # home-scoped agent config such as ~/.codex or ~/.claude. - result = run( - [str(lean_ctx), "init", "--agent", agent], - capture_output=True, - text=True, - timeout=30, - cwd=setup_cwd, - ) - except Exception as e: - click.echo(f" lean-ctx setup failed — continuing without it: {e}") - return None - - if result.returncode != 0: - detail = (result.stderr or result.stdout).strip() - suffix = f": {detail}" if detail else "" - click.echo(f" lean-ctx setup failed — continuing without it{suffix}") - return None - - if verbose: - detail = result.stdout.strip() - if detail: - click.echo(f" lean-ctx configured for {agent}: {detail}") - else: - click.echo(f" lean-ctx configured for {agent}") - return lean_ctx - - # Hook-command markers Headroom manages in Claude settings.json. unwrap drops -# any hook entry whose command contains one of these. -_HEADROOM_HOOK_MARKERS = ("rtk-rewrite", "headroom-init-claude") +# any hook entry whose command contains one of these. (Retired rtk / lean-ctx +# hooks are removed separately, by +# headroom.context_tool_cleanup.purge_context_tool_artifacts.) +_HEADROOM_HOOK_MARKERS = ("headroom-init-claude",) # Env vars Headroom's init/wrap inject into Claude settings.json; unwrap removes # them. ENABLE_TOOL_SEARCH keeps Claude Code's tool deferral on behind the proxy @@ -1021,16 +939,14 @@ _HEADROOM_ENV_KEYS = ("ANTHROPIC_BASE_URL", "ENABLE_TOOL_SEARCH") _WRAP_SELFHEAL_HOOK_MARKER = "headroom-wrap-selfheal" -def _remove_claude_rtk_hooks(settings_path: Path | None = None) -> bool: +def _remove_claude_managed_hooks(settings_path: Path | None = None) -> bool: """Remove Headroom-managed entries from Claude settings.json. - Reverses what ``headroom init claude`` and ``rtk init --auto-patch`` add: + Reverses what ``headroom init claude`` adds: * PreToolUse / SessionStart hooks whose command contains a Headroom marker - (``rtk-rewrite`` or ``headroom-init-claude``), and + (``headroom-init-claude``), and * the ``ANTHROPIC_BASE_URL`` proxy-routing env var. - Unrelated settings and user-authored hooks are left untouched. (Previously - this only matched ``rtk-rewrite`` and returned early when no hooks existed, - so init's env + hooks survived unwrap.) + Unrelated settings and user-authored hooks are left untouched. """ path = settings_path or (Path.home() / ".claude" / "settings.json") @@ -1375,14 +1291,7 @@ def _ensure_claude_wrap_selfheal_hook(settings_path: Path) -> None: call mid-session, where a transient probe blip could clear a live session. Idempotent — an existing entry carrying the marker is not duplicated. """ - payload: dict[str, Any] = {} - if settings_path.exists(): - try: - payload = json.loads(_read_text(settings_path)) - except (OSError, json.JSONDecodeError): - payload = {} - if not isinstance(payload, dict): - payload = {} + payload = _read_settings_for_write(settings_path) hooks = dict(payload.get("hooks") or {}) if isinstance(payload.get("hooks"), dict) else {} entries = ( list(hooks.get("SessionStart") or []) if isinstance(hooks.get("SessionStart"), list) else [] @@ -1419,7 +1328,7 @@ def _ensure_claude_wrap_selfheal_hook(settings_path: Path) -> None: def _remove_claude_wrap_selfheal_hook(settings_path: Path) -> bool: """Remove the SessionStart self-heal hook that ``wrap claude`` installed (#2221). - Mirrors ``_remove_claude_rtk_hooks`` but matches only the wrap self-heal + Mirrors ``_remove_claude_managed_hooks`` but matches only the wrap self-heal marker in the project-local settings.local.json. Returns True if anything was removed. Unrelated hooks and user-authored entries are left untouched. """ @@ -1497,14 +1406,7 @@ def _write_claude_wrap_base_url( detected and self-healed (issue #1768). """ path = settings_path or (Path.cwd() / ".claude" / "settings.local.json") - payload: dict[str, Any] = {} - if path.exists(): - try: - payload = json.loads(_read_text(path)) - except (OSError, json.JSONDecodeError): - payload = {} - if not isinstance(payload, dict): - payload = {} + payload = _read_settings_for_write(path) env_map = dict(payload.get("env") or {}) if isinstance(payload.get("env"), dict) else {} key = _claude_wrap_base_url_env_key(foundry_mode=foundry_mode, vertex_mode=vertex_mode) previous = env_map.get(key) @@ -1683,8 +1585,8 @@ def _ensure_serena_dashboard_disabled(*, verbose: bool = False) -> None: # Marker-fenced guidance steering the agent toward Serena's symbol tools. -# Injected only when Serena is the active code-memory engine. Mirrors the RTK -# instruction block (idempotent, marker-guarded). +# Injected only when Serena is the active code-memory engine (idempotent, +# marker-guarded). _SERENA_MARKER = "" SERENA_INSTRUCTIONS_BLOCK = """\ @@ -2030,57 +1932,6 @@ def _setup_coding_compressor(registrar: Any, *, serena_context: str, **kwargs: A _CBM_MCP_SERVER_NAME = "codebase-memory-mcp" -# rtk instructions for tools without hook support (Codex, Cursor, Aider). -# These get injected into AGENTS.md / .cursorrules so the LLM voluntarily -# uses rtk-prefixed commands. Kept concise to minimize instruction overhead. -RTK_INSTRUCTIONS_BLOCK = """\ - -# RTK (Rust Token Killer) - Token-Optimized Commands - -When running shell commands, **always prefix with `rtk`**. This reduces context -usage by 60-90% with zero behavior change. If rtk has no filter for a command, -it passes through unchanged — so it is always safe to use. - -## Key Commands -```bash -# Git (59-80% savings) -rtk git status rtk git diff rtk git log - -# Files & Search (60-75% savings) -rtk ls rtk read rtk grep -rtk find rtk diff - -# Test (90-99% savings) — shows failures only -rtk pytest tests/ rtk cargo test rtk test - -# Build & Lint (80-90% savings) — shows errors only -rtk tsc rtk lint rtk cargo build -rtk prettier --check rtk mypy rtk ruff check - -# Analysis (70-90% savings) -rtk err rtk log rtk json -rtk summary rtk deps rtk env - -# GitHub (26-87% savings) -rtk gh pr view rtk gh run list rtk gh issue list - -# Infrastructure (85% savings) -rtk docker ps rtk kubectl get rtk docker logs - -# Package managers (70-90% savings) -rtk pip list rtk pnpm install rtk npm run