mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
10 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
JD Davis
|
632cb81dbe
|
fix(learn): surface Codex analysis failures (#3016)
## Description `headroom learn` could invoke Codex CLI from a non-Git working directory without Codex’s required bypass flag. The resulting backend error was then swallowed by the analyzer and rendered as “No actionable patterns found” with exit code 0. This fixes both coupled defects so Codex can run from discovered project locations and genuine analysis failures remain visible and machine-detectable. Closes #3008 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `--skip-git-repo-check` to the Codex CLI analysis backend command. - Added an explicit `analysis_error` result field instead of conflating backend failure with an empty recommendation set. - Kept multi-project analysis best-effort, while returning exit code 1 after any project analysis fails. - Prevented failed analysis from printing a misleading no-pattern success message. - Added analyzer and CLI regression coverage for the command and failure-propagation contracts. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run pytest -q tests/test_learn/test_analyzer.py tests/test_cli_learn.py 102 passed in 2.34s uv run pytest -q tests/test_learn tests/test_cli_learn.py 257 passed, 7 skipped in 3.11s uv run mypy headroom Success: no issues found in 520 source files uv run ruff check <changed files> All checks passed! uv run ruff format --check <changed files> 5 files already formatted uv run pytest tests scripts/tests --splits 4 --group N --tb=short -q shard 1: 2766 passed, 140 skipped in 174.08s shard 2: 2699 passed, 207 skipped in 60.00s shard 3: 2822 passed, 84 skipped in 76.10s shard 4: 2734 passed, 172 skipped in 80.29s ``` ## Real Behavior Proof - Environment: macOS arm64, Python 3.13, Codex CLI 0.147.0-compatible command surface, current `main` including #2996. - Exact command / steps: verified `codex exec --help`; exercised `_call_cli_llm` with a captured subprocess command; invoked the Click command with a simulated Codex nonzero backend result. - Observed result: the subprocess command is `codex exec --skip-git-repo-check`; backend failure text is printed as `Analysis failed`, the misleading no-pattern message is absent, and the CLI exits 1. - Not tested: live paid Codex analysis against production account credentials; subprocess and CLI behavior are covered deterministically. ## Runtime Rollout Safety - Rollout-managed feature(s): none; this is CLI-only failure handling. - Minimum rollout channel: normal patch release after exact-head CI is entirely green. - Stable/default behavior changed: failed LLM analysis now exits nonzero instead of reporting success; successful and genuinely empty analyses are unchanged. - Kill switch / disable path: select another backend with `HEADROOM_LEARN_CLI` or `--model` if Codex CLI is unavailable. - Unsafe override required: none. - Qualification impact: all four Python CI shards, static checks, security checks, and command-level regression tests must pass. - Rollback path: fix forward through a human-reviewed corrective PR; no persisted data or migration is involved. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project’s style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation — inline result-contract documentation; no separate user guide change is required - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable; command-line backend and exit semantics only. ## Additional Notes Human review only. No merge or auto-merge is configured. This corrects the root failure and exit semantics without extending any timeout. |
||
|
JD Davis
|
3077ac81e8
|
feat: add deterministic runtime rollout controls (#1490)
## Description Establish one centrally resolved, observable, deterministic, versioned runtime rollout-control mechanism for Headroom. Runtime rollout controls which behaviors an already-built artifact may expose; it does not select or qualify a Headroom release/version. ## Type of Change - [x] New feature (non-breaking change that adds functionality) - [x] Bug fix (non-breaking change that fixes rollout enforcement regressions) - [x] Documentation update - [x] Code refactoring (no functional changes) ## Changes Made - Added `RolloutChannel`, `HEADROOM_ROLLOUT_CHANNEL`, `--rollout-channel`, and a versioned immutable `RolloutSnapshot` shared by Python configuration boundaries. - Added schema/policy versions, canonical registry and snapshot SHA-256 identities, per-feature decision reasons, disable precedence, unsafe qualification poisoning, strict CLI validation, and fail-closed environment handling. - Added `headroom rollout status --json`, Python `/stats.rollout`, and Rust `/rollout/status` runtime provenance. - Added equivalent Rust snapshot semantics and shared Python/Rust policy vectors while retaining language-specific feature registries. - Enforced rollout policy at alternate Python server composition roots so `HEADROOM_READ_MATURATION=1` cannot bypass its beta gate. - Preserved typed rollout snapshots across multi-worker serialization with schema, policy, registry, snapshot-digest, type, and feature-name validation. - Made loopback runtime output-shaper updates replace the immutable snapshot atomically for request readers, retain explicit request/disable provenance, preserve channel and kill-switch precedence, invalidate cached stats, and return the effective rollout decision. - Made `headroom learn --verbosity --apply` report a channel-blocked update instead of claiming the shaper is live. - Made explicit CLI feature flags fail loudly when their current channel blocks them. - Made persistent interceptor installation select canary automatically, or reject an explicitly insufficient channel unless the break-glass override is set. - Updated architecture, proxy, rollout, learn, and output-shaper documentation with required channels and hot-reload semantics. ## Testing - [x] Unit tests pass - [x] Linting passes (`ruff check .` and `ruff format --check .`) - [x] Type checking passes (`mypy headroom --ignore-missing-imports`) - [x] New regression tests added for every corrected behavior - [x] Rust tests and production-target Clippy pass - [x] Documentation build passes ### Test Output ```text Focused rollout coverage suite 57 passed; headroom.rollout + rollout CLI: 98% coverage Affected proxy/rollout/transform/governance suites 222 passed; 0 failed Final changed regression suites 100 passed; 0 failed Cross-module hot-reload isolation regression 6 passed; 0 failed cargo test -p headroom-core -p headroom-proxy --quiet headroom-core: 924 passed; 1 ignored headroom-proxy and integration suites: all passed cargo clippy -p headroom-core -p headroom-proxy --lib --bins -- -D warnings cargo fmt --all -- --check ruff check . ruff format --check . mypy headroom --ignore-missing-imports git diff --check All passed cd docs && npm run build Compiled successfully; 164 static pages generated ``` The unsharded Windows-only CI selection exposed unrelated baseline failures, principally the existing `sqlite:///C:\\...` URL parser producing an invalid `\\C:\\...` path. At commit ` |
||
|
gglucass
|
27a5468349
|
fix(learn): aggregate verbosity baselines across projects instead of overwriting (#1288)
## Description `headroom learn --verbosity --apply --all` was building the output-shaper's savings baseline from only **one** project. `_run_verbosity` wrote the savings ledger *inside* the per-project loop (`ledger.baseline = baseline; ledger.save(...)`), so each project replaced the previous baseline and only the last project processed survived — frequently a near-empty one. The synthetic-control estimate that `/stats` exposes (`savings.by_layer.output_shaping`) was then computed against a tiny, unrepresentative sample. This PR makes `--all` aggregate across every targeted project and write the ledger **once**, after the loop. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `BaselineModel.merge()` / `_Accum.merge()` (`headroom/proxy/output_savings.py`): fold one baseline into another. The accumulators hold additive online stats (`n` / `sum` / `sumsq`), so merging is element-wise and order-independent — identical to having observed both corpora against a single model. - `_run_verbosity` (`headroom/cli/learn.py`): accumulate a single `BaselineModel` across all targeted projects and persist it once after the loop, instead of overwriting per project. The applied verbosity level now comes from the project with the most samples (strongest signal) rather than whichever sorted last. Single-project runs are unchanged (an aggregate of one). When no transcripts are found, it prints a clear message and writes nothing. - Tests: unit test for `BaselineModel.merge`; CLI test that `--all --apply` across two projects aggregates both strata (totals summed, not last-wins) and applies the busier project's level. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_output_savings.py tests/test_cli_learn.py tests/test_verbosity_learn.py -q tests/test_output_savings.py ............................... [ 54%] tests/test_cli_learn.py ........... [ 73%] tests/test_verbosity_learn.py ............... [100%] ============================== 57 passed in 0.51s ============================== $ uv run ruff check headroom/cli/learn.py headroom/proxy/output_savings.py All checks passed! $ uv run mypy headroom/cli/learn.py headroom/proxy/output_savings.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: macOS, Python 3.12.13, this branch off `upstream/main`. - Exact command / steps: `headroom learn --verbosity --apply --all` (run across a multi-project transcript corpus), then inspect `~/.headroom/output_savings.json` (`baseline.glob.n`); compared against `headroom learn --verbosity --apply` for a single busy project. - Symptom (pre-fix, installed build): `headroom learn --verbosity --apply --all` across a multi-project transcript corpus wrote `~/.headroom/output_savings.json` with `baseline.glob.n = 2` (the last project processed was a near-empty `…/venv/bin` dir), while targeting a single busy project gave `baseline.glob.n = 15658`. - With this change: the new CLI test (`test_verbosity_all_apply_aggregates_baselines_across_projects`) drives `--all --apply` over two projects (3 samples + 1 sample) and asserts the persisted ledger has `total_samples == 4` with both strata present, plus the busier project's level applied. - Observed result: aggregated baseline persisted once; both strata retained; level taken from the higher-sample project. - Not tested: re-running the patched `--all` end-to-end on a live multi-project machine (covered instead by the unit merge-math test and the faked-`analyze` CLI test). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — CLI/behavioral change. ## Additional Notes - No linked issue (`Closes #` left blank intentionally). - Documentation checklist item is N/A — no user-facing docs describe the per-project overwrite behavior. - Level-selection note: for `--all`, the applied verbosity level is now deterministic (most-samples project) instead of last-processed; this is the intended improvement, not a behavior to preserve. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
Focused Instability
|
ced75e4718
|
feat(learn): write per-project learnings to CLAUDE.local.md by default (#1115)
## Description `headroom learn` wrote per-project learnings into the project's `CLAUDE.md`, which Claude Code treats as team-shared and git-tracked. That meant machine-specific absolute paths and tool-discovery byproducts polluted the shared file for every teammate. This switches the default to the personal, gitignored `CLAUDE.local.md`, adds a `--target` override, and migrates any stale block out of `CLAUDE.md`. Closes #1072. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `ClaudeCodeWriter` now writes CONTEXT_FILE recommendations to `CLAUDE.local.md` by default instead of `CLAUDE.md` (the home-directory case still uses `~/.claude/CLAUDE.md`, which is personal global memory). - Added a `--target` flag (Claude Code only) and `set_context_target()` to override the destination — e.g. `--target CLAUDE.md` to opt back into the shared file, or any relative/absolute path. - On first run after upgrade, a stale Headroom block left in `CLAUDE.md` is moved into `CLAUDE.local.md` and stripped from `CLAUDE.md`, with a warning surfaced by the CLI. If `CLAUDE.md` held nothing but the block, the empty file is removed. - `WriteResult` carries `warnings`; the `learn` CLI prints them. - Updated docs (`failure-learning.mdx`) and `CHANGELOG.md`. This implements the maintainer's stated preference order from the issue (default → `CLAUDE.local.md`, plus a `--target` flag), scoped to the Claude writer only — `AGENTS.md`/`GEMINI.md` have no `.local` convention and are untouched. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_learn/ tests/test_cli_learn.py -q 196 passed, 2 skipped in 17.80s $ ruff check headroom/learn/writer.py headroom/cli/learn.py All checks passed! $ mypy headroom/learn/writer.py headroom/cli/learn.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: macOS (Darwin), Python 3.11, headroom on rebased upstream/main - Exact command / steps: ran ClaudeCodeWriter against a temp project whose `CLAUDE.md` held hand-written content plus a legacy Headroom block, then `writer.write([...], dry_run=False)` - Observed result: `CLAUDE.md` kept its hand-written content with the block removed; `CLAUDE.local.md` gained both the migrated `### Old` section and the new `### Env` section; `result.warnings` contained the "Moved Headroom learnings out of …" notice. A block-only `CLAUDE.md` was deleted and a "Removed …" warning emitted. - Not tested: live end-to-end `headroom learn --apply` against real LLM analysis (writer + CLI plumbing covered by unit/CLI tests with mocked analysis) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated CHANGELOG.md if applicable ## Additional Notes Scoped to the Claude Code writer per the issue. After migration, `discover_projects` may briefly re-surface a section the LLM re-derives, but the write-side merge dedups by section name so the file stays correct. |
||
|
Tejas Chopra
|
0ddd4ed9e9
|
fix(learn): scan subagent and workflow transcripts (#1045)
## Description `headroom learn` only scanned top-level main Claude Code sessions (`<project>/<uuid>.jsonl`). Nested subagent and workflow transcripts under `<project>/<uuid>/subagents/**` were not opened, which hid a large amount of tool-call failure and token-spend activity from failure mining and downstream analysis. This change makes the Claude scanner descend into nested transcripts by default and tag each `SessionData` with its source. `--main-only` restores the previous top-level-only scan scope. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Updated `ClaudeCodePlugin.scan_project` to discover nested subagent and workflow transcripts by default. - Added source tagging for `main`, `subagent`, and `workflow` sessions. - Added `--main-only` and `include_subagents` plumbing so callers can opt back into top-level-only scanning. - Added the `include_subagents` scanner parameter to Codex/Gemini as a documented no-op because those scanners use flat session layouts. - Added regression tests for nested discovery, source tagging, parallel scanning, and CLI flag threading. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output ```text Full learn + CLI suite: # 186 passed, 2 skipped GitHub Actions CI for this PR: # build, build-wheel, lint, tests, e2e, CodeQL, and native wrapper checks passed ``` ## Real Behavior Proof - Environment: local Claude Code corpus with nested subagent/workflow transcripts. - Exact command / steps: Scanned the corpus with the previous top-level-only behavior and then with nested transcript discovery enabled. - Observed result: The scanner saw 24 sessions before and 306 sessions after descending into nested transcripts. - Not tested: Codex/Gemini nested transcript discovery, because those providers currently use flat session layouts and treat `include_subagents` as a no-op. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Tejas Chopra
|
22e750c78f | Fix learn CLI worker expectation in CI | ||
|
Tejas Chopra
|
155c069176 |
fix: stabilize learn write-failure worker test
Pin os.cpu_count in the write-failure regression test so the expected default worker cap is independent of the GitHub runner CPU count. |
||
|
Tejas Chopra
|
5ceca13c65 | fix: harden learn path handling across platforms | ||
|
Garm
|
efd2ac1ca4 |
chore: renormalize line endings to LF
`.gitattributes` declares `*.py text eol=lf` and `*.sh text eol=lf`, but 74 files (73 .py, 1 .sh) are stored in the index with CRLF line endings, violating that contract. Every macOS/Linux clone reports these files as "modified" on fresh checkout because git's diff engine sees the stored bytes don't match the attribute contract, even though the working tree and index match byte-for-byte. Running `git add --renormalize .` rewrites each affected blob so the stored form matches the attribute declaration. No semantic changes — every affected file's diff is "N insertions, N deletions" with inserts and deletes being the same lines modulo line endings. Follow-up commit adds `.git-blame-ignore-revs` so `git blame` / GitHub blame skip this mechanical commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
JerrettDavis
|
38bf3e639c |
test: expand coverage across helper slices
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |