headroom

2630 commits 410 branches 169 tags 142 MiB

Author SHA1 Message Date

Author	SHA1	Message	Date
JD Davis	3077ac81e8	feat: add deterministic runtime rollout controls (#1490 ) ## Description Establish one centrally resolved, observable, deterministic, versioned runtime rollout-control mechanism for Headroom. Runtime rollout controls which behaviors an already-built artifact may expose; it does not select or qualify a Headroom release/version. ## Type of Change - [x] New feature (non-breaking change that adds functionality) - [x] Bug fix (non-breaking change that fixes rollout enforcement regressions) - [x] Documentation update - [x] Code refactoring (no functional changes) ## Changes Made - Added `RolloutChannel`, `HEADROOM_ROLLOUT_CHANNEL`, `--rollout-channel`, and a versioned immutable `RolloutSnapshot` shared by Python configuration boundaries. - Added schema/policy versions, canonical registry and snapshot SHA-256 identities, per-feature decision reasons, disable precedence, unsafe qualification poisoning, strict CLI validation, and fail-closed environment handling. - Added `headroom rollout status --json`, Python `/stats.rollout`, and Rust `/rollout/status` runtime provenance. - Added equivalent Rust snapshot semantics and shared Python/Rust policy vectors while retaining language-specific feature registries. - Enforced rollout policy at alternate Python server composition roots so `HEADROOM_READ_MATURATION=1` cannot bypass its beta gate. - Preserved typed rollout snapshots across multi-worker serialization with schema, policy, registry, snapshot-digest, type, and feature-name validation. - Made loopback runtime output-shaper updates replace the immutable snapshot atomically for request readers, retain explicit request/disable provenance, preserve channel and kill-switch precedence, invalidate cached stats, and return the effective rollout decision. - Made `headroom learn --verbosity --apply` report a channel-blocked update instead of claiming the shaper is live. - Made explicit CLI feature flags fail loudly when their current channel blocks them. - Made persistent interceptor installation select canary automatically, or reject an explicitly insufficient channel unless the break-glass override is set. - Updated architecture, proxy, rollout, learn, and output-shaper documentation with required channels and hot-reload semantics. ## Testing - [x] Unit tests pass - [x] Linting passes (`ruff check .` and `ruff format --check .`) - [x] Type checking passes (`mypy headroom --ignore-missing-imports`) - [x] New regression tests added for every corrected behavior - [x] Rust tests and production-target Clippy pass - [x] Documentation build passes ### Test Output ```text Focused rollout coverage suite 57 passed; headroom.rollout + rollout CLI: 98% coverage Affected proxy/rollout/transform/governance suites 222 passed; 0 failed Final changed regression suites 100 passed; 0 failed Cross-module hot-reload isolation regression 6 passed; 0 failed cargo test -p headroom-core -p headroom-proxy --quiet headroom-core: 924 passed; 1 ignored headroom-proxy and integration suites: all passed cargo clippy -p headroom-core -p headroom-proxy --lib --bins -- -D warnings cargo fmt --all -- --check ruff check . ruff format --check . mypy headroom --ignore-missing-imports git diff --check All passed cd docs && npm run build Compiled successfully; 164 static pages generated ``` The unsharded Windows-only CI selection exposed unrelated baseline failures, principally the existing `sqlite:///C:\\...` URL parser producing an invalid `\\C:\\...` path. At commit ``8e793a80``, all 52 completed GitHub checks passed; the only other conclusions are expected skips and superseded governance jobs. ## Real Behavior Proof - Environment: Windows checkout on Python 3.13.3 and the current Rust workspace, based on upstream `main` at ``93f2d7a2``. - Exact command / steps: Exercised canary and beta feature requests through CLI status, Python `/stats.rollout`, Rust `/rollout/status`, multi-worker payload round trips, loopback `/admin/runtime-env`, real proxy request shaping before/after hot reload, installer manifest generation, and shared Python/Rust policy vectors. - Observed result: Stable blocks unstable requests; disable wins over explicit/default/legacy/unsafe paths; unsafe state reports `qualification_eligible=false`; worker handoff rejects tampering; running output shaping changes only when the effective beta policy permits it; explicit blocked flags fail with actionable diagnostics. - Not tested: Live production traffic requiring provider credentials, or future artifact qualification/promotion automation (intentionally out of scope). ## Runtime Rollout Safety - Rollout-managed features: Python `tool_result_interceptors`, `proxy_output_shaper`, `read_maturation`; Rust `native_bedrock`, `openai_responses_streaming`, `canary_probe`. - Minimum rollout channel: Registry-defined per feature; process default is `stable`. - Stable/default behavior changed: No unstable feature becomes enabled by default. Explicit blocked CLI flags now fail instead of silently doing nothing. - Kill switch / disable path: `HEADROOM_DISABLE_FEATURES=<comma-separated feature names>`; explicit disable has highest precedence, including over the unsafe override. - Unsafe override required: No. `HEADROOM_UNSAFE_ALLOW_UNSTABLE_FEATURES=1` is break-glass only and makes qualification evidence ineligible. - Qualification impact: Adds machine-readable policy/snapshot identities and eligibility; does not implement qualification itself. - Rollback path: Set the named disable list for operational rollback, lower the channel, or revert this PR. ## Review Readiness - [x] I have performed a full diff review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented hard-to-understand areas - [x] I have made corresponding documentation changes - [x] My changes generate no new warnings - [x] I added tests that reproduce and prevent every regression fixed during review - [x] New and existing affected tests pass locally - [x] I did not edit `CHANGELOG.md`; release-please generates it from the Conventional Commit PR title ## Additional Notes Out of scope: artifact candidates, benchmark orchestration, qualification manifests/gates, promotion automation, release branches, publication guards, and release-risk classification. Those workflows can consume the rollout registry digest, runtime snapshot digest, decision reasons, and qualification eligibility through supported black-box interfaces. --------- Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net> Co-authored-by: JD Davis <jd@JDH-AIR-00.local>	2026-08-12 23:16:54 -05:00
Tejas Chopra	bd76235f5c	fix(cli): harden all CLI surfaces + fix docs accuracy (#1491 ) ## Summary Full CLI audit + documentation accuracy pass. All 5 commits on this branch: ### CLI Hardening (4 commits) - Clean errors instead of tracebacks: corrupt manifests, missing Docker, malformed JSONL, bad `--profile`, invalid env-var values all now raise `click.ClickException` with helpful messages - Range validation: ~25 numeric flags across 10 files now use `click.IntRange`/`FloatRange` — `--port 0`, `--hours -1`, `--limit 0` etc. produce clean usage errors instead of silent wrong behavior - Flag combination warnings: conflicting combos (`--no-rate-limit` + `--rpm`, `--no-optimize` + `--target-ratio`, `--telemetry` + `--no-telemetry`) emit yellow warnings on stderr - `memory --db-path` default fixed: was resolving to `headroom_memory.db` (wrong bare file); now uses project store `./.headroom/memory.db` if present, else `~/.headroom/memory.db` - `memory list --search` + filters: `--scope`/`--session`/`--since` were silently ignored when `--search` was also set; now filters are applied to search results - `learn --verbosity --apply` now works: the output shaper is off by default (`HEADROOM_OUTPUT_SHAPER`); `--apply` now hot-enables it via `POST /admin/runtime-env` on a running proxy, or prints explicit `export HEADROOM_OUTPUT_SHAPER=1` instructions when no proxy is running - `perf --hours` overflow: `1e9` hours no longer raises `OverflowError`; treated as "all data" - `evals memory --categories` invalid input: `abc,1,2` now raises `BadParameter` instead of a raw `ValueError` traceback ### Documentation (1 commit, 20 files) Corrected factual errors found by 3 parallel audit agents across root docs, wiki, and the published Fumadocs site: Critical (caused runtime errors or wrong behavior if followed): - `simulation.mdx`: `plan.transforms_applied` -> `plan.transforms`; `plan.savings_percent` -> computed from available fields (both raised `AttributeError`) - `shared-context.mdx`: `import { SharedContext } from "headroom"` -> `"headroom-ai"` (5x `ImportError`) - `claude-code-azure-foundry.mdx`: `pip install headroom` -> `pip install headroom-ai` - `api-reference.mdx` + `configuration.mdx`: `from headroom import GoogleProvider` -> `from headroom.providers import GoogleProvider` - `ccr.mdx`: CCR TTL default 300s -> 1800s (30 min) Fabricated flags removed: - `wiki/proxy.md` + `wiki/cli.md`: `--no-intelligent-context`, `--no-intelligent-scoring`, `--no-compress-first` (none exist); replaced with real CCR flags - `wiki/configuration.md`: `--no-ccr-responses`, `--no-ccr-expansion` (none exist); replaced with real flags - `wiki/troubleshooting.md`, `wiki/metrics.md`, `docs/troubleshooting.mdx`: `headroom proxy --log-level debug` (flag doesn't exist) Stale content corrected: - `llms.txt`: telemetry stated as enabled-by-default (it's opt-in); wrap list had 5 tools (now 11) - `README.md`: compatibility matrix added 5 missing `wrap` targets; `unwrap`, `doctor`, `init`/`install`, savings-analytics now mentioned - `SECURITY.md`: supported version table showed 0.2.x (current: 0.27.x) - `wiki/learn.md`: 5 missing flags added; verbosity shaper-off behavior documented - `wiki/quickstart.md`: "Configuration Reference" linked to `api.md` (wrong) -> `configuration.md` - `CacheAlignerConfig.enabled` default corrected: `True` -> `False` - `opencode.mdx`: `--port` default wrong ("random") -> 8787; `openai` backend removed - `CONTRIBUTING.md`: broken Markdown table cell fixed - `docs/meta.json`: `claude-code-azure-foundry` added to nav (was unreachable orphan page) - `configuration.mdx`: SDK modes vs proxy `--mode` now clearly distinguished ## Test plan - [x] `python -m pytest tests/ -x -q` — 857 passed, 0 failures - [x] 41-combination CLI smoke test (all flag combos across 8 commands) — 0 tracebacks - [x] `ruff check` on all modified Python files — clean - [x] Docs changes are removals/corrections of fabricated or stale content; no new claims introduced	2026-06-27 14:48:43 -07:00
Adib Mohsin	911eb85a44	new docs UI + ts doc coverage	2026-04-12 13:15:58 +06:00

JD Davis

3077ac81e8

feat: add deterministic runtime rollout controls (#1490 )

## Description

Establish one centrally resolved, observable, deterministic, versioned
runtime rollout-control mechanism for Headroom. Runtime rollout controls
which behaviors an already-built artifact may expose; it does not select
or qualify a Headroom release/version.

## Type of Change

- [x] New feature (non-breaking change that adds functionality)
- [x] Bug fix (non-breaking change that fixes rollout enforcement
regressions)
- [x] Documentation update
- [x] Code refactoring (no functional changes)

## Changes Made

- Added `RolloutChannel`, `HEADROOM_ROLLOUT_CHANNEL`,
`--rollout-channel`, and a versioned immutable `RolloutSnapshot` shared
by Python configuration boundaries.
- Added schema/policy versions, canonical registry and snapshot SHA-256
identities, per-feature decision reasons, disable precedence, unsafe
qualification poisoning, strict CLI validation, and fail-closed
environment handling.
- Added `headroom rollout status --json`, Python `/stats.rollout`, and
Rust `/rollout/status` runtime provenance.
- Added equivalent Rust snapshot semantics and shared Python/Rust policy
vectors while retaining language-specific feature registries.
- Enforced rollout policy at alternate Python server composition roots
so `HEADROOM_READ_MATURATION=1` cannot bypass its beta gate.
- Preserved typed rollout snapshots across multi-worker serialization
with schema, policy, registry, snapshot-digest, type, and feature-name
validation.
- Made loopback runtime output-shaper updates replace the immutable
snapshot atomically for request readers, retain explicit request/disable
provenance, preserve channel and kill-switch precedence, invalidate
cached stats, and return the effective rollout decision.
- Made `headroom learn --verbosity --apply` report a channel-blocked
update instead of claiming the shaper is live.
- Made explicit CLI feature flags fail loudly when their current channel
blocks them.
- Made persistent interceptor installation select canary automatically,
or reject an explicitly insufficient channel unless the break-glass
override is set.
- Updated architecture, proxy, rollout, learn, and output-shaper
documentation with required channels and hot-reload semantics.

## Testing

- [x] Unit tests pass
- [x] Linting passes (`ruff check .` and `ruff format --check .`)
- [x] Type checking passes (`mypy headroom --ignore-missing-imports`)
- [x] New regression tests added for every corrected behavior
- [x] Rust tests and production-target Clippy pass
- [x] Documentation build passes

### Test Output

```text
Focused rollout coverage suite
57 passed; headroom.rollout + rollout CLI: 98% coverage

Affected proxy/rollout/transform/governance suites
222 passed; 0 failed

Final changed regression suites
100 passed; 0 failed

Cross-module hot-reload isolation regression
6 passed; 0 failed

cargo test -p headroom-core -p headroom-proxy --quiet
headroom-core: 924 passed; 1 ignored
headroom-proxy and integration suites: all passed

cargo clippy -p headroom-core -p headroom-proxy --lib --bins -- -D warnings
cargo fmt --all -- --check
ruff check .
ruff format --check .
mypy headroom --ignore-missing-imports
git diff --check
All passed

cd docs && npm run build
Compiled successfully; 164 static pages generated
```

The unsharded Windows-only CI selection exposed unrelated baseline
failures, principally the existing `sqlite:///C:\\...` URL parser
producing an invalid `\\C:\\...` path. At commit `8e793a80`, all 52
completed GitHub checks passed; the only other conclusions are expected
skips and superseded governance jobs.

## Real Behavior Proof

- **Environment:** Windows checkout on Python 3.13.3 and the current
Rust workspace, based on upstream `main` at `93f2d7a2`.
- **Exact command / steps:** Exercised canary and beta feature requests
through CLI status, Python `/stats.rollout`, Rust `/rollout/status`,
multi-worker payload round trips, loopback `/admin/runtime-env`, real
proxy request shaping before/after hot reload, installer manifest
generation, and shared Python/Rust policy vectors.
- **Observed result:** Stable blocks unstable requests; disable wins
over explicit/default/legacy/unsafe paths; unsafe state reports
`qualification_eligible=false`; worker handoff rejects tampering;
running output shaping changes only when the effective beta policy
permits it; explicit blocked flags fail with actionable diagnostics.
- **Not tested:** Live production traffic requiring provider
credentials, or future artifact qualification/promotion automation
(intentionally out of scope).

## Runtime Rollout Safety

- **Rollout-managed features:** Python `tool_result_interceptors`,
`proxy_output_shaper`, `read_maturation`; Rust `native_bedrock`,
`openai_responses_streaming`, `canary_probe`.
- **Minimum rollout channel:** Registry-defined per feature; process
default is `stable`.
- **Stable/default behavior changed:** No unstable feature becomes
enabled by default. Explicit blocked CLI flags now fail instead of
silently doing nothing.
- **Kill switch / disable path:**
`HEADROOM_DISABLE_FEATURES=<comma-separated feature names>`; explicit
disable has highest precedence, including over the unsafe override.
- **Unsafe override required:** No.
`HEADROOM_UNSAFE_ALLOW_UNSTABLE_FEATURES=1` is break-glass only and
makes qualification evidence ineligible.
- **Qualification impact:** Adds machine-readable policy/snapshot
identities and eligibility; does not implement qualification itself.
- **Rollback path:** Set the named disable list for operational
rollback, lower the channel, or revert this PR.

## Review Readiness

- [x] I have performed a full diff review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented hard-to-understand areas
- [x] I have made corresponding documentation changes
- [x] My changes generate no new warnings
- [x] I added tests that reproduce and prevent every regression fixed
during review
- [x] New and existing affected tests pass locally
- [x] I did **not** edit `CHANGELOG.md`; release-please generates it
from the Conventional Commit PR title

## Additional Notes

Out of scope: artifact candidates, benchmark orchestration,
qualification manifests/gates, promotion automation, release branches,
publication guards, and release-risk classification. Those workflows can
consume the rollout registry digest, runtime snapshot digest, decision
reasons, and qualification eligibility through supported black-box
interfaces.

---------

Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
Co-authored-by: JD Davis <jd@JDH-AIR-00.local>

2026-08-12 23:16:54 -05:00

Tejas Chopra

bd76235f5c

fix(cli): harden all CLI surfaces + fix docs accuracy (#1491 )

## Summary

Full CLI audit + documentation accuracy pass. All 5 commits on this
branch:

### CLI Hardening (4 commits)
- **Clean errors instead of tracebacks**: corrupt manifests, missing
Docker, malformed JSONL, bad `--profile`, invalid env-var values all now
raise `click.ClickException` with helpful messages
- **Range validation**: ~25 numeric flags across 10 files now use
`click.IntRange`/`FloatRange` — `--port 0`, `--hours -1`, `--limit 0`
etc. produce clean usage errors instead of silent wrong behavior
- **Flag combination warnings**: conflicting combos (`--no-rate-limit` +
`--rpm`, `--no-optimize` + `--target-ratio`, `--telemetry` +
`--no-telemetry`) emit yellow warnings on stderr
- **`memory --db-path` default fixed**: was resolving to
`headroom_memory.db` (wrong bare file); now uses project store
`./.headroom/memory.db` if present, else `~/.headroom/memory.db`
- **`memory list --search` + filters**: `--scope`/`--session`/`--since`
were silently ignored when `--search` was also set; now filters are
applied to search results
- **`learn --verbosity --apply` now works**: the output shaper is off by
default (`HEADROOM_OUTPUT_SHAPER`); `--apply` now hot-enables it via
`POST /admin/runtime-env` on a running proxy, or prints explicit `export
HEADROOM_OUTPUT_SHAPER=1` instructions when no proxy is running
- **`perf --hours` overflow**: `1e9` hours no longer raises
`OverflowError`; treated as "all data"
- **`evals memory --categories` invalid input**: `abc,1,2` now raises
`BadParameter` instead of a raw `ValueError` traceback

### Documentation (1 commit, 20 files)

Corrected factual errors found by 3 parallel audit agents across root
docs, wiki, and the published Fumadocs site:

**Critical (caused runtime errors or wrong behavior if followed):**
- `simulation.mdx`: `plan.transforms_applied` -> `plan.transforms`;
`plan.savings_percent` -> computed from available fields (both raised
`AttributeError`)
- `shared-context.mdx`: `import { SharedContext } from "headroom"` ->
`"headroom-ai"` (5x `ImportError`)
- `claude-code-azure-foundry.mdx`: `pip install headroom` -> `pip
install headroom-ai`
- `api-reference.mdx` + `configuration.mdx`: `from headroom import
GoogleProvider` -> `from headroom.providers import GoogleProvider`
- `ccr.mdx`: CCR TTL default 300s -> 1800s (30 min)

**Fabricated flags removed:**
- `wiki/proxy.md` + `wiki/cli.md`: `--no-intelligent-context`,
`--no-intelligent-scoring`, `--no-compress-first` (none exist); replaced
with real CCR flags
- `wiki/configuration.md`: `--no-ccr-responses`, `--no-ccr-expansion`
(none exist); replaced with real flags
- `wiki/troubleshooting.md`, `wiki/metrics.md`,
`docs/troubleshooting.mdx`: `headroom proxy --log-level debug` (flag
doesn't exist)

**Stale content corrected:**
- `llms.txt`: telemetry stated as enabled-by-default (it's opt-in); wrap
list had 5 tools (now 11)
- `README.md`: compatibility matrix added 5 missing `wrap` targets;
`unwrap`, `doctor`, `init`/`install`, savings-analytics now mentioned
- `SECURITY.md`: supported version table showed 0.2.x (current: 0.27.x)
- `wiki/learn.md`: 5 missing flags added; verbosity shaper-off behavior
documented
- `wiki/quickstart.md`: "Configuration Reference" linked to `api.md`
(wrong) -> `configuration.md`
- `CacheAlignerConfig.enabled` default corrected: `True` -> `False`
- `opencode.mdx`: `--port` default wrong ("random") -> 8787; `openai`
backend removed
- `CONTRIBUTING.md`: broken Markdown table cell fixed
- `docs/meta.json`: `claude-code-azure-foundry` added to nav (was
unreachable orphan page)
- `configuration.mdx`: SDK modes vs proxy `--mode` now clearly
distinguished

## Test plan

- [x] `python -m pytest tests/ -x -q` — 857 passed, 0 failures
- [x] 41-combination CLI smoke test (all flag combos across 8 commands)
— 0 tracebacks
- [x] `ruff check` on all modified Python files — clean
- [x] Docs changes are removals/corrections of fabricated or stale
content; no new claims introduced

2026-06-27 14:48:43 -07:00

Adib Mohsin

911eb85a44

new docs UI + ts doc coverage

2026-04-12 13:15:58 +06:00

Renamed from docs/learn.md (Browse further)

3 commits