feat: add deterministic runtime rollout controls (#1490)

## Description

Establish one centrally resolved, observable, deterministic, versioned
runtime rollout-control mechanism for Headroom. Runtime rollout controls
which behaviors an already-built artifact may expose; it does not select
or qualify a Headroom release/version.

## Type of Change

- [x] New feature (non-breaking change that adds functionality)
- [x] Bug fix (non-breaking change that fixes rollout enforcement
regressions)
- [x] Documentation update
- [x] Code refactoring (no functional changes)

## Changes Made

- Added `RolloutChannel`, `HEADROOM_ROLLOUT_CHANNEL`,
`--rollout-channel`, and a versioned immutable `RolloutSnapshot` shared
by Python configuration boundaries.
- Added schema/policy versions, canonical registry and snapshot SHA-256
identities, per-feature decision reasons, disable precedence, unsafe
qualification poisoning, strict CLI validation, and fail-closed
environment handling.
- Added `headroom rollout status --json`, Python `/stats.rollout`, and
Rust `/rollout/status` runtime provenance.
- Added equivalent Rust snapshot semantics and shared Python/Rust policy
vectors while retaining language-specific feature registries.
- Enforced rollout policy at alternate Python server composition roots
so `HEADROOM_READ_MATURATION=1` cannot bypass its beta gate.
- Preserved typed rollout snapshots across multi-worker serialization
with schema, policy, registry, snapshot-digest, type, and feature-name
validation.
- Made loopback runtime output-shaper updates replace the immutable
snapshot atomically for request readers, retain explicit request/disable
provenance, preserve channel and kill-switch precedence, invalidate
cached stats, and return the effective rollout decision.
- Made `headroom learn --verbosity --apply` report a channel-blocked
update instead of claiming the shaper is live.
- Made explicit CLI feature flags fail loudly when their current channel
blocks them.
- Made persistent interceptor installation select canary automatically,
or reject an explicitly insufficient channel unless the break-glass
override is set.
- Updated architecture, proxy, rollout, learn, and output-shaper
documentation with required channels and hot-reload semantics.

## Testing

- [x] Unit tests pass
- [x] Linting passes (`ruff check .` and `ruff format --check .`)
- [x] Type checking passes (`mypy headroom --ignore-missing-imports`)
- [x] New regression tests added for every corrected behavior
- [x] Rust tests and production-target Clippy pass
- [x] Documentation build passes

### Test Output

```text
Focused rollout coverage suite
57 passed; headroom.rollout + rollout CLI: 98% coverage

Affected proxy/rollout/transform/governance suites
222 passed; 0 failed

Final changed regression suites
100 passed; 0 failed

Cross-module hot-reload isolation regression
6 passed; 0 failed

cargo test -p headroom-core -p headroom-proxy --quiet
headroom-core: 924 passed; 1 ignored
headroom-proxy and integration suites: all passed

cargo clippy -p headroom-core -p headroom-proxy --lib --bins -- -D warnings
cargo fmt --all -- --check
ruff check .
ruff format --check .
mypy headroom --ignore-missing-imports
git diff --check
All passed

cd docs && npm run build
Compiled successfully; 164 static pages generated
```

The unsharded Windows-only CI selection exposed unrelated baseline
failures, principally the existing `sqlite:///C:\\...` URL parser
producing an invalid `\\C:\\...` path. At commit `8e793a80`, all 52
completed GitHub checks passed; the only other conclusions are expected
skips and superseded governance jobs.

## Real Behavior Proof

- **Environment:** Windows checkout on Python 3.13.3 and the current
Rust workspace, based on upstream `main` at `93f2d7a2`.
- **Exact command / steps:** Exercised canary and beta feature requests
through CLI status, Python `/stats.rollout`, Rust `/rollout/status`,
multi-worker payload round trips, loopback `/admin/runtime-env`, real
proxy request shaping before/after hot reload, installer manifest
generation, and shared Python/Rust policy vectors.
- **Observed result:** Stable blocks unstable requests; disable wins
over explicit/default/legacy/unsafe paths; unsafe state reports
`qualification_eligible=false`; worker handoff rejects tampering;
running output shaping changes only when the effective beta policy
permits it; explicit blocked flags fail with actionable diagnostics.
- **Not tested:** Live production traffic requiring provider
credentials, or future artifact qualification/promotion automation
(intentionally out of scope).

## Runtime Rollout Safety

- **Rollout-managed features:** Python `tool_result_interceptors`,
`proxy_output_shaper`, `read_maturation`; Rust `native_bedrock`,
`openai_responses_streaming`, `canary_probe`.
- **Minimum rollout channel:** Registry-defined per feature; process
default is `stable`.
- **Stable/default behavior changed:** No unstable feature becomes
enabled by default. Explicit blocked CLI flags now fail instead of
silently doing nothing.
- **Kill switch / disable path:**
`HEADROOM_DISABLE_FEATURES=<comma-separated feature names>`; explicit
disable has highest precedence, including over the unsafe override.
- **Unsafe override required:** No.
`HEADROOM_UNSAFE_ALLOW_UNSTABLE_FEATURES=1` is break-glass only and
makes qualification evidence ineligible.
- **Qualification impact:** Adds machine-readable policy/snapshot
identities and eligibility; does not implement qualification itself.
- **Rollback path:** Set the named disable list for operational
rollback, lower the channel, or revert this PR.

## Review Readiness

- [x] I have performed a full diff review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented hard-to-understand areas
- [x] I have made corresponding documentation changes
- [x] My changes generate no new warnings
- [x] I added tests that reproduce and prevent every regression fixed
during review
- [x] New and existing affected tests pass locally
- [x] I did **not** edit `CHANGELOG.md`; release-please generates it
from the Conventional Commit PR title

## Additional Notes

Out of scope: artifact candidates, benchmark orchestration,
qualification manifests/gates, promotion automation, release branches,
publication guards, and release-risk classification. Those workflows can
consume the rollout registry digest, runtime snapshot digest, decision
reasons, and qualification eligibility through supported black-box
interfaces.

---------

Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
Co-authored-by: JD Davis <jd@JDH-AIR-00.local>

scripts/pr-governance.py

@@ -22,6 +22,7 @@ REQUIRED_SECTIONS = (
     "Changes Made",
     "Testing",
     "Real Behavior Proof",
+    "Runtime Rollout Safety",
     "Review Readiness",
 )
 PROOF_FIELDS = (
@@ -30,6 +31,15 @@ PROOF_FIELDS = (
     "Observed result",
     "Not tested",
 )
+ROLLOUT_FIELDS = (
+    "Rollout-managed feature(s)",
+    "Minimum rollout channel",
+    "Stable/default behavior changed",
+    "Kill switch / disable path",
+    "Unsafe override required",
+    "Qualification impact",
+    "Rollback path",
+)
 
 SECTION_RE = re.compile(r"^##\s+(.+?)\s*$", re.MULTILINE)
 CHECKBOX_RE = re.compile(r"^- \[(?P<checked>[ xX])\] (?P<label>.+)$", re.MULTILINE)
@@ -193,6 +203,12 @@ def validate_pull_request(event: dict[str, Any]) -> GovernanceReport:
         if proof_section and not proof_values.get(field_name):
             problems.append(f"Fill in `Real Behavior Proof` → `{field_name}`.")
 
+    rollout_section = sections.get("Runtime Rollout Safety", "")
+    rollout_values = proof_field_values(rollout_section)
+    for field_name in ROLLOUT_FIELDS:
+        if rollout_section and not rollout_values.get(field_name):
+            problems.append(f"Fill in `Runtime Rollout Safety` → `{field_name}`.")
+
     readiness_checked = normalize_checkbox_map(checked_items(sections.get("Review Readiness", "")))
     has_self_review = "i have performed a self-review" in readiness_checked
     has_ready_checkbox = "this pr is ready for human review" in readiness_checked

scripts/tests/test_pr_governance.py

@@ -64,6 +64,16 @@ pytest scripts/tests/test_pr_governance.py -q
 - Observed result: The governance check fails and the PR gets a needs-author-action label.
 - Not tested: Automatic Copilot review rulesets in repository settings.
 
+## Runtime Rollout Safety
+
+- Rollout-managed feature(s): None.
+- Minimum rollout channel: Stable.
+- Stable/default behavior changed: No.
+- Kill switch / disable path: Not applicable.
+- Unsafe override required: No.
+- Qualification impact: None.
+- Rollback path: Revert the workflow and script changes.
+
 ## Review Readiness
 
 - [x] I have performed a self-review
@@ -186,6 +196,16 @@ Fixes #123
 - Observed result:
 - Not tested:
 
+## Runtime Rollout Safety
+
+- Rollout-managed feature(s):
+- Minimum rollout channel:
+- Stable/default behavior changed:
+- Kill switch / disable path:
+- Unsafe override required:
+- Qualification impact:
+- Rollback path:
+
 ## Review Readiness
 
 - [ ] I have performed a self-review
@@ -201,6 +221,7 @@ Fixes #123
     assert any("Type of Change" in problem for problem in report.problems)
     assert any("Test Output" in problem for problem in report.problems)
     assert any("Real Behavior Proof" in problem for problem in report.problems)
+    assert any("Runtime Rollout Safety" in problem for problem in report.problems)
 
 
 def test_validate_pull_request_skips_bot_authored_prs() -> None: