headroom/wiki
JD Davis 3077ac81e8
feat: add deterministic runtime rollout controls (#1490)
## Description

Establish one centrally resolved, observable, deterministic, versioned
runtime rollout-control mechanism for Headroom. Runtime rollout controls
which behaviors an already-built artifact may expose; it does not select
or qualify a Headroom release/version.

## Type of Change

- [x] New feature (non-breaking change that adds functionality)
- [x] Bug fix (non-breaking change that fixes rollout enforcement
regressions)
- [x] Documentation update
- [x] Code refactoring (no functional changes)

## Changes Made

- Added `RolloutChannel`, `HEADROOM_ROLLOUT_CHANNEL`,
`--rollout-channel`, and a versioned immutable `RolloutSnapshot` shared
by Python configuration boundaries.
- Added schema/policy versions, canonical registry and snapshot SHA-256
identities, per-feature decision reasons, disable precedence, unsafe
qualification poisoning, strict CLI validation, and fail-closed
environment handling.
- Added `headroom rollout status --json`, Python `/stats.rollout`, and
Rust `/rollout/status` runtime provenance.
- Added equivalent Rust snapshot semantics and shared Python/Rust policy
vectors while retaining language-specific feature registries.
- Enforced rollout policy at alternate Python server composition roots
so `HEADROOM_READ_MATURATION=1` cannot bypass its beta gate.
- Preserved typed rollout snapshots across multi-worker serialization
with schema, policy, registry, snapshot-digest, type, and feature-name
validation.
- Made loopback runtime output-shaper updates replace the immutable
snapshot atomically for request readers, retain explicit request/disable
provenance, preserve channel and kill-switch precedence, invalidate
cached stats, and return the effective rollout decision.
- Made `headroom learn --verbosity --apply` report a channel-blocked
update instead of claiming the shaper is live.
- Made explicit CLI feature flags fail loudly when their current channel
blocks them.
- Made persistent interceptor installation select canary automatically,
or reject an explicitly insufficient channel unless the break-glass
override is set.
- Updated architecture, proxy, rollout, learn, and output-shaper
documentation with required channels and hot-reload semantics.

## Testing

- [x] Unit tests pass
- [x] Linting passes (`ruff check .` and `ruff format --check .`)
- [x] Type checking passes (`mypy headroom --ignore-missing-imports`)
- [x] New regression tests added for every corrected behavior
- [x] Rust tests and production-target Clippy pass
- [x] Documentation build passes

### Test Output

```text
Focused rollout coverage suite
57 passed; headroom.rollout + rollout CLI: 98% coverage

Affected proxy/rollout/transform/governance suites
222 passed; 0 failed

Final changed regression suites
100 passed; 0 failed

Cross-module hot-reload isolation regression
6 passed; 0 failed

cargo test -p headroom-core -p headroom-proxy --quiet
headroom-core: 924 passed; 1 ignored
headroom-proxy and integration suites: all passed

cargo clippy -p headroom-core -p headroom-proxy --lib --bins -- -D warnings
cargo fmt --all -- --check
ruff check .
ruff format --check .
mypy headroom --ignore-missing-imports
git diff --check
All passed

cd docs && npm run build
Compiled successfully; 164 static pages generated
```

The unsharded Windows-only CI selection exposed unrelated baseline
failures, principally the existing `sqlite:///C:\\...` URL parser
producing an invalid `\\C:\\...` path. At commit `8e793a80`, all 52
completed GitHub checks passed; the only other conclusions are expected
skips and superseded governance jobs.

## Real Behavior Proof

- **Environment:** Windows checkout on Python 3.13.3 and the current
Rust workspace, based on upstream `main` at `93f2d7a2`.
- **Exact command / steps:** Exercised canary and beta feature requests
through CLI status, Python `/stats.rollout`, Rust `/rollout/status`,
multi-worker payload round trips, loopback `/admin/runtime-env`, real
proxy request shaping before/after hot reload, installer manifest
generation, and shared Python/Rust policy vectors.
- **Observed result:** Stable blocks unstable requests; disable wins
over explicit/default/legacy/unsafe paths; unsafe state reports
`qualification_eligible=false`; worker handoff rejects tampering;
running output shaping changes only when the effective beta policy
permits it; explicit blocked flags fail with actionable diagnostics.
- **Not tested:** Live production traffic requiring provider
credentials, or future artifact qualification/promotion automation
(intentionally out of scope).

## Runtime Rollout Safety

- **Rollout-managed features:** Python `tool_result_interceptors`,
`proxy_output_shaper`, `read_maturation`; Rust `native_bedrock`,
`openai_responses_streaming`, `canary_probe`.
- **Minimum rollout channel:** Registry-defined per feature; process
default is `stable`.
- **Stable/default behavior changed:** No unstable feature becomes
enabled by default. Explicit blocked CLI flags now fail instead of
silently doing nothing.
- **Kill switch / disable path:**
`HEADROOM_DISABLE_FEATURES=<comma-separated feature names>`; explicit
disable has highest precedence, including over the unsafe override.
- **Unsafe override required:** No.
`HEADROOM_UNSAFE_ALLOW_UNSTABLE_FEATURES=1` is break-glass only and
makes qualification evidence ineligible.
- **Qualification impact:** Adds machine-readable policy/snapshot
identities and eligibility; does not implement qualification itself.
- **Rollback path:** Set the named disable list for operational
rollback, lower the channel, or revert this PR.

## Review Readiness

- [x] I have performed a full diff review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented hard-to-understand areas
- [x] I have made corresponding documentation changes
- [x] My changes generate no new warnings
- [x] I added tests that reproduce and prevent every regression fixed
during review
- [x] New and existing affected tests pass locally
- [x] I did **not** edit `CHANGELOG.md`; release-please generates it
from the Conventional Commit PR title

## Additional Notes

Out of scope: artifact candidates, benchmark orchestration,
qualification manifests/gates, promotion automation, release branches,
publication guards, and release-risk classification. Those workflows can
consume the rollout registry digest, runtime snapshot digest, decision
reasons, and qualification eligibility through supported black-box
interfaces.

---------

Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
Co-authored-by: JD Davis <jd@JDH-AIR-00.local>
2026-08-12 23:16:54 -05:00
..
overrides new docs UI + ts doc coverage 2026-04-12 13:15:58 +06:00
plans fix: add anthropic pre-upstream timeouts 2026-04-21 09:48:32 +07:00
screenshots new docs UI + ts doc coverage 2026-04-12 13:15:58 +06:00
stylesheets new docs UI + ts doc coverage 2026-04-12 13:15:58 +06:00
agno.md docs: sync README + benchmarks with code (drop retired IntelligentContext/RollingWindow) (#1545) 2026-06-28 22:36:41 -07:00
api.md docs: sync README + benchmarks with code (drop retired IntelligentContext/RollingWindow) (#1545) 2026-06-28 22:36:41 -07:00
ARCHITECTURE.md docs: sync README + benchmarks with code (drop retired IntelligentContext/RollingWindow) (#1545) 2026-06-28 22:36:41 -07:00
benchmarks.md fix(telemetry): switch anonymous telemetry to opt-in (off by default) (#1223) 2026-06-20 21:26:04 -07:00
ccr.md docs: sync README + benchmarks with code (drop retired IntelligentContext/RollingWindow) (#1545) 2026-06-28 22:36:41 -07:00
cli.md fix: remove rtk and lean-ctx CLI context tools (#2677) 2026-07-30 22:59:41 -07:00
compression.md fix(cli): proxy/perf/wrap UX cleanup + perf --hours correctness 2026-05-07 16:43:35 -07:00
configuration.md feat: add deterministic runtime rollout controls (#1490) 2026-08-12 23:16:54 -05:00
docker-install.md fix: remove rtk and lean-ctx CLI context tools (#2677) 2026-07-30 22:59:41 -07:00
errors.md new docs UI + ts doc coverage 2026-04-12 13:15:58 +06:00
filesystem-contract.md fix: remove rtk and lean-ctx CLI context tools (#2677) 2026-07-30 22:59:41 -07:00
getting-started.md Clarify uv tool install path on macOS (#1196) 2026-07-15 21:06:30 +00:00
image-compression.md new docs UI + ts doc coverage 2026-04-12 13:15:58 +06:00
index.md Clarify uv tool install path on macOS (#1196) 2026-07-15 21:06:30 +00:00
integration-guide.md fix: remove rtk and lean-ctx CLI context tools (#2677) 2026-07-30 22:59:41 -07:00
langchain.md new docs UI + ts doc coverage 2026-04-12 13:15:58 +06:00
LATENCY_BENCHMARKS.md new docs UI + ts doc coverage 2026-04-12 13:15:58 +06:00
learn.md feat: add deterministic runtime rollout controls (#1490) 2026-08-12 23:16:54 -05:00
LIMITATIONS.md docs: sync README + benchmarks with code (drop retired IntelligentContext/RollingWindow) (#1545) 2026-06-28 22:36:41 -07:00
macos-deployment.md Clarify uv tool install path on macOS (#1196) 2026-07-15 21:06:30 +00:00
mcp.md docs: document HEADROOM_CONFIG_DIR / HEADROOM_WORKSPACE_DIR filesystem contract 2026-04-16 19:24:17 -05:00
memory.md feat(memory): add opt-in Apple-GPU (MPS) embedding runtime (#766) 2026-06-11 12:59:20 -05:00
metrics.md fix(cli): harden all CLI surfaces + fix docs accuracy (#1491) 2026-06-27 14:48:43 -07:00
network-diff-capture.md feat: add differential network capture harness (#761) 2026-06-08 22:18:31 -07:00
persistent-installs.md docs: document HEADROOM_CONFIG_DIR / HEADROOM_WORKSPACE_DIR filesystem contract 2026-04-16 19:24:17 -05:00
proxy.md fix(compress): resolve the /v1/compress tokenizer per model, and document the real contract (#2743) 2026-08-03 12:20:33 -07:00
quickstart.md Clarify uv tool install path on macOS (#1196) 2026-07-15 21:06:30 +00:00
sdk.md new docs UI + ts doc coverage 2026-04-12 13:15:58 +06:00
shared-context.md new docs UI + ts doc coverage 2026-04-12 13:15:58 +06:00
strands.md docs: sync README + benchmarks with code (drop retired IntelligentContext/RollingWindow) (#1545) 2026-06-28 22:36:41 -07:00
text-compression.md new docs UI + ts doc coverage 2026-04-12 13:15:58 +06:00
transforms.md docs: sync README + benchmarks with code (drop retired IntelligentContext/RollingWindow) (#1545) 2026-06-28 22:36:41 -07:00
troubleshooting.md fix: skip Magika backend on x86 CPUs without AVX2 (#1162) 2026-06-30 13:34:17 -05:00
typescript-sdk.md fix(compress): resolve the /v1/compress tokenizer per model, and document the real contract (#2743) 2026-08-03 12:20:33 -07:00
vertex.md chore: remove committed node_modules + stray/internal markdown (repo hygiene) (#1528) 2026-06-27 23:32:54 -07:00