headroom/scripts
JD Davis 3077ac81e8
feat: add deterministic runtime rollout controls (#1490)
## Description

Establish one centrally resolved, observable, deterministic, versioned
runtime rollout-control mechanism for Headroom. Runtime rollout controls
which behaviors an already-built artifact may expose; it does not select
or qualify a Headroom release/version.

## Type of Change

- [x] New feature (non-breaking change that adds functionality)
- [x] Bug fix (non-breaking change that fixes rollout enforcement
regressions)
- [x] Documentation update
- [x] Code refactoring (no functional changes)

## Changes Made

- Added `RolloutChannel`, `HEADROOM_ROLLOUT_CHANNEL`,
`--rollout-channel`, and a versioned immutable `RolloutSnapshot` shared
by Python configuration boundaries.
- Added schema/policy versions, canonical registry and snapshot SHA-256
identities, per-feature decision reasons, disable precedence, unsafe
qualification poisoning, strict CLI validation, and fail-closed
environment handling.
- Added `headroom rollout status --json`, Python `/stats.rollout`, and
Rust `/rollout/status` runtime provenance.
- Added equivalent Rust snapshot semantics and shared Python/Rust policy
vectors while retaining language-specific feature registries.
- Enforced rollout policy at alternate Python server composition roots
so `HEADROOM_READ_MATURATION=1` cannot bypass its beta gate.
- Preserved typed rollout snapshots across multi-worker serialization
with schema, policy, registry, snapshot-digest, type, and feature-name
validation.
- Made loopback runtime output-shaper updates replace the immutable
snapshot atomically for request readers, retain explicit request/disable
provenance, preserve channel and kill-switch precedence, invalidate
cached stats, and return the effective rollout decision.
- Made `headroom learn --verbosity --apply` report a channel-blocked
update instead of claiming the shaper is live.
- Made explicit CLI feature flags fail loudly when their current channel
blocks them.
- Made persistent interceptor installation select canary automatically,
or reject an explicitly insufficient channel unless the break-glass
override is set.
- Updated architecture, proxy, rollout, learn, and output-shaper
documentation with required channels and hot-reload semantics.

## Testing

- [x] Unit tests pass
- [x] Linting passes (`ruff check .` and `ruff format --check .`)
- [x] Type checking passes (`mypy headroom --ignore-missing-imports`)
- [x] New regression tests added for every corrected behavior
- [x] Rust tests and production-target Clippy pass
- [x] Documentation build passes

### Test Output

```text
Focused rollout coverage suite
57 passed; headroom.rollout + rollout CLI: 98% coverage

Affected proxy/rollout/transform/governance suites
222 passed; 0 failed

Final changed regression suites
100 passed; 0 failed

Cross-module hot-reload isolation regression
6 passed; 0 failed

cargo test -p headroom-core -p headroom-proxy --quiet
headroom-core: 924 passed; 1 ignored
headroom-proxy and integration suites: all passed

cargo clippy -p headroom-core -p headroom-proxy --lib --bins -- -D warnings
cargo fmt --all -- --check
ruff check .
ruff format --check .
mypy headroom --ignore-missing-imports
git diff --check
All passed

cd docs && npm run build
Compiled successfully; 164 static pages generated
```

The unsharded Windows-only CI selection exposed unrelated baseline
failures, principally the existing `sqlite:///C:\\...` URL parser
producing an invalid `\\C:\\...` path. At commit `8e793a80`, all 52
completed GitHub checks passed; the only other conclusions are expected
skips and superseded governance jobs.

## Real Behavior Proof

- **Environment:** Windows checkout on Python 3.13.3 and the current
Rust workspace, based on upstream `main` at `93f2d7a2`.
- **Exact command / steps:** Exercised canary and beta feature requests
through CLI status, Python `/stats.rollout`, Rust `/rollout/status`,
multi-worker payload round trips, loopback `/admin/runtime-env`, real
proxy request shaping before/after hot reload, installer manifest
generation, and shared Python/Rust policy vectors.
- **Observed result:** Stable blocks unstable requests; disable wins
over explicit/default/legacy/unsafe paths; unsafe state reports
`qualification_eligible=false`; worker handoff rejects tampering;
running output shaping changes only when the effective beta policy
permits it; explicit blocked flags fail with actionable diagnostics.
- **Not tested:** Live production traffic requiring provider
credentials, or future artifact qualification/promotion automation
(intentionally out of scope).

## Runtime Rollout Safety

- **Rollout-managed features:** Python `tool_result_interceptors`,
`proxy_output_shaper`, `read_maturation`; Rust `native_bedrock`,
`openai_responses_streaming`, `canary_probe`.
- **Minimum rollout channel:** Registry-defined per feature; process
default is `stable`.
- **Stable/default behavior changed:** No unstable feature becomes
enabled by default. Explicit blocked CLI flags now fail instead of
silently doing nothing.
- **Kill switch / disable path:**
`HEADROOM_DISABLE_FEATURES=<comma-separated feature names>`; explicit
disable has highest precedence, including over the unsafe override.
- **Unsafe override required:** No.
`HEADROOM_UNSAFE_ALLOW_UNSTABLE_FEATURES=1` is break-glass only and
makes qualification evidence ineligible.
- **Qualification impact:** Adds machine-readable policy/snapshot
identities and eligibility; does not implement qualification itself.
- **Rollback path:** Set the named disable list for operational
rollback, lower the channel, or revert this PR.

## Review Readiness

- [x] I have performed a full diff review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented hard-to-understand areas
- [x] I have made corresponding documentation changes
- [x] My changes generate no new warnings
- [x] I added tests that reproduce and prevent every regression fixed
during review
- [x] New and existing affected tests pass locally
- [x] I did **not** edit `CHANGELOG.md`; release-please generates it
from the Conventional Commit PR title

## Additional Notes

Out of scope: artifact candidates, benchmark orchestration,
qualification manifests/gates, promotion automation, release branches,
publication guards, and release-risk classification. Those workflows can
consume the rollout registry digest, runtime snapshot digest, decision
reasons, and qualification eligibility through supported black-box
interfaces.

---------

Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
Co-authored-by: JD Davis <jd@JDH-AIR-00.local>
2026-08-12 23:16:54 -05:00
..
ci ci: harden PR governance and model cache checks (#1401) 2026-06-26 21:34:34 -07:00
fixtures feat(scripts): add Codex proxy reconnect-storm repro harness 2026-04-20 22:02:02 +07:00
tests feat: add deterministic runtime rollout controls (#1490) 2026-08-12 23:16:54 -05:00
audit_wheel_glibc_symbols.py fix(crusher): shim __libc_single_threaded for glibc < 2.32 + extend audit 2026-05-05 13:59:21 -07:00
bootstrap-windows-dev.ps1 chore(release): harden local artifact smokes (#1824) 2026-07-14 16:07:34 -04:00
build_npm_release_assets.mjs chore(release): harden local artifact smokes (#1824) 2026-07-14 16:07:34 -04:00
build_python_release_smoke.py chore(release): harden local artifact smokes (#1824) 2026-07-14 16:07:34 -04:00
build_rust_extension.sh refactor: single-wheel maturin build backend (fixes #355) 2026-05-03 13:16:41 -07:00
changelog-gen.py chore: renormalize line endings to LF 2026-04-24 15:33:30 +02:00
eval_output_shaper.py feat: output-token reduction — verbosity shaper, per-user learning, counterfactual savings (#965) 2026-06-16 21:06:43 -07:00
export_kompress_v2_onnx.py feat: switch Kompress default to kompress-v2-base with weight-only int8 ONNX (#799) 2026-06-09 23:28:40 -07:00
install-git-hooks.sh Fix CI lint failure by formatting PR governance scripts (#933) 2026-06-12 17:11:39 -05:00
install.ps1 fix(install): don't crash the PowerShell installer when $PROFILE is unset (#2469) 2026-08-12 00:12:30 -05:00
install.sh fix(install): trust Docker bridge for dashboard metadata 2026-08-11 14:25:29 -07:00
pr-governance.py feat: add deterministic runtime rollout controls (#1490) 2026-08-12 23:16:54 -05:00
README.md chore(release): harden local artifact smokes (#1824) 2026-07-14 16:07:34 -04:00
record_code_compressor_fixtures.py feat(rust): port CodeCompressor AST compressor to Rust (parity-only) (#1154) 2026-07-27 09:21:57 -07:00
record_fixtures.py feat(rust): scaffold workspace + parity harness (phase-0) 2026-04-24 13:39:48 -07:00
record_kompress_fixtures.py feat(rust): port Kompress ML prose compressor to Rust (parity-only) (#1153) 2026-07-27 08:17:53 -07:00
refresh_model_limits.sh fix(rust): wire ICM compressor into Rust proxy on /v1/messages 2026-05-01 16:44:44 -07:00
release_smoke_all.py chore(release): harden local artifact smokes (#1824) 2026-07-14 16:07:34 -04:00
replay_codex_ws_load.py fix(tests): ship scripts/replay_codex_ws_load.py so CI can import it 2026-05-14 13:44:41 -07:00
repro_codex_replay.py fix: replace asyncio.timeout with 3.10-compat shim in repro harness 2026-04-20 13:41:02 -05:00
smoke_issue_327.py fix(proxy): remove content-keyed TTL walker that conflated content with positional cache (#327) 2026-05-01 12:04:28 -07:00
sync-plugin-versions.py fix(proxy): lazy-import server to avoid fastapi crash (#442) 2026-06-10 12:44:23 -05:00
validate-workflows.sh ci: scope PR workflow runs by changed paths (#1067) 2026-06-16 19:11:45 -07:00
verify-ruff-version.py fix(ci): align Ruff tooling versions (#2406) 2026-07-18 20:55:20 -07:00
verify-versions.py fix: publish headroom-opencode in release workflow (#2372) 2026-08-11 23:56:40 -05:00
verify_npm_release_assets.mjs chore(release): harden local artifact smokes (#1824) 2026-07-14 16:07:34 -04:00
version-sync.py fix: publish headroom-opencode in release workflow (#2372) 2026-08-11 23:56:40 -05:00

scripts/

Utility scripts bundled with the Headroom repo. Most are one-off operator tools; a few are runnable as part of development workflows.

Reproducing the reconnect storm

repro_codex_replay.py reproduces the multi-agent Codex reconnect/retry storm against a local Headroom proxy (default http://127.0.0.1:8787). Use it to:

  • Regression-check that /livez stays responsive under a cold-start storm.
  • Empirically tune the Unit 4 pre-upstream semaphore default (HEADROOM_ANTHROPIC_PRE_UPSTREAM_CONCURRENCY).
  • Exercise the Codex WS lifecycle + Anthropic HTTP path simultaneously without needing to replay captured production traffic.

Run

# Default: 8 WS + 4 HTTP clients, 30s storm, p99 /livez must stay <= 500ms.
python scripts/repro_codex_replay.py

# Tighter budget, shorter run:
python scripts/repro_codex_replay.py \
    --url http://127.0.0.1:8787 \
    --ws-clients 16 \
    --anthropic-clients 8 \
    --duration 60 \
    --livez-threshold-ms 100

# Dump the full summary as JSON for downstream tooling:
python scripts/repro_codex_replay.py --json

Exit code:

  • 0 — warmup succeeded (or was skipped), storm ran for the requested duration, and /livez p99 stayed under --livez-threshold-ms.
  • 1 — soft assertion failed, proxy unreachable, or unhandled exception. Proxy-unreachable is detected and reported within ~5 seconds.

Fixtures

The script loads two hand-crafted, fully synthetic JSON fixtures:

  • scripts/fixtures/anthropic_replay_body.json — shape of a large agent reconnect replay /v1/messages?beta=true POST body.
  • scripts/fixtures/codex_response_create_frame.json — first Codex WS frame with the {"type": "response.create", "response": {...}} envelope.

Override via --ws-frame-fixture / --anthropic-body-fixture if you have captured traffic to replay instead.

Interpretation

  • /livez p99 under threshold means the event loop is not starved during the storm. If it rises with the semaphore unbounded (HEADROOM_ANTHROPIC_PRE_UPSTREAM_CONCURRENCY=10000) and drops back under the default, Unit 4's backpressure is working.
  • Codex WS: opened should equal --ws-clients. response.completed typically stays low when upstream auth isn't configured locally — the goal is handshake + relay wiring, not real upstream traffic.
  • Anthropic HTTP: ok_2xx + non_2xx + timed_out + errors should roughly equal attempted. Sustained non-zero timed_out during the storm is the failure signal the plan targets.

A smoke test at tests/test_scripts/test_repro_codex_replay_smoke.py exercises the script against a mock FastAPI server on every PR.

Install scripts

  • install.sh — POSIX installer.
  • install.ps1 — Windows PowerShell installer.

These are generated by the release pipeline; edit with care.

Windows development bootstrap

bootstrap-windows-dev.ps1 prepares a Windows development checkout. It resolves or creates a repo-local Python virtual environment, checks for Rust, installs Python build/test tooling, installs npm dependencies for the TypeScript SDK and OpenClaw plugin, and runs a small smoke set.

powershell -ExecutionPolicy Bypass -File scripts/bootstrap-windows-dev.ps1

Use -CheckOnly to print detected tool versions without installing packages. Use -SkipSmoke, -SkipDocs, -SkipNode, or -SkipRust when intentionally debugging one part of the environment.

npm release asset smoke

build_npm_release_assets.mjs locally reproduces the release workflow's npm asset build. It builds the TypeScript SDK tarball, installs that tarball into OpenClaw, rewrites OpenClaw's release dependency to the same version, regenerates dist/package.json, packs OpenClaw, and then runs verify_npm_release_assets.mjs.

node scripts/build_npm_release_assets.mjs <version>

By default, output goes into a timestamped release-assets-local/<version>-* directory. Pass an explicit empty directory when you want a predictable path:

node scripts/build_npm_release_assets.mjs <version> release-assets-local/smoke

Expected tarballs:

  • headroom-ai-<version>.tgz
  • headroom-openclaw-<version>.tgz

The script restores package metadata after it finishes so the source tree keeps the registry-installable development dependency range.

Python release artifact smoke

build_python_release_smoke.py locally reproduces the Python artifact smoke: it builds a wheel with maturin, builds an sdist, verifies the sdist License-File metadata against tarball contents, installs the wheel into a fresh python -m venv environment, and imports the native headroom._core extension from that installed wheel.

python scripts/build_python_release_smoke.py

By default, the wheel uses the faster Cargo ci profile and output goes into a timestamped release-assets-local/python-<version>-* directory. Use --release when you want the slower shipped-wheel profile:

python scripts/build_python_release_smoke.py --release --out release-assets-local/python-release-smoke

Expected artifacts:

  • headroom_ai-<version>-*.whl
  • headroom_ai-<version>.tar.gz

Full local release smoke

release_smoke_all.py is the one-command local release gate. It first runs scripts/verify-versions.py, then runs the npm release asset smoke and the Python wheel/sdist smoke into sibling output directories.

python scripts/release_smoke_all.py

By default, output goes into release-assets-local/all-<version>-*/npm and release-assets-local/all-<version>-*/python. Pass an explicit empty output directory for a predictable evidence path:

python scripts/release_smoke_all.py --out release-assets-local/full-release-smoke

Use --python-release when the Python smoke should build with maturin's slower release profile. Use --skip-npm or --skip-python only when intentionally debugging one side of the artifact pipeline.