mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
## Description Establish one centrally resolved, observable, deterministic, versioned runtime rollout-control mechanism for Headroom. Runtime rollout controls which behaviors an already-built artifact may expose; it does not select or qualify a Headroom release/version. ## Type of Change - [x] New feature (non-breaking change that adds functionality) - [x] Bug fix (non-breaking change that fixes rollout enforcement regressions) - [x] Documentation update - [x] Code refactoring (no functional changes) ## Changes Made - Added `RolloutChannel`, `HEADROOM_ROLLOUT_CHANNEL`, `--rollout-channel`, and a versioned immutable `RolloutSnapshot` shared by Python configuration boundaries. - Added schema/policy versions, canonical registry and snapshot SHA-256 identities, per-feature decision reasons, disable precedence, unsafe qualification poisoning, strict CLI validation, and fail-closed environment handling. - Added `headroom rollout status --json`, Python `/stats.rollout`, and Rust `/rollout/status` runtime provenance. - Added equivalent Rust snapshot semantics and shared Python/Rust policy vectors while retaining language-specific feature registries. - Enforced rollout policy at alternate Python server composition roots so `HEADROOM_READ_MATURATION=1` cannot bypass its beta gate. - Preserved typed rollout snapshots across multi-worker serialization with schema, policy, registry, snapshot-digest, type, and feature-name validation. - Made loopback runtime output-shaper updates replace the immutable snapshot atomically for request readers, retain explicit request/disable provenance, preserve channel and kill-switch precedence, invalidate cached stats, and return the effective rollout decision. - Made `headroom learn --verbosity --apply` report a channel-blocked update instead of claiming the shaper is live. - Made explicit CLI feature flags fail loudly when their current channel blocks them. - Made persistent interceptor installation select canary automatically, or reject an explicitly insufficient channel unless the break-glass override is set. - Updated architecture, proxy, rollout, learn, and output-shaper documentation with required channels and hot-reload semantics. ## Testing - [x] Unit tests pass - [x] Linting passes (`ruff check .` and `ruff format --check .`) - [x] Type checking passes (`mypy headroom --ignore-missing-imports`) - [x] New regression tests added for every corrected behavior - [x] Rust tests and production-target Clippy pass - [x] Documentation build passes ### Test Output ```text Focused rollout coverage suite 57 passed; headroom.rollout + rollout CLI: 98% coverage Affected proxy/rollout/transform/governance suites 222 passed; 0 failed Final changed regression suites 100 passed; 0 failed Cross-module hot-reload isolation regression 6 passed; 0 failed cargo test -p headroom-core -p headroom-proxy --quiet headroom-core: 924 passed; 1 ignored headroom-proxy and integration suites: all passed cargo clippy -p headroom-core -p headroom-proxy --lib --bins -- -D warnings cargo fmt --all -- --check ruff check . ruff format --check . mypy headroom --ignore-missing-imports git diff --check All passed cd docs && npm run build Compiled successfully; 164 static pages generated ``` The unsharded Windows-only CI selection exposed unrelated baseline failures, principally the existing `sqlite:///C:\\...` URL parser producing an invalid `\\C:\\...` path. At commit `8e793a80`, all 52 completed GitHub checks passed; the only other conclusions are expected skips and superseded governance jobs. ## Real Behavior Proof - **Environment:** Windows checkout on Python 3.13.3 and the current Rust workspace, based on upstream `main` at `93f2d7a2`. - **Exact command / steps:** Exercised canary and beta feature requests through CLI status, Python `/stats.rollout`, Rust `/rollout/status`, multi-worker payload round trips, loopback `/admin/runtime-env`, real proxy request shaping before/after hot reload, installer manifest generation, and shared Python/Rust policy vectors. - **Observed result:** Stable blocks unstable requests; disable wins over explicit/default/legacy/unsafe paths; unsafe state reports `qualification_eligible=false`; worker handoff rejects tampering; running output shaping changes only when the effective beta policy permits it; explicit blocked flags fail with actionable diagnostics. - **Not tested:** Live production traffic requiring provider credentials, or future artifact qualification/promotion automation (intentionally out of scope). ## Runtime Rollout Safety - **Rollout-managed features:** Python `tool_result_interceptors`, `proxy_output_shaper`, `read_maturation`; Rust `native_bedrock`, `openai_responses_streaming`, `canary_probe`. - **Minimum rollout channel:** Registry-defined per feature; process default is `stable`. - **Stable/default behavior changed:** No unstable feature becomes enabled by default. Explicit blocked CLI flags now fail instead of silently doing nothing. - **Kill switch / disable path:** `HEADROOM_DISABLE_FEATURES=<comma-separated feature names>`; explicit disable has highest precedence, including over the unsafe override. - **Unsafe override required:** No. `HEADROOM_UNSAFE_ALLOW_UNSTABLE_FEATURES=1` is break-glass only and makes qualification evidence ineligible. - **Qualification impact:** Adds machine-readable policy/snapshot identities and eligibility; does not implement qualification itself. - **Rollback path:** Set the named disable list for operational rollback, lower the channel, or revert this PR. ## Review Readiness - [x] I have performed a full diff review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented hard-to-understand areas - [x] I have made corresponding documentation changes - [x] My changes generate no new warnings - [x] I added tests that reproduce and prevent every regression fixed during review - [x] New and existing affected tests pass locally - [x] I did **not** edit `CHANGELOG.md`; release-please generates it from the Conventional Commit PR title ## Additional Notes Out of scope: artifact candidates, benchmark orchestration, qualification manifests/gates, promotion automation, release branches, publication guards, and release-risk classification. Those workflows can consume the rollout registry digest, runtime snapshot digest, decision reasons, and qualification eligibility through supported black-box interfaces. --------- Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net> Co-authored-by: JD Davis <jd@JDH-AIR-00.local>
235 lines
6.2 KiB
Python
235 lines
6.2 KiB
Python
"""Tests for pr-governance.py."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import importlib.util
|
|
import json
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
|
|
def _load_module():
|
|
script = Path(__file__).parent.parent / "pr-governance.py"
|
|
spec = importlib.util.spec_from_file_location("pr_governance", script)
|
|
assert spec is not None
|
|
assert spec.loader is not None
|
|
module = importlib.util.module_from_spec(spec)
|
|
sys.modules[spec.name] = module
|
|
spec.loader.exec_module(module)
|
|
return module
|
|
|
|
|
|
def _event(body: str, *, draft: bool = False, login: str = "octocat") -> dict[str, object]:
|
|
return {
|
|
"pull_request": {
|
|
"number": 42,
|
|
"draft": draft,
|
|
"body": body,
|
|
"user": {"login": login},
|
|
}
|
|
}
|
|
|
|
|
|
VALID_BODY = """## Description
|
|
|
|
Add a required PR-governance gate for template validation and review readiness.
|
|
|
|
Closes #123
|
|
|
|
## Type of Change
|
|
|
|
- [x] New feature (non-breaking change that adds functionality)
|
|
- [ ] Documentation update
|
|
|
|
## Changes Made
|
|
|
|
- Added a workflow-backed PR template validator.
|
|
- Added local commit message linting in the commit-msg hook.
|
|
|
|
## Testing
|
|
|
|
- [x] Unit tests pass (`pytest`)
|
|
- [x] Manual testing performed
|
|
|
|
### Test Output
|
|
|
|
```text
|
|
pytest scripts/tests/test_pr_governance.py -q
|
|
```
|
|
|
|
## Real Behavior Proof
|
|
|
|
- Environment: Ubuntu runner, Python 3.12
|
|
- Exact command / steps: Open a PR, remove the ready checkbox, re-run the workflow.
|
|
- Observed result: The governance check fails and the PR gets a needs-author-action label.
|
|
- Not tested: Automatic Copilot review rulesets in repository settings.
|
|
|
|
## Runtime Rollout Safety
|
|
|
|
- Rollout-managed feature(s): None.
|
|
- Minimum rollout channel: Stable.
|
|
- Stable/default behavior changed: No.
|
|
- Kill switch / disable path: Not applicable.
|
|
- Unsafe override required: No.
|
|
- Qualification impact: None.
|
|
- Rollback path: Revert the workflow and script changes.
|
|
|
|
## Review Readiness
|
|
|
|
- [x] I have performed a self-review
|
|
- [x] This PR is ready for human review
|
|
|
|
## Additional Notes
|
|
|
|
- Maintainers can optionally enable Copilot code review from repository rulesets.
|
|
"""
|
|
|
|
|
|
def test_validate_pull_request_marks_ready_pr_valid() -> None:
|
|
module = _load_module()
|
|
|
|
report = module.validate_pull_request(_event(VALID_BODY))
|
|
|
|
assert report.valid is True
|
|
assert report.ready_for_review is True
|
|
assert report.needs_author_action is False
|
|
assert report.problems == []
|
|
assert report.labels_to_add == [module.READY_LABEL]
|
|
assert module.AUTHOR_ACTION_LABEL in report.labels_to_remove
|
|
|
|
|
|
def test_validate_pull_request_accepts_crlf_test_output_code_block() -> None:
|
|
module = _load_module()
|
|
body = VALID_BODY.replace("\n", "\r\n")
|
|
|
|
report = module.validate_pull_request(_event(body))
|
|
|
|
assert report.valid is True
|
|
assert report.problems == []
|
|
|
|
|
|
def test_validate_pull_request_body_override_uses_live_body() -> None:
|
|
module = _load_module()
|
|
stale_event_body = ""
|
|
|
|
report = module.validate_pull_request_body(_event(stale_event_body), VALID_BODY)
|
|
|
|
assert report.valid is True
|
|
assert report.ready_for_review is True
|
|
assert report.problems == []
|
|
|
|
|
|
def test_cli_body_file_override_uses_live_body(tmp_path: Path, monkeypatch) -> None:
|
|
module = _load_module()
|
|
event_path = tmp_path / "event.json"
|
|
body_path = tmp_path / "body.md"
|
|
report_path = tmp_path / "report.json"
|
|
event_path.write_text(
|
|
json.dumps(_event("")),
|
|
encoding="utf-8",
|
|
)
|
|
body_path.write_text(VALID_BODY, encoding="utf-8")
|
|
monkeypatch.delenv("GITHUB_OUTPUT", raising=False)
|
|
|
|
exit_code = module.main(
|
|
[
|
|
"--event",
|
|
str(event_path),
|
|
"--body-file",
|
|
str(body_path),
|
|
"--report",
|
|
str(report_path),
|
|
]
|
|
)
|
|
|
|
assert exit_code == 0
|
|
report = json.loads(report_path.read_text(encoding="utf-8"))
|
|
assert report["valid"] is True
|
|
assert report["ready_for_review"] is True
|
|
|
|
|
|
def test_validate_pull_request_allows_draft_without_ready_checkboxes() -> None:
|
|
module = _load_module()
|
|
body = VALID_BODY.replace(
|
|
"- [x] I have performed a self-review", "- [ ] I have performed a self-review"
|
|
)
|
|
body = body.replace(
|
|
"- [x] This PR is ready for human review",
|
|
"- [ ] This PR is ready for human review",
|
|
)
|
|
|
|
report = module.validate_pull_request(_event(body, draft=True))
|
|
|
|
assert report.valid is True
|
|
assert report.ready_for_review is False
|
|
assert report.needs_author_action is False
|
|
assert report.labels_to_add == []
|
|
assert module.READY_LABEL in report.labels_to_remove
|
|
|
|
|
|
def test_validate_pull_request_fails_on_missing_required_content() -> None:
|
|
module = _load_module()
|
|
body = """## Description
|
|
|
|
Fixes #123
|
|
|
|
## Type of Change
|
|
|
|
- [ ] New feature (non-breaking change that adds functionality)
|
|
|
|
## Changes Made
|
|
|
|
- Change 1
|
|
|
|
## Testing
|
|
|
|
### Test Output
|
|
|
|
```text
|
|
# Paste relevant command output or artifact links here
|
|
```
|
|
|
|
## Real Behavior Proof
|
|
|
|
- Environment:
|
|
- Exact command / steps:
|
|
- Observed result:
|
|
- Not tested:
|
|
|
|
## Runtime Rollout Safety
|
|
|
|
- Rollout-managed feature(s):
|
|
- Minimum rollout channel:
|
|
- Stable/default behavior changed:
|
|
- Kill switch / disable path:
|
|
- Unsafe override required:
|
|
- Qualification impact:
|
|
- Rollback path:
|
|
|
|
## Review Readiness
|
|
|
|
- [ ] I have performed a self-review
|
|
- [ ] This PR is ready for human review
|
|
"""
|
|
|
|
report = module.validate_pull_request(_event(body))
|
|
|
|
assert report.valid is False
|
|
assert report.needs_author_action is True
|
|
assert module.AUTHOR_ACTION_LABEL in report.labels_to_add
|
|
assert any("Description" in problem for problem in report.problems)
|
|
assert any("Type of Change" in problem for problem in report.problems)
|
|
assert any("Test Output" in problem for problem in report.problems)
|
|
assert any("Real Behavior Proof" in problem for problem in report.problems)
|
|
assert any("Runtime Rollout Safety" in problem for problem in report.problems)
|
|
|
|
|
|
def test_validate_pull_request_skips_bot_authored_prs() -> None:
|
|
module = _load_module()
|
|
|
|
report = module.validate_pull_request(_event("", login="dependabot[bot]"))
|
|
|
|
assert report.valid is True
|
|
assert report.is_bot_pr is True
|
|
assert report.needs_author_action is False
|
|
assert report.labels_to_add == []
|