headroom/scripts/tests/test_ci_workflow.py

"""Tests for CI workflow hardening contracts."""

from __future__ import annotations

from pathlib import Path


def test_sharded_ci_verifies_offline_huggingface_cache_before_pytest() -> None:
    workflow = Path(".github/workflows/ci.yml").read_text(encoding="utf-8")
    verify_step = "Verify offline HuggingFace model cache"
    pytest_step = "Run test shard ${{ matrix.shard }}/4"

    assert verify_step in workflow
    assert "python scripts/ci/verify_hf_model_cache.py" in workflow
    assert workflow.index(verify_step) < workflow.index(pytest_step)


def test_sharded_ci_uploads_only_explicit_coverage_reports() -> None:
    workflow = Path(".github/workflows/ci.yml").read_text(encoding="utf-8")
    upload_step = workflow[workflow.index("Upload coverage shard") :]

    assert "files: coverage-${{ matrix.shard }}.xml" in upload_step
    assert "disable_search: true" in upload_step
ci: harden PR governance and model cache checks (#1401) ## Description Hardens two routine PR-review pain points from the recent open-PR sweep: - PR Governance reruns could keep validating the stale `pull_request_target` event body even after the live PR description had been fixed. - Main CI model-cache misses could surface as dozens of unrelated memory-test failures instead of one clear cache-preflight failure. This intentionally avoids PyPI/package-bloat and release/nightly workflow changes so the PR stays scoped to review and CI stabilization. ## Type of Change - [x] Bug fix - [ ] New feature - [ ] Documentation - [x] Refactor - [x] Tests only ## Changes Made - Added `--body-file` support to `scripts/pr-governance.py` so workflows can validate the current PR body rather than stale rerun payloads. - Updated PR Governance to fetch the live PR body via the GitHub API before validating template fields. - Added a CI preflight script that loads the default sentence-transformer model in offline mode and verifies the expected embedding dimension. - Wired that preflight into the sharded CI job before pytest starts, turning missing/corrupt Hugging Face caches into one early, actionable failure. - Added workflow/script regression tests for the live-body override and model-cache preflight placement. ## Testing - [x] Unit tests - [x] Lint/static checks - [ ] Integration tests - [ ] Manual testing ### Test Output ```text uv run --with pytest --with pytest-asyncio python -m pytest scripts/tests/test_pr_governance.py scripts/tests/test_pr_health_workflow.py scripts/tests/test_ci_workflow.py -q 9 passed in 0.04s uv run ruff check scripts/pr-governance.py scripts/ci/verify_hf_model_cache.py scripts/tests/test_pr_governance.py scripts/tests/test_pr_health_workflow.py scripts/tests/test_ci_workflow.py All checks passed! python -m py_compile scripts\ci\verify_hf_model_cache.py scripts\pr-governance.py # passed ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13.3, isolated worktree `C:\git\headroom\.worktrees\stabilization-hardening`. - Exact command / steps: Ran the focused governance/workflow tests, ruff on touched Python files, and `py_compile` for the executable scripts. - Observed result: Governance tests prove a stale event body can be overridden by the live PR body; workflow tests prove CI validates live PR body and runs the Hugging Face offline-cache preflight before pytest shards. - Not tested: Full GitHub CI before PR creation; that will run on this PR. The new Hugging Face preflight itself is intentionally not run locally because it depends on the CI-warmed offline model cache. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review 2026-06-26 23:34:34 -05:00			`"""Tests for CI workflow hardening contracts."""`

			`from __future__ import annotations`

			`from pathlib import Path`


			`def test_sharded_ci_verifies_offline_huggingface_cache_before_pytest() -> None:`
			`workflow = Path(".github/workflows/ci.yml").read_text(encoding="utf-8")`
			`verify_step = "Verify offline HuggingFace model cache"`
			`pytest_step = "Run test shard ${{ matrix.shard }}/4"`

			`assert verify_step in workflow`
			`assert "python scripts/ci/verify_hf_model_cache.py" in workflow`
			`assert workflow.index(verify_step) < workflow.index(pytest_step)`
fix(ci): restrict Codecov shard uploads (#2745) ## Description Closes #2744 Restrict each Codecov Action v5 matrix upload to its declared `coverage-${{ matrix.shard }}.xml` report. This prevents automatic discovery from uploading the unsharded `coverage.xml` alongside every shard. ## Type of Change - [x] Bug fix (non-breaking change fixes an issue) ## Changes Made - Set Codecov Action `disable_search: true` for Python shard uploads. - Add a CI workflow contract test that protects the explicit-report-only setup. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy headroom`) (not applicable: workflow/test-only change) - [x] New tests added new functionality - [x] Manual testing performed (not applicable: GitHub Actions will execute the workflow) ### Test Output ```text $ uv run --with ruff ruff format --check scripts/tests/test_ci_workflow.py 1 file already formatted $ uv run --with ruff ruff check scripts/tests/test_ci_workflow.py All checks passed! $ uv run --with pytest pytest scripts/tests/test_ci_workflow.py -q 2 passed ``` ## Real Behavior Proof - Environment: GitHub Actions Ubuntu runner using Python 3.12.13; Codecov Action v5. - Exact command / steps: Run the CI Python test matrix, which writes `coverage-${{ matrix.shard }}.xml`, then runs the Codecov Action upload step. Inspect the uploader's discovered/uploaded report list. - Observed result: Before this change, raw CI logs showed the Action explicitly uploading `coverage-2.xml` and additionally discovering/uploading `coverage.xml`. This PR configures `disable_search: true`; the workflow contract test confirms the explicit report setting and search disablement. Runtime upload evidence will be added from this draft PR's CI run. - Not tested: Codecov's final cross-shard patch calculation; that depends on Codecov processing the reports after CI completes. ## Review Readiness - [x] I have performed a self-review - [x] This PR ready for human review ## Checklist - [x] My code follows project's style guidelines - [x] I have performed a self-review - [x] I commented my code, particularly in hard-to-understand areas - [x] I made corresponding changes documentation (not applicable) - [x] My changes generate no new warnings - [x] I added tests prove my fix is effective or feature works - [x] New and existing unit tests pass locally changes - [x] I did not edit `CHANGELOG.md` — generated by release-please from Conventional Commit PR title (a CI guard enforces this) ## Additional Notes This is intentionally limited to the Codecov upload configuration and its workflow contract test. It does not include the unrelated Copilot Keychain fix. 2026-08-03 23:20:17 +02:00

			`def test_sharded_ci_uploads_only_explicit_coverage_reports() -> None:`
			`workflow = Path(".github/workflows/ci.yml").read_text(encoding="utf-8")`
			`upload_step = workflow[workflow.index("Upload coverage shard") :]`

			`assert "files: coverage-${{ matrix.shard }}.xml" in upload_step`
			`assert "disable_search: true" in upload_step`