fix(release): keep prereleases out of downstream publishing (#1120)

* fix(release): keep prereleases out of downstream publishing

* fix(ci): budget exhaustive Swift cold builds

* fix(ci): align Windows cache version paths
This commit is contained in:
Nick DiZazzo 2026-07-30 06:47:59 -04:00 committed by GitHub
parent f7517f8b77
commit a6f6f83c6c
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
12 changed files with 101 additions and 26 deletions

View file

@ -31,7 +31,7 @@ the commands at the end before operational changes.
| `windows-warm-caches.yml` | Main path push, dispatch | Trusted Windows ABI cache warming |
| `website-pages.yml` | Main website path push, dispatch | Public website Pages build/deploy |
| `fly-deploy-console.yml` | Dispatch | `fly-console` environment deployment |
| `release.yml` | `v*` tag, dispatch | Release builds, attestations, publishing, downstream package/image/npm dispatch |
| `release.yml` | `v*` tag, dispatch | Release builds, attestations, publishing, stable-only downstream package/image/npm dispatch |
| `reset-caches.yml` | Confirmed dispatch | Destructive repository cache reset |
| `stale-prs.yml` | Schedule, dispatch | PR warning/closure maintenance |
@ -62,6 +62,12 @@ manifest and SHA-256 sidecar. The release publisher requires all five producers
and attaches those exact artifacts; downstream packaging verifies and assembles
them without recompiling native source.
Only a successful, complete stable release dispatches downstream package,
image, and npm publication. Prereleases publish their immutable GitHub Release
inputs but never invoke `mesh-packaging`; this provides a safe artifact
validation boundary without exposing prerelease inputs to production
promotion.
The Windows host input also carries the checksum-protected `xtask` executable
that performed producer-side attestation. Windows product composers invoke that
prebuilt verifier for the immutable host instead of compiling workspace code.
@ -88,6 +94,12 @@ or manual dispatch. Every composed backend product requires `runtime list`
plus no-driver client readiness; hosted GPU rows neither inject a driver stub
nor skip startup because no device is present.
The exhaustive Swift producer has a 180-minute main/release cold-start budget
because it serially builds seven Apple target ABIs. PR host-only calls retain
their shorter budget. Exact native ABI and compiler caches remain responsible
for reducing the warm path; the timeout is only the reliability ceiling for an
unseeded cache.
`pr_builds.yml` uses the same split producer/composer shape for Linux CPU/GPU
and macOS Metal products while retaining debug-profile hosts for lightweight
PR iteration. Windows broad-Rust validation stays at lightweight Cargo checks;
@ -154,7 +166,9 @@ Local actions:
CUDA, ROCm, and Vulkan ABI cache identity shared by the trusted warmer and
PR/main/release runtime producers. The hosted-image epoch, architecture sets,
and toolchain versions are compatibility boundaries; the action requires the
key epoch to equal the build-stamp epoch and never uses restore prefixes.
key epoch to equal the build-stamp epoch, includes the publication action in
the key hash, exports one validated absolute path for both restore and save,
and never uses restore prefixes.
- `.github/actions/save-and-verify-actions-cache` snapshots existing exact
key/ref cache entries before saving a trusted miss, then requires a new,
non-empty entry to appear and performs a lookup-only restore with the same

View file

@ -315,7 +315,7 @@ runs:
if [[ "$ALL_RUST" == "true" ]]; then
BACKEND_CHANGED="true"
elif [[ -n "$CHANGED_FILES" ]]; then
BACKEND_INPUTS=$(echo "$CHANGED_FILES" | grep -E '(^third_party/llama\.cpp/|^crates/skippy-ffi/|^scripts/(build-llama|prepare-llama|build-linux|build-linux-rocm|build-mac|build-windows|install-windows-sdk|build-host|build-release|package-release|package-native-runtime|verify-native-runtime-package|verify-checksum-sidecar|safe-extract-tar|compose-product-bundle|ci-compose-product-input|ci-client-readiness-smoke)\.|^\.github/actions/(prepare-host-input|prepare-windows-host-input|prepare-native-runtime-input|compose-product-input|resolve-native-toolchain-epoch|restore-smoke-inputs|restore-windows-abi-cache|setup-windows-rocm-sdk)/|^\.github/workflows/(ci|pr_builds|release|sdk-smoke|smoke)\.yml$|^\.github/cache-version\.txt$)' || true)
BACKEND_INPUTS=$(echo "$CHANGED_FILES" | grep -E '(^third_party/llama\.cpp/|^crates/skippy-ffi/|^scripts/(build-llama|prepare-llama|build-linux|build-linux-rocm|build-mac|build-windows|install-windows-sdk|build-host|build-release|package-release|package-native-runtime|verify-native-runtime-package|verify-checksum-sidecar|safe-extract-tar|compose-product-bundle|ci-compose-product-input|ci-client-readiness-smoke)\.|^\.github/actions/(prepare-host-input|prepare-windows-host-input|prepare-native-runtime-input|compose-product-input|resolve-native-toolchain-epoch|restore-smoke-inputs|restore-windows-abi-cache|save-and-verify-actions-cache|setup-windows-rocm-sdk)/|^\.github/workflows/(ci|pr_builds|release|sdk-smoke|smoke)\.yml$|^\.github/cache-version\.txt$)' || true)
if [[ -n "$BACKEND_INPUTS" ]] || [[ "$BACKEND_RECIPE_CHANGED" == "true" ]]; then
BACKEND_CHANGED="true"
fi
@ -327,8 +327,8 @@ runs:
WINDOWS_CPU_BUILD_REQUIRED="true"
WINDOWS_GPU_BUILD_REQUIRED="true"
elif [[ -n "$CHANGED_FILES" ]]; then
WINDOWS_CPU_INPUTS=$(echo "$CHANGED_FILES" | grep -E '(^crates/mesh-llm-release-footer/|^crates/mesh-llm-nodejs/|^crates/skippy-ffi/|^scripts/(build-windows|package-release)\.ps1$|^scripts/verify-host-dependencies\.py$|^scripts/(package-native-runtime|verify-native-runtime-package|verify-checksum-sidecar|safe-extract-tar|compose-product-bundle|ci-compose-product-input|ci-client-readiness-smoke)\.|^third_party/llama\.cpp/|^Cargo\.toml$|^Cargo\.lock$|^\.github/cache-version\.txt$|^\.github/workflows/(ci|pr_builds|release|windows-warm-caches)\.yml$|^\.github/actions/(compute-changes/action\.yml$|prepare-windows-host-input/|prepare-native-runtime-input/|compose-product-input/|resolve-native-toolchain-epoch/|restore-windows-abi-cache/))' || true)
WINDOWS_GPU_INPUTS=$(echo "$CHANGED_FILES" | grep -E '(^crates/skippy-ffi/|^scripts/(build-windows|install-windows-sdk|package-release)\.ps1$|^scripts/verify-host-dependencies\.py$|^scripts/(package-native-runtime|verify-native-runtime-package|verify-checksum-sidecar|safe-extract-tar|compose-product-bundle|ci-compose-product-input|ci-client-readiness-smoke)\.|^scripts/windows-native-runtime-deps\.py$|^scripts/tests/test_windows_native_runtime_deps\.py$|^third_party/llama\.cpp/|^\.github/cache-version\.txt$|^\.github/workflows/(ci|pr_builds|release|windows-warm-caches)\.yml$|^\.github/actions/(compute-changes/action\.yml$|prepare-windows-host-input/|prepare-native-runtime-input/|compose-product-input/|resolve-native-toolchain-epoch/|restore-windows-abi-cache/|setup-windows-rocm-sdk/))' || true)
WINDOWS_CPU_INPUTS=$(echo "$CHANGED_FILES" | grep -E '(^crates/mesh-llm-release-footer/|^crates/mesh-llm-nodejs/|^crates/skippy-ffi/|^scripts/(build-windows|package-release)\.ps1$|^scripts/verify-host-dependencies\.py$|^scripts/(package-native-runtime|verify-native-runtime-package|verify-checksum-sidecar|safe-extract-tar|compose-product-bundle|ci-compose-product-input|ci-client-readiness-smoke)\.|^third_party/llama\.cpp/|^Cargo\.toml$|^Cargo\.lock$|^\.github/cache-version\.txt$|^\.github/workflows/(ci|pr_builds|release|windows-warm-caches)\.yml$|^\.github/actions/(compute-changes/action\.yml$|prepare-windows-host-input/|prepare-native-runtime-input/|compose-product-input/|resolve-native-toolchain-epoch/|restore-windows-abi-cache/|save-and-verify-actions-cache/))' || true)
WINDOWS_GPU_INPUTS=$(echo "$CHANGED_FILES" | grep -E '(^crates/skippy-ffi/|^scripts/(build-windows|install-windows-sdk|package-release)\.ps1$|^scripts/verify-host-dependencies\.py$|^scripts/(package-native-runtime|verify-native-runtime-package|verify-checksum-sidecar|safe-extract-tar|compose-product-bundle|ci-compose-product-input|ci-client-readiness-smoke)\.|^scripts/windows-native-runtime-deps\.py$|^scripts/tests/test_windows_native_runtime_deps\.py$|^third_party/llama\.cpp/|^\.github/cache-version\.txt$|^\.github/workflows/(ci|pr_builds|release|windows-warm-caches)\.yml$|^\.github/actions/(compute-changes/action\.yml$|prepare-windows-host-input/|prepare-native-runtime-input/|compose-product-input/|resolve-native-toolchain-epoch/|restore-windows-abi-cache/|save-and-verify-actions-cache/|setup-windows-rocm-sdk/))' || true)
if [[ -n "$WINDOWS_CPU_INPUTS" ]] || [[ "$BACKEND_RECIPE_CHANGED" == "true" ]]; then
WINDOWS_CPU_BUILD_REQUIRED="true"
fi

View file

@ -35,6 +35,9 @@ outputs:
cache-primary-key:
description: Exact primary key used for restore and a later cache save.
value: ${{ steps.restore.outputs.cache-primary-key }}
cache-path:
description: Validated absolute build path shared by restore and save.
value: ${{ steps.identity.outputs.build-dir }}
runs:
using: composite
@ -50,7 +53,7 @@ runs:
INPUT_CUDA_TOOLCHAIN_VERSION: ${{ inputs.cuda_toolchain_version }}
INPUT_VULKAN_TOOLCHAIN_VERSION: ${{ inputs.vulkan_toolchain_version }}
INPUT_ROCM_TOOLCHAIN_VERSION: ${{ inputs.rocm_toolchain_version }}
CACHE_INPUT_HASH: ${{ hashFiles('.github/actions/restore-windows-abi-cache/action.yml', '.github/actions/resolve-native-toolchain-epoch/action.yml', '.github/actions/prepare-native-runtime-input/action.yml', '.github/actions/setup-windows-rocm-sdk/action.yml', 'scripts/build-llama.sh', 'scripts/prepare-llama.sh', 'scripts/package-native-runtime.sh', 'third_party/llama.cpp/upstream.txt', 'third_party/llama.cpp/patches/**', '.github/cache-version.txt') }}
CACHE_INPUT_HASH: ${{ hashFiles('.github/actions/restore-windows-abi-cache/action.yml', '.github/actions/save-and-verify-actions-cache/action.yml', '.github/actions/resolve-native-toolchain-epoch/action.yml', '.github/actions/prepare-native-runtime-input/action.yml', '.github/actions/setup-windows-rocm-sdk/action.yml', 'scripts/build-llama.sh', 'scripts/prepare-llama.sh', 'scripts/package-native-runtime.sh', 'third_party/llama.cpp/upstream.txt', 'third_party/llama.cpp/patches/**', '.github/cache-version.txt') }}
run: |
$ErrorActionPreference = "Stop"

View file

@ -723,7 +723,7 @@ jobs:
mode: full
artifact_name: ci-swift-sdk-input
retention_days: 1
timeout_minutes: 90
timeout_minutes: 180
swift_sdk_smoke:
needs: [changes, macos_cpu_artifact, swift_sdk_input]

View file

@ -726,7 +726,7 @@ jobs:
update_release_manifest: ${{ github.event_name == 'workflow_dispatch' }}
require_tracked_console: ${{ github.event_name != 'workflow_dispatch' }}
retention_days: 90
timeout_minutes: 90
timeout_minutes: 180
build_linux_arm64:
name: Build immutable host Linux ARM64
@ -1733,7 +1733,7 @@ jobs:
dispatch_packaging_release:
name: Dispatch package and image publishing
needs: [metadata, publish]
if: ${{ needs.publish.result == 'success' && needs.metadata.outputs.skip_gpu_bundles != 'true' }}
if: ${{ needs.publish.result == 'success' && needs.metadata.outputs.prerelease != 'true' && needs.metadata.outputs.skip_gpu_bundles != 'true' }}
runs-on: ubuntu-24.04
permissions:
contents: read

View file

@ -117,7 +117,7 @@ jobs:
if: steps.llama_cache.outputs.cache-hit != 'true'
uses: ./.github/actions/save-and-verify-actions-cache
with:
path: ${{ env.LLAMA_STAGE_BUILD_DIR }}
path: ${{ steps.llama_cache.outputs.cache-path }}
cache-key: ${{ steps.llama_cache.outputs.cache-primary-key }}
cache-ref: ${{ github.ref }}
cache-label: Windows CPU ABI
@ -267,7 +267,7 @@ jobs:
if: steps.llama_cache.outputs.cache-hit != 'true'
uses: ./.github/actions/save-and-verify-actions-cache
with:
path: ${{ env.LLAMA_STAGE_BUILD_DIR }}
path: ${{ steps.llama_cache.outputs.cache-path }}
cache-key: ${{ steps.llama_cache.outputs.cache-primary-key }}
cache-ref: ${{ github.ref }}
cache-label: Windows ${{ matrix.name }} ABI

View file

@ -8,13 +8,14 @@ Releases are normally cut by running the **Release** workflow
dispatched workflow bumps versions, generates and patches the SwiftPM
manifest, packages SDK console assets, creates and pushes the release tag,
builds all platform bundles, and publishes the GitHub release. After a complete
non-canary release succeeds, it dispatches `Mesh-LLM/mesh-packaging` to
package the verified release archives, publish the native package release
assets, publish the supported GHCR image matrix, and assemble and publish the
Node SDK to npm. Dispatch inputs include
`skip_gpu_bundles` and `canary` (dry-run: build and smoke everything without
publishing). Releases that intentionally skip GPU bundles do not dispatch the
full packaging matrix.
stable, non-canary release with the full GPU matrix succeeds, it dispatches
`Mesh-LLM/mesh-packaging` to package the verified release archives, publish the
native package release assets, publish the supported GHCR image matrix, and
assemble and publish the Node SDK to npm. Prereleases publish their immutable
GitHub Release inputs without invoking downstream publication. Dispatch inputs
include `skip_gpu_bundles` and `canary` (dry-run: build and smoke everything
without publishing). Releases that intentionally skip GPU bundles do not
dispatch the full packaging matrix.
The sections below document the underlying steps. They matter when releasing
manually via a tag push, debugging the workflow, or validating bundles
@ -216,10 +217,11 @@ Verify:
Push a `v*` tag to run `.github/workflows/release.yml`. The upstream release
workflow owns release archive production, but it does not publish OCI images.
`Mesh-LLM/mesh-packaging` is the canonical package, GHCR, and npm producer and starts only
after the GitHub release and its complete archive set have published
successfully. The upstream `docker.yml` workflow performs Dockerfile validation
only and is not a distribution channel.
`Mesh-LLM/mesh-packaging` is the canonical package, GHCR, and npm producer. It
starts only after a stable GitHub release and its complete CPU/GPU archive set
have published successfully. Prereleases never dispatch it. The upstream
`docker.yml` workflow performs Dockerfile validation only and is not a
distribution channel.
On non-prerelease tags, the release workflow also publishes the Rust SDK crate
chain to crates.io in dependency order:

View file

@ -189,7 +189,9 @@ flowchart TD
smokes consume the staged runtime instead of compiling a private replacement.
The Swift XCFramework is also built by the same typed producer used by PR and
release: main requests exhaustive `full` mode and its smoke only verifies and
consumes that immutable artifact.
consumes that immutable artifact. Main and release give the seven-target
full producer a 180-minute cold-start ceiling; PR host-only validation keeps
its shorter iteration budget.
- Main builds immutable Linux, macOS, and Windows release hosts independently
from their CPU, Metal, CUDA, ROCm, and Vulkan runtimes. Composition-only jobs
verify and combine those exact producer inputs. Each Linux GPU backend has
@ -205,6 +207,9 @@ flowchart TD
manifest-bound, checksummed archive. Release publication requires all five
artifacts, and `mesh-packaging` consumes those exact release assets instead
of compiling addon source again.
- Release publication dispatches downstream package/image/npm promotion only
for stable versions. Prereleases retain the complete immutable GitHub Release
artifact graph for validation but never invoke `mesh-packaging`.
- `.github/actions/prepare-host-input`,
`.github/actions/prepare-windows-host-input`,
`.github/actions/prepare-native-runtime-input`,
@ -222,7 +227,10 @@ flowchart TD
afterward, then performs a lookup-only restore with the same path and key to
prove the current cache version exists. A cache-service reservation warning
therefore cannot leave the warmer green without publishing a reusable ABI
input.
input. The restore action exports the normalized absolute cache path used by
the save action, and publication-action changes participate in the exact key,
preventing an incompatible opaque cache version from blocking its
replacement under the same key.
### Current PR Builds contract
@ -331,8 +339,10 @@ flowchart TD
re-stamping them. The Windows host input includes a checksum-protected
producer-built attestation verifier so Windows composers do not compile
workspace code. Product consumers never rebuild a missing producer. It
dispatches the completed release to `Mesh-LLM/mesh-packaging`,
which owns package, GHCR, and npm publication.
dispatches a completed stable release with the full GPU matrix to
`Mesh-LLM/mesh-packaging`, which owns package, GHCR, and npm publication.
Prereleases publish immutable GitHub Release inputs but never dispatch
downstream publication.
- `fly-deploy-console.yml` is a manual (`workflow_dispatch`) deploy of the
`mesh-llm-console` Fly app. It builds the image on Fly's remote builders from
`fly/Dockerfile` and authenticates with the app-scoped `FLY_API_TOKEN` repo

View file

@ -141,6 +141,16 @@ class BuildWindowsScriptTests(unittest.TestCase):
),
2,
)
self.assertEqual(
warmer.count(
"path: ${{ steps.llama_cache.outputs.cache-path }}",
),
2,
)
self.assertNotIn(
"path: ${{ env.LLAMA_STAGE_BUILD_DIR }}",
warmer,
)
self.assertEqual(
warmer.count("cache-ref: ${{ github.ref }}"),
2,

View file

@ -448,6 +448,7 @@ class CiArtifactActionTests(unittest.TestCase):
"prepare-windows-host-input",
"prepare-native-runtime-input",
"compose-product-input",
"save-and-verify-actions-cache",
"package-native-runtime",
"verify-native-runtime-package",
"verify-checksum-sidecar",
@ -507,6 +508,7 @@ class CiArtifactActionTests(unittest.TestCase):
expected_hash = (
"${{ hashFiles("
"'.github/actions/restore-windows-abi-cache/action.yml', "
"'.github/actions/save-and-verify-actions-cache/action.yml', "
"'.github/actions/resolve-native-toolchain-epoch/action.yml', "
"'.github/actions/prepare-native-runtime-input/action.yml', "
"'.github/actions/setup-windows-rocm-sdk/action.yml', "
@ -540,6 +542,10 @@ class CiArtifactActionTests(unittest.TestCase):
"value: ${{ steps.restore.outputs.cache-primary-key }}",
action,
)
self.assertIn(
"value: ${{ steps.identity.outputs.build-dir }}",
action,
)
def test_native_toolchain_epoch_is_exact_and_shared_with_build_stamp(
self,

View file

@ -487,6 +487,7 @@ class CiWorkflowArtifactTests(unittest.TestCase):
)
self.assertIn("mode: full", producer)
self.assertIn("artifact_name: ci-swift-sdk-input", producer)
self.assertIn("timeout_minutes: 180", producer)
self.assertNotIn("macos_runner:", producer)
self.assertNotIn("macos_cpu_artifact", producer)
self.assertNotIn("macos_unit_tests", producer)

View file

@ -205,6 +205,7 @@ class ReleaseWorkflowArtifactTests(unittest.TestCase):
)
self.assertIn("mode: full", producer)
self.assertIn("artifact_name: release-swift-sdk", producer)
self.assertIn("timeout_minutes: 180", producer)
self.assertNotIn("macos_runner:", producer)
self.assertIn(
"release_tag: ${{ needs.metadata.outputs.tag }}",
@ -269,6 +270,34 @@ class ReleaseWorkflowArtifactTests(unittest.TestCase):
self.assertIn("if: ${{ !cancelled()", publish)
self.assertNotIn("always()", publish)
def test_prereleases_never_dispatch_downstream_publication(self) -> None:
workflow = RELEASE_WORKFLOW.read_text(encoding="utf-8")
metadata = job_block(
workflow,
"metadata",
"build",
)
dispatch = job_block(
workflow,
"dispatch_packaging_release",
"publish_crates_preflight",
)
self.assertIn('if [[ "$version" == *-* ]]', metadata)
self.assertIn("prerelease=true", metadata)
self.assertIn(
"needs.metadata.outputs.prerelease != 'true'",
dispatch,
)
self.assertIn(
"needs.metadata.outputs.skip_gpu_bundles != 'true'",
dispatch,
)
self.assertIn("dry_run: false", dispatch)
self.assertIn("publish_images: true", dispatch)
self.assertIn("publish_release_assets: true", dispatch)
self.assertIn("publish_npm: true", dispatch)
def test_release_assets_and_manual_tags_are_immutable(self) -> None:
workflow = RELEASE_WORKFLOW.read_text(encoding="utf-8")
publish = job_block(