## Description
Fixes two release-automation defects exposed by the 0.35.0 release:
1. Release Please grouped the single package into a PR titled `chore:
release main`, which could not be matched back to the `headroom-ai`
component/version and therefore never emitted the release event that
starts PyPI publishing.
2. Docker manifest jobs downloaded digest artifacts with overlapping
variant globs. For example, `digests-code-*` also selected code-nonroot,
code-slim, and code-slim-nonroot artifacts, yielding eight markers where
exactly two were required.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Route the single root package through Release Please's normal
versioned-PR path.
- Preserve both `${component}` and `${version}` in generated release PR
titles.
- Download Docker amd64 and arm64 digest artifacts by exact name instead
of an overlapping variant glob.
- Add regression assertions for both Release Please title matching and
Docker artifact isolation.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
python -m pytest tests/test_release_workflows.py -q
44 passed in 2.29s
pre-commit hooks:
Sync plugin versions........................Passed
Verify Ruff version alignment...............Passed
check for merge conflicts...................Passed
ruff (legacy alias).........................Passed
ruff format.................................Passed
mypy........................................Passed
```
## Real Behavior Proof
- Environment: GitHub Actions release runs 31653073583 and 31664975515,
`main` at the 0.35.0 release merge (`93f2d7a2`).
- Exact command / steps: Inspected the Release Please rerun and each
failed Docker manifest job; enumerated the digest artifacts downloaded
by their configured patterns.
- Observed result: Release Please logged `There are untagged, merged
release PRs outstanding - aborting`. Docker's `slim`, `code`, and
`code-slim` manifest cells found 4, 8, and 4 digest markers respectively
instead of 2 because their prefix globs included related variants. The
new Docker workflow requests `digests-<variant>-amd64` and
`digests-<variant>-arm64` by exact name.
- Not tested: A synthetic release was not published because registry
versions/tags are irreversible. Both configuration invariants are
covered by regression tests, and GitHub's PR workflow validation runs
against this branch.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A — workflow configuration and regression-test changes only.
## Additional Notes
- Documentation is N/A: these are internal release workflow corrections
with no user-facing command or API changes.
- The 0.35.0 Python release recovery is proceeding separately through
the existing wheel smoke-test and PyPI publication gates.
- The already-started 0.35.0 Docker run used the old workflow from tag
commit `93f2d7a2`; this PR prevents the artifact collision in subsequent
Docker runs.
## Description
A successful root Docker image can miss `:latest` when any optional
variant manifest fails. The release workflow currently gates the
standalone `promote-latest` job on the aggregate `docker-manifest`
matrix, so one sibling failure skips promotion even when the signed root
amd64+arm64 manifest exists.
This moves `:latest` promotion into the successful root manifest cell.
Optional variant failures remain visible and continue to fail their
jobs, but they no longer suppress the image used by `headroom install`,
which defaults to `ghcr.io/headroomlabs-ai/headroom:latest`.
Refs #1583
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Publish `:latest` from the root `docker-manifest` matrix cell after
its versioned multi-architecture manifest is created and signed.
- Remove the aggregate `promote-latest` dependency that allowed
unrelated variant failures to suppress publication.
- Keep all existing root, slim, code, and nonroot variants.
- Preserve native linux/amd64 and linux/arm64 manifest assembly.
- Add a focused workflow-contract regression test.
## Testing
- [x] Unit tests pass (`uv run pytest tests/test_release_workflows.py -q
-k "docker or latest"`)
- [x] Linting passes (`uv run ruff check
tests/test_release_workflows.py`)
- [x] Formatting passes (`uv run ruff format --check
tests/test_release_workflows.py`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
Focused checks pass: `5 passed, 34 deselected` for `uv run pytest tests/test_release_workflows.py -q -k "docker or latest"`; the full test file has one unrelated Windows `FileNotFoundError` in `test_no_native_tls_in_wheel_build_tree` because its external command is unavailable. `uv run ruff check tests/test_release_workflows.py` and `uv run ruff format tests/test_release_workflows.py --check` pass. The repository-wide format check reports eight pre-existing files outside this target. Proof report: `D:\Repos\.claude\pr-sweep\headroom-PR-TARGET-1583-PROOF.md`.
```
## Real Behavior Proof
- Environment: Windows, Python managed by `uv`, repository
workflow-contract tests; production publication owned by GitHub Actions
and GHCR.
- Exact command / steps: run the focused release-workflow tests; after
merge, inspect the next Docker release run and execute `docker buildx
imagetools inspect ghcr.io/headroomlabs-ai/headroom:latest` without
registry login.
- Observed result: local workflow-contract proof passes for root-owned
promotion and both native architecture inputs; live GHCR publication
remains unverified until the next release.
- Not tested: production GHCR publication before merge.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [ ] My code follows the project's style guidelines
- [ ] I have performed a self-review of my code
- [x] Workflow comments explain the non-obvious root-only promotion
boundary
- [x] Documentation outside the changelog is unchanged because the CLI
image reference is already correct
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing focused tests pass locally with my changes
## Screenshots (if applicable)
Not applicable.
## Additional Notes
Release run
https://github.com/headroomlabs-ai/headroom/actions/runs/28404020512
demonstrated the cascade: the root manifest succeeded, a nonroot
manifest failed during Buildx setup, and `promote-latest` was skipped.
PR CI can prove the workflow dependency and architecture-preservation
contracts. GHCR availability and anonymous package visibility require
the next production release plus an unauthenticated registry inspection.
## Description
`docker.yml` builds a full multi-arch image on **every push to `main`**
with **no concurrency group**, so a merge spree stacks one build per
commit. Against the Free-plan **20 concurrent-job cap**, those long
builds hog the runner pool and starve the `CI` (`test`/`lint`/`build`)
jobs that PR merges actually depend on. This adds concurrency-cancel so
only the latest build per ref runs — `cancel-in-progress` scoped to
`main` so a release tag's publish (its own ref) is never killed. Same
fix for the per-PR **Merge Conflicts** check.
## Type of Change
- [x] Repo infrastructure / CI (no functional change)
## Changes Made
- `.github/workflows/docker.yml`: `concurrency: docker-${{ github.ref
}}`, cancel-in-progress on `main` only.
- `.github/workflows/merge-conflicts.yml`: per-PR concurrency-cancel.
## Testing
- [x] YAML validated (`yaml.safe_load` on both) — release/tag builds
unaffected (separate ref group).
## Real Behavior Proof
- Before: 24 merges → 24 stacked Docker builds queued against 20 slots.
- After: only the latest `main` Docker build runs; older superseded ones
auto-cancel; release publishes untouched.
## Summary
- Adds `push: branches: [main]` trigger to `docker.yml` so every merge
to main builds and tags all image variants.
- Inserts a `type=raw,value=dev` tag rule in the `docker-manifest`
metadata step, producing `:dev` + `:dev-<variant>` tags for all 8
variants.
- Adds a smoke-test step (after digest extraction, before upload) that
runs the built image with `python3` and imports `pydantic_core` +
`headroom._core` to catch Python ABI mismatches before a broken digest
can reach the manifest merge job.
## Tags produced on every `main` push
| Variant | Tag |
|---|---|
| root | `:dev` |
| nonroot | `:dev-nonroot` |
| code | `:dev-code` |
| code-nonroot | `:dev-code-nonroot` |
| slim | `:dev-slim` |
| slim-nonroot | `:dev-slim-nonroot` |
| code-slim | `:dev-code-slim` |
| code-slim-nonroot | `:dev-code-slim-nonroot` |
## Guard logic
```
enable=${{ inputs.enable_ref_tags != 'false' && github.event_name == 'push' }}
```
- **Push to main** → `'' != 'false'` = true AND `push == push` = true →
`:dev` fires
- **Release** (`workflow_call` with `enable_ref_tags: false`) → `'false'
!= 'false'` = false → skips
- **PR dry-run** (same `workflow_call` path) → skips
`promote-latest` runs but its re-tag step self-skips (no version set on
push events) — no `:latest` churn.
## Test plan
- [ ] Merge to main; confirm all 8 `:dev-*` tags appear in GHCR
- [ ] Trigger a release; confirm `:dev-*` tags are NOT overwritten or
re-emitted
- [ ] Confirm `actionlint` passes: `actionlint
.github/workflows/docker.yml`
Closes#530
## Problem
The release pipeline's `docker-manifest` jobs fail at action resolution:
```
Unable to resolve action `sigstore/cosign-installer@v4`, unable to find version `v4`
```
`sigstore/cosign-installer` has no `v4`; its current major is `v3`. This
broke the multi-arch manifest assembly and `promote-latest` on the
v0.24.0 release run (and would break every release). Per-arch image
builds and **PyPI/npm/GitHub-Packages publishing were unaffected**.
## Fix
`.github/workflows/docker.yml`: `sigstore/cosign-installer@v4` → `@v3`.
## Verification
Resolves the only failing jobs in release run
[27184823371](https://github.com/chopratejas/headroom/actions/runs/27184823371).
After merge, the docker-manifest + promote-latest steps will resolve the
action and run.
* ci: speed up GitHub Actions - path filters, caching, timeouts, version upgrades
Performance improvements:
- init-e2e.yml, wrap-e2e.yml: add path filters so e2e Docker builds only run when
e2e-related files change (saves ~10 min per irrelevant PR push)
- init-e2e.yml, wrap-e2e.yml: add concurrency groups to cancel superseded PR runs
- ci.yml: add pip caching to lint and build jobs
- ci.yml: cache actionlint + act binaries in workflow-validation (skip curl on hits)
- eval.yml: add pip caching to smoke-test and weekly-suite jobs
- docs.yml: add pip caching for mkdocs-material install
- rust.yml: replace cargo install --locked cargo-audit/deny with taiki-e/install-action
(prebuilt binaries; saves 2-5 min per audit run)
Bug fixes:
- docker.yml: fix actions/checkout@v6 -> @v4 (v6 does not exist; would break all
Docker builds on every release/PR touching docker paths)
Version upgrades:
- wagoid/commitlint-github-action: @v5 -> @v6
- devcontainers.yml: docker/setup-buildx-action@v3 -> @v4 (align with docker.yml)
Safety improvements:
- ci.yml: add timeout-minutes to all 13 jobs (changes, lint, build-wheel,
prefetch-model, test x4, test-extras, test-agno, commitlint, build,
workflow-validation, docker-native-e2e, windows-native-wrapper, macos-native-wrapper)
- docker.yml: add timeout-minutes to docker-build (75m), docker-manifest (20m),
promote-latest (10m)
- eval.yml: add timeout-minutes to smoke-test (30m); bump weekly-suite 60->90m
- rust.yml: add timeout-minutes to test (30m), wheels (45m), audit (20m)
Observed wall-clock impact on recent PRs:
- Init E2E and Wrap E2E were running on every single PR push regardless of content
- CI workflow was taking 12-17 min; path filters reduce unnecessary e2e runs to 0
* fix(ci): bust actionlint+act cache when workflow file changes
Static cache key 'ci-tools-actionlint-act-v1' never invalidated on
tool version updates. Switched to hashFiles('.github/workflows/ci.yml')
so the cache busts automatically whenever the download scripts are
updated to point at a newer release.
Flagged by adversarial review (Architecture + Testing/Reliability personas).
* fix(ci): add missing Dockerfile COPY paths to e2e path filters
e2e/init/Dockerfile and e2e/wrap/Dockerfile COPY files not covered
by the initial path filter set:
init-e2e: Cargo.toml, Cargo.lock, rust-toolchain.toml, uv.lock,
.claude-plugin, .github/plugin/**, plugins/headroom-agent-hooks/**
wrap-e2e: Cargo.toml, Cargo.lock, rust-toolchain.toml, uv.lock,
sdk/typescript/**, plugins/openclaw/**
Without these, a Rust toolchain bump or SDK change on a PR would
skip the e2e gate entirely, only catching it on the merge to main.
Flagged by adversarial review (Domain/Correctness persona).
* fix(devcontainer): upgrade uv to >=0.7.0 to parse uv.lock revision=3
* fix(devcontainer): set UV_SKIP_WHEEL_FILENAME_CHECK=1 in post-create.sh for gitpython wheel
* ci: bump actions/checkout and actions/setup-node to v5 (Node.js 20 EOL Jun 16)
* fix(devcontainer): export UV_SKIP_WHEEL_FILENAME_CHECK so uv run also skips wheel check
* ci: bump all GitHub Actions to latest versions (Node.js 24)
* fix(test): accept release-please-action v4 or v5 in workflow assertion
* fix(format): ruff format test_release_workflows.py
PR #376's per-arch fan-out correctly removed `bake-file-tags` from
the docker-build step (tags belong on the multi-arch manifest, not
on per-arch images). But that left bake without ANY reference for
the push target — no tags AND no explicit `name=` in the output
spec. Every release-time docker-build job failed with the
misleading message:
ERROR: tag is needed when pushing to registry
Buildx's actual constraint is "no tags AND no `name=` in the
output = no push target." Since push-by-digest discards tags
anyway, the fix is to specify `name=<registry>/<image>` directly
in the `*.output` spec. Tags are still applied later, only on the
multi-arch manifest by `docker-manifest`.
Regression test in tests/test_release_workflows.py:
`test_docker_per_arch_build_specifies_image_name_in_output`
pins the `name=` substring so a future "the labels block already
has the registry, surely buildx can figure it out" refactor will
fail at PR time rather than 2 minutes into release.
GitHub-hosted Linux arm64 runners (`ubuntu-24.04-arm`) went GA in Aug
2025 and are free for public repositories. Switching the aarch64
wheel + the multi-arch docker matrix off `ubuntu-latest`+QEMU onto
the native runner cuts wall-clock on both surfaces.
release.yml — build-wheels matrix
* `aarch64-unknown-linux-gnu`: `ubuntu-latest` → `ubuntu-24.04-arm`.
maturin-action still runs inside `quay.io/pypa/manylinux_2_28_aarch64`,
but the container now executes natively on an aarch64 kernel
instead of through QEMU emulation. Aarch64 wheel build drops
from ~50–60 min to ~10 min.
* `x86_64-unknown-linux-gnu`: `ubuntu-latest` → `ubuntu-24.04`
(pin the moving alias for reproducibility; no semantic change).
docker.yml — fan-out + manifest merge
* Pre-#377: one `docker-variant-tags` matrix job per variant on
`ubuntu-latest`, using bake's `platforms = [amd64, arm64]` with
QEMU for the arm64 leg. ~1h per variant, 8 variants.
* Post-#377: split into `docker-build` (variant × arch = 16
parallel jobs, each on its native runner, single-platform
push-by-digest) and `docker-manifest` (per variant, merges the
two arch digests into a multi-arch tagged manifest with
`docker buildx imagetools create`, signs the index manifest
with cosign). Wall-clock drops from ~1h per variant to ~10 min.
* `docker/setup-qemu-action` removed — there's no QEMU left.
* Per-(variant, arch) GHA cache scopes so the two arches don't
collide on cache keys.
* `promote-latest` rewired to depend on `docker-manifest`.
Behavior change: cosign now signs only the multi-arch index digest
per variant, not each per-platform image. `cosign verify <repo>:tag`
(the typical flow) is unchanged because cosign resolves the tag to
the index digest. Verifiers pinning a specific per-arch digest will
need to verify the index digest instead.
Regression tests in tests/test_release_workflows.py:
* `test_aarch64_wheel_uses_native_arm64_runner` — pins the
aarch64 row to `ubuntu-24.04-arm` (and the amd64 row to
`ubuntu-24.04`, not `-latest`), so a future "let me unify on
ubuntu-latest" refactor surfaces the QEMU regression at PR time.
* `test_docker_workflow_builds_on_native_arch_runners` — pins the
fan-out matrix's arch entries, asserts push-by-digest, asserts
`setup-qemu-action` is absent from non-comment lines, asserts
the manifest-merge job exists.
Verified:
* Both workflow files parse as valid YAML with the expected job
graph (`docker-build` → `docker-manifest` → `promote-latest`,
16 fan-out jobs, 8 manifest jobs).
* `docker buildx imagetools inspect <tag> --format '{{ json . }}'`
exposes the index digest at `.manifest.digest` (confirmed via
Docker's official reference).
* `ubuntu-24.04-arm` is the correct GitHub-hosted runner label
(GA 2025-08-07, free for public repos).
* `make ci-precheck-rust` and `make ci-precheck-python` both pass
locally; `tests/test_release_workflows.py` is 15/15 green
(13 existing + 2 new).
The cosign signing step passed bake metadata via env var:
env:
BAKE_META: ${{ steps.bake.outputs.metadata }}
run: echo "$BAKE_META" | jq ...
For large bake targets (code-nonroot, runtime-code-nonroot) the
metadata JSON is large enough that combined argv+env at bash spawn
exceeds Linux ARG_MAX (~128 KiB on ubuntu-latest), so bash dies with
E2BIG before the script even runs.
Switch to writing metadata into a heredoc-backed temp file, then read
it via jq file input. Heredocs put the JSON in the script body itself,
which bash reads from a temp file (no ARG_MAX limit), bypassing the
env-size ceiling entirely.
Module: .github/workflows/docker.yml
Adds 0.10.7-ab46594 (root) and 0.10.7-<variant>-<sha> (variants) so
images can be referenced by an exact version+commit pair without
relying on the moving variant or :latest tags.
When the docker workflow is triggered directly by release.published
(rather than via workflow_call from the Release parent), inputs.enable_ref_tags
is null and produced an empty enable= attribute that the metadata-action
rejected. Default to true on non-release triggers and skip ref/pr tags
on release events where they don't apply anyway.
- Replace full-sha image tags with type=sha,format=short (7-char) so the
primary package versions list stops accumulating long sha-only entries.
- Route cosign signatures into a sibling GHCR package via
COSIGN_REPOSITORY=<image>-signatures, so the main image's package
version list stays clean. GHCR does not yet implement the OCI 1.1
Distribution Referrers API (community discussion #163029, June 2025),
so legacy signature mode is used here -- OCI 1.1 mode would force the
signature manifest's subject into the same repo as the image and
override COSIGN_REPOSITORY. Verifiers must export the same
COSIGN_REPOSITORY value when running 'cosign verify'.
- Add a promote-latest job that runs after the variant matrix and
re-pushes the :latest tag pointing at the root image with a unique
index annotation. This forces a fresh manifest digest, generating a
new GHCR package version with current timestamp so :latest sits at
the top of the version listing instead of whichever variant happened
to finish last.
Call the Docker workflow from the release pipeline so Docker publishes in
the same run, build npm tarballs alongside Python distributions, and
attach those artifacts to the GitHub release page.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Derive the exact Docker image version from the release tag or manual
workflow input, sync versioned files in the build workspace before the
image build, and publish an explicit matching image tag.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
