mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
bf651c3dc1
## Description Closes #3150. `ghcr.io/headroomlabs-ai/headroom:latest` resolved to the distroless `code-slim` build, whose `import onnxruntime` segfaults on arm64. The proxy imports onnxruntime at startup in cache mode, so the container never bound its port and `headroom deploy` crash-looped (exit 139) on Apple Silicon. @ricwo's report is exceptionally good — it isolates the base image with a copy-`site-packages`-onto-`debian:trixie-slim` experiment, and explicitly retracts an earlier wrong theory about the `cpuid_info` line. I verified the tagging half independently against the live registry: ``` latest sha256:6b34905489e3... <- identical 0.36.0-code-slim sha256:6b34905489e3... <- identical 0.36.0 sha256:bb8e77d01b54... ``` **Root cause, proven from the job log rather than inferred.** `docker/metadata-action` defaults to `latest=auto`, which appends a bare `latest` for any semver release — and its own log line reads `suffixLatest=false`, meaning the per-tag `suffix=` that keeps every other tag variant-scoped never reaches it. All eight variant cells therefore emitted `:latest`, and the last to finish won. From the 0.36.0 `code-slim` cell: ``` latest=auto suffixLatest=false tags: [..."ghcr.io/headroomlabs-ai/headroom:code-slim", "ghcr.io/headroomlabs-ai/headroom:latest"] pushing sha256:fbcbb68... to ghcr.io/headroomlabs-ai/headroom:latest ``` It landed on `code-slim` by scheduling luck. Any of the eight could have won on any release. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - **`flavor: latest=false`** on the `docker-manifest` metadata-action. Stops the tag being generated at all, leaving the root-cell promotion step as the single writer of `:latest`. - **A runtime guard** in `Create multi-arch manifest`: if a suffixed variant reaches the push carrying a bare `latest`, the job fails instead of publishing. `VARIANT_NAME` is passed via `env:` rather than spliced inline. - **A test that encodes the missing half of the contract.** `test_docker_latest_promotion_is_owned_by_root_manifest_cell` already existed and passed throughout — it asserted the *intended* writer was the root cell but never the *absence of unintended ones*. The new test asserts exclusivity: `latest=false` is set, no tag rule reintroduces `value=latest`, and the guard runs before anything is pushed. ## Testing - [x] Unit tests pass (`pytest`) ### Test Output ```text tests/test_release_workflows.py 48 passed, 1 skipped, 1 failed The failure is test_no_native_tls_in_wheel_build_tree: FileNotFoundError: [Errno 2] No such file or directory: 'cargo' Pre-existing and environmental — cargo is not installed on this machine; it fails identically on a clean main checkout. ruff check: All checks passed ruff format --check: 1 file already formatted YAML parses; flavor='latest=false', env keys ['IMAGE','DIGEST_DIR','VARIANT_NAME']. ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0), worktree off `main`. Live registry queried anonymously via the GHCR token endpoint. - Exact command / steps: (1) resolved `latest`, `0.36.0` and all four variant tags to manifest digests directly from `ghcr.io/v2/.../manifests/*` to confirm the aliasing; (2) pulled the `docker-manifest (code-slim)` job log from the 0.36.0 release run to see which tags that cell actually pushed; (3) applied the fix and ran the workflow test suite; (4) **removed `latest=false` again and re-ran the new test** to confirm it reproduces the bug. - Observed result: `:latest` and `:0.36.0-code-slim` share digest `sha256:6b34905489e3...` while `:0.36.0` is `sha256:bb8e77d01b54...`, exactly as reported. The code-slim job log shows `latest=auto` / `suffixLatest=false` and `pushing ... to ghcr.io/headroomlabs-ai/headroom:latest`. With the fix removed the new test fails on `assert 'latest=false' in ''`; with it restored, it passes. - Not tested: I could not exercise the arm64 segfault or a real multi-arch push from here — no ghcr write credential and no arm64 runner. The tagging fix is verified at the config layer plus the registry evidence above; the end-to-end proof is the re-run described below. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: n/a - Stable/default behavior changed: Yes, and that is the fix — `:latest` will track the plain Debian-based build instead of whichever variant cell happened to finish last. - Kill switch / disable path: n/a (CI tagging policy). - Unsafe override required: No. - Qualification impact: A variant cell that would publish a bare `latest` now fails the Docker job loudly rather than silently repointing the default tag. - Rollback path: Revert the commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes **The live `:latest` is still wrong until the images are re-tagged.** Merging this fixes future releases but does not touch the registry. Once merged, run `docker.yml` via `workflow_dispatch` with `version=0.36.0` to rebuild and repoint `:latest` at the plain build. I don't hold a `write:packages` credential, so that step needs a maintainer. **Not fixed here, and it outlives this PR:** the distroless arm64 segfault itself. After this change `:latest` points at the Debian build that works, but `0.36.0-slim` and `0.36.0-code-slim` remain broken on arm64 for anyone selecting them explicitly. @ricwo's evidence points squarely at the distroless base — same wheel, same numpy 2.5.2, same Python 3.13.5, works on `debian:trixie-slim` and segfaults on distroless. That deserves its own issue; the two failures are independent and this one is a release-tagging bug, exactly as the report says. Related but separate, from an earlier audit of this same file: the four bare variants set `RUNTIME_USER = "root"` in `docker-bake.hcl` while `Dockerfile:162` defaults to `nonroot`, and the `runtime-default` (nonroot) bake target is referenced by the docs but by no workflow. Worth its own change. Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> |
||
|---|---|---|
| .. | ||
| act | ||
| actions/headroom-e2e-setup | ||
| ISSUE_TEMPLATE | ||
| plugin | ||
| pr-images | ||
| scripts | ||
| workflows | ||
| CODEOWNERS | ||
| copilot-instructions.md | ||
| dependabot.yml | ||
| PULL_REQUEST_TEMPLATE.md | ||