From f1aa12cebfbd5ae4657133ec11efecf7ad6f433c Mon Sep 17 00:00:00 2001
From: chopratejas <chopratejas@gmail.com>
Date: Sat, 2 May 2026 18:55:09 -0700
Subject: [PATCH] fix(ci): reorder Dockerfile so wheel installs before
 headroom-ai
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

PR #350 CI: docker-native-e2e's wheel install succeeded but the
build-stage verify (`from headroom._core import hello`) failed with
`ModuleNotFoundError: No module named 'headroom._core'`. Same failure
mode the customer hit in production (Finding #2) — but in CI we have
the full layer trace.

Root cause: the headroom-core-py wheel claims ownership of both
`headroom/__init__.py` (stub from maturin's python-source layout)
AND `headroom/_core.cpython-*.so`. The previous Dockerfile installed
headroom-ai FIRST (which laid down the real `headroom/` tree), then
the wheel SECOND with `--force-reinstall`. pip's --force-reinstall
uninstalls the wheel's previously installed files before reinstalling
— but the wheel's stub `__init__.py` had already overwritten
headroom-ai's at first install. Net result: pip deleted
`headroom/__init__.py` and `headroom/_core.so` ownership records
got into a state where the .so wasn't present after the install.

Fix: swap the order. Install the wheel first (lays down stub
`__init__.py` + `_core.so`), then install headroom-ai (overwrites the
stub with the real `__init__.py` and adds the rest of the
`headroom/` tree). `_core.so` survives because headroom-ai doesn't
claim ownership of it. Drop `--force-reinstall` from the wheel step
since nothing is installing the wheel before it.

This is the exact failure A0 was designed to catch — a deployment
that ships without `_core` working. CI is now serving as a
regression gate for the production install path.

The remaining 3 PR check failures (validate × 3 / Dev Containers)
are environmental: the runner's PyPI mirror (`pypi.netflix.net`)
times out fetching `cuda-bindings==12.9.4` /
`nvidia-cuda-cupti-cu12==12.8.90` / `safetensors==0.7.0`. These come
from `headroom-ai[dev]` → `sentence-transformers` → `torch` → CUDA
deps. Not caused by the realignment branch; the post-create script
needs a `--extra dev-light` profile or the mirror needs the packages
cached. Tracking separately.
---
 Dockerfile | 39 +++++++++++++++++++++++++++++++--------
 1 file changed, 31 insertions(+), 8 deletions(-)

diff --git a/Dockerfile b/Dockerfile
index 553de90f0..3bccc3e4f 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -55,13 +55,28 @@ ARG HEADROOM_EXTRAS=proxy,code
 RUN --mount=type=cache,target=/root/.cache/uv \
     uv pip install --system ".[${HEADROOM_EXTRAS}]"
 
-# Layer 2: copy real source, reinstall only headroom-ai (no deps)
-COPY headroom/ headroom/
-RUN --mount=type=cache,target=/root/.cache/uv \
-    uv pip install --system --no-deps --reinstall-package headroom-ai .
-
-# Layer 3 (Hotfix-A0): build and install the Rust extension wheel. uv
-# already installed `maturin` as a transitive of the [proxy]/[code]
+# Layer 2 (Hotfix-A0): build and install the Rust extension wheel
+# BEFORE installing headroom-ai source. Why this order:
+#
+#   * The headroom-core-py wheel includes a stub `headroom/__init__.py`
+#     plus `headroom/_core.cpython-*.so` (maturin's `python-source`
+#     layout — see `crates/headroom-py/pyproject.toml`).
+#   * The headroom-ai install also writes files under `headroom/`.
+#   * If headroom-ai is installed FIRST and the wheel goes second with
+#     `--force-reinstall`, pip uninstalls the wheel's previously
+#     installed files, deleting `headroom/__init__.py` (which the wheel
+#     also claims). headroom-ai's __init__.py was already overwritten
+#     by the wheel's empty stub at install-time, so the deletion leaves
+#     no `__init__.py` at all — `from headroom._core import hello`
+#     then fails with `ModuleNotFoundError: No module named
+#     'headroom._core'`. Observed in PR #350 CI before this reorder.
+#   * Installing the wheel FIRST means: wheel lays down stub
+#     `__init__.py` + `_core.so`. Then headroom-ai install OVERWRITES
+#     `__init__.py` with the real one and adds the rest of the
+#     `headroom/` tree. `_core.so` survives because headroom-ai
+#     doesn't claim ownership of it.
+#
+# uv already installed `maturin` as a transitive of the [proxy]/[code]
 # extras; if it didn't, install it explicitly here so the build never
 # silently skips.
 COPY crates/ crates/
@@ -71,7 +86,15 @@ RUN --mount=type=cache,target=/root/.cache/uv \
     --mount=type=cache,target=/build/target \
     uv pip install --system maturin \
     && maturin build --release -m crates/headroom-py/Cargo.toml --out /build/wheels \
-    && uv pip install --system --force-reinstall --no-deps /build/wheels/headroom_core_py-*.whl
+    && uv pip install --system --no-deps /build/wheels/headroom_core_py-*.whl
+
+# Layer 3: copy real source, install headroom-ai (no deps). This
+# overwrites the wheel's stub `headroom/__init__.py` with the real one
+# and adds the full `headroom/` tree alongside the surviving
+# `_core.so` from Layer 2.
+COPY headroom/ headroom/
+RUN --mount=type=cache,target=/root/.cache/uv \
+    uv pip install --system --no-deps --reinstall-package headroom-ai .
 
 # Layer 4 (Hotfix-A0): verify the extension actually loads end-to-end
 # inside the build image. If this fails, the runtime image would fail