From 73a4782917cf393bfb71e240cc66e6629ef31702 Mon Sep 17 00:00:00 2001 From: chopratejas Date: Sun, 3 May 2026 14:30:46 -0700 Subject: [PATCH] fix(ci): switch e2e runtime to python:3.11-slim (trixie, glibc 2.41) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR #360's previous attempt (multi-stage manylinux_2_28 build) still failed with the same `__isoc23_strtoll` undefined-symbol ImportError. Local repro showed the wheel built inside manylinux_2_28 has THREE glibc 2.38+ C23 symbol references (`__isoc23_strtol`, `__isoc23_strtoll`, `__isoc23_strtoull`) embedded by one of our transitive C/C++ deps during cc-rs compilation — most likely libstdc++'s `` resolving `std::strtoll` to the C23 variant when the manylinux toolchain has newer-glibc-aware headers. We can't easily fix the source of that emission downstream. Path of least resistance: switch the e2e runtime stage from a glibc-2.36 base to one with glibc 2.38+. Verified on Mac (linux/arm64 native): the same wheel that fails on `node:22-bookworm` (glibc 2.36) imports cleanly on `python:3.11-slim` (now trixie, glibc 2.41). ## Changes - e2e/init/Dockerfile: stage 2 base `node:22-trixie` → `python:3.11-slim`. The init harness only needs Python; no Node 22. Drops apt-get install of python3/python3-pip/python3-venv (already in the base image) and the `ln -sf` python alias. - e2e/wrap/Dockerfile: stage 2 base `node:22-bookworm` → `python:3.11-slim`. The wrap harness needs both Python 3.11 (aider-chat==0.86.2 requires Python <3.12) AND Node 22 (codex, openclaw). Trixie's default python3 is 3.13 — too new for aider — so we build on top of `python:3.11-slim` (trixie + py 3.11) and install Node 22 from NodeSource. - Both: stage 1 `--interpreter` reverted from python3.13 to python3.11 to match the runtime. ## Verification (local, linux/arm64) docker buildx build -f e2e/wrap/Dockerfile.aarch64-test \ --platform linux/arm64 -t headroom-wrap-test . → stage 1 manylinux build green → stage 2 `from headroom._core import DiffCompressor` → OK → stage 2 aider-chat install in progress (separate venv) ## Production-side note (out of scope for this PR) `pip install headroom-ai` from PyPI on a glibc-2.36 host (e.g. Debian 12, Ubuntu 22.04) will hit the same ImportError once the wheel matrix publishes. python:3.X-slim is now trixie (glibc 2.41) for ALL of 3.10/3.11/3.12/3.13, so users on those base images are unaffected. Tracking the underlying cc-rs symbol-emission bug as a separate issue. --- e2e/init/Dockerfile | 17 +++++++++-------- e2e/wrap/Dockerfile | 33 ++++++++++++++++++++++++--------- 2 files changed, 33 insertions(+), 17 deletions(-) diff --git a/e2e/init/Dockerfile b/e2e/init/Dockerfile index d51fa20dd..3dd825712 100644 --- a/e2e/init/Dockerfile +++ b/e2e/init/Dockerfile @@ -22,11 +22,16 @@ COPY crates/ crates/ COPY headroom/ headroom/ ENV PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 +# Build for Python 3.11 to match Stage 2's `python:3.11-slim` runtime. RUN /opt/python/cp311-cp311/bin/pip install 'maturin>=1.5,<2.0' && \ /opt/python/cp311-cp311/bin/maturin build --release --out /dist --interpreter python3.11 -# ─── Stage 2: node-based init e2e runtime ────────────────────────────────── -FROM node:22-bookworm +# ─── Stage 2: python runtime ─────────────────────────────────────────────── +# `python:3.11-slim` (now trixie, glibc 2.41) — see e2e/wrap/Dockerfile +# for why we need glibc ≥ 2.38 (the wheel's `_core.so` references C23 +# symbols that bookworm's 2.36 doesn't export). The init e2e harness +# only needs python (no node) — `headroom init -g ` is Python. +FROM python:3.11-slim ENV DEBIAN_FRONTEND=noninteractive \ PATH="/opt/headroom-venv/bin:${PATH}" \ @@ -38,11 +43,7 @@ ENV DEBIAN_FRONTEND=noninteractive \ RUN apt-get update && \ apt-get install -y --no-install-recommends \ ca-certificates \ - git \ - python3 \ - python3-pip \ - python3-venv && \ - ln -sf /usr/bin/python3 /usr/local/bin/python && \ + git && \ rm -rf /var/lib/apt/lists/* WORKDIR /workspace @@ -61,7 +62,7 @@ COPY e2e/_lib ./e2e/_lib COPY e2e/init ./e2e/init RUN python -m venv /opt/headroom-venv && \ - /opt/headroom-venv/bin/python -m pip install --upgrade "pip<25" && \ + /opt/headroom-venv/bin/python -m pip install --upgrade pip && \ /opt/headroom-venv/bin/python -m pip install "$(ls /tmp/wheels/headroom_ai-*.whl)[proxy]" && \ /opt/headroom-venv/bin/python -c "from headroom._core import DiffCompressor; print('headroom._core OK')" diff --git a/e2e/wrap/Dockerfile b/e2e/wrap/Dockerfile index e974f7010..1881ff2e6 100644 --- a/e2e/wrap/Dockerfile +++ b/e2e/wrap/Dockerfile @@ -33,14 +33,27 @@ COPY headroom/ headroom/ # Build the wheel with maturin. PYO3_USE_ABI3_FORWARD_COMPATIBILITY allows # building against Python 3.14+ until we bump pyo3 past 0.22. ENV PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 +# Build for Python 3.11 — aider-chat==0.86.2 requires Python <3.12, and +# trixie's default python3.13 is too new for aider. The manylinux image +# has python interpreters for every version at /opt/python/cpXY-cpXY/. +# Stage 2 uses `python:3.11-slim` (now trixie-based, glibc 2.41). RUN /opt/python/cp311-cp311/bin/pip install 'maturin>=1.5,<2.0' && \ /opt/python/cp311-cp311/bin/maturin build --release --out /dist --interpreter python3.11 -# ─── Stage 2: node-based e2e runtime ──────────────────────────────────────── -# Wraps Aider, Codex, OpenClaw on top of the prebuilt headroom-ai wheel. -# No rust/maturin needed at this stage — we just `pip install` the wheel -# from stage 1. -FROM node:22-bookworm +# ─── Stage 2: python+node runtime ─────────────────────────────────────────── +# Base on `python:3.11-slim` (now trixie, glibc 2.41) instead of +# `node:22-bookworm` (glibc 2.36): the wheel's `_core.so` references +# three glibc 2.38+ C23 wrappers (`__isoc23_strtol{,l,ul}`) that one of +# our transitive C/C++ deps emits during cc-rs compilation even when +# built inside manylinux_2_28 (probably libstdc++'s `` resolving +# `std::strtoll` to the C23 variant when the toolchain has newer +# headers). Bookworm's libc.so.6 doesn't export these wrappers, so +# `import headroom._core` fails at runtime. Trixie's glibc 2.41 does. +# +# We need Python 3.11 here (aider-chat==0.86.2 requires Python <3.12). +# Trixie's default `python3` is 3.13 — too new for aider. python:3.11-slim +# gives us Python 3.11 + trixie glibc + apt access for installing Node. +FROM python:3.11-slim ENV DEBIAN_FRONTEND=noninteractive \ AIDER_CHAT_VERSION=0.86.2 \ @@ -52,15 +65,17 @@ ENV DEBIAN_FRONTEND=noninteractive \ PYTHONUNBUFFERED=1 \ PYTHONDONTWRITEBYTECODE=1 +# Install Node.js 22 from NodeSource (matches what node:22-trixie gives, +# but layered onto python:3.11-slim so we get py3.11 + node22 + glibc 2.41 +# in one image). Also install git for any package that clones at install. RUN apt-get update && \ apt-get install -y --no-install-recommends \ ca-certificates \ curl \ git \ - python3 \ - python3-pip \ - python3-venv && \ - ln -sf /usr/bin/python3 /usr/local/bin/python && \ + gnupg && \ + curl -fsSL https://deb.nodesource.com/setup_22.x | bash - && \ + apt-get install -y --no-install-recommends nodejs && \ rm -rf /var/lib/apt/lists/* WORKDIR /workspace