mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
fix(ci): rustls-everywhere — eliminate openssl-sys from build tree
# Root cause of the wheel-build cascade We have shipped 5 release-pipeline hot-fixes in 12 hours, each addressing a different symptom of the same architectural problem: 1. PR #363 — npm artifact downloads + tried `yum openssl-devel` 2. PR #367 — vendored OpenSSL in `headroom-proxy` + dropped Intel mac 3. PR #369 — Debian-cross perl install (`perl` not `libipc-cmd-perl`) 4. PR #370 — moved `openssl/vendored` from headroom-proxy to headroom-py 5. (this PR) — ELIMINATE OpenSSL entirely Each fix exposed a different missing system package or feature flag in a different build surface (manylinux x86_64 vs aarch64-cross-Debian vs macOS Intel vs e2e/wrap Dockerfile vs e2e/init Dockerfile vs main Dockerfile vs devcontainer). We were playing whack-a-mole because every Cargo dep change to the OpenSSL surface required matching system-package updates in 6+ different Dockerfiles and workflows, and the PR-level CI didn't exercise all of them. # Why this PR is the structural fix `fastembed` exposes clean rustls feature flags: - `hf-hub-rustls-tls` (replaces default `hf-hub-native-tls`) - `ort-download-binaries-rustls-tls` (replaces default `…native-tls`) By disabling fastembed's default features and enabling the rustls variants explicitly, we remove `native-tls` (and therefore `openssl-sys`, `openssl`, `openssl-src`, perl modules, OpenSSL build-time deps, vendored OpenSSL ~30s build cost) from the entire workspace dep tree. Verified locally: $ cargo tree -p headroom-py -i openssl-sys error: package ID specification `openssl-sys` did not match any packages $ cargo tree -p headroom-py -i native-tls error: package ID specification `native-tls` did not match any packages $ cargo build --release -p headroom-py Finished `release` profile [optimized] target(s) in 25.57s (Down from 1m+ with vendored OpenSSL.) # Cleanups enabled by this change - crates/headroom-py/Cargo.toml — dropped the `openssl/vendored` workaround from PR #370. - crates/headroom-proxy/Cargo.toml — same dep removed. - e2e/wrap/Dockerfile — dropped `yum install openssl-devel pkgconfig perl-IPC-Cmd`. Comment retained explaining why. - e2e/init/Dockerfile — same. - Dockerfile (main) — dropped `pkg-config libssl-dev` from apt-get. - .devcontainer/Dockerfile — dropped `pkg-config libssl-dev`. - .github/workflows/release.yml — removed the entire before-script-linux block (perl install probe + multi-package-manager dispatch + fail-loud assertion). No longer needed. # Regression gate Three new structural tests in tests/test_release_workflows.py: - test_no_openssl_sys_in_wheel_build_tree — runs `cargo tree -p <crate> -i openssl-sys` for headroom-py / headroom-proxy / headroom-core. If openssl-sys reappears (a future native-tls enabler creeping in via a new dep), this fails AT PR TIME with an actionable message. - test_no_native_tls_in_wheel_build_tree — same shape, native-tls is the proximate cause. - test_fastembed_uses_rustls_features — checks the Cargo.toml so a future "let me bump fastembed and forget the features" doesn't silently re-introduce OpenSSL. Plus two cleanup gates: - test_dockerfiles_no_longer_install_openssl_devel - test_release_yml_does_not_install_openssl_or_perl_for_wheels All 13 release-workflow tests pass. `make ci-precheck` PASSED. # What this teaches us about rollouts (per user's ultrathink ask) The 5-fix cascade exposed three meta-problems: 1. PR checks don't block merges. PR #370 had docker-init-e2e, docker-wrap-e2e, docker-native-e2e all FAILED yet got merged. Branch protection should require these checks. Operator action needed (cannot fix in code). 2. Local validation is misleading. `cargo build -p headroom-py` from the workspace root used the workspace lockfile and looked green; CI did fresh resolution against headroom-py's manifest alone where the feature wasn't enabled. Lesson: verify structural invariants with `cargo tree -e features` before trusting that a build "works." 3. 6+ build surfaces with independent system-dep state. Every Cargo change required matching updates in 6 places. The structural answer (this PR) is to NOT depend on system OpenSSL at all. Where structural fixes are not possible, the answer is a single shared scripts/install-rust-build-deps.sh — but with this PR there's nothing left to install.
This commit is contained in:
chopratejas
parent
a5c7f6fed9
commit
6f2c0a8400
10 changed files with 206 additions and 388 deletions
|
|
@ -3,8 +3,9 @@ FROM mcr.microsoft.com/devcontainers/python:1-${VARIANT}
|
|||
|
||||
# Single-wheel architecture (post-#355): `uv sync` builds `headroom-ai`
|
||||
# from the local pyproject.toml using maturin (declared in build-system).
|
||||
# Maturin needs rust + cargo + pkg-config + libssl-dev (transitive
|
||||
# openssl-sys via fastembed/hf-hub).
|
||||
# Maturin needs rust + cargo. The rustls-everywhere refactor (PR #371)
|
||||
# eliminated `openssl-sys` from our build tree, so this image no longer
|
||||
# needs `pkg-config` or `libssl-dev`.
|
||||
#
|
||||
# Rust toolchain is provisioned via the official devcontainer feature
|
||||
# (`ghcr.io/devcontainers/features/rust:1`) declared in devcontainer.json
|
||||
|
|
@ -18,11 +19,6 @@ FROM mcr.microsoft.com/devcontainers/python:1-${VARIANT}
|
|||
# base image's yarnpkg.com source has an expired GPG key that aborts
|
||||
# the whole RUN with "NO_PUBKEY 62D54FD4003F6525". The maturin build
|
||||
# doesn't need yarn.
|
||||
RUN rm -f /etc/apt/sources.list.d/yarn.list && \
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
pkg-config \
|
||||
libssl-dev && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
RUN rm -f /etc/apt/sources.list.d/yarn.list
|
||||
|
||||
RUN python -m pip install --no-cache-dir uv==0.6.17 'maturin>=1.5,<2.0'
|
||||
|
|
|
|||
50
.github/workflows/release.yml
vendored
50
.github/workflows/release.yml
vendored
|
|
@ -226,48 +226,14 @@ jobs:
|
|||
target: ${{ matrix.target }}
|
||||
args: --release --out dist --interpreter python3.10 python3.11 python3.12 python3.13
|
||||
manylinux: ${{ matrix.manylinux }}
|
||||
# OpenSSL is vendored at the cargo level via headroom-proxy's
|
||||
# `openssl = { features = ["vendored"] }` dep — that compiles
|
||||
# OpenSSL from source as part of the cargo build, so we no
|
||||
# longer need to install system OpenSSL in the manylinux
|
||||
# container or via Homebrew on macOS. We DO need `IPC::Cmd`
|
||||
# at build time because OpenSSL's vendored Configure script
|
||||
# imports it (without it the build fails with "Can't locate
|
||||
# IPC/Cmd.pm").
|
||||
#
|
||||
# `IPC::Cmd` has been a Perl core module since 5.10, so any
|
||||
# working `perl` install provides it. The aarch64-cross
|
||||
# container the maturin-action uses is Debian/Ubuntu-based
|
||||
# (apt), the x86_64 native container is AlmaLinux 8 (yum/dnf).
|
||||
# Probe the IPC::Cmd module directly first; only install when
|
||||
# missing, using whichever package manager exists. The legacy
|
||||
# `libipc-cmd-perl` alias on Debian is no longer in default
|
||||
# sources — installing plain `perl` pulls IPC::Cmd via
|
||||
# perl-modules-* on every Debian/Ubuntu version we'll see.
|
||||
before-script-linux: |
|
||||
set -euo pipefail
|
||||
if perl -MIPC::Cmd -e 1 >/dev/null 2>&1; then
|
||||
echo "IPC::Cmd already available"
|
||||
exit 0
|
||||
fi
|
||||
if command -v dnf >/dev/null 2>&1; then
|
||||
dnf install -y perl-IPC-Cmd
|
||||
elif command -v yum >/dev/null 2>&1; then
|
||||
yum install -y perl-IPC-Cmd
|
||||
elif command -v apt-get >/dev/null 2>&1; then
|
||||
apt-get update -qq
|
||||
# `perl` is the meta-package that pulls perl-modules-* containing IPC::Cmd.
|
||||
apt-get install -y --no-install-recommends perl
|
||||
elif command -v apk >/dev/null 2>&1; then
|
||||
apk add --no-cache perl-utils
|
||||
else
|
||||
echo "::error::No supported package manager (dnf/yum/apt-get/apk) in container" >&2
|
||||
exit 1
|
||||
fi
|
||||
# Final assertion — fail the build now (not 5 minutes later
|
||||
# in the openssl-src compile step) if IPC::Cmd is still not
|
||||
# importable. No silent fallback per project guideline.
|
||||
perl -MIPC::Cmd -e 'print "IPC::Cmd loaded OK\n"'
|
||||
# No before-script-linux needed: the rustls-everywhere refactor
|
||||
# (PR #371) switched fastembed to `hf-hub-rustls-tls` +
|
||||
# `ort-download-binaries-rustls-tls` features, which removed
|
||||
# `openssl-sys` from our build tree entirely. The wheel build
|
||||
# no longer needs system OpenSSL or any perl modules.
|
||||
# `cargo tree -p headroom-py -i openssl-sys` returns "not
|
||||
# found" — that's the regression gate (see test_release_workflows
|
||||
# ::test_no_openssl_sys_in_wheel_build_tree).
|
||||
env:
|
||||
# PyO3 0.22 supports up to Python 3.13; allow forward-compat
|
||||
# builds for 3.14+ until we bump PyO3 (tracked separately).
|
||||
|
|
|
|||
215
Cargo.lock
generated
215
Cargo.lock
generated
|
|
@ -647,12 +647,6 @@ dependencies = [
|
|||
"vsimd",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "base64ct"
|
||||
version = "1.8.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06"
|
||||
|
||||
[[package]]
|
||||
name = "bit-set"
|
||||
version = "0.8.0"
|
||||
|
|
@ -1008,16 +1002,6 @@ dependencies = [
|
|||
"url",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "core-foundation"
|
||||
version = "0.9.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "91e195e091a93c46f7102ec7818a2aa394e1e1771c3ab4825963fa03e45afb8f"
|
||||
dependencies = [
|
||||
"core-foundation-sys",
|
||||
"libc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "core-foundation"
|
||||
version = "0.10.1"
|
||||
|
|
@ -1238,16 +1222,6 @@ version = "0.1.4"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "092966b41edc516079bdf31ec78a2e0588d1d0c08f78b91d8307215928642b2b"
|
||||
|
||||
[[package]]
|
||||
name = "der"
|
||||
version = "0.8.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "71fd89660b2dc699704064e59e9dba0147b903e85319429e131620d022be411b"
|
||||
dependencies = [
|
||||
"pem-rfc7468",
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "deranged"
|
||||
version = "0.5.8"
|
||||
|
|
@ -1547,21 +1521,6 @@ version = "0.2.0"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb"
|
||||
|
||||
[[package]]
|
||||
name = "foreign-types"
|
||||
version = "0.3.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f6f339eb8adc052cd2ca78910fda869aefa38d22d5cb648e6485e4d3fc06f3b1"
|
||||
dependencies = [
|
||||
"foreign-types-shared",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "foreign-types-shared"
|
||||
version = "0.1.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "00b0228411908ca8685dba7fc2cdd70ec9990a6e753e89b6ac91a84c40fbaf4b"
|
||||
|
||||
[[package]]
|
||||
name = "form_urlencoded"
|
||||
version = "1.2.2"
|
||||
|
|
@ -1866,7 +1825,6 @@ dependencies = [
|
|||
"humantime",
|
||||
"hyper",
|
||||
"hyper-util",
|
||||
"openssl",
|
||||
"pin-project-lite",
|
||||
"proptest",
|
||||
"reqwest",
|
||||
|
|
@ -1892,7 +1850,6 @@ name = "headroom-py"
|
|||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"headroom-core",
|
||||
"openssl",
|
||||
"pyo3",
|
||||
"serde_json",
|
||||
]
|
||||
|
|
@ -1946,7 +1903,6 @@ dependencies = [
|
|||
"indicatif 0.18.4",
|
||||
"libc",
|
||||
"log",
|
||||
"native-tls",
|
||||
"rand 0.9.4",
|
||||
"reqwest",
|
||||
"serde",
|
||||
|
|
@ -2092,22 +2048,6 @@ dependencies = [
|
|||
"webpki-roots 1.0.7",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "hyper-tls"
|
||||
version = "0.6.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "70206fc6890eaca9fde8a0bf71caa2ddfc9fe045ac9e5c70df101a7dbde866e0"
|
||||
dependencies = [
|
||||
"bytes",
|
||||
"http-body-util",
|
||||
"hyper",
|
||||
"hyper-util",
|
||||
"native-tls",
|
||||
"tokio",
|
||||
"tokio-native-tls",
|
||||
"tower-service",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "hyper-util"
|
||||
version = "0.1.20"
|
||||
|
|
@ -2126,11 +2066,9 @@ dependencies = [
|
|||
"percent-encoding",
|
||||
"pin-project-lite",
|
||||
"socket2",
|
||||
"system-configuration",
|
||||
"tokio",
|
||||
"tower-service",
|
||||
"tracing",
|
||||
"windows-registry",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
@ -2695,23 +2633,6 @@ dependencies = [
|
|||
"pxfm",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "native-tls"
|
||||
version = "0.2.18"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "465500e14ea162429d264d44189adc38b199b62b1c21eea9f69e4b73cb03bbf2"
|
||||
dependencies = [
|
||||
"libc",
|
||||
"log",
|
||||
"openssl",
|
||||
"openssl-probe",
|
||||
"openssl-sys",
|
||||
"schannel",
|
||||
"security-framework",
|
||||
"security-framework-sys",
|
||||
"tempfile",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ndarray"
|
||||
version = "0.17.2"
|
||||
|
|
@ -2897,60 +2818,12 @@ version = "11.1.5"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d6790f58c7ff633d8771f42965289203411a5e5c68388703c06e14f24770b41e"
|
||||
|
||||
[[package]]
|
||||
name = "openssl"
|
||||
version = "0.10.78"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f38c4372413cdaaf3cc79dd92d29d7d9f5ab09b51b10dded508fb90bb70b9222"
|
||||
dependencies = [
|
||||
"bitflags",
|
||||
"cfg-if",
|
||||
"foreign-types",
|
||||
"libc",
|
||||
"once_cell",
|
||||
"openssl-macros",
|
||||
"openssl-sys",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "openssl-macros"
|
||||
version = "0.1.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a948666b637a0f465e8564c73e89d4dde00d72d4d473cc972f390fc3dcee7d9c"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "openssl-probe"
|
||||
version = "0.2.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe"
|
||||
|
||||
[[package]]
|
||||
name = "openssl-src"
|
||||
version = "300.6.0+3.6.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a8e8cbfd3a4a8c8f089147fd7aaa33cf8c7450c4d09f8f80698a0cf093abeff4"
|
||||
dependencies = [
|
||||
"cc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "openssl-sys"
|
||||
version = "0.9.114"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "13ce1245cd07fcc4cfdb438f7507b0c7e4f3849a69fd84d52374c66d83741bb6"
|
||||
dependencies = [
|
||||
"cc",
|
||||
"libc",
|
||||
"openssl-src",
|
||||
"pkg-config",
|
||||
"vcpkg",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "option-ext"
|
||||
version = "0.2.0"
|
||||
|
|
@ -3012,15 +2885,6 @@ version = "0.1.1"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "35fb2e5f958ec131621fdd531e9fc186ed768cbe395337403ae56c17a74c68ec"
|
||||
|
||||
[[package]]
|
||||
name = "pem-rfc7468"
|
||||
version = "1.0.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a6305423e0e7738146434843d1694d621cce767262b2a86910beab705e4493d9"
|
||||
dependencies = [
|
||||
"base64ct",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "percent-encoding"
|
||||
version = "2.3.2"
|
||||
|
|
@ -3582,7 +3446,6 @@ checksum = "eddd3ca559203180a307f12d114c268abf583f59b03cb906fd0b3ff8646c1147"
|
|||
dependencies = [
|
||||
"base64 0.22.1",
|
||||
"bytes",
|
||||
"encoding_rs",
|
||||
"futures-core",
|
||||
"futures-util",
|
||||
"h2",
|
||||
|
|
@ -3591,12 +3454,9 @@ dependencies = [
|
|||
"http-body-util",
|
||||
"hyper",
|
||||
"hyper-rustls",
|
||||
"hyper-tls",
|
||||
"hyper-util",
|
||||
"js-sys",
|
||||
"log",
|
||||
"mime",
|
||||
"native-tls",
|
||||
"percent-encoding",
|
||||
"pin-project-lite",
|
||||
"quinn",
|
||||
|
|
@ -3607,7 +3467,6 @@ dependencies = [
|
|||
"serde_urlencoded",
|
||||
"sync_wrapper",
|
||||
"tokio",
|
||||
"tokio-native-tls",
|
||||
"tokio-rustls",
|
||||
"tokio-util",
|
||||
"tower",
|
||||
|
|
@ -3805,7 +3664,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
|||
checksum = "b7f4bc775c73d9a02cde8bf7b2ec4c9d12743edf609006c7facc23998404cd1d"
|
||||
dependencies = [
|
||||
"bitflags",
|
||||
"core-foundation 0.10.1",
|
||||
"core-foundation",
|
||||
"core-foundation-sys",
|
||||
"libc",
|
||||
"security-framework-sys",
|
||||
|
|
@ -4076,27 +3935,6 @@ dependencies = [
|
|||
"syn",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "system-configuration"
|
||||
version = "0.7.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a13f3d0daba03132c0aa9767f98351b3488edc2c100cda2d2ec2b04f3d8d3c8b"
|
||||
dependencies = [
|
||||
"bitflags",
|
||||
"core-foundation 0.9.4",
|
||||
"system-configuration-sys",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "system-configuration-sys"
|
||||
version = "0.6.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8e1d1b10ced5ca923a1fcb8d03e96b8d3268065d724548c0211415ff6ac6bac4"
|
||||
dependencies = [
|
||||
"core-foundation-sys",
|
||||
"libc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "target-lexicon"
|
||||
version = "0.12.16"
|
||||
|
|
@ -4321,16 +4159,6 @@ dependencies = [
|
|||
"syn",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tokio-native-tls"
|
||||
version = "0.3.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "bbae76ab933c85776efabc971569dd6119c580d8f5d448769dec1764bf796ef2"
|
||||
dependencies = [
|
||||
"native-tls",
|
||||
"tokio",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tokio-rustls"
|
||||
version = "0.26.4"
|
||||
|
|
@ -4677,10 +4505,8 @@ checksum = "dea7109cdcd5864d4eeb1b58a1648dc9bf520360d7af16ec26d0a9354bafcfc0"
|
|||
dependencies = [
|
||||
"base64 0.22.1",
|
||||
"cookie_store",
|
||||
"der",
|
||||
"flate2",
|
||||
"log",
|
||||
"native-tls",
|
||||
"percent-encoding",
|
||||
"rustls",
|
||||
"rustls-pki-types",
|
||||
|
|
@ -4689,7 +4515,6 @@ dependencies = [
|
|||
"socks",
|
||||
"ureq-proto",
|
||||
"utf8-zero",
|
||||
"webpki-root-certs",
|
||||
"webpki-roots 1.0.7",
|
||||
]
|
||||
|
||||
|
|
@ -4967,15 +4792,6 @@ dependencies = [
|
|||
"wasm-bindgen",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "webpki-root-certs"
|
||||
version = "1.0.7"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f31141ce3fc3e300ae89b78c0dd67f9708061d1d2eda54b8209346fd6be9a92c"
|
||||
dependencies = [
|
||||
"rustls-pki-types",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "webpki-roots"
|
||||
version = "0.26.11"
|
||||
|
|
@ -5037,35 +4853,6 @@ version = "0.2.1"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
|
||||
|
||||
[[package]]
|
||||
name = "windows-registry"
|
||||
version = "0.6.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "02752bf7fbdcce7f2a27a742f798510f3e5ad88dbe84871e5168e2120c3d5720"
|
||||
dependencies = [
|
||||
"windows-link",
|
||||
"windows-result",
|
||||
"windows-strings",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "windows-result"
|
||||
version = "0.4.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5"
|
||||
dependencies = [
|
||||
"windows-link",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "windows-strings"
|
||||
version = "0.5.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091"
|
||||
dependencies = [
|
||||
"windows-link",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "windows-sys"
|
||||
version = "0.52.0"
|
||||
|
|
|
|||
10
Dockerfile
10
Dockerfile
|
|
@ -14,18 +14,16 @@ ARG UV_VERSION
|
|||
|
||||
# build-essential / g++ for any C extension wheels uv may need to build
|
||||
# from source. curl + ca-certificates are required by the rustup
|
||||
# bootstrap below. pkg-config + libssl-dev for `openssl-sys` (transitive
|
||||
# from fastembed/hf-hub/ureq → native-tls — the workspace `rustls-tls`
|
||||
# pin loses to cargo feature unification). patchelf for maturin's
|
||||
# wheel-link repair on linux.
|
||||
# bootstrap below. patchelf for maturin's wheel-link repair on linux.
|
||||
# No OpenSSL system deps required: the rustls-everywhere refactor
|
||||
# eliminated `openssl-sys` from our build tree by switching fastembed
|
||||
# to `hf-hub-rustls-tls` + `ort-download-binaries-rustls-tls`.
|
||||
RUN apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
build-essential \
|
||||
g++ \
|
||||
curl \
|
||||
ca-certificates \
|
||||
pkg-config \
|
||||
libssl-dev \
|
||||
patchelf \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
|
|
|
|||
|
|
@ -53,7 +53,19 @@ flate2 = "1"
|
|||
# int8-quantized ONNX) auto-downloads from HuggingFace Hub on first use.
|
||||
# Same library + same model = byte-equal embeddings between Python and Rust
|
||||
# (both call into ONNX Runtime over the identical ONNX file).
|
||||
fastembed = "5"
|
||||
#
|
||||
# `default-features = false` + explicit rustls features eliminates the
|
||||
# transitive `native-tls` → `openssl-sys` dependency that fastembed's
|
||||
# default features pull in (via `hf-hub-native-tls` + `ort-download-
|
||||
# binaries-native-tls`). Without this, every wheel-build surface
|
||||
# (release matrix, e2e/wrap, e2e/init, devcontainer, ci.yml) needs
|
||||
# system OpenSSL + perl modules for openssl-src vendored compile. Using
|
||||
# rustls everywhere removes the entire OpenSSL build-deps surface.
|
||||
fastembed = { version = "5", default-features = false, features = [
|
||||
"hf-hub-rustls-tls",
|
||||
"ort-download-binaries-rustls-tls",
|
||||
"image-models",
|
||||
] }
|
||||
# `magika` is Google's ONNX-backed content classifier (Tier 1 of the
|
||||
# Stage-3d ContentRouter detection arch). Bundled standard-model is
|
||||
# loaded once per process via `OnceLock` and shared across calls. The
|
||||
|
|
|
|||
|
|
@ -24,18 +24,6 @@ tracing = { workspace = true }
|
|||
tracing-subscriber = { version = "0.3", features = ["json", "env-filter", "fmt"] }
|
||||
reqwest = { version = "0.12", default-features = false, features = ["stream", "rustls-tls", "http2"] }
|
||||
tokio-tungstenite = { version = "0.24", default-features = false, features = ["connect", "rustls-tls-webpki-roots"] }
|
||||
# `hf-hub` (transitive via `fastembed`) hard-codes `native-tls` as a default
|
||||
# feature; Cargo's feature unification then enables `openssl-sys` for the
|
||||
# whole workspace despite our `reqwest`/`tokio-tungstenite`/`tokio-rustls`
|
||||
# preferences. The wheel build matrix breaks on every Linux + Intel-macOS
|
||||
# target where the manylinux container's system OpenSSL is too old or
|
||||
# absent (manylinux2014 ships 1.0.2k; aarch64 cross sysroot has none).
|
||||
# Enabling `openssl/vendored` instructs `openssl-sys` to compile OpenSSL
|
||||
# from source as part of the cargo build, removing the system-OpenSSL
|
||||
# dependency entirely. ~30s extra one-time build cost; works on every
|
||||
# target uniformly. The crate is referenced here only to enable the
|
||||
# feature via Cargo's unification — we don't use the API directly.
|
||||
openssl = { version = "0.10", features = ["vendored"] }
|
||||
clap = { workspace = true, features = ["derive", "env"] }
|
||||
serde = { workspace = true }
|
||||
serde_json = { workspace = true }
|
||||
|
|
|
|||
|
|
@ -25,19 +25,3 @@ extension-module = ["pyo3/extension-module"]
|
|||
headroom-core = { path = "../headroom-core" }
|
||||
pyo3 = { workspace = true }
|
||||
serde_json = { workspace = true }
|
||||
# `headroom-core` pulls `openssl-sys` transitively via fastembed →
|
||||
# hf-hub → ureq → native-tls. The wheel matrix's manylinux_2_28
|
||||
# x86_64 container has no `openssl-devel`, and even with one the
|
||||
# manylinux2014 fallback ships OpenSSL 1.0.2k which `openssl-sys 0.9`
|
||||
# rejects. We force the `vendored` feature on `openssl` so cargo
|
||||
# compiles OpenSSL from source as part of the wheel build, removing
|
||||
# the system-OpenSSL dependency entirely.
|
||||
#
|
||||
# Why this dep lives HERE (the wheel-producing crate) rather than
|
||||
# headroom-proxy: cargo's feature unification only applies to the
|
||||
# resolution graph that's actually built. `maturin build --manifest-path
|
||||
# crates/headroom-py/Cargo.toml` resolves from headroom-py's deps
|
||||
# only — headroom-proxy's `openssl/vendored` enable doesn't propagate
|
||||
# because headroom-py never depends on headroom-proxy. Pinning the
|
||||
# vendored feature on headroom-py's own deps is the right fix.
|
||||
openssl = { version = "0.10", features = ["vendored"] }
|
||||
|
|
|
|||
|
|
@ -6,7 +6,8 @@
|
|||
# 2.28 baseline) yields a wheel portable to any glibc 2.28+ runtime.
|
||||
FROM quay.io/pypa/manylinux_2_28_x86_64 AS builder
|
||||
|
||||
RUN yum install -y openssl-devel pkgconfig perl-IPC-Cmd
|
||||
# No OpenSSL system deps required (rustls-everywhere refactor). See
|
||||
# e2e/wrap/Dockerfile for the full rationale.
|
||||
|
||||
ENV CARGO_HOME=/usr/local/cargo \
|
||||
RUSTUP_HOME=/usr/local/rustup \
|
||||
|
|
|
|||
|
|
@ -9,10 +9,15 @@
|
|||
# on any glibc 2.28+ runtime, including bookworm.
|
||||
FROM quay.io/pypa/manylinux_2_28_x86_64 AS builder
|
||||
|
||||
# `openssl-sys` (transitive via fastembed/hf-hub/ureq → native-tls) needs
|
||||
# OpenSSL dev headers. perl-IPC-Cmd is required by openssl-sys's autotools
|
||||
# probe.
|
||||
RUN yum install -y openssl-devel pkgconfig perl-IPC-Cmd
|
||||
# No OpenSSL system deps required. As of the rustls-everywhere refactor,
|
||||
# all HTTP+TLS in our Rust crates goes through `rustls`. fastembed's
|
||||
# `hf-hub-rustls-tls` + `ort-download-binaries-rustls-tls` features
|
||||
# replace the default native-tls path; `cargo tree -p headroom-py -i
|
||||
# openssl-sys` returns "not found", confirming this Dockerfile no
|
||||
# longer needs `openssl-devel`, `pkgconfig`, or any perl modules
|
||||
# (Time::Piece, IPC::Cmd) for the openssl-src vendored Configure
|
||||
# script. The historical `yum install openssl-devel pkgconfig
|
||||
# perl-IPC-Cmd` line is intentionally absent.
|
||||
|
||||
# Install rust toolchain into the manylinux container. Match the
|
||||
# rust-toolchain.toml at repo root (1.95.0 + rustfmt + clippy) so cargo
|
||||
|
|
|
|||
|
|
@ -69,103 +69,184 @@ def test_ci_commitlint_skips_default_github_merge_commits() -> None:
|
|||
assert "!startsWith(github.event.head_commit.message, 'Merge pull request ')" in content
|
||||
|
||||
|
||||
def test_headroom_py_vendors_openssl() -> None:
|
||||
"""`hf-hub` (transitive via `fastembed`) hard-codes `native-tls` as a
|
||||
default feature, forcing `openssl-sys` for the entire build graph
|
||||
despite our `reqwest`/`tokio-tungstenite`/`tokio-rustls` preferences.
|
||||
def test_no_openssl_sys_in_wheel_build_tree() -> None:
|
||||
"""STRUCTURAL INVARIANT: openssl-sys must NOT appear in the wheel
|
||||
build's resolved dependency graph.
|
||||
|
||||
Critically: this dep MUST live in `headroom-py/Cargo.toml` (the
|
||||
wheel-producing crate) — NOT in `headroom-proxy/Cargo.toml`. Cargo's
|
||||
feature unification only applies to the resolution graph that's
|
||||
actually built. `maturin build --manifest-path crates/headroom-py/
|
||||
Cargo.toml` resolves headroom-py's deps only; headroom-py does NOT
|
||||
depend on headroom-proxy, so a `vendored` feature enable in
|
||||
headroom-proxy never propagates to the wheel build. The earlier
|
||||
hot-fix put it in headroom-proxy and the wheel still failed with
|
||||
"Could not find directory of OpenSSL installation" — fixed by
|
||||
moving the dep to headroom-py.
|
||||
This is the load-bearing assertion for the entire build pipeline.
|
||||
If openssl-sys is in the wheel-build resolution graph, every
|
||||
Linux/macOS surface that builds from source needs system OpenSSL
|
||||
+ perl modules + pkg-config — and we've spent five hot-fixes
|
||||
chasing whichever combination of perl modules / OpenSSL versions
|
||||
/ pkg-config paths was missing in each manylinux/Dockerfile/
|
||||
devcontainer surface. The cleanest fix is to NOT depend on
|
||||
OpenSSL at all.
|
||||
|
||||
fastembed exposes `hf-hub-rustls-tls` and
|
||||
`ort-download-binaries-rustls-tls` features that replace its
|
||||
default `native-tls` path. With `default-features = false` plus
|
||||
those rustls features enabled in headroom-core, our entire build
|
||||
tree uses rustls and no crate pulls openssl-sys.
|
||||
|
||||
This test runs `cargo tree` (so it actually exercises the
|
||||
resolved feature graph, not just declared Cargo.toml features).
|
||||
A future refactor that adds a transitive native-tls user will
|
||||
fail here, surfaced at PR time rather than 5 minutes into a CI
|
||||
wheel-build error.
|
||||
"""
|
||||
py_cargo = (ROOT / "crates" / "headroom-py" / "Cargo.toml").read_text(encoding="utf-8")
|
||||
import subprocess
|
||||
|
||||
# Guard against a future refactor that re-removes the vendored dep
|
||||
# from headroom-py (would silently break the wheel build again).
|
||||
assert 'openssl = { version = "0.10", features = ["vendored"] }' in py_cargo, (
|
||||
"headroom-py/Cargo.toml must declare openssl with the vendored feature; "
|
||||
"without it the manylinux wheel build cannot find system OpenSSL."
|
||||
)
|
||||
for crate in ("headroom-py", "headroom-proxy", "headroom-core"):
|
||||
result = subprocess.run(
|
||||
[
|
||||
"cargo",
|
||||
"tree",
|
||||
"--target",
|
||||
"x86_64-unknown-linux-gnu",
|
||||
"-p",
|
||||
crate,
|
||||
"-i",
|
||||
"openssl-sys",
|
||||
],
|
||||
cwd=str(ROOT),
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
# `cargo tree -i <pkg>` exits 101 with "did not match any
|
||||
# packages" when the package is NOT in the tree — the GREEN
|
||||
# case. Exit 0 with a tree of consumers means it IS pulled.
|
||||
not_in_tree = result.returncode != 0 and "did not match any packages" in result.stderr
|
||||
assert not_in_tree, (
|
||||
f"openssl-sys is back in {crate}'s build tree:\n"
|
||||
f"stdout:\n{result.stdout}\n"
|
||||
f"stderr:\n{result.stderr}\n"
|
||||
"Find the new native-tls user (likely a default-features=true "
|
||||
"on a transitive crate) and disable it. Switching every "
|
||||
"transitive HTTP+TLS consumer to rustls is the load-bearing "
|
||||
"invariant that keeps wheel builds working without system "
|
||||
"OpenSSL or perl modules."
|
||||
)
|
||||
|
||||
|
||||
def test_build_wheels_installs_perl_ipc_cmd_for_vendored_openssl() -> None:
|
||||
"""OpenSSL's vendored `Configure` script needs `IPC::Cmd`. Without
|
||||
it the build fails with `Can't locate IPC/Cmd.pm`. The before-script
|
||||
must (a) probe the module first to skip a no-op install when the
|
||||
container already has it, (b) cover RHEL family (dnf/yum) AND
|
||||
Debian/Ubuntu (apt-get) AND musllinux (apk) since the maturin-action
|
||||
uses different containers per target, and (c) fail loud with an
|
||||
explicit `perl -MIPC::Cmd -e 1` assertion if the install path
|
||||
didn't actually resolve the module — no silent fallback into a
|
||||
later confusing openssl-src build error.
|
||||
def test_no_native_tls_in_wheel_build_tree() -> None:
|
||||
"""The dual of the openssl-sys gate: native-tls is the proximate
|
||||
cause of openssl-sys being pulled. Catch it earlier with a more
|
||||
specific error message so future debugging starts at the right
|
||||
place.
|
||||
"""
|
||||
import subprocess
|
||||
|
||||
The previous shape used `libipc-cmd-perl` on the apt branch, which
|
||||
is a deprecated alias and is not in the default sources of the
|
||||
aarch64-cross container; that broke the aarch64 wheel build.
|
||||
for crate in ("headroom-py", "headroom-proxy", "headroom-core"):
|
||||
result = subprocess.run(
|
||||
[
|
||||
"cargo",
|
||||
"tree",
|
||||
"--target",
|
||||
"x86_64-unknown-linux-gnu",
|
||||
"-p",
|
||||
crate,
|
||||
"-i",
|
||||
"native-tls",
|
||||
],
|
||||
cwd=str(ROOT),
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
not_in_tree = result.returncode != 0 and "did not match any packages" in result.stderr
|
||||
assert not_in_tree, (
|
||||
f"native-tls is back in {crate}'s build tree — likely some "
|
||||
f"crate's `default-features = true` re-enabled native-tls "
|
||||
f"transitively:\n{result.stdout}"
|
||||
)
|
||||
|
||||
|
||||
def test_fastembed_uses_rustls_features() -> None:
|
||||
"""The mechanism that keeps openssl-sys out of the build is
|
||||
fastembed's explicit rustls feature selection in headroom-core.
|
||||
fastembed's default features include `hf-hub-native-tls` and
|
||||
`ort-download-binaries-native-tls` — both pull openssl-sys.
|
||||
Disabling defaults and enabling the rustls equivalents removes
|
||||
the OpenSSL surface entirely.
|
||||
"""
|
||||
cargo = (ROOT / "crates" / "headroom-core" / "Cargo.toml").read_text(encoding="utf-8")
|
||||
|
||||
assert "default-features = false" in cargo
|
||||
assert '"hf-hub-rustls-tls"' in cargo
|
||||
assert '"ort-download-binaries-rustls-tls"' in cargo
|
||||
# `image-models` is in default; we re-enable it explicitly so we
|
||||
# don't lose the image-embedding capability when defaults are off.
|
||||
assert '"image-models"' in cargo
|
||||
|
||||
|
||||
def test_dockerfiles_no_longer_install_openssl_devel() -> None:
|
||||
"""Once openssl-sys is out of the build tree, every Dockerfile
|
||||
that used to install `openssl-devel` / `libssl-dev` for the Rust
|
||||
build can drop those packages. This test enforces the cleanup so
|
||||
a future refactor doesn't carry the old packages forward "just
|
||||
in case".
|
||||
|
||||
The check looks only at non-comment lines so explanatory comments
|
||||
that mention the historical packages don't false-positive.
|
||||
"""
|
||||
targets = [
|
||||
ROOT / "e2e" / "wrap" / "Dockerfile",
|
||||
ROOT / "e2e" / "init" / "Dockerfile",
|
||||
ROOT / "Dockerfile",
|
||||
ROOT / ".devcontainer" / "Dockerfile",
|
||||
]
|
||||
|
||||
forbidden = ["openssl-devel", "libssl-dev"]
|
||||
|
||||
for target in targets:
|
||||
content = target.read_text(encoding="utf-8")
|
||||
non_comment = "\n".join(
|
||||
line for line in content.splitlines() if not line.lstrip().startswith("#")
|
||||
)
|
||||
for pkg in forbidden:
|
||||
assert pkg not in non_comment, (
|
||||
f"{target.relative_to(ROOT)} still installs {pkg!r} on a "
|
||||
f"non-comment line. The rustls-everywhere refactor removed "
|
||||
f"openssl-sys from the build tree; this package is no "
|
||||
f"longer needed."
|
||||
)
|
||||
|
||||
|
||||
def test_release_yml_does_not_install_openssl_or_perl_for_wheels() -> None:
|
||||
"""With openssl-sys out of the build tree (verified by
|
||||
test_no_openssl_sys_in_wheel_build_tree), the previous
|
||||
before-script-linux that installed perl-IPC-Cmd / perl /
|
||||
perl-utils for the openssl-src vendored Configure script is
|
||||
obsolete. Removing it speeds the wheel build and keeps the
|
||||
Linux entry honest — every package install we keep here
|
||||
represents a hidden assumption about the manylinux container.
|
||||
"""
|
||||
content = (ROOT / ".github" / "workflows" / "release.yml").read_text(encoding="utf-8")
|
||||
|
||||
assert "before-script-linux:" in content
|
||||
|
||||
# Pre-probe so we no-op when IPC::Cmd is already importable.
|
||||
assert "perl -MIPC::Cmd -e 1" in content
|
||||
|
||||
# All three RHEL-family + Debian-family + Alpine package managers covered.
|
||||
assert "dnf install -y perl-IPC-Cmd" in content
|
||||
assert "yum install -y perl-IPC-Cmd" in content
|
||||
# The plain `perl` meta-package pulls perl-modules-* on Debian/Ubuntu,
|
||||
# which contains IPC::Cmd. The legacy `libipc-cmd-perl` alias must NOT
|
||||
# be referenced as an installable target — it is no longer in default
|
||||
# Debian/Ubuntu sources. We allow the string in YAML/shell comments
|
||||
# (operator hints) by checking only non-comment lines.
|
||||
assert "apt-get install -y --no-install-recommends perl" in content
|
||||
assert "apk add --no-cache perl-utils" in content
|
||||
|
||||
non_comment_lines: list[str] = []
|
||||
for raw in content.splitlines():
|
||||
stripped = raw.lstrip()
|
||||
# Drop YAML and shell `#` comments. Mid-line `#` comments would
|
||||
# require a real YAML/shell tokenizer; the file's actual install
|
||||
# commands are never on the same physical line as a comment.
|
||||
if stripped.startswith("#"):
|
||||
continue
|
||||
non_comment_lines.append(raw)
|
||||
code_only = "\n".join(non_comment_lines)
|
||||
assert "libipc-cmd-perl" not in code_only, (
|
||||
"libipc-cmd-perl must not appear as an apt-get target — it is a "
|
||||
"deprecated alias not in default Debian/Ubuntu sources."
|
||||
)
|
||||
|
||||
# Final fail-loud assertion (no silent fallback).
|
||||
assert "perl -MIPC::Cmd -e 'print \"IPC::Cmd loaded OK" in content
|
||||
|
||||
|
||||
def test_build_wheels_does_not_set_openssl_dir() -> None:
|
||||
"""`OPENSSL_DIR` (and the related `OPENSSL_LIB_DIR` /
|
||||
`OPENSSL_INCLUDE_DIR`) DEFEATS the `vendored` feature: openssl-sys
|
||||
will use the system OpenSSL at that path instead of compiling from
|
||||
source. The earlier macOS hot-fix exported these env vars; with
|
||||
vendored enabled they must NOT be set, otherwise we silently
|
||||
regress to the system-OpenSSL path that broke originally.
|
||||
"""
|
||||
content = (ROOT / ".github" / "workflows" / "release.yml").read_text(encoding="utf-8")
|
||||
|
||||
# Locate the build-wheels job and assert it does not export OPENSSL_DIR.
|
||||
bw_start = content.index("\n build-wheels:")
|
||||
bw_end = content.index("\n collect-dist:")
|
||||
body = content[bw_start:bw_end]
|
||||
|
||||
assert "OPENSSL_DIR" not in body, (
|
||||
"build-wheels must not set OPENSSL_DIR — it disables openssl-sys's "
|
||||
"vendored feature and reintroduces the system-OpenSSL dependency."
|
||||
)
|
||||
non_comment = "\n".join(line for line in body.splitlines() if not line.lstrip().startswith("#"))
|
||||
|
||||
# No legacy install commands or env vars must appear on a non-comment
|
||||
# line. Each forbidden token represents an assumption about system
|
||||
# OpenSSL that the rustls refactor removed.
|
||||
forbidden = [
|
||||
"openssl-devel",
|
||||
"libssl-dev",
|
||||
"perl-IPC-Cmd",
|
||||
"libipc-cmd-perl",
|
||||
"OPENSSL_DIR",
|
||||
]
|
||||
for token in forbidden:
|
||||
assert token not in non_comment, (
|
||||
f"release.yml build-wheels job still references {token!r} on "
|
||||
f"a non-comment line. The rustls-everywhere refactor removed "
|
||||
f"openssl-sys from the build tree; this command/env is now "
|
||||
f"obsolete."
|
||||
)
|
||||
|
||||
|
||||
def test_build_wheels_matrix_excludes_intel_macos() -> None:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue