mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
ci: require ONNX Runtime >= 1.24 and fail fast when it is missing or too old (#2591)
CI installed 'onnxruntime>=1.16.0' for the Rust ort runtime. That floor is 8 minor versions too low, and the failure mode below it is a silent hang. Why 1.24: ort-sys computes ORT_API_VERSION = 17 + one per enabled api-N feature, and Cargo features are additive across the graph. fastembed 5.17.3 enables api-24, so the constant resolves to 24 and ort rejects any lower runtime. Why it hangs instead of failing: on rejection ort calls Error::new() from inside load_dylib_from_path, which already runs inside the Once that setup_api() is initialising. Building the error re-enters that Once, and std::sync::Once blocks forever on re-entry. Reproduced in isolation with a bare Session::builder() and onnxruntime 1.21.1 - killed after >1h at 0% CPU, no output; ORT_DYLIB_PATH makes no difference. With 1.24.4 the same call returns in 1.2s and the kompress parity fixtures pass 21/21. Per @RubenAAA this is not limited to old runtimes: a box that resolves no libonnxruntime at all hangs identically (0.0% CPU, threads in futex_wait_queue, nothing onnx-shaped in /proc/<pid>/maps). Any failure inside load_dylib_from_path re-enters the Once, so a pin alone cannot close it. So this adds a pre-flight to the dylib step asserting, before any test runs, that onnxruntime imports, that its minor is >= 24, and that a libonnxruntime object exists under capi/. Each failure exits 1 with an ::error:: annotation naming the cause, instead of burning the 30-minute timeout with an empty log. Verified all three branches locally (absent -> rc=1, 1.21.1 -> rc=1, 1.24.4 -> rc=0) and in CI, where it resolved onnxruntime 1.28.0 and exported the .so path. pyproject.toml is deliberately untouched: bumping the floor there makes headroom-ai[all] unsatisfiable via a pillow chain (onnxruntime>=1.24 forces pillow>=10.3.0,<12.0 while [all] requires pillow>=12.3.0). The user-facing hazard via headroom/_ort.py remains open and needs its own change.
This commit is contained in:
parent
e562d007d8
commit
a30305bc4c
2 changed files with 45 additions and 4 deletions
4
.github/workflows/network-diff-capture.yml
vendored
4
.github/workflows/network-diff-capture.yml
vendored
|
|
@ -43,7 +43,7 @@ jobs:
|
|||
'magika>=0.6.0' \
|
||||
'zstandard>=0.20.0' \
|
||||
'websockets>=13.0' \
|
||||
'onnxruntime>=1.16.0' \
|
||||
'onnxruntime>=1.24' \
|
||||
'transformers>=4.30.0' \
|
||||
'watchdog>=4.0.0' \
|
||||
'sqlite-vec>=0.1.6' \
|
||||
|
|
@ -107,7 +107,7 @@ jobs:
|
|||
'magika>=0.6.0' \
|
||||
'zstandard>=0.20.0' \
|
||||
'websockets>=13.0' \
|
||||
'onnxruntime>=1.16.0' \
|
||||
'onnxruntime>=1.24' \
|
||||
'transformers>=4.30.0' \
|
||||
'watchdog>=4.0.0' \
|
||||
'sqlite-vec>=0.1.6'
|
||||
|
|
|
|||
45
.github/workflows/rust.yml
vendored
45
.github/workflows/rust.yml
vendored
|
|
@ -95,8 +95,49 @@ jobs:
|
|||
# at it — same contract `headroom/_ort.py` fulfills for Python
|
||||
# users via the pip `onnxruntime` package.
|
||||
run: |
|
||||
pip install 'onnxruntime>=1.16.0'
|
||||
echo "ORT_DYLIB_PATH=$(python -c "import onnxruntime, pathlib; p = pathlib.Path(onnxruntime.__file__).parent / 'capi'; print(next(iter(sorted(p.glob('libonnxruntime.so*')))))")" >> "$GITHUB_ENV"
|
||||
# >= 1.24, not 1.16: `ort`'s ORT_API_VERSION resolves to 24 because
|
||||
# `fastembed` enables its `api-24` feature.
|
||||
pip install 'onnxruntime>=1.24'
|
||||
# Pre-flight, not just a pin. `ort` deadlocks rather than errors on
|
||||
# ANY failure inside `load_dylib_from_path` — a version mismatch and
|
||||
# a library that cannot be resolved at all both re-enter the `Once`
|
||||
# that `setup_api()` is initialising, and `std::sync::Once` blocks
|
||||
# forever on re-entry. Either way the job burns its full 30-minute
|
||||
# timeout at 0% CPU with nothing in the log. Assert both conditions
|
||||
# here so a bad runner fails in seconds with a readable message.
|
||||
python - <<'PY' >> "$GITHUB_ENV"
|
||||
import pathlib, sys
|
||||
|
||||
def die(msg: str) -> None:
|
||||
print(f"::error::{msg}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
try:
|
||||
import onnxruntime
|
||||
except Exception as exc: # noqa: BLE001 - any import failure is fatal here
|
||||
die(f"onnxruntime is not importable: {exc}")
|
||||
|
||||
version = onnxruntime.__version__
|
||||
try:
|
||||
major, minor = (int(part) for part in version.split(".")[:2])
|
||||
except ValueError:
|
||||
die(f"cannot parse onnxruntime version {version!r}")
|
||||
if (major, minor) < (1, 24):
|
||||
die(
|
||||
f"onnxruntime {version} is too old: ort requires >= 1.24 "
|
||||
"(ORT_API_VERSION=24, set by fastembed's api-24 feature). "
|
||||
"ort DEADLOCKS instead of erroring below this, so the tests "
|
||||
"would hang rather than fail."
|
||||
)
|
||||
|
||||
capi = pathlib.Path(onnxruntime.__file__).parent / "capi"
|
||||
libs = sorted(capi.glob("libonnxruntime.so*")) or sorted(capi.glob("libonnxruntime*.dylib"))
|
||||
if not libs:
|
||||
die(f"no libonnxruntime shared library under {capi}")
|
||||
|
||||
print(f"ORT_DYLIB_PATH={libs[0]}")
|
||||
print(f"onnxruntime {version} -> {libs[0]}", file=sys.stderr)
|
||||
PY
|
||||
- name: cargo fmt --check
|
||||
run: cargo fmt --all -- --check
|
||||
- name: cargo clippy
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue