mirror of
https://github.com/Mesh-LLM/mesh-llm.git
synced 2026-08-08 22:23:19 -04:00
fix(mesh): enable QUIC keep-alive on mesh transport (was: connections dropping mid-inference)
PR #566 review feedback flagged MoA returning early with 0/N workers under load. Live debug on a 2-node mesh (M4 Max + Mac Studio M3 Ultra) running an OpenCode agent against `model: "mesh"` showed repeated: WARN noq_proto::connection: failed closing path err=LastOpenPath INFO mesh: Connection to <peer> closed: timed out WARN moa: reducer ... failed: recv: read error: connection lost happening 30-60s into otherwise healthy inference calls, including plain non-MoA `stream: false` requests through `model: "auto"`. Root cause: noq-proto's default `max_idle_timeout` is 30s and `keep_alive_interval` is `None` (the spec, RFC 9000 §10.1.2, makes keep-alive opt-in; quinn / noq follow that). Non-streaming inference requests send no application bytes while the remote model is generating tokens, so the wire is idle. Under concurrent load (parallel MoA workers + reducer + gossip + heartbeats), noq's multipath bookkeeping closes the idle path, and when it is the last open path the entire connection drops mid-stream. The in-flight HTTP tunnel errors with `connection lost` and the caller must retry. This only became visible recently because: * Streaming OpenAI clients (Goose, Claude Code, pi, the web UI) all set `stream: true` by default. SSE chunks flow continuously and reset the idle timer, so the bug never manifests for them. * MoA `RemoteModelBackend` is the first significant non-streaming long-running RPC in the codebase (`stream: false` hardcoded in `crates/mesh-mixture-of-agents/src/backend.rs`). * Reasoning models with big agent prompts (MiniMax-M2.5 on a 13k OpenCode system prompt, Qwen3-32B class reducers) routinely take 30-90s for a first useful response. That is the combination that exceeds the default 30s idle window. Fix: set `keep_alive_interval = 10s` and `max_idle_timeout = 5m` on the mesh QUIC transport config, plus the matching multipath `default_path_keep_alive_interval` and `default_path_max_idle_timeout` so individual paths don't get torn down while the connection-level idle timer is fine. Cost: one QUIC PING (~30-60 bytes) every 10s per connection only when no other application data has been sent for that long. In a typical mesh with periodic gossip and heartbeats this fires rarely. `keep_alive` is opportunistic, not unconditional. Live verification on the same 2-node mesh: * 60s idle test, before fix: 2x `Connection to <peer> closed: timed out`. After fix: 0x. Connection stays healthy. * 75s of mixed non-streaming inference (53s `auto` to MiniMax + 21s `mesh` 2-worker fanout), before fix: multiple `LastOpenPath` + `connection lost` errors. After fix: 0x. Both completed successfully with finish_reason=stop and full content. * OpenCode `model: mesh` agent loop, before fix: 0 of 2 turns landed. After fix: 2 of 3 turns landed (the 3rd hit a separate KV cache exhaustion in the local stage runtime, tracked independently). Validation: `cargo fmt --all -- --check` clean, `cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings` clean, `cargo test -p mesh-llm-host-runtime --lib` 1435/1435 pass.
This commit is contained in:
parent
d527965612
commit
f5cf4b8678
1 changed files with 29 additions and 0 deletions
|
|
@ -1380,8 +1380,37 @@ async fn startup_secret_key(role: &NodeRole) -> Result<SecretKey> {
|
|||
}
|
||||
|
||||
fn startup_transport_config() -> iroh::endpoint::QuicTransportConfig {
|
||||
// Keep QUIC connections alive during long inference calls.
|
||||
//
|
||||
// noq-proto's default `max_idle_timeout` is ~30s and `keep_alive_interval`
|
||||
// is `None`. A non-streaming inference request (e.g. MoA reducer or any
|
||||
// `stream:false` call) sends nothing on the wire while the remote model is
|
||||
// generating tokens. Under concurrent load (multiple in-flight model
|
||||
// requests + gossip + heartbeats) noq's multipath bookkeeping will close
|
||||
// an idle path, and if it is the last open path the whole connection
|
||||
// drops mid-stream. The in-flight stream errors with `connection lost`
|
||||
// and the caller has to retry from scratch.
|
||||
//
|
||||
// A 10s keep-alive sends a small PING every 10s on each path, keeping
|
||||
// paths and the connection healthy during long compute. The 5-minute idle
|
||||
// timeout is defense in depth for truly silent connections (paused
|
||||
// agents, suspended laptops); short-term silence is handled by
|
||||
// keep-alive.
|
||||
let max_idle = iroh::endpoint::IdleTimeout::try_from(std::time::Duration::from_secs(300))
|
||||
.expect("5-minute idle timeout fits in a VarInt");
|
||||
let keep_alive = std::time::Duration::from_secs(10);
|
||||
let path_idle = std::time::Duration::from_secs(300);
|
||||
iroh::endpoint::QuicTransportConfig::builder()
|
||||
.max_concurrent_bidi_streams(1024u32.into())
|
||||
.keep_alive_interval(keep_alive)
|
||||
.max_idle_timeout(Some(max_idle))
|
||||
// noq-proto's multipath uses per-path idle timers independent of the
|
||||
// connection-level idle. Without these, a path can be torn down while
|
||||
// the connection idle timer is fine, and when the last path closes the
|
||||
// connection dies with `LastOpenPath`. Mirror connection-level
|
||||
// settings onto the default per-path config.
|
||||
.default_path_max_idle_timeout(path_idle)
|
||||
.default_path_keep_alive_interval(keep_alive)
|
||||
.build()
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue