fix(mesh): enable QUIC keep-alive on mesh transport (was: connections dropping mid-inference)

PR #566 review feedback flagged MoA returning early with 0/N workers
under load. Live debug on a 2-node mesh (M4 Max + Mac Studio M3 Ultra)
running an OpenCode agent against `model: "mesh"` showed repeated:

  WARN noq_proto::connection: failed closing path err=LastOpenPath
  INFO mesh: Connection to <peer> closed: timed out
  WARN moa: reducer ... failed: recv: read error: connection lost

happening 30-60s into otherwise healthy inference calls, including
plain non-MoA `stream: false` requests through `model: "auto"`.

Root cause: noq-proto's default `max_idle_timeout` is 30s and
`keep_alive_interval` is `None` (the spec, RFC 9000 §10.1.2, makes
keep-alive opt-in; quinn / noq follow that). Non-streaming inference
requests send no application bytes while the remote model is
generating tokens, so the wire is idle. Under concurrent load
(parallel MoA workers + reducer + gossip + heartbeats), noq's
multipath bookkeeping closes the idle path, and when it is the last
open path the entire connection drops mid-stream. The in-flight HTTP
tunnel errors with `connection lost` and the caller must retry.

This only became visible recently because:
* Streaming OpenAI clients (Goose, Claude Code, pi, the web UI) all
  set `stream: true` by default. SSE chunks flow continuously and
  reset the idle timer, so the bug never manifests for them.
* MoA `RemoteModelBackend` is the first significant non-streaming
  long-running RPC in the codebase (`stream: false` hardcoded in
  `crates/mesh-mixture-of-agents/src/backend.rs`).
* Reasoning models with big agent prompts (MiniMax-M2.5 on a 13k
  OpenCode system prompt, Qwen3-32B class reducers) routinely take
  30-90s for a first useful response. That is the combination that
  exceeds the default 30s idle window.

Fix: set `keep_alive_interval = 10s` and `max_idle_timeout = 5m`
on the mesh QUIC transport config, plus the matching multipath
`default_path_keep_alive_interval` and
`default_path_max_idle_timeout` so individual paths don't get torn
down while the connection-level idle timer is fine.

Cost: one QUIC PING (~30-60 bytes) every 10s per connection only
when no other application data has been sent for that long. In a
typical mesh with periodic gossip and heartbeats this fires rarely.
`keep_alive` is opportunistic, not unconditional.

Live verification on the same 2-node mesh:

* 60s idle test, before fix: 2x `Connection to <peer> closed:
  timed out`. After fix: 0x. Connection stays healthy.
* 75s of mixed non-streaming inference (53s `auto` to MiniMax +
  21s `mesh` 2-worker fanout), before fix: multiple `LastOpenPath`
  + `connection lost` errors. After fix: 0x. Both completed
  successfully with finish_reason=stop and full content.
* OpenCode `model: mesh` agent loop, before fix: 0 of 2 turns
  landed. After fix: 2 of 3 turns landed (the 3rd hit a separate
  KV cache exhaustion in the local stage runtime, tracked
  independently).

Validation: `cargo fmt --all -- --check` clean,
`cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings`
clean, `cargo test -p mesh-llm-host-runtime --lib` 1435/1435 pass.
This commit is contained in:
Michael Neale 2026-05-20 20:00:08 +10:00
parent d527965612
commit f5cf4b8678

View file

@ -1380,8 +1380,37 @@ async fn startup_secret_key(role: &NodeRole) -> Result<SecretKey> {
}
fn startup_transport_config() -> iroh::endpoint::QuicTransportConfig {
// Keep QUIC connections alive during long inference calls.
//
// noq-proto's default `max_idle_timeout` is ~30s and `keep_alive_interval`
// is `None`. A non-streaming inference request (e.g. MoA reducer or any
// `stream:false` call) sends nothing on the wire while the remote model is
// generating tokens. Under concurrent load (multiple in-flight model
// requests + gossip + heartbeats) noq's multipath bookkeeping will close
// an idle path, and if it is the last open path the whole connection
// drops mid-stream. The in-flight stream errors with `connection lost`
// and the caller has to retry from scratch.
//
// A 10s keep-alive sends a small PING every 10s on each path, keeping
// paths and the connection healthy during long compute. The 5-minute idle
// timeout is defense in depth for truly silent connections (paused
// agents, suspended laptops); short-term silence is handled by
// keep-alive.
let max_idle = iroh::endpoint::IdleTimeout::try_from(std::time::Duration::from_secs(300))
.expect("5-minute idle timeout fits in a VarInt");
let keep_alive = std::time::Duration::from_secs(10);
let path_idle = std::time::Duration::from_secs(300);
iroh::endpoint::QuicTransportConfig::builder()
.max_concurrent_bidi_streams(1024u32.into())
.keep_alive_interval(keep_alive)
.max_idle_timeout(Some(max_idle))
// noq-proto's multipath uses per-path idle timers independent of the
// connection-level idle. Without these, a path can be torn down while
// the connection idle timer is fine, and when the last path closes the
// connection dies with `LastOpenPath`. Mirror connection-level
// settings onto the default per-path config.
.default_path_max_idle_timeout(path_idle)
.default_path_keep_alive_interval(keep_alive)
.build()
}