fix(moa): bump worker/reducer timeouts to 60s for agent-scale prompts

PR #566 review feedback (Apr 2026) flagged that MoA worker accounting
sometimes reported zero successful workers, especially under churn or
load. Investigating from a real 2-node mesh (Mac M4 Max + Mac Studio
M3 Ultra) with an OpenCode agent driving `model: "mesh"` showed
that the root cause was the 15s worker_timeout being too tight for
agent-scale prompts:

* OpenCode's default system prompt is ~13.7k tokens.
* Large strong-tier models (MiniMax-M2.5 Q4_K_M, Qwen3-32B+) at 13k+
  prompts + tool schemas take 20\u201340s for a first useful response \u2014
  the reasoning preamble alone often eats the 15s budget.
* The MoA gateway killed the strong worker at exactly 15s every turn:
    moa: worker unsloth/MiniMax-M2.5-GGUF:Q4_K_M (strong) failed
         after 15001ms: remote timeout after 15s
* The arbiter then early-exited on the surviving small worker, never
  giving the strong worker a chance to land. The strong worker was
  effectively unreachable for OpenCode/Goose-style flows.

Bump both `worker_timeout` and `reducer_timeout` from 15s \u2192 60s
in `build_moa_config`. Live verification on the same 2-node mesh:

* With 15s: `model: mesh` from OpenCode finished 0 of 3 turns
  successfully. Every turn returned 1/2 workers, strong worker
  timeout, no useful response.
* With 60s: `model: mesh` from OpenCode finished 2 of 3 turns
  successfully \u2014 strong worker landed, MoA produced the structured
  `tool_calls` field, OpenCode invoked the file-read tool correctly.
  (The 3rd turn hit a separate llama_decode / connection-lost issue
  in the local stage runtime that is unrelated to MoA timing.)

The trade-off is that a single hung remote worker can stall a turn
for 60s instead of 15s. That is acceptable for an interactive agent
loop where the alternative is consistent failure to land the strong
worker at all. The hedged-reducer ladder (`hedge_delay` = 5s)
still keeps end-to-end latency bounded when only the *reducer* is
slow.

`cargo test -p mesh-llm-host-runtime --lib` \u2014 1435/1435 pass.
`cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings`
\u2014 clean. `cargo fmt --all -- --check` \u2014 clean.
This commit is contained in:
Michael Neale 2026-05-20 19:21:46 +10:00
parent f3355bfd34
commit d527965612

View file

@ -204,10 +204,17 @@ pub async fn build_moa_config(
Some(moa::GatewayConfig {
backends,
models,
worker_timeout: std::time::Duration::from_secs(15),
// Bumped from 15s → 60s. 15s was tight for big-context interactive
// turns: a large model with a 1020k-token prompt and tool schema
// (typical for agent harnesses like OpenCode/Goose) can need 2030s
// just to produce a first tool-call. Workers were getting killed
// mid-inference and MoA reported `kind=early-exit` with the small
// worker, never the strong one. 60s gives the strong worker room
// to land without making the no-progress wait painful.
worker_timeout: std::time::Duration::from_secs(60),
// Per-attempt cap; hedged_reducer_call hedges across candidates so the
// end-to-end wait is roughly reducer_timeout + a couple of hedge delays.
reducer_timeout: std::time::Duration::from_secs(15),
reducer_timeout: std::time::Duration::from_secs(60),
// Start a second reducer candidate after 5s if the first hasn't replied
// (or sooner on outright failure). Cheap on the happy path, big win on
// the cold-KV / stale-peer tail.