mirror of
https://github.com/Mesh-LLM/mesh-llm.git
synced 2026-08-08 22:23:19 -04:00
fix(moa): bump worker/reducer timeouts to 60s for agent-scale prompts
PR #566 review feedback (Apr 2026) flagged that MoA worker accounting sometimes reported zero successful workers, especially under churn or load. Investigating from a real 2-node mesh (Mac M4 Max + Mac Studio M3 Ultra) with an OpenCode agent driving `model: "mesh"` showed that the root cause was the 15s worker_timeout being too tight for agent-scale prompts: * OpenCode's default system prompt is ~13.7k tokens. * Large strong-tier models (MiniMax-M2.5 Q4_K_M, Qwen3-32B+) at 13k+ prompts + tool schemas take 20\u201340s for a first useful response \u2014 the reasoning preamble alone often eats the 15s budget. * The MoA gateway killed the strong worker at exactly 15s every turn: moa: worker unsloth/MiniMax-M2.5-GGUF:Q4_K_M (strong) failed after 15001ms: remote timeout after 15s * The arbiter then early-exited on the surviving small worker, never giving the strong worker a chance to land. The strong worker was effectively unreachable for OpenCode/Goose-style flows. Bump both `worker_timeout` and `reducer_timeout` from 15s \u2192 60s in `build_moa_config`. Live verification on the same 2-node mesh: * With 15s: `model: mesh` from OpenCode finished 0 of 3 turns successfully. Every turn returned 1/2 workers, strong worker timeout, no useful response. * With 60s: `model: mesh` from OpenCode finished 2 of 3 turns successfully \u2014 strong worker landed, MoA produced the structured `tool_calls` field, OpenCode invoked the file-read tool correctly. (The 3rd turn hit a separate llama_decode / connection-lost issue in the local stage runtime that is unrelated to MoA timing.) The trade-off is that a single hung remote worker can stall a turn for 60s instead of 15s. That is acceptable for an interactive agent loop where the alternative is consistent failure to land the strong worker at all. The hedged-reducer ladder (`hedge_delay` = 5s) still keeps end-to-end latency bounded when only the *reducer* is slow. `cargo test -p mesh-llm-host-runtime --lib` \u2014 1435/1435 pass. `cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings` \u2014 clean. `cargo fmt --all -- --check` \u2014 clean.
This commit is contained in:
parent
f3355bfd34
commit
d527965612
1 changed files with 9 additions and 2 deletions
|
|
@ -204,10 +204,17 @@ pub async fn build_moa_config(
|
|||
Some(moa::GatewayConfig {
|
||||
backends,
|
||||
models,
|
||||
worker_timeout: std::time::Duration::from_secs(15),
|
||||
// Bumped from 15s → 60s. 15s was tight for big-context interactive
|
||||
// turns: a large model with a 10–20k-token prompt and tool schema
|
||||
// (typical for agent harnesses like OpenCode/Goose) can need 20–30s
|
||||
// just to produce a first tool-call. Workers were getting killed
|
||||
// mid-inference and MoA reported `kind=early-exit` with the small
|
||||
// worker, never the strong one. 60s gives the strong worker room
|
||||
// to land without making the no-progress wait painful.
|
||||
worker_timeout: std::time::Duration::from_secs(60),
|
||||
// Per-attempt cap; hedged_reducer_call hedges across candidates so the
|
||||
// end-to-end wait is roughly reducer_timeout + a couple of hedge delays.
|
||||
reducer_timeout: std::time::Duration::from_secs(15),
|
||||
reducer_timeout: std::time::Duration::from_secs(60),
|
||||
// Start a second reducer candidate after 5s if the first hasn't replied
|
||||
// (or sooner on outright failure). Cheap on the happy path, big win on
|
||||
// the cold-KV / stale-peer tail.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue