mesh-llm

mirror of https://github.com/Mesh-LLM/mesh-llm.git synced 2026-08-08 22:23:19 -04:00

No description

Find a file

Michael Neale 96ef6227f2 MoA: mesh answers improve as capable nodes join (#1116 ) * feat(moa): give MoA workers tools, synthesize disagreement, survive flaky peers Agentic MoA turns now reliably produce tool calls, and diverging answers are synthesized instead of relaying one arbitrarily-chosen worker's text. Fixes found by replaying recorded traces from 9 open-weight models: * Tools were withheld from workers. `query_uses_tools` was derived from `looks_like_tool_intent`, an English keyword match on the user's text. "The test suite is failing. Find out which test fails and why" matches nothing, so workers were dispatched without tool schemas and the arbiter ran with has_tools=false — a unanimous tool proposal then fell through to the answer path and leaked "calling search" as prose. 5 of 10 recorded scenarios hit this. Tool availability is now the caller's declaration. * Argument outliers could win a unanimous tool call. Native tool calls are all normalized to a fixed 0.9 confidence, so the confidence-only tiebreak could not separate 8 workers proposing {"path":"src"} from one hallucinating {"path":"rust_project/src"}. Added argument clustering (key-order independent); the largest cluster wins and argument-free calls never outvote filled-in ones. * Early exit committed on tied arguments. Early exit runs on whoever has arrived, so two fast models agreeing on a tool name with different arguments was a coin flip that also aborted the workers who would have broken the tie. It now waits. * Truncation was invisible. `finish_reason` was never read; 39/140 recorded responses came back "length" and 24 carried partial text, which parsed as a normal answer and could be returned verbatim. Plumbed through as BackendReply.truncated -> WorkerOutput.truncated; such answers are barred from consensus and verbatim output but still feed synthesis, labelled incomplete. * A strict endpoint could kill a worker permanently. minimax-m2.5 returns HTTP 400 "Reasoning is mandatory for this endpoint" and failed 12/12 requests; HttpBackend now drops the thinking flags and retries once. Policy changes: * Thinking is always off for MoA workers, not merely defaulted off. The previous escape hatch only let callers request the broken configuration: qwen3-32b spent 408 reasoning tokens against a 384-token cap and returned null content. Ignored overrides are logged. * Diverging answers go to the reducer. Returning the top-confidence answer was near-arbitrary because models rarely emit our confidence envelope, so everything defaults to 0.5 and max_by returns whichever worker finished first. Agreement still short-circuits without paying for synthesis. * Reducer prompt adopts Together's aggregator framing (synthesize, critically evaluate, agreement is not proof), keeping our per-worker attribution, structured tool proposals, and 500-char payload bound. Tests: new sim_real_traces.rs replays 56 recorded cases from 9 models through handle_turn, plus regression unit tests for each bug. Recorders and corpora live in evals/moa-openrouter/. * feat(moa): asymmetric tool turns — references advise, best tool-caller acts Tool-bearing turns now use the Hermes/Nous-style asymmetric shape instead of a majority vote across workers: - references run TOOL-FREE and only advise in prose - the single best tool-caller (the "actor") acts on that advice with the real tools and emits the tool call Tool authority now tracks capability, not popularity. The old path let several weak models proposing a popular-but-wrong tool outvote the one strong tool-caller that picked correctly (observed: qwen3-32b alone chose run_command for a failing-test triage while the smaller models chose list_dir, and the vote shipped list_dir). An actor model removes that failure class instead of patching the vote arithmetic. Stays a pure stateless /v1/chat/completions turn — references are regenerated from the caller's transcript each request, and the external client still owns tool execution. Text-only turns are unchanged (symmetric fan-out + synthesis-on-divergence). Actor selection is capability-first: the host ranks candidates by gossiped `tool_use` level (Supported > Likely > None), then model size tier, then stable order, and passes the ordering to the engine via the new `GatewayConfig.actor_candidates`. Empty ordering falls back to the engine's name-derived size tier, so existing callers and tests are unaffected. Mesh guardrails for mixed/public meshes: references are gathered with a bounded wait (proceed at a majority of advisors, never block on the slow tail), and the actor is called through the existing hedged ladder so a slow/broken best candidate falls through to the next tool-capable peer. New: - crates/mesh-mixture-of-agents/src/tool_turn.rs — asymmetric tool-turn handler - context::pack_for_actor — "advise, then act" framing (vs synthesis framing) - fanout::gather_references — bounded, tool-free reference collection - GatewayConfig.actor_candidates + reducer_candidates honours it - host compute_actor_candidates from gossiped tool_use + size The dead majority-vote code in the arbiter (now unreachable — tool turns bypass it) is removed in the follow-up commit. Tests: eval_openrouter.rs adds a live OpenRouter harness (ignored by default) that runs the real handle_turn over 6 open-weight models with mesh-realism latency/failure injection, for benchmarking asymmetric MoA vs single models. 192 engine tests pass; clippy -D warnings and fmt clean on both crates. * refactor(moa): remove dead majority tool-vote from the arbiter Tool turns now take the asymmetric actor path (previous commit), so the arbiter's tool-proposal voting is unreachable — and it was the majority-of-weakness bug: a popular-but-wrong tool choice from several weak models could outvote the one strong tool-caller that picked correctly. Removes it rather than leaving dead code: - delete best_tool_proposal, best_tool_proposal_by_consensus, and decisive_argument_cluster - drop the tool-arbitration and tool-vs-answer branches from arbitrate, and the tool-consensus branch from try_early_decision - drop the now-unused has_tools parameter from arbitrate, try_early_decision, single_output_decision, and gather_workers_incremental - the arbiter is now purely an answer/critique/uncertainty arbiter; tool-shaped text on the text path is already demoted to Uncertainty by enforce_tool_call_contract (tools disabled ⇒ empty allow-list) - drop the arbiter tests that pinned the removed tool-vote behavior; keep and re-point every answer-consensus, truncation, synthesis, and tier-gate test No behavior change for text turns. 178 engine tests pass; host-runtime lib tests (1851) pass; clippy -D warnings and fmt clean on mesh-mixture-of-agents, mesh-llm-host-runtime, and mesh-llm. * test(moa): controlled actor-ablation eval — do references actually help? Replaces the confounded "MoA vs best single model" comparison with a within-actor ablation that isolates the one variable that matters: the references. One pinned actor, identical sampling / token budget / system prompt across all arms; only the advice changes: A. actor alone (no references) B. actor + real references (the production tool path) C. actor + shuffled references (advice from a different task, length-similar) Metric: rescue (A✗→B✓) minus harm (A✓→B✗). Arm C is the key control — it separates "useful information" from "extra tokens + a think-carefully prompt". Why the change: the earlier best-single comparison ran solo models at different sampling (0.8/512) than the actor (0.3/2048) and with a different system prompt, and scored transient 429/502/504 and tool-unsupported endpoints as capability failures. Those numbers measured sampling+prompt+infra, not the design, and are not citable. This ablation removes all of that by construction: the same scorer hits all three arms of the same actor, so an imperfect label largely cancels in the rescue-minus-harm delta. Also: - chat_completion_retrying: predeclared retry on transient infra errors, which are then excluded (∅) from the capability analysis rather than scored as wrong answers. - MOA_ABLATION_ACTOR / MOA_ABLATION_DRAWS env overrides. - the old best-single test is kept but documented as a confounded smoke test, not evidence. Pilot findings (5 draws, 4 tasks, ignored/live): - Strong actor (qwen3-32b): 20/20 all arms — aces every task alone, so references are inert-but-harmless (no headroom to measure rescue). - Weak actor (qwen3-8b): A 15/20, B 20/20, C 15/20 — net uplift +5, all on the one task with headroom (triage). Real advice rescued the weak actor every draw; SHUFFLED advice did not (C=A), so the gain is advice CONTENT, not token count or the decision prompt. Zero harm. Reading: reference value tracks actor-alone headroom — a safety net when the actor is weak, dead weight (not damage) when strong. Suggests gating the reference phase on actor strength. This is a directional pilot, not the merge-blocking study (that needs ~40 preregistered stratified tasks, k>=10 draws, a paired hierarchical bootstrap CI, and the production-selected actor). * test(moa): preregistered scaled actor-ablation study + bootstrap analyzer The defensible version of the pilot, committed BEFORE running so the labels are preregistered and can't be seen to chase results. - tests/fixtures/ablation_tasks.json: 40 preregistered tasks, 4 strata x 10 (inspect / search / execute / no_tool). Set-valued accept labels (accept_tools is a SET; empty = "no tool call"), with optional arg substring constraints — addresses the brittle single-tool label from the pilot. - ablation_scaled_study: loads the fixture, runs A/B/C arms at k draws (default 10) with bounded concurrency, writes one JSONL trial per (draw,task,arm) to MOA_ABLATION_OUT. Live measurement only; no stats here. - evals/moa-openrouter/analyze_ablation.py: deterministic paired HIERARCHICAL bootstrap — resample tasks within stratum, then draws within task — for a 95% CI on net uplift = P(rescue) - P(harm), plus the shuffled-arm control (content effect = B_uplift - C_uplift). Validated on synthetic data: 3/40 rescue tasks -> +0.075 point estimate, shuffled +0.000, and the CI honestly spans 0 when rescue is sparse (no over-claiming). Arms (one pinned actor, identical sampling/prompt; only references vary): A actor alone · B actor + real references · C actor + shuffled references. Env: MOA_ABLATION_ACTOR, MOA_ABLATION_DRAWS, MOA_ABLATION_CONCURRENCY, MOA_ABLATION_OUT. Ignored by default (live network + cost). * docs(moa): trim over-long rationale comments in the asymmetric tool path Cut the essay-length doc/inline comments added with the actor design down to the non-obvious "why". No code change. - tool_turn.rs: 26-line module preamble -> 6 lines; inline comments tightened (20% -> 10% comment density) - context.rs: pack_for_actor docstring 16 -> 5 lines; reducer synthesis-framing block 14 -> 6 lines - workers.rs: compute_actor_candidates docstring 21 -> 6 lines (kept the terse sort-key comments, which are load-bearing) * test(moa): pre-hoc structured-proposal and post-hoc correction studies Two more ablation arms to answer "can peers help tool selection at all?", reusing the A/B/C JSONL schema + analyze_ablation.py. matched_peer_structured_study — do similar-strength, different-family peers help a fixed finalizer via STRUCTURED candidate tool calls (not prose)? A solo · B diverse (2 different-family peers) · C homogeneous (2 resamples of the finalizer's own model). B-C isolates cross-family diversity from extra sampling; also records oracle-union. correction_rescues_weak_tool_caller — the mesh scenario neither Hermes nor Together handles: a weak tool-caller with no strong peer. Tests correction of the CONCRETE drafted call instead of pre-hoc advice. A draft-alone · B deterministic (schema-validate + re-prompt on structural failure) · C semantic (different-family critic reviews the concrete call, finalizer revises once). Findings (live, directional): every tool-selection intervention was inert-to-harmful vs routing to a capable model. Pre-hoc structured proposals were flat (37-39/40 all arms). Deterministic correction fired ~never (qwen3-8b already drafts structurally valid calls ~95%); the residual failures are semantic (wrong tool choice), which neither validation nor a strong critic fixed because the revision still runs through the weak actor. Conclusion: tool selection is a "best capable model acts" task; MoA's value is on the answer/chat path (untested). * test(moa): committee study on realistic reasoning/answer turns Tests where MoA's value should live per Together's validated claim: open-ended answer QUALITY on realistic agent-session turns (reason-over-tool-output, planning, explanation) — not tool selection. Fixed aggregator; only its input varies: A alone · B committee (synthesize 3 diverse-peer drafts, 1 round) C layered (peers refine seeing each other first, then synthesize — Together's `layers`) Judged pairwise by an out-of-pool different-family judge (gpt-4o-mini), position-swapped (win only if consistent both orders), output lengths logged. - tests/fixtures/committee_tasks.json: 15 realistic reasoning/answer turns - committee_beats_solo_on_reasoning: writes per-trial JSONL Pilot findings (2 draws, live; DIRECTIONAL, underpowered): - committee(B) vs solo(A): win 6 / tie 2 / loss 2 — positive but sign-test p=0.29, NOT significant at n=10. Every B win was also the longer answer, so length is not cleanly ruled out. - layered(C) vs committee(B): loss 6 / tie 2 / win 2 — Together's extra refinement round is negative value on these tasks, at extra cost. - 20/30 trials skipped because the aggregator (qwen3-32b) returned empty content (reasoning-budget exhaustion — the content:null bug Hermes' troubleshooting.md documents). A flaky aggregator degrades the committee; the instrument needs empty-output handling before a real run. Net: first directionally-positive result for MoA in this investigation, but unproven. Contrast with 5 tool-selection experiments that were all null-to-harmful. Supports the task-split: route tools to one caller, convene a (single-round) committee on reasoning turns. * feat(moa): Hermes-style reference packing — strip system prompt + tool transcript Our references were packed very differently from Hermes', and it was measurably costly. Adds `pack_for_reference`, which gives advisors only the conversation's user/assistant prose: - strips the agent system prompt (an advisor told "you are a coding agent, run the tests" role-plays the actor instead of advising it) - strips the tool transcript (prior tool_calls + results anchored every advisor on the trajectory already taken, collapsing the error-independence that makes aggregation worth anything) - drops the "respond with your best answer or tool call" instruction: advisors hold no schemas, so asking for a tool call yields tool-shaped prose — exactly the advice that pulled the actor off its own better choice - uniform view across advisors (no per-role trimming), so the packing is a stable function of history and caches across iterations - caps advisor output at 600 tokens (advisor generation dominates turn latency; the turn waits for the slowest advisor) Head-to-head on the same preregistered study (strong actor qwen3-32b, 40 tasks x 10 draws, identical everything except packing): packing B pass net uplift 95% CI original 359/400 -0.102 [-0.170, -0.045] hermes 385/400 -0.037 [-0.090, -0.003] Harm cut by ~64%. Per-category: search -14 -> -2, execute -25 -> -13, inspect -2 -> 0. The content-specific component (B-C differential) fell from -0.075 to -0.015, i.e. with correct packing the residual harm is no longer mostly "bad advice content". Honest read: most of the harm I previously attributed to "references" was an artifact of how WE packed them, not a property of reference-based MoA. The direction still stands though — even correctly packed, references remain a small but statistically real regression for a STRONG actor on tool selection (CI still entirely below zero). Adds 4 unit tests pinning the packing contract (no tool-call request, no system prompt leak, no tool-transcript leak, prose preserved), and MOA_REFERENCE_PACKING=hermes to select the style in the eval harness. * docs(moa): record the MoA evidence — packing bug, actor headroom, what failed Writes up every live study in one place (evals/moa-openrouter/RESULTS.md) so the conclusions and their caveats survive the investigation. Headline 2x2 (40 preregistered tasks x 10 draws, paired hierarchical bootstrap): actor packing B pass net uplift 95% CI strong original 359/400 -0.102 [-0.170, -0.045] <- the bug strong hermes 385/400 -0.037 [-0.090, -0.003] weak original 365/400 -0.013 [-0.090, +0.070] weak hermes 377/400 +0.017 [-0.053, +0.100] Two monotonic effects: fixing the packing helps in both actor conditions, and references are worth more to a weaker actor. The only significant cell is the original-packing strong-actor harm — i.e. our bug, not a property of MoA. Per-stratum (weak + hermes) shows references help exactly where the actor has headroom (search +10, execute +4) and hurt where it was already perfect (inspect -7). That is a gating signal. Also records what did NOT work for tool selection (pre-hoc structured proposals: flat; deterministic correction: never fires, the weak actor already emits valid calls ~95%; semantic correction: negative, since the revision still runs through the weak actor), and the committee pilot on reasoning turns (directionally positive, n=10, not significant; Together's layering loses to single-round). * fix(moa): use the validated reference packing in production The Hermes-style packing was measured and unit-tested but only wired into the eval harness — the production tool path still used pack_for_worker_selected, the exact packing measured at -0.102 net uplift. tool_turn now calls pack_for_reference: conversation prose only, no agent system prompt, no tool transcript, no request for a tool call. Same head-to-head (strong actor, 40 tasks x 10 draws) puts this at -0.037 vs -0.102, and it is the only configuration where references show a positive point estimate for a weak actor (+0.017). See evals/moa-openrouter/RESULTS.md. * feat(moa): gate advisory references on actor headroom References are not universally good or bad — their value tracks how much headroom the acting model has. Measured over 40 preregistered tool tasks x 10 draws with correct advisor packing (evals/moa-openrouter/RESULTS.md): weak actor (qwen3-8b): +0.017 net uplift -> advice helps strong actor (qwen3-32b): -0.037 net uplift -> advice costs Per-stratum the split is sharper: references gained where the actor had headroom (search +10, execute +4) and lost where it was already perfect (inspect -7). Adds `GatewayConfig.reference_policy`: Auto (default) - advise a small-tier actor, let a big-tier actor act alone Always - always fan out advisors Never - actor acts alone (Hermes' `enabled: false`) A one-model pool never gathers references under any policy (nobody left to advise once the actor is excluded), and an unknown actor keeps the prior advise-by-default behaviour rather than silently degrading. The host sets `Auto`. sim_real_traces pins `Always` because that suite exists to verify fan-out accounting; the gate itself is covered by six new unit tests in tool_turn. 189 tests pass; clippy -D warnings and fmt clean on mesh-mixture-of-agents and mesh-llm-host-runtime. * test(moa): scale the committee study — 40 preregistered prompts, concurrent, empty-output fix Prepares the reasoning-turn committee study for a defensible run. Committed before executing so the prompts are preregistered. Three fixes to the pilot's weaknesses: 1. Empty-output bug. `response_text` read only `/message/content`, so a reasoning model that spends its budget in `reasoning` and returns `content: null` looked like an empty answer — that silently dropped 20/30 pilot trials. Now falls back to `reasoning`, mirroring the in-tree backend's chain. Remaining skips are counted and reported rather than hidden. 2. Serial execution. ~18 calls x 30 trials was far too slow to scale. Trials are independent, so they now run under a bounded semaphore (MOA_COMMITTEE_CONCURRENCY, default 4), with peer drafts and refinements gathered concurrently within each trial. Output is sorted for stable reporting. 3. Fixture too small. 15 -> 40 prompts, 4 strata x 10: reason_over_output, planning, explain, code_review. All realistic agent-session reasoning/answer turns (several embed simulated tool output or code, as an agent loop would). Arms and controls are unchanged: fixed aggregator, A alone / B committee / C layered, judged pairwise by an out-of-pool different-family judge with position swap, output lengths logged for the length-vs-preference check. * docs(moa): committee wins decisively on reasoning turns (120 trials, p<1e-11) Scaled the reasoning-turn committee study from the 15-prompt pilot to 40 preregistered prompts (4 strata x 10) x 3 draws = 120 trials, 0 skipped. comparison win/tie/loss mean 95% CI sign p committee(B) vs solo(A) 86/16/18 +0.567 [+0.392,+0.733] 8.2e-12 layered(C) vs solo(A) 90/11/19 +0.592 [+0.408,+0.758] 3.1e-12 layered(C) vs committee(B) 57/30/33 +0.200 [+0.008,+0.392] 1.5e-02 Consistent across every stratum. The length confound runs the OPPOSITE way and strengthens the result: the committee's answers are SHORTER than solo (2548 vs 3136 chars) and still win; restricted to the 61 trials where B was shorter, B wins 40-14 (p=5.4e-4). This reverses two pilot conclusions, both of which were artifacts of the empty-output bug that silently dropped 20/30 pilot trials (biased toward exactly the cases where the aggregator struggled): - "committee is directionally positive but not significant" -> it is significant by a wide margin; - "Together's layering is negative value, don't build it" -> RETRACTED; layered beats solo about as strongly as single-round, and edges single-round itself (though that is the weakest result and costs an extra peer round). Net across the whole investigation: tool selection shows no win for multi-model (route to the best tool-caller); reasoning/answer turns show a clear one (convene the committee). That task split is now evidence-backed rather than assumed. * feat(moa): cross-peer refinement round — a small-model mesh beats its best member The mesh value proposition, measured: a pool of four 8B-class models (qwen3-8b aggregating llama-3.1-8b + granite-4.1-8b + ministral-8b) produces better answers than the best single member — but ONLY with a refinement round. 40 preregistered reasoning prompts x 3 draws = 120 trials, judged pairwise by an out-of-pool different-family judge (gpt-4o-mini), position-swapped: small pool (all 8B): committee (1 round) vs solo 26/75/19 +0.058 p=0.37 ns layered (2 rounds) vs solo 42/66/12 +0.250 p=5.2e-05 * layered vs committee 39/73/ 8 +0.258 p=5.5e-06 * Single-round synthesis does nothing for a small pool; the cross-peer refinement round is what creates the gain. With a 32B aggregator both work and the extra round adds much less (p=0.015), so it isn't worth a second fan-out there. Adds `refinement::refine_round`: every worker sees all round-1 drafts (anonymized, length-bounded) and rewrites its own answer, then the reducer synthesizes the refined set. Runs on the text path only, and only when we were going to synthesize anyway — an early-exit consensus already has its answer. `GatewayConfig.refinement_policy`: Auto (default) - refine when the pool is all small-tier Always / Never - explicit override Mesh flavour: the round is best-effort. It bounds its own wait at worker_timeout, refines with whichever drafts arrived, tolerates workers that fail or time out, and returns the round-1 outputs unchanged on any shortfall — so a slow or absent peer can never cost the turn. This also retracts, for the second time, the pilot claim that "Together's layering is negative value". It is essential for exactly the pool shape a consumer mesh has. 194 tests pass; clippy -D warnings and fmt clean on both crates. * test(moa): pin refinement behaviour under mesh conditions; record small-pool evidence Refinement runs on exactly the hardware where peers are slowest and least reliable, and a second fan-out is a second chance to hang a turn. Four deterministic sim tests pin it as strictly best-effort: - refined drafts (not round-1) are what reach the client - a refiner that never returns cannot extend the turn past worker_timeout - every refiner failing degrades to the round-1 drafts, turn still answers - a big-tier pool skips the extra fan-out entirely under Auto Two of these failed first and both were my test's fault, not the engine's: identical round-1 text clustered into consensus so the turn early-exited before synthesis (peers now return distinct answers), and the "reducer must synthesize" assertion was too narrow — when refined drafts agree, the arbiter takes consensus and returns refined text directly, which is equally correct. The contract pinned is now "refined text reaches the client, round-1 text does not". RESULTS.md records the small-pool study that motivated the feature: an all-8B diverse pool (qwen3-8b aggregating llama-3.1-8b + granite-4.1-8b + ministral-8b) beats its best member only with the refinement round — single-round 26/75/19 (p=0.37, ns) vs layered 42/66/12 (p=5.2e-05), with layered over single-round at p=5.5e-06. Includes the honest caveats: ties dominate for the small pool, and unlike the strong-pool result the length control is not clean (answers are longer than solo). * fix(moa): bound the refinement round to half the worker budget Refinement sits between round 1 and the reducer, so giving it the full `worker_timeout` let one text turn pay three sequential worker budgets — worst on exactly the slow, high-latency meshes the feature exists to serve. The round now gets half the budget: enough for a pool that answered round 1 promptly, and it falls back to the round-1 drafts if the pool can't refine in time. Worst case drops from ~3x a plain turn to ~2.5x. * feat(moa): shorter patience on public meshes A public mesh is a pathological availability case — unknown peers, wider latency spread, more churn — not a trust case. Both patience knobs are "hold a usable answer hoping for a better one", which is exactly the wait that hurts when the tail is long. knob private public first_answer_grace 3s 1.5s strong_patience 20s 8s Hard bounds (worker_timeout, reducer_timeout) are unchanged; only the optional waiting shrinks, so quality paths still run when peers are prompt. Sourced from the node's existing `public_mesh` flag — no new config. Three tests pin the contract: public waits less on both knobs, public still gives a prompt strong peer a real chance (grace non-zero, patience >= 5s), and the private defaults don't drift silently. 1854 host-runtime tests pass; clippy -D warnings and fmt clean. * test(moa): prove refinement runs under the production grace setting Every refinement test so far set `first_answer_grace: ZERO`, but production uses 3s (1.5s public). Grace produces an `early_decision`, and refinement is skipped whenever one exists — so if grace fired on a fast all-small pool the whole feature would have been dead code on real traffic. That was untested. Two tests close the gap: - `refinement_still_runs_under_production_grace` — at the real 3s default, a prompt all-small pool still refines and the refined drafts reach the client. Grace only arms once its window has elapsed, so a pool that answers promptly reaches synthesis first. - `grace_shipping_a_slow_answer_skips_refinement` — pins the opposite case honestly: when round-1 answers straggle past the grace window, grace ships an early answer and refinement does not run. That tradeoff is now visible in a test rather than assumed. 195 engine tests pass; clippy -D warnings and fmt clean. * fix(moa): straggling peers must not cost the refinement round The answer grace was silently disabling the feature that makes a small mesh worth running. Grace fires at 3s (1.5s public) and produces an `early_decision`; refinement is skipped whenever one exists. On an all-small pool a single synthesis round is worth nothing (26/75/19, p=0.37) — refinement is the only step that beats the best member (42/66/12, p=5.2e-05). So one fast peer answering while others straggle would ship a lone small-model answer and skip the round, while still paying for the fan-out. Variable peer latency is the norm on consumer hardware, so this hit exactly the case the feature exists for. Now: when refinement is expected (policy + pool shape, decidable before dispatch via `refinement_expected`), the answer grace is disabled for that turn. Round 1 is still bounded by `worker_timeout` and refinement by its own half-budget deadline, so this costs bounded latency, never an unbounded wait. Pools that don't refine — anything with a big-tier model — keep the fast chat path untouched. Found by re-reading my own test against the goal: the previous `grace_shipping_a_slow_answer_skips_refinement` pinned this as an acceptable tradeoff. It wasn't; it was the feature not firing. Replaced with `straggling_peers_do_not_cost_the_refinement_round`, plus `grace_still_short_circuits_when_refinement_is_not_expected` to pin that the fast path survives where refinement is off. 202 engine tests + 1854 host tests pass; clippy -D warnings and fmt clean. * fix(moa): grace must bound the wait, not forfeit the refinement round On an all-small pool the answer grace could ship a lone fast answer and skip refinement — the only step that makes such a pool beat its best member (26/75/19, p=0.37 without it vs 42/66/12 with). Variable peer latency is the norm on consumer hardware, so this silently disabled the feature exactly where it exists to help, while still paying for the fan-out. Fix: `GatherPolicy.min_grace_answers`. Normally 1 (ship the good answer, stop waiting for the tail). When refinement is expected it becomes MIN_DRAFTS, so grace still caps the round-1 wait but will not finalize before refinement is even possible. Two earlier attempts were wrong and are worth recording: - Disabling grace outright for all-small pools removed the only straggler bound on round 1 (worst case round1 + refine + reducer), which is backwards on a public mesh whose whole profile is "wait less". - Letting refinement override an existing `early_decision` broke `same_tier_pool_keeps_early_exit`, and rightly: that decision is consensus (workers agreed — a real signal and the cheap path), not a timeout. Grace and consensus both surface as `early_decision` but mean opposite things; only the timeout should be prevented from pre-empting refinement. 202 engine tests + 1854 host tests pass; clippy -D warnings and fmt clean. * fix(moa): let the reducer and refiners actually see the answers Eval-vs-production fidelity gap, same class as the reference-packing bug. The small-pool result (42/66/12, p=5.2e-05) was measured on untruncated drafts averaging ~3.8k chars. Production truncated much harder: refinement input 1200 chars -> refiners saw ~30% of each peer draft reducer payload 500 chars -> reducer saw ~13% of each refined answer So the shipped path discarded most of exactly the content the refinement round produces. The measured gain could not have survived it. Both budgets are now 4000 chars, which passes realistic answers intact while still bounding a pathological worker. Tool turns keep the tight 500-char bound on purpose: there the signal is the proposal itself, and long prose crowds out the tool schemas. Three tests pin the contract (realistic text answers survive, tool payloads stay tight, refiners see full peer drafts), and the existing `reducer_truncates_long_worker_payloads` now probes above the new bound so it still guards the pathological case it was written for. 205 engine tests pass; clippy -D warnings and fmt clean. * fix(moa): ship the refinement prompt that was actually measured Final eval-vs-production fidelity check. Sampling already matched exactly (SamplingParams::worker, thinking off, 1024 tokens), but the prompt did not: measured (eval) "You have been given a user request and several candidate responses... Synthesize them into one high-quality response..." + "Candidate responses:" + [Response N] shipped (before) "Several models independently answered... write a better answer of your own..." + [Answer N] The eval wording is also what Together's advanced-moa.py uses — it reuses the aggregator prompt for refinement layers. A refinement-specific wording may well read better, but the eval wording is the one with evidence behind it, and "surely equivalent" prompt differences have already caused two wrong conclusions in this branch (the reference-packing bug and the layering dismissal). Changing it should be a measured change, not an assumed improvement. Production now matches the measured configuration exactly: prompt text, header, and per-draft labels. The sim fixture detected refinement calls by matching the old prompt string, so it needed updating too — it now keys on the "Candidate responses:" header. 205 engine tests pass; clippy -D warnings and fmt clean. * docs(moa): record the eval-vs-production fidelity gaps and their fixes A measured gain only counts if the shipped path reproduces the measured configuration. Three gaps were found after the numbers were collected: refinement input truncated to 1200 chars (~30% of a real draft), reducer payload to 500 (~13%), and a different refinement prompt than the one measured. All closed; tool turns keep the tight 500-char bound on purpose. Also states the honest limit: production now matches the measured configuration, but the small-pool result has not been re-measured through the shipped path since. The gain should carry — that is an expectation, not an observation. * fix(guardrails): panic slicing tool-call text with a multi-byte prefix `parse_parenthesized_tool_call` resumed parsing at `after_open[json_text.len()..]`, but `first_balanced_object` returns the object starting at its `{` — so any characters before the brace skew the index. With a multi-byte character in that prefix the index lands mid-character and slicing panics. Hit for real during an end-to-end MoA eval against live models: "start byte index 13 is not a char boundary; it is inside '‑'" (U+2011, non-breaking hyphen) — a worker task panic that would have taken down the turn. Adds `first_balanced_object_span` returning the byte span, so callers resume at the object's end rather than guessing from its length. Three regression tests: multi-byte prefix must not panic, happy path still parses, multi-byte content inside the arguments round-trips. 19 guardrails tests pass; clippy -D warnings and fmt clean. * fix(moa): stop telling the text reducer to "Be concise" End-to-end through `handle_turn`, MoA lost 7/8/65 to a single small model. Two causes; this is the second. The text reducer's instruction framed the turn as reconciling a disagreement and ended with "Be concise". Measured output: ~2.0k chars against a ~4.1k-char solo baseline — and the judge preferred the fuller answer. Terseness is not the goal on a reasoning turn; accuracy and completeness are. Text turns now use the wording the committee study actually measured (evals/moa-openrouter/RESULTS.md): "produce the most accurate, well-structured reply. Be direct." Tool turns keep the tight framing — there the output is an action and the reducer must stay free to emit a tool call. Also adds the e2e harness (`e2e_handle_turn_beats_best_single_small_model`) that found this: it drives the real `moa::handle_turn` over a live all-8B pool and judges it against the pool's strongest member. The committee study measures the mechanism with its own helpers; this measures the shipped path, which is where all three fidelity bugs actually hid. 205 engine tests pass; clippy -D warnings and fmt clean. * fix(moa): give every peer the full budget when refinement is expected Third and last cause of the end-to-end loss. Role tiers (Fast 256 tokens, Specialist 512, Strong 1024) encode a capability spread so the cheap worker can answer the grace path quickly. A homogeneous all-8B pool has no such spread, and when refinement is expected every draft is an input to the round — a 256-token draft is a ~1000-char stub that drags the refined answer down. Measured: production's tiered budgets produced ~3.1k-char answers against a ~4.1k-char solo baseline. The study that showed the +0.250 gain gave every peer the full 1024. When refinement is expected, all workers now pack as Generalist (full budget). Role identity is unchanged, so tier-gating and accounting still see the real roles; only the packing budget is uniform. 205 engine tests pass; clippy -D warnings and fmt clean. * fix(moa): text workers get a text preamble, not the tool-turn one Fourth divergence between the measured study and the shipped path. Every production worker was prepended MOA_PREAMBLE: "Respond with your best answer or tool call. Be direct." On a text turn that is wrong twice: there is no tool to call, and "be direct" pushes workers toward stubs. These drafts are the input to the refinement round, so brevity compounds — measured end-to-end, MoA answered ~3.3k chars against a ~4.1k-char solo baseline and lost on judged quality. The study that showed the gain gave workers no such instruction. Text turns now get a preamble that says the parts will be combined and asks for the most accurate and complete answer. Tool turns keep the original wording. 205 engine tests pass; clippy -D warnings and fmt clean. * fix(moa): don't give the reducer the worker preamble Fifth divergence between the measured study and the shipped path. The reducer's system prompt began with the worker preamble — "multiple models are answering this in parallel, give your most accurate answer" — and then immediately contradicted it with "you have been given several candidate responses; synthesize them". The combiner was being told it is also one of the answerers. The measured study gave the synthesizer the synthesis framing alone. `reducer_system_prompt` now passes the agent's own system prompt (tool guidance stripped on text turns) without the worker preamble. 205 engine tests pass; clippy -D warnings and fmt clean. * Revert "fix(moa): don't give the reducer the worker preamble" This reverts commit fb1281a0. Measured, the change made things worse, not better: with preamble (17/39/24) -> without (15/28/37) MoA mean output 3697 chars -> 3534 chars My reasoning was that the worker preamble ("multiple models are answering in parallel") gave the reducer the wrong role and contradicted the synthesis instruction. The e2e says otherwise: that preamble also says "the best parts of each will be combined; give your most accurate and complete answer", and the reducer was evidently using it to produce fuller answers. Removing it shortened the output and lost ground. Fifth hypothesis, first one the measurement rejected. Keeping the behaviour the evidence supports. * fix(eval): the e2e judge was scoring length, not quality Analysis of the 80-trial e2e run, conditioned on relative output length: MoA LONGER than solo: n=25 win 13 tie 12 loss 0 (100% winrate, p=2e-4) MoA SHORTER than solo: n=55 win 4 tie 27 loss 24 ( 14% winrate, p=2e-4) point-biserial r(length delta, verdict) = +0.681 mean delta when MoA wins: +682 chars; when it loses: -1337 chars Zero losses whenever MoA was longer. The judge prompt asked for "accurate, complete, and useful" — "complete" reads as "longer" — so the metric was verbosity, not quality. This is the exact bias the committee study controls for (position swap + length reporting) and that I failed to carry into the e2e harness. Consequence: "MoA loses 17/39/24 end-to-end" is NOT a valid quality finding, and four of the five fixes measured against it were partly optimising toward a biased judge. The correctness fixes stand on their own merits (a real panic, truncation destroying content, prompts contradicting the evidence); the scoreboard they were measured against did not. Judge now scores correctness/relevance only and is explicitly told length is not quality. * docs(moa): withdraw the length-biased numbers, record the corrected ones The judge that produced the small-pool and e2e results scored length, not quality (r = +0.681 between length delta and verdict; the longer answer won 13-0). Re-run with a length-controlled judge, r fell to +0.132. Corrected small-pool result (n=80, length-controlled): committee (1 round) vs solo 6 / 73 / 1 p = 0.125 ns layered (2 rounds) vs solo 11 / 68 / 1 p = 0.0063 significant layered vs committee 3 / 77 / 0 p = 0.25 ns The core claim survives: a small-model mesh beats its best member, but only with the refinement round. The magnitude does not — 42/66/12 is withdrawn, most of those "wins" were length artifacts and are now ties. Ties dominate (68/80): on most prompts a small mesh and a single small model are indistinguishable. The strong-pool section keeps its numbers and gains a note explaining why: the bias ran against the winner there (the committee's answers were shorter and won anyway, 40-14 on the shorter-only subset), so that result is unaffected. Status now records end-to-end parity (5/65/10, p=0.30) rather than the earlier loss, and lists closing the harness-vs-production gap as outstanding. * fix(moa): commit the grace_finalizes field definition HEAD did not compile. `lib.rs` set `GatherPolicy.grace_finalizes` (committed in "grace must bound the wait, not forfeit the refinement round") but the field definition in `fanout.rs` was never staged — it sat unstaged in my working tree across several commits, so every local build passed while the pushed tree was broken. Adds the field, its use in the gather loop (grace expiry stops collecting but does not finalize when refinement is expected), and `grace_finalizes: true` at the eight fanout test sites. Verified against a clean worktree checkout of HEAD, which is how the breakage was found: `cargo build -p mesh-mixture-of-agents` now succeeds there. * fix(moa): anonymize reducer inputs on text turns Sixth divergence between the measured study and the shipped path. Production labelled reducer inputs "[Worker N — model_name]"; the study that measured the gain used anonymous "[Response N]". Hermes anonymizes reference outputs for the same reason — a named model invites the aggregator to defer to the name rather than the content. Text turns now use "[Response N]". Tool turns keep attribution: there the reducer arbitrates between proposals, provenance is genuinely useful, and the tool-path tests pin it. The "## Worker outputs" header is unchanged — two sim fixtures use it to identify reducer calls. 205 engine tests pass; clippy -D warnings and fmt clean. * docs(moa): record that anonymization did not close the harness gap Sixth divergence fixed (named -> anonymous reducer inputs) and measured: before 5 / 65 / 10 p=0.30 after 9 / 59 / 12 p=0.66 Decided trials went 5W/10L -> 9W/12L. That is not an improvement at this sample size; both runs are parity. Recorded as such rather than as progress. Also records the gap that remains unexplained — harness 11/68/1 (p=0.0063) vs shipped 9/59/12 (p=0.66) on the same models, prompts, judge and packing — and lists the candidates not yet ruled out. Adds a caution that r(length,verdict) was +0.465 in the latest e2e run vs +0.132 in the small-pool study, so length bias is not fully suppressed even with the corrected judge; single-run e2e deltas are directional only. * fix(moa): text reducer gets only the synthesis framing Seventh divergence between the measured study and the shipped path, and the last known one in the reducer prompt. Production prepended two things the harness never had: * the worker preamble ("several models are answering this in parallel") — the reducer is the combiner, not one of the answerers; * "Reason for synthesis: {reason}" — an internal arbiter string such as "3 workers answered with no agreement", which reads to the model as a warning that its inputs are unreliable. Text turns now get exactly what the study measured: the synthesis instruction plus the candidate answers. Tool turns keep both extras — there the reducer is arbitrating a real disagreement, the reason is informative, and the agent's tool guidance matters. Note: an earlier commit removed only the preamble and appeared to make things worse, but that was measured with the length-biased judge, so the result was invalid and was reverted. This re-tests both extras together under the corrected judge. 205 engine tests pass; clippy -D warnings and fmt clean. * Revert "fix(moa): text reducer gets only the synthesis framing" This reverts commit 031a349a. Measurement rejected it, and this time the finding replicates. v7 preamble + reason kept 5/65/10 decided 33% MoA len 3606 v8 + anonymized inputs 9/59/12 decided 43% MoA len 3679 v9 - preamble - reason 8/57/15 decided 35% MoA len 3314 Removing the worker preamble shortened MoA's output on both occasions it was tried (v6 3534, v9 3314) versus keeping it (v7 3606, v8 3679), and lost ground on decided trials both times. The first rejection was under the length-biased judge so it did not count; this one is under the corrected judge and agrees. Reading: the preamble ("the best parts of each will be combined; give your most accurate and complete answer") is doing useful work on the reducer even though it is nominally addressed to a worker. Matching the harness exactly is not automatically right — the harness had no system prompt at all, while production has one, and the preamble evidently compensates. Second hypothesis rejected by measurement rather than argument. Keeping the behaviour the evidence supports. * docs(moa): record the two rejected hypotheses for the harness gap Both prompt-level explanations for the harness-vs-shipped gap were tested and rejected: baseline (v7) decided 5/15 = 33% MoA len 3606 anonymize reducer inputs (v8) decided 9/21 = 43% MoA len 3679 also drop preamble + reason (v9) decided 8/23 = 35% MoA len 3314 Dropping the worker preamble shortened output and lost ground both times it was tried (v6, v9) and was reverted both times. The preamble does useful work on the reducer even though it is nominally addressed to a worker — so matching the harness exactly is not automatically right: the harness sent no system prompt at all, production sends one, and the preamble compensates. Anonymization is retained (Hermes does it, the study measured it, it did not hurt) but 43% vs 33% on ~20 decided trials is not a result. Notes what remains untested (arbiter short-circuit on converged drafts, normalize_worker_output reshaping prose) and the honest conclusion: after two rejected guesses, the gap needs a byte-level diff of the prompts each path actually sends, not more hypotheses. * test(moa): diagnostic that dumps every prompt handle_turn actually sends Two prompt-level hypotheses for the harness-vs-shipped gap were tested live and rejected, so rather than guess a third this captures what production actually sends on a text turn — every role/content pair, in order. Deterministic and offline (canned backends, no network). What it shows for a 4-model all-small pool: calls 1-4 round-1 workers — text preamble + user prompt, max_tokens=1024 calls 5-8 refinement round — byte-identical to the harness's refine() call 9 reducer — harness synthesis prompt, plus the preamble, a "Reason for synthesis" line and a "## Worker outputs" header So the refinement round now matches the harness exactly, and the round-1 and reducer calls differ only by the preamble and the reducer scaffolding — both of which were removed in an experiment and measured worse (v9), then reverted. This is the instrument for the next person: run it, diff against the harness block it prints alongside, and the remaining delta is visible without guessing. 206 engine tests pass; clippy -D warnings and fmt clean. * fix(moa): refine over all drafts, not the bare minimum Eighth eval-vs-production divergence, found by reading the gather policy rather than guessing at prompts. `min_grace_answers` was set to MIN_DRAFTS (2) — the minimum for the refinement round to run. On a 4-model pool that let grace stop collecting at 2 of 4 drafts, so production refined over half the perspectives while the study that measured the gain always refined over all of them. Refinement quality scales with how many drafts it sees, so this directly undercuts the mechanism. Now waits for all but one draft (floored at MIN_DRAFTS), so a single straggler still cannot hold the turn and `worker_timeout` still bounds the wait. 206 engine tests pass; clippy -D warnings and fmt clean. * fix(moa): always synthesize after a refinement round Ninth divergence, and the first one found from outcome data rather than by reading code. Pooling the e2e runs showed the shipped path wins at roughly the harness rate (22/240 vs 11/80) but loses 32 times where the harness loses once. Splitting those turns by whether the reducer actually ran located the asymmetry: reducer ran n=66-74 win 8-9 loss 6-10 loss rate 9-14% reducer skipped n= 6-14 win 0 loss 2- 4 loss rate 29-33% MoA won 0 of 20 turns where synthesis was skipped, at triple the loss rate. That is a pure loss mode. Cause: `arbitrate` returns `Answer(payload)` when drafts agree, shipping one worker's text verbatim. That is the right cheap path for raw round-1 answers, but after refinement it discards the round we just paid for and returns whichever single small model represented the cluster. Refined drafts agreeing is not a reason to skip synthesis — it is the best possible input to it. Text turns that refined now always route to the reducer. Three sim assertions updated: the client now sees the reducer's synthesis of refined drafts rather than raw refined text. 206 engine tests pass; clippy -D warnings and fmt clean. * docs(moa): N=2 is not enough — pool-size comparison Direct test of "do 2 different models beat 1 of the same strength?", same 40 prompts / same judge / same harness, only pool size differs: N=2 (qwen3-8b + llama-3.1-8b) 2W 75T 3L p=1.00 5 decided N=4 (+ granite-4.1-8b, ministral) 11W 68T 1L p=0.0063 12 decided Fisher exact on decided win/loss, N=2 vs N=4: p=0.053. At 8B scale two peers produced no measurable gain — 2 wins against 3 losses. The same code with four models across four families wins 11-1. Two quantities move together as peers are added: decided trials rise (5 -> 12; more peers produce differentiated output instead of near-identical answers the judge ties) and the win share among them rises (40% -> 92%). Caveats recorded: 5 decided trials is weak evidence of absence, not evidence of no effect; and this is a claim about small models — Hermes reports a two-model frontier preset beating its stronger member by ~6 points, which is untested here. * docs(moa): N=2 null was scale, not count — mid-scale N=2 wins cleanly Mid-scale re-run refutes the "two peers isn't enough" reading. Same prompts, judge and shipped path, layered arm: N=2 8B (qwen3-8b + llama-3.1-8b) 2W 75T 3L p=1.00 N=4 8B (+ granite, ministral) 11W 68T 1L p=0.006 N=2 mid (qwen3-32b + mistral-24b) 49W 24T 6L p=2e-9 length r=-0.04 N=4 str (32b agg + 14b/24b/minimax) 90W 11T 19L p=3e-12 length r=+0.30 The N=2-mid result is the study's cleanest: no length confound, MoA answers shorter than solo, still won 93% of shorter-MoA trials. So member capability, not raw count, is the active variable — 8B needs ~4 models, 24-32B needs 2. Records the deciding caveat: this shows ensembling beats the best single member, not that cross-model diversity causes it — the baseline is strength-matched, not compute-matched. A compute-matched Self-MoA arm (two samples of the same model) is running to settle it; if it ties Mixed, the claim becomes "test-time ensembling beats one draw". * docs(moa): Self-MoA settles it — ensembling, not diversity, is the mechanism Compute-matched test (both spend 2 drafts + refine + synthesize; only member identity differs): Mixed (qwen3-32b + mistral-24b) 49W 24T 6L p=1.8e-9 length r=-0.04 Self (qwen3-32b x2) 48W 23T 2L p=2.3e-12 length r=-0.04 Mixed vs Self: Fisher exact p=0.27 — indistinguishable. Both crush the single best member, both confound-free (MoA answers shorter than solo, ~93% winrate on shorter-MoA trials). Different-family membership is not required. The active ingredient is test-time ensembling — several sampled drafts (temp 0.8), a refinement round, synthesis — and it works from repeated samples of one model as well as from different ones. Matches the Self-MoA paper (arXiv:2502.00674). Practical: the mesh does not need a curated diverse pool; any >=2 sufficiently capable participants beat picking one, distinct models or repeated instances. * docs(moa): the 8B ladder — need ~4, and same-vs-different doesn't matter Same prompts/judge/harness, layered arm: 2x 8B different (Qwen+Meta) 2W 75T 3L p=1.00 2x 8B same (Qwen x2) 2W 78T 0L p=0.50 3x 8B different 3W 76T 1L p=0.63 4x 8B different 11W 68T 1L p=0.006 Two 8B instances of the SAME model do exactly what two different ones do (nothing) — confirming count/strength drive the result, not family identity. Four 8B models is where it starts winning. Open gap: 4x same-model 8B not run, so whether the 8B N=4 win needs distinct models or just four drafts is extrapolated (mid-scale Self-MoA says four drafts alone should suffice). * fix(moa): refine on correlated drafts (homogeneous pools), not just small ones The refinement gate keyed on tier: `Auto` refined only when every model was small, and skipped whenever a big-tier model was present. That is wrong for the case the mesh goal cares about — a pool of one model repeated across nodes. Measured (evals/moa-openrouter/RESULTS.md), refinement's value tracks draft correlation, not size: same-model 32B x2: 48/2 with refinement vs 35/10 without (decisive) diverse mid 32B+24B: 49/6 layered vs 47/4 single (~no gain) all-small 8B x4: needs the round to beat its best member So `Auto` now refines when the pool is homogeneous (same canonical base — the repeated-instance / quant case) OR all small-tier, and skips the extra fan-out only for a diverse pool that already has a big-tier synthesizer. Adds `worker::canonical_base_name` (mirrors the host's dedup) and `pool_is_homogeneous`, with unit tests; refinement tests updated to pin the new homogeneous-big-tier behaviour. 212 engine tests pass; clippy -D warnings and fmt clean. * feat(moa): admission control — weak workers don't join a strong pool A modest node must not drag down a committee that already has a stronger member. Aggregation quality tracks proposal quality (Self-MoA, arXiv:2502.00674), and an 8B draft added to a 24-32B pool is expected noise-to-harm. `build_moa_config` now applies admission control before the >=2 check: when the pool mixes tiers, small-tier workers are dropped and only the big-tier ones kept. Consequences: * mixed pool (32B + 8B) -> 8B excluded, 32B committee (or solo if alone) * all-small (8B x4) -> untouched (the case that wins) * all-big / homogeneous -> untouched A lone strong model after exclusion falls through the >=2 check and simply serves solo — the safe outcome. `backends`/`models` are parallel vecs linked by backend_index, so both are rebuilt and reindexed together. Four unit tests pin the matrix (drops small when big present, keeps all-small, keeps all-big, keeps homogeneous-big). 68 moa_gateway tests pass; clippy -D warnings and fmt clean. * feat(moa): same-model mesh forms a pool (self-fill from extra instances) A mesh where every node serves the same model got no MoA at all: build_moa_config resolves one worker per canonical base, so N same-model nodes collapsed to 1 worker and failed the >=2 check. Self-MoA (arXiv:2502.00674) shows repeated sampling of one model ensembles as well as different models, so this is exactly the "add a modest node and it helps" case that was missing. When distinct-model resolution leaves a single model, `self_fill_from_extra_ instances` now adds extra reachable nodes serving that same model as workers (distinct remote endpoints only, never the same node twice, capped at 2). So: 1 node, 1 model -> 1 worker -> solo (unchanged, no surprise cost) 2 nodes, same model -> 2 workers -> MoA committee Combined with the refinement-gate fix (homogeneous pools now refine), a same-model pool runs the layered path that measured 48/2 for 32B x2. Extracted pool assembly (resolve loop + admission + self-fill) into `assemble_worker_pool` to keep build_moa_config under the cognitive-complexity limit. 68 moa_gateway tests pass; clippy -D warnings and fmt clean. * chore(moa): drop diag_prompt_diff scratch test It was an assertion-free diagnostic that dumps prompts to stdout — useful during the harness-vs-shipped investigation, but it always passes and does not belong in the merge set. The prompts it printed are now understood and the findings are recorded in evals/moa-openrouter/RESULTS.md. * docs(moa): admission control validated — weak node in strong pool is no-upside Arm C (32B x2 + one 8B, weak node admitted) vs arm B (32B x2): B 48W 23T 2L p=2e-12 decided winrate 96% C 50W 25T 5L p=2e-10 decided winrate 91% Fisher B-vs-C p=0.44 (not separable at n=80) but one-way: admitting the weak node never helped and raised losses 2 -> 5. Both still beat solo, so a weak node doesn't collapse the pool, but it adds cost for no upside and a small tail risk. This is the measured basis for tier-based apply_admission_control. * fix(moa): admission control must not collapse a lone-strong pool to solo You asked the right question: we showed admitting a weak node HURTS when the pool already has two strong (arm C), but never tested where admitting HELPS. The missing cell — one strong + one weak — is decisive and overturns the first-cut rule: 32B + 8B layered vs solo 32B: 47W 27T 5L p=1.3e-9 length-clean (r=-0.01) Admission control rejected the 8B here, collapsing the pool to a solo 32B. But the mixed committee beats solo decisively, so rejecting it threw away MoA in exactly the core "modest node joins a strong node and helps" case. Rule corrected: drop small-tier workers only when >=2 big-tier remain (a real committee survives). If dropping would collapse to solo, keep the mix. 32B x2 + 8B -> drop 8B (committee survives; 8B adds nothing, arm C) 32B + 8B -> keep both (dropping = solo, and the mix wins 47/5) Two admission tests updated/added to pin both cases. 69 moa_gateway tests pass; clippy -D warnings and fmt clean. * feat(moa): model=mesh gracefully degrades to single-model instead of 503 Before: a `model=mesh` request on a lone node (or a mesh with <2 workers) returned 503 "MoA requires ≥2 models". That makes `mesh` unusable as a default model — it only worked once a committee could form. Now: when `build_moa_config` can't form a committee but a real model is available, `try_handle_moa` rewrites `model=mesh` to that model and hands the stream back to normal single-model routing. `mesh` works everywhere — passthrough on one node, committee once a second worker joins. Only a node with no model at all still 503s. Both call sites handle the new fall-through: - passive/transport path already routes a returned `Some(stream)` normally; - host/ingress path: `try_handle_moa_intercept` gained a `Degraded { stream, model }` result. The pre-computed `decision.effective_model` is stale ("mesh") after degradation, so the degraded model name is threaded into `route_request`, and the pipeline classifier (computed against "mesh") is skipped for degraded turns. Unchanged from main was the hard 503; this is the missing piece for `mesh` as a superior always-on drop-in. 1859 host tests pass; clippy -D warnings and fmt clean. * feat(moa): cap the committee at 4 workers on large meshes Fan-out cost is ~2N+1 model calls per turn (N drafts + N refines + 1 synthesis), and measured quality is flat past ~4 workers while latency and spend keep climbing. On a big shared mesh — say 20 nodes — an uncapped pool fanned out to all of them: 41 calls for no quality gain. `cap_committee` now trims the assembled pool to the best MAX_COMMITTEE_WORKERS (4) by the same capability ranking used for actor selection (gossiped tool_use, then size, then stable index). Nodes beyond the cap are standbys — they still serve direct traffic, just not this committee. Runs after admission control and self-fill, so the cap applies to the final worker set. Extracted as a helper to keep assemble_worker_pool under the cognitive-complexity limit. 69 moa_gateway tests pass; clippy -D warnings and fmt clean. * test(moa): pin partial-worker-survival robustness sim_all_workers_fail covers total failure (clean structured error). This covers the common mesh reality: nodes flicker, so a subset of the committee dies mid-turn while the rest answer. MoA must degrade to survivors, not the error path. Three cases pinned: - half the committee dies, rest answer -> turn completes, all 4 accounted (2 succeeded, 2 recorded failed, none silently dropped) - lone survivor (3 of 4 die) -> turn still answers - slow-failing worker -> does not stall the turn past the survivors Locks the pre-existing best-effort fanout behaviour against regression now that assembly (admission, self-fill, cap) is more complex. 214 moa tests pass; clippy -D warnings and fmt clean. * refactor(moa): split pool assembly out of workers.rs into pool.rs workers.rs had grown to 1215 lines (over the 1k guideline) mixing two responsibilities: worker-pool assembly/selection and the model backends + config orchestration. Per CodeRabbit review, extract the assembly half. pool.rs (658 lines) now owns: assemble_worker_pool, resolve_one_worker_from_ aliases, add_worker_backend, group_aliases_by_canonical_base, is_locally_served, apply_admission_control, self_fill_from_extra_instances, cap_committee, compute_actor_candidates, canonical_base_name, WorkerBackendResolution, the two size caps, and their tests. workers.rs (578 lines) keeps: build_moa_config (orchestrator), the thinking-override helpers, patience_profile, the Local/Remote model backends + parse_quic_http_response, and their tests. Pure mechanical move, no behavior change. Backends and canonical_base_name made pub(super) for cross-module use. 69 moa_gateway tests pass (unchanged); clippy -D warnings and fmt clean. * test(moa): loosen partial-survival assertion that raced under early-exit `turn_completes_when_some_workers_die` asserted exactly succeeded==2/failed==2, but early-exit consensus can abort a live worker once a usable answer is in hand, so the succeeded/aborted split is timing-dependent. It passed locally but raced red in CI (Rust crate tests shard 0). The robustness contract is unchanged and still pinned: all four dispatched workers appear in summaries (none silently dropped) and the two dead ones are recorded as failed (>=2). Dropped the exact success-count assertion. Stable across 5 consecutive local runs. * fix(moa): model=mesh degrades to single-model serving, verified live `model=mesh` now works as a universal virtual model from one node upward: 1 node / 1 model -> serves that model (single-model passthrough) >=2 workers -> real MoA committee (fan-out + synthesis) 0 models -> 503 (correct: nothing to serve) Two bugs found and fixed against a live 2-node mesh (MacBook 3B + mini 1B): 1. request.raw not rewritten. degrade patched model_name/body_json but the forwarded request rides on request.raw bytes, which still said "mesh", so the embedded openai-frontend 404'd. Now uses rewrite_model_field (patches raw + body + Content-Length together; already unit-tested). 2. wrong model source. degrade read models_being_served(), empty on a fresh serve node. /v1/models draws from three sources; degrade now tries them in order: callable_models(targets) -> models_being_served() -> serving_models(). Live-verified: single node model=mesh returns HTTP 200 with a real answer (was 503/404); 2-node mesh model=mesh returns x-moa-turn=fanout workers=2. Known cold-start transient: the very first mesh request on a fresh serve node can 404 for ~1-2s until the routing table populates; solid once warm. * fix(moa): tier from verified GGUF/gossiped size, not name guessing Addresses two i386 P1s: destructive admission and committee-cap decisions were made from a name heuristic that treats every unparseable name as big-tier, so a small fine-tune/alias (`my-assistant`) could bypass the weak-worker filter or become the reducer — contradicting the branch's core "weak can't soil strong" claim. Producer (each serving node, has its own GGUF): scan_gguf_total_parameters sums tensor element counts for the exact stored-parameter count and publishes it via the existing ServedModelMetadata.parameter_count_b. profile.rs prefers this over name parsing; missing => None (unknown), never a guess. Consumer (MoA orchestrator, has NO peer GGUFs): reads the gossiped parameter_count_b off served descriptors. New SizeTier {Small,Big,Unknown}: * admission excludes only VERIFIED small, counts only VERIFIED big toward the ">=2 big remain" gate; Unknown is never strong and never filtered. * cap_committee ranks by verified size, NOT compute_actor_candidates (tool_use-first), which could evict a big model for tool-advertising small ones on an answer turn. Name parsing (model_param_size_b) remains a lower-confidence fallback only. 71 moa_gateway tests (2 new: unknown-never-filtered, name-parse fallback); model-artifact + host clippy -D warnings and fmt clean. * fix(moa): drop name-based size fallback — no GGUF size ⇒ weakest Per i386: parsing NNb from a served name is brittle and a destructive admission decision must not rest on an unverified label. Removed the name fallback entirely. Producer (profile.rs): parameter_count_b comes ONLY from the GGUF tensor sum. No local GGUF to sum ⇒ None (no guessed count). Deleted the now-dead parameter_count_b_from_text + its test. Consumer (pool.rs): tier_for uses only the gossiped verified size. Dropped SizeTier::Unknown — a model with no verified size is Small (weakest), so an unverifiable label can never masquerade as big and displace a real strong worker. cap_committee ranking updated (no-size ranks last). Admission excludes only verified-small; with no verified big there is nothing to protect so an all-unsized pool is untouched. Tests updated: unsized-worker-is-weakest (excluded next to verified big), no-verified-sizes keeps all. 71 moa_gateway + 28 profile tests pass; clippy -D warnings and fmt clean. * fix(moa): iron law — single physical endpoint never fakes a committee Self-fill could re-add the same physical node (or an endpoint already backing the sole worker), turning ONE box into a fake 2-worker committee that hits it twice for near-identical drafts. Catastrophic for mesh mode: a lone node must degrade to single-model serving, never pretend to be a committee. Rewrote self_fill_from_extra_instances to build the pool from DISTINCT physical endpoints only: the local skippy port (if this node serves the model and context fits) plus each distinct remote peer from hosts_for_model. If fewer than two distinct endpoints serve the model, the pool stays the single worker and MoA degrades to single-model. A single endpoint can no longer appear twice. Also fixes the CodeRabbit self-fill bugs (re-adds same peer; skipped context eligibility) — context fit is now checked on the local endpoint, and endpoints are distinct by construction. 71 moa_gateway tests pass; clippy -D warnings and fmt clean. * fix(moa): reject missing/malformed messages with 400 (homelab API-1) A `model=mesh` request with no `messages`, a non-array `messages`, or an empty array fell through to the workers and fabricated a 200 answer from nothing (homelab validation API-1). try_handle_moa now requires a present, non-empty `messages` array and returns 400 otherwise, before any model call. Verified live: {"model":"mesh"}, string messages, and [] all now 400; a valid request still returns 200. 71 moa_gateway tests pass; clippy -D warnings + fmt clean. * feat(moa): tier-aware committee width — small pools fan out to 6 Width sprint (evals/moa-openrouter, aggregator qwen3-8b, 8B peers, shipped committee path) shows the fan-out cap was throttling the pools that benefit most: 6× diverse 8B vs best member: 12W/65T/2L, p=0.013 (wins) 4× diverse 8B: 5W/73T/0L, p=0.06 (marginal) 2× diverse 8B: 2W/77T/1L, p=1.0 (null) Small, weak drafts need WIDTH — more independent proposals — before aggregation clears the best member. A verified big model, by contrast, wins at 2 and gains nothing past ~4. - committee_cap is now tier-aware: COMMITTEE_CAP_SMALL=6 for all-small pools, COMMITTEE_CAP_BIG=4 when a verified big model is present (replaces the flat MAX_COMMITTEE_WORKERS=4). - eval harness: the three per-trial judge comparisons now run concurrently (tokio::join!) instead of three serial awaits — ~3x faster judging phase, no behaviour change. Also measured (recorded in RESULTS.md): the refine round never beats single aggregation at 8B (refine-vs-single null in every cell) — Hermes' cheaper single-synth cadence matches Together's layered shape here; and at 8B diversity matters (6 diverse >> 6 same), unlike mid-scale. 72 moa_gateway tests (new: committee_cap_is_wide_for_small_pools_tight_for_big); clippy -D warnings and fmt clean. * fix(moa): don't refine small pools — width + single-agg wins, refine is dead cost The width sprint measured refine-vs-single-aggregation as null in every 8B cell (2/4/6 workers, diverse and same). Small pools win by WIDTH under a single aggregation, not by the extra serial refine pass. Previously RefinementPolicy:: Auto ran the round for ALL-small pools — paying two synthesis passes for no measured gain. Auto now refines only for a homogeneous pool that is NOT all-small (i.e. same-model at real scale, where correlated drafts + the round measured 48/2 vs 35/10). All-small and diverse pools skip it — matching Hermes' cheaper single-synth cadence, where refine buys nothing. Effect: an all-small mesh turn drops from 2 serial synthesis passes to 1, roughly halving added latency, with no measured quality loss. Tests: auto_skips_an_all_small_pool (was auto_refines_...); the 5 all-small mechanics sim tests (straggler/grace/degradation) pinned with Always so they still exercise the round; big-pool gate tests keep Auto. 178 + sim tests pass; clippy -D warnings and fmt clean. * fix(moa): make the shipped path deliver the measured win Three shipped-path bugs meant `handle_turn` never ran the committee it was supposed to. Found by measuring the shipped entrypoint at capable scale. 1. Grace pre-empted synthesis. `first_answer_grace` armed on the FIRST answer and finalized the turn, so on fast backends it fired ~every turn and shipped one worker's text. Measured 80/80 EarlyExit at 32B scale. Now grace finalizes on TOOL turns only; on answer turns it is a collection deadline — stop waiting for the tail, then synthesize what arrived. Private-mesh window widened 3s -> 10s so a normal committee completes before it arms. 2. Role-tiered draft budgets (Fast 256 / Specialist 512) truncated the drafts that synthesis consumes. Those tiers existed only to make the grace fast-path cheap; grace no longer finalizes answer turns, so every answer-turn worker now gets the full budget. 3. Answer turns shipped one worker verbatim when drafts agreed. Agreeing drafts are the best input to synthesis, not a reason to skip it. Answer turns with >=2 workers always synthesize now. Also drops the worker preamble from the reducer prompt — the synthesizer is not one of the parallel answerers, and the contradictory framing cost a weak aggregator. Measured through `moa::handle_turn` at production defaults, 40 prompts x 2 draws, out-of-family judge, length-controlled: 32B/24B/35B-MoE/14B pool vs best member: 71W / 8T / 1L p<0.0001 (was 26W/14T/40L before these fixes; wins 28/29 even when MoA is SHORTER, so it is not a length artifact) Robustness contracts unchanged and still verified: slow worker cannot stall a turn, lone survivor answers, hanging refiner cannot hold the turn, patience expiry releases held consensus. * fix(moa): all-small pools serve the best member instead of a committee Measured through the shipped path (OpenRouter, 8B-class peers with an 8B reducer, 40 preregistered prompts x 2 draws, out-of-family position-swapped judge, vs the pool's best member alone): 2x 8B: 0W/43T/37L p<0.0001 6x 8B: 5W/52T/23L p=0.0009 The committee never won and lost about a third of decided trials, with consistently shorter answers (3236-3372 chars vs ~4070 solo). A capable pool is the opposite (71W/8T/1L, p<0.0001), so this is a statement about this configuration -- a weak reducer synthesizing weak drafts -- not about small-model MoA in general. The untested cell is small peers with a strong reducer; if a mesh gains a big-tier model the pool is no longer all-small and MoA engages again. Rather than ship a measured regression, an all-small pool now collapses to its single strongest member, so the caller degrades to serving that model directly. Also corrects the private-mesh grace assertion to the widened 10s default. The earlier "6x 8B beats its best member (12W/2L, p=0.013)" harness result did not replicate: the same rig on the same 40 tasks now gives 3W/76T/1L (p=0.63), with 11 of 40 tasks flipping verdict. It rested on ~14 decided trials out of 80 (the rest ties) and was a single unreplicated run. Withdrawn. * docs(moa): withdraw the unreplicated 6x8B win, record all-small shipped results The width-sprint headline (12W/2L, p=0.013) did not replicate: same rig, same tasks, same pool now gives 3W/76T/1L (p=0.63), 11 of 40 tasks flipping. It rested on ~14 decided trials from a single unreplicated run. Records the shipped-path all-small measurements that motivated the best-member gate (2x: 0W/37L, 4x: 5W/12L, 6x: 5W/23L), scopes the claim to weak-reducer configurations, and documents known methodology limitations (tie bucket conflates failures, per-draw vs per-prompt significance, unverified best member, single judge). * fix(moa): min_grace_answers back to 1 — N-1 stalled turns to worker_timeout Public-mesh comparison against released v0.74.0 (same 3B model, same prompts, `model=mesh`) exposed a liveness regression I introduced earlier today: released v0.74.0: 4.0s / 3.2s / 11.3s (reducer never ran) branch (N-1 gate): 61.0s / 6.7s / 61.7s (reducer ran, SHORTER answers) `min_grace_answers` gates whether grace can ARM at all, so any value above 1 is a liveness hazard. With N-1 on a 6-worker public-mesh pool where two peers never returned, only 4 answers ever arrived, grace could never arm, and the turn rode `worker_timeout` (60s) instead. The two 61s turns had 6 workers/4 ok and 6/3; the fast 6.7s turn had 4 workers/3 ok, where N-1=3 was satisfiable. 3/3 consistent. Width comes from the grace WINDOW (10s), not a count gate: healthy peers land inside it and are all synthesized, while a dead peer costs 10s rather than 60s. This also restores the exact configuration the capable-pool win was measured under (71W/8T/1L, p<0.0001) — that run predates the N-1 gate — and the gate bought nothing on small pools either (8W/17L with it, 9W/17L without). * docs(moa): correct the gateway doc and flag the withdrawn small-pool claim MOA_GATEWAY.md described behaviour that no longer holds: - "requires >=2 distinct models, returns 503 if fewer" -> a single model (or an all-small pool collapsed to its best member) now degrades: the virtual `mesh` name is rewritten to a real served model and routed normally. Only a node serving nothing returns 503. - name-derived tiering -> tiering now comes from verified GGUF tensor sums gossiped as `parameter_count_b`, split at 10B, with unverified sizes ranking small so an unparseable alias cannot pose as big-tier. Name parsing mis-tiered real models (gemma-4-E4B stores 7.5B, not 4B). - documents pool shaping in order (admission, all-small best-member fallback, committee cap) with the measured numbers behind each. - records that answer turns pack every worker at the full budget. - stale peer-timeout row (15s) -> 60s worker timeout with the 10s grace window. RESULTS.md: the width-sprint section now carries an explicit WITHDRAWN banner pointing at the replication failure, so the 6x8B number cannot be cited from the middle of the file. * test(moa): pin the tool-turn contract against a live multi-model pool The tool path is deliberately asymmetric — route to the single best tool-caller rather than fan out and vote, because voting on tool calls measured null-to-harmful. That is only safe if structured `tool_calls` still come back intact when several models are on the wire, so this asserts the contract through the shipped `handle_turn` with a real pool: a tool prompt yields exactly one well-formed call whose arguments parse as JSON and whose name is an offered tool, and a no-tool prompt does not invent one. Measured live (OpenRouter, both cells pass): 4x 8B pool -> dispatched=4 ok=3 reducer=true list_dir({"path":"src"}) search({"path":".","pattern":"MeshError::Timeout\\(.*?\\)"}) run_command({"cmd":"pytest --verbose"}) no_tool_concept: no call invented 4x 24-35B pool -> dispatched=1 ok=1 reducer=true same shapes, single actor The dispatch difference is `ReferencePolicy::Auto` working as measured: a small-tier actor gets tool-free advisors (+0.017 uplift), a big-tier actor acts alone (advisors measured -0.037 there). The 4-worker cell also survived a worker flaking (ok=3 of 4) and still emitted a valid call, which is the ensemble-active case that was previously only covered by replay fixtures. Asserts the contract, not a quality delta — the tool chosen may legitimately differ from a single model's first move, so only a non-offered tool or malformed arguments fail. --------- Co-authored-by: Michael Neale <14976+michaelneale@users.noreply.github.com>		2026-08-06 18:41:34 +10:00
.agents/skills	fix(skippy): preserve native tool-call semantics (#1144 )	2026-08-04 04:28:34 -04:00
.cargo	mockup: Reserves high-fidelity UI mockup (#560 )	2026-05-20 17:17:42 -04:00
.github	fix(skippy): preserve native tool-call semantics (#1144 )	2026-08-04 04:28:34 -04:00
.skills	Consolidate agent skills and fix stale docs (Windows deploy, repo map, design docs) (#836 )	2026-06-12 18:44:36 +10:00
.well-known	fix(config): fundation configuration setup	2026-06-12 03:08:48 -04:00
ci	fix(skippy): preserve native tool-call semantics (#1144 )	2026-08-04 04:28:34 -04:00
contrib/windows	Add split readiness doctor (#723 )	2026-05-29 10:44:16 +10:00
crates	MoA: mesh answers improve as capable nodes join (#1116 )	2026-08-06 18:41:34 +10:00
dist	Load startup models from config.toml	2026-04-06 10:08:01 +10:00
docker	fix: speed up dependency-heavy container rebuilds (#1154 )	2026-08-02 16:50:20 -04:00
docs	MoA: mesh answers improve as capable nodes join (#1116 )	2026-08-06 18:41:34 +10:00
evals	MoA: mesh answers improve as capable nodes join (#1116 )	2026-08-06 18:41:34 +10:00
fly	fix: speed up dependency-heavy container rebuilds (#1154 )	2026-08-02 16:50:20 -04:00
recipes/quantization	Add direct quant layer package tooling (#1031 )	2026-07-22 14:44:22 +10:00
schemas	feat: unify release hosts and native runtimes	2026-07-29 10:16:23 -04:00
scripts	feat(skippy): expose authoritative generation lifecycle events (#1149 )	2026-08-04 17:13:09 -04:00
sdk	fix(swift): synchronize generated binding checksums (#1122 )	2026-07-30 09:51:59 -04:00
third_party/llama.cpp	fix(skippy): isolate chat grammar during speculative verification (#1172 )	2026-08-05 06:13:55 +00:00
tools	feat(skippy): expose the loaded tokenizer capability (#1147 )	2026-08-04 13:22:46 -04:00
website	feat(skippy): expose authoritative generation lifecycle events (#1149 )	2026-08-04 17:13:09 -04:00
.dockerignore	docker: multi-stage images for most architectures, CI, and fly rewrite (#220 )	2026-04-12 16:17:59 -04:00
.gitignore	Add meshllm.cloud website, catalog viewer, and onboarding docs (#806 )	2026-06-10 05:18:25 -04:00
.nvmrc	Upgrade GitHub Actions and enforce Node 24	2026-03-30 10:35:30 +11:00
AGENTS.md	feat: unify release hosts and native runtimes	2026-07-29 10:16:23 -04:00
build-unification.md	feat: unify release hosts and native runtimes	2026-07-29 10:16:23 -04:00
Cargo.lock	Provide the V1 tokenizer inventory activation bridge (#1168 )	2026-08-05 05:24:24 +00:00
Cargo.toml	feat(skippy): expose authoritative generation lifecycle events (#1149 )	2026-08-04 17:13:09 -04:00
clippy.toml	chore(code-quality): Set max 200 line limit for long methods (#595 )	2026-05-19 18:22:47 -04:00
CONTRIBUTING.md	feat: unify release hosts and native runtimes	2026-07-29 10:16:23 -04:00
install.md	Document canonical Homebrew tap (#1102 )	2026-07-28 14:26:26 -04:00
install.ps1	ci: compose reusable products and add Depot routing (#1113 )	2026-07-30 04:02:53 -04:00
install.sh	feat: unify release hosts and native runtimes	2026-07-29 10:16:23 -04:00
Justfile	feat(skippy): expose the loaded tokenizer capability (#1147 )	2026-08-04 13:22:46 -04:00
LICENSE	Add Apache 2.0 license	2026-04-07 10:52:27 +10:00
mesh.png	chore(docs): update docs to match planned updates	2026-05-12 17:52:17 -04:00
package-lock.json	Add meshllm.cloud website, catalog viewer, and onboarding docs (#806 )	2026-06-10 05:18:25 -04:00
Package.swift	feat: native rust SDK for rust consumers. (#736 )	2026-06-03 12:39:19 +10:00
playwright.config.js	mesh-llm crate decomposition (#459 )	2026-05-08 12:11:14 +10:00
README.md	feat(mesh): add pinned model-only OpenAI serving (#1148 )	2026-08-04 15:49:49 -04:00
RELEASE.md	fix(release): keep prereleases out of downstream publishing (#1120 )	2026-07-30 06:47:59 -04:00
ROADMAP.md	Upgrade demanded local models during reconciliation (#754 )	2026-05-30 15:00:35 -07:00
SKIPPY_PROTOCOL_TODO.md	🚬 Add bounded MTP + N-gram pipelining for latency-limited split inference (#1026 )	2026-07-22 18:32:34 +10:00

README.md

Mesh LLM

Mesh LLM pools GPUs and memory across machines and exposes the result as one OpenAI-compatible API at http://localhost:9337/v1. Start one node, add more nodes later, and let the mesh decide whether a model runs locally, routes to a peer, or uses Skippy stage splits for models that are too large for one box.

Quick start

Install the latest release executable:

curl -fsSL https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.sh | bash

On Windows, use PowerShell:

irm https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.ps1 | iex

Install the Apple Silicon Homebrew formula with brew install Mesh-LLM/tap/mesh-llm. Versioned formulas, Ubuntu and Arch packages, checksums, SBOMs, and OCI images are produced by the public Mesh-LLM/mesh-packaging repository. See the platform install guides for the supported package matrix and install commands.

Finish setup:

mesh-llm setup

On Windows PowerShell, use mesh-llm.exe setup. (If you plan to run MeshLLM inside WSL2 for CUDA 13+ support or multi-node LAN clustering, see the Windows & WSL2 Troubleshooting Guide.)

To remove an executable install later, preview the cleanup first:

mesh-llm uninstall --dry-run
mesh-llm uninstall --yes

Uninstall preserves ~/.mesh-llm configuration and identity data unless you explicitly pass --purge-config.

Join the public mesh and start serving:

mesh-llm serve --auto

That command chooses a backend flavor, downloads a suitable model if needed, joins the best discovered public mesh, starts the local API on port 9337, and starts the web console on port 3131.

Check available models:

curl -s http://localhost:9337/v1/models | jq '.data[].id'

Send an OpenAI-compatible request:

curl http://localhost:9337/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"GLM-4.7-Flash-Q4_K_M","messages":[{"role":"user","content":"hello"}]}'

For server deployments, add --headless to hide the web UI while keeping the management API on the --console port:

mesh-llm serve --auto --headless

Pick the workflow you need

Goal	Command	Full guide
Try the public mesh	`mesh-llm serve --auto`	docs/MESHES.md
Start a private mesh	`mesh-llm serve --model Qwen3-8B-Q4_K_M`	docs/MESHES.md
Serve one model without mesh networking	`mesh-llm serve --local-model-only --model /models/model.gguf`	OpenAI API defaults to `127.0.0.1:9337` (`--port` and `--listen-all` change it)
Publish your own mesh	`mesh-llm serve --model Qwen3-8B-Q4_K_M --publish`	docs/MESHES.md
Join by invite token	`mesh-llm serve --join <token>`	docs/MESHES.md
Run an API-only client	`mesh-llm client --auto`	docs/MESHES.md
Run a big model with splits	`mesh-llm serve --model hf://meshllm/<repo>@<rev> --split`	docs/SKIPPY_SPLITS.md
Attach a Flash-MoE SSD backend	`mesh-llm serve` with `[[plugin]] name = "flash-moe"`	docs/plugins/flash-moe.md
Fan out one prompt to every model in the mesh	`curl ... -d '{"model":"mesh", ...}'`	docs/design/MOA_GATEWAY.md
Use Goose, OpenCode, Claude Code, or Pi	`mesh-llm goose`, `mesh-llm opencode`, `mesh-llm claude`, `mesh-llm pi`	docs/AGENTS.md
Build or contribute	`just build`	CONTRIBUTING.md

How the mesh works

Single-machine fit first. If one node can host the full model, it serves the model locally without stage traffic.
Mesh routing. Every node exposes the same /v1 API. Requests are routed by the model field to the peer that can serve that model.
Owner-control plane. Operator config and inventory actions use an additive mesh-llm-control/1 lane with explicit endpoint bootstrap, while public mesh join, gossip, routing, and inference stay on the public mesh plane for mixed-version compatibility.
Skippy stage splits. Large dense models can load as package-backed layer stages. The coordinator plans contiguous layer ranges, starts downstream stages first, waits for readiness, then publishes the stage-0 route.
Layer packages. Package repositories contain model-package.json plus GGUF fragments so peers fetch only the pieces needed for their assigned stage.
Public discovery. Published meshes advertise through Nostr discovery; private meshes stay invite-token based.

For a deeper operator guide, see docs/USAGE.md. For every CLI command and switch, see docs/CLI.md.

Local model-only serving

Use the direct topology when a process should expose one complete local model through the OpenAI API without becoming a mesh node:

mesh-llm serve \
  --local-model-only \
  --model /models/model.gguf \
  --port 9337

This mode starts the OpenAI frontend and one local Skippy model runtime. It does not start QUIC, discovery, peer maintenance, split planning, plugins, release lookup, the web console, or the management API. Add --listen-all only when the OpenAI endpoint must bind beyond loopback. Startup fails if the complete model does not fit within detected local capacity (or --max-vram); it never falls back to distributed serving.

For --local-model-only, --model, --gguf, and --mmproj values must be absolute paths and must not be symlinks.

Mixture-of-Agents (`model: "mesh"`) — experimental

⚠️ Experimental. The MoA gateway is new in this release. Behavior, routing heuristics, error shapes, and tuning knobs may change between versions while we tune it. Treat model: "mesh" as a preview feature rather than a stable production path; use a specific model id when you need stable semantics.

Send a request with "model": "mesh" and the proxy fans it out to every model available in the mesh in parallel, arbitrates their responses with deterministic logic, and returns one OpenAI-compatible reply. The arbiter runs in code (not as another model call) and only escalates to a reducer LLM on genuine conflict. Tool calls flow through the full pipeline.

curl http://localhost:9337/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"mesh","messages":[{"role":"user","content":"What is the capital of Japan?"}]}'

Requires at least two distinct models in the mesh. See docs/design/MOA_GATEWAY.md for the architecture, arbitration rules, and tuning knobs.

Supported model families

Mesh LLM's Skippy runtime tracks llama.cpp family parity with reviewed GGUF representatives. The current reviewed support set covers 72 P0/P1 family rows, with 89 certified rows in the full parity inventory, including Qwen, Llama, Gemma, Mistral, DeepSeek, GLM, MiniMax, Phi, Granite, Hunyuan, EXAONE, Cohere, Falcon, RWKV, and many others.

Split multimodal serving is certified for Qwen2-VL, Qwen3-VL, Qwen3-VL-MoE, HunyuanOCR/Hunyuan-VL, and DeepSeek-OCR using real GGUF plus projector fixtures. DeepSeek3 and EXAONE-MoE use package-backed stages because the full GGUFs are too large for the cheap local baseline.

See docs/skippy/FAMILY_STATUS.md for the full artifact, split, wire dtype, cache policy, and exception matrix. See docs/skippy/LLAMA_PARITY.md for the remaining llama.cpp parity queue.

Install and build notes

Tagged releases publish macOS bundles plus Linux CPU, Linux ARM64 CPU, Linux ARM64 CUDA, Linux CUDA, Linux CUDA Blackwell, Linux ROCm, Linux Vulkan, Windows CPU, Windows CUDA, Windows ROCm, and Windows Vulkan bundles. Metal is macOS-only. Every flavor is composed from the same backend-neutral host for its OS/architecture plus one versioned native runtime. The Linux ARM64 CPU artifact is mesh-llm-aarch64-unknown-linux-gnu.tar.gz; the Linux ARM64 CUDA artifact is mesh-llm-aarch64-unknown-linux-gnu-cuda.tar.gz. In install and release contexts, arm64 and aarch64 mean the same 64-bit ARM target. Portable archives work offline: the host discovers the adjacent native-runtimes/<runtime-id> tree before consulting the user cache.

Build from source with just:

git clone https://github.com/Mesh-LLM/mesh-llm
cd mesh-llm
just build

Source builds require just, cmake, Rust, and Node.js 24 + npm. To exercise the release boundary locally, build the neutral host and one runtime:

just release-host-build
just release-runtime-build metal # or cpu, cuda, rocm, vulkan
MESH_LLM_NATIVE_RUNTIME_BUNDLE_DIR="$PWD/dist/native-runtimes" \
MESH_LLM_NATIVE_RUNTIME_CACHE_DIR="$(mktemp -d)" \
  ./target/release/mesh-llm runtime list

CUDA runtimes need nvcc, ROCm runtimes need ROCm/HIP, and Vulkan runtimes need Vulkan development files plus glslc. See docs/design/NATIVE_RUNTIMES.md for the manifest, discovery, and compatibility contract.

The shipped mesh-llm executable uses embedded release attestation for provenance and admission hardening only. It does not apply to SDK, XCFramework, or other native artifacts, and it is not a runtime integrity proof. Verify a stamped packaged executable with cargo run -p xtask -- release-attestation inspect --binary <path-to-packaged-mesh-llm> --public-key-file <release-signing-public-key.json>. A packaged release binary reports valid, an unstamped local or dev build reports missing, and a binary that changed after packaging reports invalid. Bare inspect --binary ... is only enough to classify an unstamped binary as missing; stamped binaries require --public-key-file and otherwise report invalid with an explicit error. Post-download mutation can flip a stamped binary to invalid, but default startup still allows it.

🪟 Windows & WSL2 Troubleshooting

Running on Windows with CUDA 13+ Drivers

If you are running Windows with an NVIDIA CUDA 13.x driver (e.g., Driver version 595+) and mesh-llm reports 0 GPUs or falls back to Vulkan, use WSL2 (Windows Subsystem for Linux) with --llama-flavor cuda to access Linux CUDA 13 runtimes.

1. Install CUDA 13.0 Toolkit inside WSL2

Inside your Ubuntu WSL2 terminal, install cuda-toolkit-13-0 to supply libcudart.so.13:

wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin
sudo mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/13.0.0/local_installers/cuda-repo-wsl-ubuntu-13-0-local_13.0.0-1_amd64.deb
sudo dpkg -i cuda-repo-wsl-ubuntu-13-0-local_13.0.0-1_amd64.deb
sudo cp /var/cuda-repo-wsl-ubuntu-13-0-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update && sudo apt-get -y install cuda-toolkit-13-0

echo 'export LD_LIBRARY_PATH=/usr/local/cuda-13.0/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

2. Enable Hyper-V & Windows Firewall for WSL2 Mirrored Mode

If you use WSL2 networkingMode=mirrored in %UserProfile%\.wslconfig, Windows 11 manages a separate Hyper-V VM Firewall that defaults to Block for inbound network traffic when third-party security software (e.g. Norton, McAfee) is present.

2.1 Configure Mirrored Networking (`.wslconfig`) Part 1

Create or edit C:\Users\<username>\.wslconfig on the Windows host:

notepad $env:USERPROFILE\.wslconfig

2.2 Configure Mirrored Networking (`.wslconfig`) Part 2

[wsl2]
networkingMode=mirrored
autoProxy=true

2.3 Restart WSL in Powershell

Restart WSL in PowerShell:

wsl --shutdown

2.4 Enable Windows Firewall Needs

To allow incoming LAN connections to the Web UI (3131) and P2P QUIC mesh transport (9337), run PowerShell as Administrator on the Windows host:

# 1. Allow Inbound Traffic through the WSL Hyper-V VM Firewall Container
$vmCreatorId = '{40E0AC32-46A5-438A-A0B2-2B479E8F2E90}'
Set-NetFirewallHyperVVMSetting -Name $vmCreatorId -DefaultInboundAction Allow

# 2. Allow MeshLLM Ports in Windows Defender Firewall
New-NetFirewallRule -DisplayName "MeshLLM TCP In" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 3131,9337 -Profile Any
New-NetFirewallRule -DisplayName "MeshLLM UDP In" -Direction Inbound -Action Allow -Protocol UDP -LocalPort 9337,5353 -Profile Any

3. Match Model Paths for Direct LAN Reading

To ensure worker nodes load GGUF model shards directly off local NVMe/SSD storage without streaming tens of gigabytes over the network, ensure the --gguf file path string is identical across all nodes (or use symlinks/bind mounts):

# Example: Mount or symlink model path on worker nodes
sudo mkdir -p /mnt/d/models/
sudo mount --bind /path/to/local/fast/nvme/ /mnt/d/models/

# Launch Master and Worker with matching path strings
mesh-llm --llama-flavor cuda serve \
  --console 3131 \
  --gguf "/mnt/d/models/model.gguf" \
  --mesh-name "MainMesh" \
  --listen-all \
  --auto

Documentation hub

Doc	Use it for
docs/MESHES.md	Private meshes, public discovery, publishing, invite tokens, API-only clients
docs/SKIPPY_SPLITS.md	Running big models with package-backed Skippy stage splits
docs/LAYER_PACKAGE_REPOS.md	Contributing and publishing layer package repositories
docs/AGENTS.md	Goose, Claude Code, OpenCode, Pi, curl, and blackboard
docs/EXO_COMPARISON.md	Balanced comparison with Exo
docs/CLI.md	Command reference and JSON automation
docs/USAGE.md	Longer operational usage guide, runtime control, owner-control operator flows
docs/design/TESTING.md	Testing playbook, mixed-version QA, remote deploy checks
docs/plugins/flash-moe.md	Optional Flash-MoE SSD expert streaming backend setup
docs/skippy/FAMILY_STATUS.md	Certified Skippy model-family status
docs/specs/layer-package-repos.md	Manifest and artifact format spec
docs/specs/mesh-setup-installer.md	Installer/bootstrap and setup command behavior spec

CI infrastructure

Mesh LLM is adopting Depot's managed GitHub Actions runners for non-GPU CI builds. Hardware-qualified GPU tests remain on dedicated runners.

Community

Mesh LLM is experimental distributed-systems software. When you report bugs, include the command you ran, platform/backend flavor, /api/status output if available, and whether the node was private, published, or joined with --auto.