168 commits

Author	SHA1	Message	Date
James Dumay	7368fb0c8e	Support Pascal GPUs in CUDA 12 runtimes (#1206)	2026-08-09 00:35:02 +00:00
James Dumay	cad3fb2996	fix(plugin): bound proposal queue deadlines (#1180) ... * fix(plugin): bound proposal queue deadlines * fix(plugin): isolate passive callbacks * fix(plugin): guarantee plugin shutdown and terminal discards Address review feedback on the bounded proposal dispatch path. - Close queues instead of enqueueing a Shutdown command, so a full queue can no longer strand a worker and skip the native shutdown callback. - Join a worker on an explicit exit signal rather than is_finished(), so the gap between the last callback and thread teardown cannot skip join. - Classify a proposal's forwarding decision and telemetry from a single timestamp, so a candidate can never be forwarded while telemetry calls it late. - Reserve queue headroom for terminal dispositions so a late candidate's discard is delivered instead of silently dropped. - Surface a failing proposal callback as SourceError with the plugin's message instead of an unexplained silent abstention. - Extract the dispatch subsystem into plugin_dispatch, taking lib.rs from 1689 to 745 lines per the repo's >1000-line rule. --------- Co-authored-by: Michael Neale <14976+michaelneale@users.noreply.github.com>	2026-08-08 16:51:13 +10:00
Michael Neale	96ef6227f2	MoA: mesh answers improve as capable nodes join (#1116) ... * feat(moa): give MoA workers tools, synthesize disagreement, survive flaky peers Agentic MoA turns now reliably produce tool calls, and diverging answers are synthesized instead of relaying one arbitrarily-chosen worker's text. Fixes found by replaying recorded traces from 9 open-weight models: * Tools were withheld from workers. `query_uses_tools` was derived from `looks_like_tool_intent`, an English keyword match on the user's text. "The test suite is failing. Find out which test fails and why" matches nothing, so workers were dispatched without tool schemas and the arbiter ran with has_tools=false — a unanimous tool proposal then fell through to the answer path and leaked "calling search" as prose. 5 of 10 recorded scenarios hit this. Tool availability is now the caller's declaration. * Argument outliers could win a unanimous tool call. Native tool calls are all normalized to a fixed 0.9 confidence, so the confidence-only tiebreak could not separate 8 workers proposing {"path":"src"} from one hallucinating {"path":"rust_project/src"}. Added argument clustering (key-order independent); the largest cluster wins and argument-free calls never outvote filled-in ones. * Early exit committed on tied arguments. Early exit runs on whoever has arrived, so two fast models agreeing on a tool name with different arguments was a coin flip that also aborted the workers who would have broken the tie. It now waits. * Truncation was invisible. `finish_reason` was never read; 39/140 recorded responses came back "length" and 24 carried partial text, which parsed as a normal answer and could be returned verbatim. Plumbed through as BackendReply.truncated -> WorkerOutput.truncated; such answers are barred from consensus and verbatim output but still feed synthesis, labelled incomplete. * A strict endpoint could kill a worker permanently. minimax-m2.5 returns HTTP 400 "Reasoning is mandatory for this endpoint" and failed 12/12 requests; HttpBackend now drops the thinking flags and retries once. Policy changes: * Thinking is always off for MoA workers, not merely defaulted off. The previous escape hatch only let callers request the broken configuration: qwen3-32b spent 408 reasoning tokens against a 384-token cap and returned null content. Ignored overrides are logged. * Diverging answers go to the reducer. Returning the top-confidence answer was near-arbitrary because models rarely emit our confidence envelope, so everything defaults to 0.5 and max_by returns whichever worker finished first. Agreement still short-circuits without paying for synthesis. * Reducer prompt adopts Together's aggregator framing (synthesize, critically evaluate, agreement is not proof), keeping our per-worker attribution, structured tool proposals, and 500-char payload bound. Tests: new sim_real_traces.rs replays 56 recorded cases from 9 models through handle_turn, plus regression unit tests for each bug. Recorders and corpora live in evals/moa-openrouter/. * feat(moa): asymmetric tool turns — references advise, best tool-caller acts Tool-bearing turns now use the Hermes/Nous-style asymmetric shape instead of a majority vote across workers: - references run TOOL-FREE and only advise in prose - the single best tool-caller (the "actor") acts on that advice with the real tools and emits the tool call Tool authority now tracks capability, not popularity. The old path let several weak models proposing a popular-but-wrong tool outvote the one strong tool-caller that picked correctly (observed: qwen3-32b alone chose run_command for a failing-test triage while the smaller models chose list_dir, and the vote shipped list_dir). An actor model removes that failure class instead of patching the vote arithmetic. Stays a pure stateless /v1/chat/completions turn — references are regenerated from the caller's transcript each request, and the external client still owns tool execution. Text-only turns are unchanged (symmetric fan-out + synthesis-on-divergence). Actor selection is capability-first: the host ranks candidates by gossiped `tool_use` level (Supported > Likely > None), then model size tier, then stable order, and passes the ordering to the engine via the new `GatewayConfig.actor_candidates`. Empty ordering falls back to the engine's name-derived size tier, so existing callers and tests are unaffected. Mesh guardrails for mixed/public meshes: references are gathered with a bounded wait (proceed at a majority of advisors, never block on the slow tail), and the actor is called through the existing hedged ladder so a slow/broken best candidate falls through to the next tool-capable peer. New: - crates/mesh-mixture-of-agents/src/tool_turn.rs — asymmetric tool-turn handler - context::pack_for_actor — "advise, then act" framing (vs synthesis framing) - fanout::gather_references — bounded, tool-free reference collection - GatewayConfig.actor_candidates + reducer_candidates honours it - host compute_actor_candidates from gossiped tool_use + size The dead majority-vote code in the arbiter (now unreachable — tool turns bypass it) is removed in the follow-up commit. Tests: eval_openrouter.rs adds a live OpenRouter harness (ignored by default) that runs the real handle_turn over 6 open-weight models with mesh-realism latency/failure injection, for benchmarking asymmetric MoA vs single models. 192 engine tests pass; clippy -D warnings and fmt clean on both crates. * refactor(moa): remove dead majority tool-vote from the arbiter Tool turns now take the asymmetric actor path (previous commit), so the arbiter's tool-proposal voting is unreachable — and it *was* the majority-of-weakness bug: a popular-but-wrong tool choice from several weak models could outvote the one strong tool-caller that picked correctly. Removes it rather than leaving dead code: - delete best_tool_proposal, best_tool_proposal_by_consensus, and decisive_argument_cluster - drop the tool-arbitration and tool-vs-answer branches from arbitrate, and the tool-consensus branch from try_early_decision - drop the now-unused has_tools parameter from arbitrate, try_early_decision, single_output_decision, and gather_workers_incremental - the arbiter is now purely an answer/critique/uncertainty arbiter; tool-shaped text on the text path is already demoted to Uncertainty by enforce_tool_call_contract (tools disabled ⇒ empty allow-list) - drop the arbiter tests that pinned the removed tool-vote behavior; keep and re-point every answer-consensus, truncation, synthesis, and tier-gate test No behavior change for text turns. 178 engine tests pass; host-runtime lib tests (1851) pass; clippy -D warnings and fmt clean on mesh-mixture-of-agents, mesh-llm-host-runtime, and mesh-llm. * test(moa): controlled actor-ablation eval — do references actually help? Replaces the confounded "MoA vs best single model" comparison with a within-actor ablation that isolates the one variable that matters: the references. One pinned actor, identical sampling / token budget / system prompt across all arms; only the advice changes: A. actor alone (no references) B. actor + real references (the production tool path) C. actor + shuffled references (advice from a different task, length-similar) Metric: rescue (A✗→B✓) minus harm (A✓→B✗). Arm C is the key control — it separates "useful information" from "extra tokens + a think-carefully prompt". Why the change: the earlier best-single comparison ran solo models at different sampling (0.8/512) than the actor (0.3/2048) and with a different system prompt, and scored transient 429/502/504 and tool-unsupported endpoints as capability failures. Those numbers measured sampling+prompt+infra, not the design, and are not citable. This ablation removes all of that by construction: the same scorer hits all three arms of the same actor, so an imperfect label largely cancels in the rescue-minus-harm delta. Also: - chat_completion_retrying: predeclared retry on transient infra errors, which are then excluded (∅) from the capability analysis rather than scored as wrong answers. - MOA_ABLATION_ACTOR / MOA_ABLATION_DRAWS env overrides. - the old best-single test is kept but documented as a confounded smoke test, not evidence. Pilot findings (5 draws, 4 tasks, ignored/live): - Strong actor (qwen3-32b): 20/20 all arms — aces every task alone, so references are inert-but-harmless (no headroom to measure rescue). - Weak actor (qwen3-8b): A 15/20, B 20/20, C 15/20 — net uplift +5, all on the one task with headroom (triage). Real advice rescued the weak actor every draw; SHUFFLED advice did not (C=A), so the gain is advice CONTENT, not token count or the decision prompt. Zero harm. Reading: reference value tracks actor-alone headroom — a safety net when the actor is weak, dead weight (not damage) when strong. Suggests gating the reference phase on actor strength. This is a directional pilot, not the merge-blocking study (that needs ~40 preregistered stratified tasks, k>=10 draws, a paired hierarchical bootstrap CI, and the production-selected actor). * test(moa): preregistered scaled actor-ablation study + bootstrap analyzer The defensible version of the pilot, committed BEFORE running so the labels are preregistered and can't be seen to chase results. - tests/fixtures/ablation_tasks.json: 40 preregistered tasks, 4 strata x 10 (inspect / search / execute / no_tool). Set-valued accept labels (accept_tools is a SET; empty = "no tool call"), with optional arg substring constraints — addresses the brittle single-tool label from the pilot. - ablation_scaled_study: loads the fixture, runs A/B/C arms at k draws (default 10) with bounded concurrency, writes one JSONL trial per (draw,task,arm) to MOA_ABLATION_OUT. Live measurement only; no stats here. - evals/moa-openrouter/analyze_ablation.py: deterministic paired HIERARCHICAL bootstrap — resample tasks within stratum, then draws within task — for a 95% CI on net uplift = P(rescue) - P(harm), plus the shuffled-arm control (content effect = B_uplift - C_uplift). Validated on synthetic data: 3/40 rescue tasks -> +0.075 point estimate, shuffled +0.000, and the CI honestly spans 0 when rescue is sparse (no over-claiming). Arms (one pinned actor, identical sampling/prompt; only references vary): A actor alone · B actor + real references · C actor + shuffled references. Env: MOA_ABLATION_ACTOR, MOA_ABLATION_DRAWS, MOA_ABLATION_CONCURRENCY, MOA_ABLATION_OUT. Ignored by default (live network + cost). * docs(moa): trim over-long rationale comments in the asymmetric tool path Cut the essay-length doc/inline comments added with the actor design down to the non-obvious "why". No code change. - tool_turn.rs: 26-line module preamble -> 6 lines; inline comments tightened (20% -> 10% comment density) - context.rs: pack_for_actor docstring 16 -> 5 lines; reducer synthesis-framing block 14 -> 6 lines - workers.rs: compute_actor_candidates docstring 21 -> 6 lines (kept the terse sort-key comments, which are load-bearing) * test(moa): pre-hoc structured-proposal and post-hoc correction studies Two more ablation arms to answer "can peers help tool selection at all?", reusing the A/B/C JSONL schema + analyze_ablation.py. matched_peer_structured_study — do similar-strength, different-family peers help a fixed finalizer via STRUCTURED candidate tool calls (not prose)? A solo · B diverse (2 different-family peers) · C homogeneous (2 resamples of the finalizer's own model). B-C isolates cross-family diversity from extra sampling; also records oracle-union. correction_rescues_weak_tool_caller — the mesh scenario neither Hermes nor Together handles: a weak tool-caller with no strong peer. Tests correction of the CONCRETE drafted call instead of pre-hoc advice. A draft-alone · B deterministic (schema-validate + re-prompt on structural failure) · C semantic (different-family critic reviews the concrete call, finalizer revises once). Findings (live, directional): every tool-selection intervention was inert-to-harmful vs routing to a capable model. Pre-hoc structured proposals were flat (37-39/40 all arms). Deterministic correction fired ~never (qwen3-8b already drafts structurally valid calls ~95%); the residual failures are semantic (wrong tool choice), which neither validation nor a strong critic fixed because the revision still runs through the weak actor. Conclusion: tool selection is a "best capable model acts" task; MoA's value is on the answer/chat path (untested). * test(moa): committee study on realistic reasoning/answer turns Tests where MoA's value should live per Together's validated claim: open-ended answer QUALITY on realistic agent-session turns (reason-over-tool-output, planning, explanation) — not tool selection. Fixed aggregator; only its input varies: A alone · B committee (synthesize 3 diverse-peer drafts, 1 round) C layered (peers refine seeing each other first, then synthesize — Together's `layers`) Judged pairwise by an out-of-pool different-family judge (gpt-4o-mini), position-swapped (win only if consistent both orders), output lengths logged. - tests/fixtures/committee_tasks.json: 15 realistic reasoning/answer turns - committee_beats_solo_on_reasoning: writes per-trial JSONL Pilot findings (2 draws, live; DIRECTIONAL, underpowered): - committee(B) vs solo(A): win 6 / tie 2 / loss 2 — positive but sign-test p=0.29, NOT significant at n=10. Every B win was also the longer answer, so length is not cleanly ruled out. - layered(C) vs committee(B): loss 6 / tie 2 / win 2 — Together's extra refinement round is negative value on these tasks, at extra cost. - 20/30 trials skipped because the aggregator (qwen3-32b) returned empty content (reasoning-budget exhaustion — the content:null bug Hermes' troubleshooting.md documents). A flaky aggregator degrades the committee; the instrument needs empty-output handling before a real run. Net: first directionally-positive result for MoA in this investigation, but unproven. Contrast with 5 tool-selection experiments that were all null-to-harmful. Supports the task-split: route tools to one caller, convene a (single-round) committee on reasoning turns. * feat(moa): Hermes-style reference packing — strip system prompt + tool transcript Our references were packed very differently from Hermes', and it was measurably costly. Adds `pack_for_reference`, which gives advisors only the conversation's user/assistant prose: - strips the agent system prompt (an advisor told "you are a coding agent, run the tests" role-plays the actor instead of advising it) - strips the tool transcript (prior tool_calls + results anchored every advisor on the trajectory already taken, collapsing the error-independence that makes aggregation worth anything) - drops the "respond with your best answer or tool call" instruction: advisors hold no schemas, so asking for a tool call yields tool-shaped prose — exactly the advice that pulled the actor off its own better choice - uniform view across advisors (no per-role trimming), so the packing is a stable function of history and caches across iterations - caps advisor output at 600 tokens (advisor generation dominates turn latency; the turn waits for the slowest advisor) Head-to-head on the same preregistered study (strong actor qwen3-32b, 40 tasks x 10 draws, identical everything except packing): packing B pass net uplift 95% CI original 359/400 -0.102 [-0.170, -0.045] hermes 385/400 -0.037 [-0.090, -0.003] Harm cut by ~64%. Per-category: search -14 -> -2, execute -25 -> -13, inspect -2 -> 0. The content-specific component (B-C differential) fell from -0.075 to -0.015, i.e. with correct packing the residual harm is no longer mostly "bad advice content". Honest read: most of the harm I previously attributed to "references" was an artifact of how WE packed them, not a property of reference-based MoA. The direction still stands though — even correctly packed, references remain a small but statistically real regression for a STRONG actor on tool selection (CI still entirely below zero). Adds 4 unit tests pinning the packing contract (no tool-call request, no system prompt leak, no tool-transcript leak, prose preserved), and MOA_REFERENCE_PACKING=hermes to select the style in the eval harness. * docs(moa): record the MoA evidence — packing bug, actor headroom, what failed Writes up every live study in one place (evals/moa-openrouter/RESULTS.md) so the conclusions and their caveats survive the investigation. Headline 2x2 (40 preregistered tasks x 10 draws, paired hierarchical bootstrap): actor packing B pass net uplift 95% CI strong original 359/400 -0.102 [-0.170, -0.045] <- the bug strong hermes 385/400 -0.037 [-0.090, -0.003] weak original 365/400 -0.013 [-0.090, +0.070] weak hermes 377/400 +0.017 [-0.053, +0.100] Two monotonic effects: fixing the packing helps in both actor conditions, and references are worth more to a weaker actor. The only significant cell is the original-packing strong-actor harm — i.e. our bug, not a property of MoA. Per-stratum (weak + hermes) shows references help exactly where the actor has headroom (search +10, execute +4) and hurt where it was already perfect (inspect -7). That is a gating signal. Also records what did NOT work for tool selection (pre-hoc structured proposals: flat; deterministic correction: never fires, the weak actor already emits valid calls ~95%; semantic correction: negative, since the revision still runs through the weak actor), and the committee pilot on reasoning turns (directionally positive, n=10, not significant; Together's layering loses to single-round). * fix(moa): use the validated reference packing in production The Hermes-style packing was measured and unit-tested but only wired into the eval harness — the production tool path still used pack_for_worker_selected, the exact packing measured at -0.102 net uplift. tool_turn now calls pack_for_reference: conversation prose only, no agent system prompt, no tool transcript, no request for a tool call. Same head-to-head (strong actor, 40 tasks x 10 draws) puts this at -0.037 vs -0.102, and it is the only configuration where references show a positive point estimate for a weak actor (+0.017). See evals/moa-openrouter/RESULTS.md. * feat(moa): gate advisory references on actor headroom References are not universally good or bad — their value tracks how much headroom the acting model has. Measured over 40 preregistered tool tasks x 10 draws with correct advisor packing (evals/moa-openrouter/RESULTS.md): weak actor (qwen3-8b): +0.017 net uplift -> advice helps strong actor (qwen3-32b): -0.037 net uplift -> advice costs Per-stratum the split is sharper: references gained where the actor had headroom (search +10, execute +4) and lost where it was already perfect (inspect -7). Adds `GatewayConfig.reference_policy`: Auto (default) - advise a small-tier actor, let a big-tier actor act alone Always - always fan out advisors Never - actor acts alone (Hermes' `enabled: false`) A one-model pool never gathers references under any policy (nobody left to advise once the actor is excluded), and an unknown actor keeps the prior advise-by-default behaviour rather than silently degrading. The host sets `Auto`. sim_real_traces pins `Always` because that suite exists to verify fan-out accounting; the gate itself is covered by six new unit tests in tool_turn. 189 tests pass; clippy -D warnings and fmt clean on mesh-mixture-of-agents and mesh-llm-host-runtime. * test(moa): scale the committee study — 40 preregistered prompts, concurrent, empty-output fix Prepares the reasoning-turn committee study for a defensible run. Committed before executing so the prompts are preregistered. Three fixes to the pilot's weaknesses: 1. Empty-output bug. `response_text` read only `/message/content`, so a reasoning model that spends its budget in `reasoning` and returns `content: null` looked like an empty answer — that silently dropped 20/30 pilot trials. Now falls back to `reasoning`, mirroring the in-tree backend's chain. Remaining skips are counted and reported rather than hidden. 2. Serial execution. ~18 calls x 30 trials was far too slow to scale. Trials are independent, so they now run under a bounded semaphore (MOA_COMMITTEE_CONCURRENCY, default 4), with peer drafts and refinements gathered concurrently within each trial. Output is sorted for stable reporting. 3. Fixture too small. 15 -> 40 prompts, 4 strata x 10: reason_over_output, planning, explain, code_review. All realistic agent-session reasoning/answer turns (several embed simulated tool output or code, as an agent loop would). Arms and controls are unchanged: fixed aggregator, A alone / B committee / C layered, judged pairwise by an out-of-pool different-family judge with position swap, output lengths logged for the length-vs-preference check. * docs(moa): committee wins decisively on reasoning turns (120 trials, p<1e-11) Scaled the reasoning-turn committee study from the 15-prompt pilot to 40 preregistered prompts (4 strata x 10) x 3 draws = 120 trials, 0 skipped. comparison win/tie/loss mean 95% CI sign p committee(B) vs solo(A) 86/16/18 +0.567 [+0.392,+0.733] 8.2e-12 layered(C) vs solo(A) 90/11/19 +0.592 [+0.408,+0.758] 3.1e-12 layered(C) vs committee(B) 57/30/33 +0.200 [+0.008,+0.392] 1.5e-02 Consistent across every stratum. The length confound runs the OPPOSITE way and strengthens the result: the committee's answers are SHORTER than solo (2548 vs 3136 chars) and still win; restricted to the 61 trials where B was shorter, B wins 40-14 (p=5.4e-4). This reverses two pilot conclusions, both of which were artifacts of the empty-output bug that silently dropped 20/30 pilot trials (biased toward exactly the cases where the aggregator struggled): - "committee is directionally positive but not significant" -> it is significant by a wide margin; - "Together's layering is negative value, don't build it" -> RETRACTED; layered beats solo about as strongly as single-round, and edges single-round itself (though that is the weakest result and costs an extra peer round). Net across the whole investigation: tool selection shows no win for multi-model (route to the best tool-caller); reasoning/answer turns show a clear one (convene the committee). That task split is now evidence-backed rather than assumed. * feat(moa): cross-peer refinement round — a small-model mesh beats its best member The mesh value proposition, measured: a pool of four 8B-class models (qwen3-8b aggregating llama-3.1-8b + granite-4.1-8b + ministral-8b) produces better answers than the best single member — but ONLY with a refinement round. 40 preregistered reasoning prompts x 3 draws = 120 trials, judged pairwise by an out-of-pool different-family judge (gpt-4o-mini), position-swapped: small pool (all 8B): committee (1 round) vs solo 26/75/19 +0.058 p=0.37 ns layered (2 rounds) vs solo 42/66/12 +0.250 p=5.2e-05 *** layered vs committee 39/73/ 8 +0.258 p=5.5e-06 *** Single-round synthesis does nothing for a small pool; the cross-peer refinement round is what creates the gain. With a 32B aggregator both work and the extra round adds much less (p=0.015), so it isn't worth a second fan-out there. Adds `refinement::refine_round`: every worker sees all round-1 drafts (anonymized, length-bounded) and rewrites its own answer, then the reducer synthesizes the refined set. Runs on the text path only, and only when we were going to synthesize anyway — an early-exit consensus already has its answer. `GatewayConfig.refinement_policy`: Auto (default) - refine when the pool is all small-tier Always / Never - explicit override Mesh flavour: the round is best-effort. It bounds its own wait at worker_timeout, refines with whichever drafts arrived, tolerates workers that fail or time out, and returns the round-1 outputs unchanged on any shortfall — so a slow or absent peer can never cost the turn. This also retracts, for the second time, the pilot claim that "Together's layering is negative value". It is essential for exactly the pool shape a consumer mesh has. 194 tests pass; clippy -D warnings and fmt clean on both crates. * test(moa): pin refinement behaviour under mesh conditions; record small-pool evidence Refinement runs on exactly the hardware where peers are slowest and least reliable, and a second fan-out is a second chance to hang a turn. Four deterministic sim tests pin it as strictly best-effort: - refined drafts (not round-1) are what reach the client - a refiner that never returns cannot extend the turn past worker_timeout - every refiner failing degrades to the round-1 drafts, turn still answers - a big-tier pool skips the extra fan-out entirely under Auto Two of these failed first and both were my test's fault, not the engine's: identical round-1 text clustered into consensus so the turn early-exited before synthesis (peers now return distinct answers), and the "reducer must synthesize" assertion was too narrow — when refined drafts agree, the arbiter takes consensus and returns refined text directly, which is equally correct. The contract pinned is now "refined text reaches the client, round-1 text does not". RESULTS.md records the small-pool study that motivated the feature: an all-8B diverse pool (qwen3-8b aggregating llama-3.1-8b + granite-4.1-8b + ministral-8b) beats its best member only with the refinement round — single-round 26/75/19 (p=0.37, ns) vs layered 42/66/12 (p=5.2e-05), with layered over single-round at p=5.5e-06. Includes the honest caveats: ties dominate for the small pool, and unlike the strong-pool result the length control is not clean (answers are longer than solo). * fix(moa): bound the refinement round to half the worker budget Refinement sits between round 1 and the reducer, so giving it the full `worker_timeout` let one text turn pay three sequential worker budgets — worst on exactly the slow, high-latency meshes the feature exists to serve. The round now gets half the budget: enough for a pool that answered round 1 promptly, and it falls back to the round-1 drafts if the pool can't refine in time. Worst case drops from ~3x a plain turn to ~2.5x. * feat(moa): shorter patience on public meshes A public mesh is a pathological availability case — unknown peers, wider latency spread, more churn — not a trust case. Both patience knobs are "hold a usable answer hoping for a better one", which is exactly the wait that hurts when the tail is long. knob private public first_answer_grace 3s 1.5s strong_patience 20s 8s Hard bounds (worker_timeout, reducer_timeout) are unchanged; only the optional waiting shrinks, so quality paths still run when peers are prompt. Sourced from the node's existing `public_mesh` flag — no new config. Three tests pin the contract: public waits less on both knobs, public still gives a prompt strong peer a real chance (grace non-zero, patience >= 5s), and the private defaults don't drift silently. 1854 host-runtime tests pass; clippy -D warnings and fmt clean. * test(moa): prove refinement runs under the production grace setting Every refinement test so far set `first_answer_grace: ZERO`, but production uses 3s (1.5s public). Grace produces an `early_decision`, and refinement is skipped whenever one exists — so if grace fired on a fast all-small pool the whole feature would have been dead code on real traffic. That was untested. Two tests close the gap: - `refinement_still_runs_under_production_grace` — at the real 3s default, a prompt all-small pool still refines and the refined drafts reach the client. Grace only arms once its window has *elapsed*, so a pool that answers promptly reaches synthesis first. - `grace_shipping_a_slow_answer_skips_refinement` — pins the opposite case honestly: when round-1 answers straggle past the grace window, grace ships an early answer and refinement does not run. That tradeoff is now visible in a test rather than assumed. 195 engine tests pass; clippy -D warnings and fmt clean. * fix(moa): straggling peers must not cost the refinement round The answer grace was silently disabling the feature that makes a small mesh worth running. Grace fires at 3s (1.5s public) and produces an `early_decision`; refinement is skipped whenever one exists. On an all-small pool a single synthesis round is worth nothing (26/75/19, p=0.37) — refinement is the only step that beats the best member (42/66/12, p=5.2e-05). So one fast peer answering while others straggle would ship a lone small-model answer and skip the round, while still paying for the fan-out. Variable peer latency is the norm on consumer hardware, so this hit exactly the case the feature exists for. Now: when refinement is expected (policy + pool shape, decidable before dispatch via `refinement_expected`), the answer grace is disabled for that turn. Round 1 is still bounded by `worker_timeout` and refinement by its own half-budget deadline, so this costs bounded latency, never an unbounded wait. Pools that don't refine — anything with a big-tier model — keep the fast chat path untouched. Found by re-reading my own test against the goal: the previous `grace_shipping_a_slow_answer_skips_refinement` pinned this as an acceptable tradeoff. It wasn't; it was the feature not firing. Replaced with `straggling_peers_do_not_cost_the_refinement_round`, plus `grace_still_short_circuits_when_refinement_is_not_expected` to pin that the fast path survives where refinement is off. 202 engine tests + 1854 host tests pass; clippy -D warnings and fmt clean. * fix(moa): grace must bound the wait, not forfeit the refinement round On an all-small pool the answer grace could ship a lone fast answer and skip refinement — the only step that makes such a pool beat its best member (26/75/19, p=0.37 without it vs 42/66/12 with). Variable peer latency is the norm on consumer hardware, so this silently disabled the feature exactly where it exists to help, while still paying for the fan-out. Fix: `GatherPolicy.min_grace_answers`. Normally 1 (ship the good answer, stop waiting for the tail). When refinement is expected it becomes MIN_DRAFTS, so grace still caps the round-1 wait but will not finalize before refinement is even possible. Two earlier attempts were wrong and are worth recording: - Disabling grace outright for all-small pools removed the only straggler bound on round 1 (worst case round1 + refine + reducer), which is backwards on a public mesh whose whole profile is "wait less". - Letting refinement override an existing `early_decision` broke `same_tier_pool_keeps_early_exit`, and rightly: that decision is *consensus* (workers agreed — a real signal and the cheap path), not a timeout. Grace and consensus both surface as `early_decision` but mean opposite things; only the timeout should be prevented from pre-empting refinement. 202 engine tests + 1854 host tests pass; clippy -D warnings and fmt clean. * fix(moa): let the reducer and refiners actually see the answers Eval-vs-production fidelity gap, same class as the reference-packing bug. The small-pool result (42/66/12, p=5.2e-05) was measured on untruncated drafts averaging ~3.8k chars. Production truncated much harder: refinement input 1200 chars -> refiners saw ~30% of each peer draft reducer payload 500 chars -> reducer saw ~13% of each refined answer So the shipped path discarded most of exactly the content the refinement round produces. The measured gain could not have survived it. Both budgets are now 4000 chars, which passes realistic answers intact while still bounding a pathological worker. Tool turns keep the tight 500-char bound on purpose: there the signal is the proposal itself, and long prose crowds out the tool schemas. Three tests pin the contract (realistic text answers survive, tool payloads stay tight, refiners see full peer drafts), and the existing `reducer_truncates_long_worker_payloads` now probes above the new bound so it still guards the pathological case it was written for. 205 engine tests pass; clippy -D warnings and fmt clean. * fix(moa): ship the refinement prompt that was actually measured Final eval-vs-production fidelity check. Sampling already matched exactly (SamplingParams::worker, thinking off, 1024 tokens), but the prompt did not: measured (eval) "You have been given a user request and several candidate responses... Synthesize them into one high-quality response..." + "Candidate responses:" + [Response N] shipped (before) "Several models independently answered... write a better answer of your own..." + [Answer N] The eval wording is also what Together's advanced-moa.py uses — it reuses the aggregator prompt for refinement layers. A refinement-specific wording may well read better, but the eval wording is the one with evidence behind it, and "surely equivalent" prompt differences have already caused two wrong conclusions in this branch (the reference-packing bug and the layering dismissal). Changing it should be a measured change, not an assumed improvement. Production now matches the measured configuration exactly: prompt text, header, and per-draft labels. The sim fixture detected refinement calls by matching the old prompt string, so it needed updating too — it now keys on the "Candidate responses:" header. 205 engine tests pass; clippy -D warnings and fmt clean. * docs(moa): record the eval-vs-production fidelity gaps and their fixes A measured gain only counts if the shipped path reproduces the measured configuration. Three gaps were found after the numbers were collected: refinement input truncated to 1200 chars (~30% of a real draft), reducer payload to 500 (~13%), and a different refinement prompt than the one measured. All closed; tool turns keep the tight 500-char bound on purpose. Also states the honest limit: production now matches the measured configuration, but the small-pool result has not been re-measured through the shipped path since. The gain should carry — that is an expectation, not an observation. * fix(guardrails): panic slicing tool-call text with a multi-byte prefix `parse_parenthesized_tool_call` resumed parsing at `after_open[json_text.len()..]`, but `first_balanced_object` returns the object starting at its `{` — so any characters before the brace skew the index. With a multi-byte character in that prefix the index lands mid-character and slicing panics. Hit for real during an end-to-end MoA eval against live models: "start byte index 13 is not a char boundary; it is inside '‑'" (U+2011, non-breaking hyphen) — a worker task panic that would have taken down the turn. Adds `first_balanced_object_span` returning the byte span, so callers resume at the object's end rather than guessing from its length. Three regression tests: multi-byte prefix must not panic, happy path still parses, multi-byte content inside the arguments round-trips. 19 guardrails tests pass; clippy -D warnings and fmt clean. * fix(moa): stop telling the text reducer to "Be concise" End-to-end through `handle_turn`, MoA lost 7/8/65 to a single small model. Two causes; this is the second. The text reducer's instruction framed the turn as reconciling a *disagreement* and ended with "Be concise". Measured output: ~2.0k chars against a ~4.1k-char solo baseline — and the judge preferred the fuller answer. Terseness is not the goal on a reasoning turn; accuracy and completeness are. Text turns now use the wording the committee study actually measured (evals/moa-openrouter/RESULTS.md): "produce the most accurate, well-structured reply. Be direct." Tool turns keep the tight framing — there the output is an action and the reducer must stay free to emit a tool call. Also adds the e2e harness (`e2e_handle_turn_beats_best_single_small_model`) that found this: it drives the real `moa::handle_turn` over a live all-8B pool and judges it against the pool's strongest member. The committee study measures the mechanism with its own helpers; this measures the shipped path, which is where all three fidelity bugs actually hid. 205 engine tests pass; clippy -D warnings and fmt clean. * fix(moa): give every peer the full budget when refinement is expected Third and last cause of the end-to-end loss. Role tiers (Fast 256 tokens, Specialist 512, Strong 1024) encode a capability spread so the cheap worker can answer the grace path quickly. A homogeneous all-8B pool has no such spread, and when refinement is expected every draft is an *input* to the round — a 256-token draft is a ~1000-char stub that drags the refined answer down. Measured: production's tiered budgets produced ~3.1k-char answers against a ~4.1k-char solo baseline. The study that showed the +0.250 gain gave every peer the full 1024. When refinement is expected, all workers now pack as Generalist (full budget). Role identity is unchanged, so tier-gating and accounting still see the real roles; only the packing budget is uniform. 205 engine tests pass; clippy -D warnings and fmt clean. * fix(moa): text workers get a text preamble, not the tool-turn one Fourth divergence between the measured study and the shipped path. Every production worker was prepended MOA_PREAMBLE: "Respond with your best answer or tool call. Be direct." On a text turn that is wrong twice: there is no tool to call, and "be direct" pushes workers toward stubs. These drafts are the *input* to the refinement round, so brevity compounds — measured end-to-end, MoA answered ~3.3k chars against a ~4.1k-char solo baseline and lost on judged quality. The study that showed the gain gave workers no such instruction. Text turns now get a preamble that says the parts will be combined and asks for the most accurate and complete answer. Tool turns keep the original wording. 205 engine tests pass; clippy -D warnings and fmt clean. * fix(moa): don't give the reducer the worker preamble Fifth divergence between the measured study and the shipped path. The reducer's system prompt began with the *worker* preamble — "multiple models are answering this in parallel, give your most accurate answer" — and then immediately contradicted it with "you have been given several candidate responses; synthesize them". The combiner was being told it is also one of the answerers. The measured study gave the synthesizer the synthesis framing alone. `reducer_system_prompt` now passes the agent's own system prompt (tool guidance stripped on text turns) without the worker preamble. 205 engine tests pass; clippy -D warnings and fmt clean. * Revert "fix(moa): don't give the reducer the worker preamble" This reverts commit fb1281a0. Measured, the change made things worse, not better: with preamble (17/39/24) -> without (15/28/37) MoA mean output 3697 chars -> 3534 chars My reasoning was that the worker preamble ("multiple models are answering in parallel") gave the reducer the wrong role and contradicted the synthesis instruction. The e2e says otherwise: that preamble also says "the best parts of each will be combined; give your most accurate and complete answer", and the reducer was evidently using it to produce fuller answers. Removing it shortened the output and lost ground. Fifth hypothesis, first one the measurement rejected. Keeping the behaviour the evidence supports. * fix(eval): the e2e judge was scoring length, not quality Analysis of the 80-trial e2e run, conditioned on relative output length: MoA LONGER than solo: n=25 win 13 tie 12 loss 0 (100% winrate, p=2e-4) MoA SHORTER than solo: n=55 win 4 tie 27 loss 24 ( 14% winrate, p=2e-4) point-biserial r(length delta, verdict) = +0.681 mean delta when MoA wins: +682 chars; when it loses: -1337 chars Zero losses whenever MoA was longer. The judge prompt asked for "accurate, complete, and useful" — "complete" reads as "longer" — so the metric was verbosity, not quality. This is the exact bias the committee study controls for (position swap + length reporting) and that I failed to carry into the e2e harness. Consequence: "MoA loses 17/39/24 end-to-end" is NOT a valid quality finding, and four of the five fixes measured against it were partly optimising toward a biased judge. The correctness fixes stand on their own merits (a real panic, truncation destroying content, prompts contradicting the evidence); the *scoreboard* they were measured against did not. Judge now scores correctness/relevance only and is explicitly told length is not quality. * docs(moa): withdraw the length-biased numbers, record the corrected ones The judge that produced the small-pool and e2e results scored length, not quality (r = +0.681 between length delta and verdict; the longer answer won 13-0). Re-run with a length-controlled judge, r fell to +0.132. Corrected small-pool result (n=80, length-controlled): committee (1 round) vs solo 6 / 73 / 1 p = 0.125 ns layered (2 rounds) vs solo 11 / 68 / 1 p = 0.0063 significant layered vs committee 3 / 77 / 0 p = 0.25 ns The core claim survives: a small-model mesh beats its best member, but only with the refinement round. The magnitude does not — 42/66/12 is withdrawn, most of those "wins" were length artifacts and are now ties. Ties dominate (68/80): on most prompts a small mesh and a single small model are indistinguishable. The strong-pool section keeps its numbers and gains a note explaining why: the bias ran against the winner there (the committee's answers were shorter and won anyway, 40-14 on the shorter-only subset), so that result is unaffected. Status now records end-to-end parity (5/65/10, p=0.30) rather than the earlier loss, and lists closing the harness-vs-production gap as outstanding. * fix(moa): commit the grace_finalizes field definition HEAD did not compile. `lib.rs` set `GatherPolicy.grace_finalizes` (committed in "grace must bound the wait, not forfeit the refinement round") but the field definition in `fanout.rs` was never staged — it sat unstaged in my working tree across several commits, so every local build passed while the pushed tree was broken. Adds the field, its use in the gather loop (grace expiry stops collecting but does not finalize when refinement is expected), and `grace_finalizes: true` at the eight fanout test sites. Verified against a clean worktree checkout of HEAD, which is how the breakage was found: `cargo build -p mesh-mixture-of-agents` now succeeds there. * fix(moa): anonymize reducer inputs on text turns Sixth divergence between the measured study and the shipped path. Production labelled reducer inputs "[Worker N — model_name]"; the study that measured the gain used anonymous "[Response N]". Hermes anonymizes reference outputs for the same reason — a named model invites the aggregator to defer to the name rather than the content. Text turns now use "[Response N]". Tool turns keep attribution: there the reducer arbitrates between proposals, provenance is genuinely useful, and the tool-path tests pin it. The "## Worker outputs" header is unchanged — two sim fixtures use it to identify reducer calls. 205 engine tests pass; clippy -D warnings and fmt clean. * docs(moa): record that anonymization did not close the harness gap Sixth divergence fixed (named -> anonymous reducer inputs) and measured: before 5 / 65 / 10 p=0.30 after 9 / 59 / 12 p=0.66 Decided trials went 5W/10L -> 9W/12L. That is not an improvement at this sample size; both runs are parity. Recorded as such rather than as progress. Also records the gap that remains unexplained — harness 11/68/1 (p=0.0063) vs shipped 9/59/12 (p=0.66) on the same models, prompts, judge and packing — and lists the candidates not yet ruled out. Adds a caution that r(length,verdict) was +0.465 in the latest e2e run vs +0.132 in the small-pool study, so length bias is not fully suppressed even with the corrected judge; single-run e2e deltas are directional only. * fix(moa): text reducer gets only the synthesis framing Seventh divergence between the measured study and the shipped path, and the last known one in the reducer prompt. Production prepended two things the harness never had: * the worker preamble ("several models are answering this in parallel") — the reducer is the combiner, not one of the answerers; * "Reason for synthesis: {reason}" — an internal arbiter string such as "3 workers answered with no agreement", which reads to the model as a warning that its inputs are unreliable. Text turns now get exactly what the study measured: the synthesis instruction plus the candidate answers. Tool turns keep both extras — there the reducer is arbitrating a real disagreement, the reason is informative, and the agent's tool guidance matters. Note: an earlier commit removed only the preamble and appeared to make things worse, but that was measured with the length-biased judge, so the result was invalid and was reverted. This re-tests both extras together under the corrected judge. 205 engine tests pass; clippy -D warnings and fmt clean. * Revert "fix(moa): text reducer gets only the synthesis framing" This reverts commit 031a349a. Measurement rejected it, and this time the finding replicates. v7 preamble + reason kept 5/65/10 decided 33% MoA len 3606 v8 + anonymized inputs 9/59/12 decided 43% MoA len 3679 v9 - preamble - reason 8/57/15 decided 35% MoA len 3314 Removing the worker preamble shortened MoA's output on both occasions it was tried (v6 3534, v9 3314) versus keeping it (v7 3606, v8 3679), and lost ground on decided trials both times. The first rejection was under the length-biased judge so it did not count; this one is under the corrected judge and agrees. Reading: the preamble ("the best parts of each will be combined; give your most accurate and complete answer") is doing useful work on the reducer even though it is nominally addressed to a worker. Matching the harness exactly is not automatically right — the harness had no system prompt at all, while production has one, and the preamble evidently compensates. Second hypothesis rejected by measurement rather than argument. Keeping the behaviour the evidence supports. * docs(moa): record the two rejected hypotheses for the harness gap Both prompt-level explanations for the harness-vs-shipped gap were tested and rejected: baseline (v7) decided 5/15 = 33% MoA len 3606 anonymize reducer inputs (v8) decided 9/21 = 43% MoA len 3679 also drop preamble + reason (v9) decided 8/23 = 35% MoA len 3314 Dropping the worker preamble shortened output and lost ground both times it was tried (v6, v9) and was reverted both times. The preamble does useful work on the reducer even though it is nominally addressed to a worker — so matching the harness exactly is not automatically right: the harness sent no system prompt at all, production sends one, and the preamble compensates. Anonymization is retained (Hermes does it, the study measured it, it did not hurt) but 43% vs 33% on ~20 decided trials is not a result. Notes what remains untested (arbiter short-circuit on converged drafts, normalize_worker_output reshaping prose) and the honest conclusion: after two rejected guesses, the gap needs a byte-level diff of the prompts each path actually sends, not more hypotheses. * test(moa): diagnostic that dumps every prompt handle_turn actually sends Two prompt-level hypotheses for the harness-vs-shipped gap were tested live and rejected, so rather than guess a third this captures what production actually sends on a text turn — every role/content pair, in order. Deterministic and offline (canned backends, no network). What it shows for a 4-model all-small pool: calls 1-4 round-1 workers — text preamble + user prompt, max_tokens=1024 calls 5-8 refinement round — byte-identical to the harness's refine() call 9 reducer — harness synthesis prompt, plus the preamble, a "Reason for synthesis" line and a "## Worker outputs" header So the refinement round now matches the harness exactly, and the round-1 and reducer calls differ only by the preamble and the reducer scaffolding — both of which were removed in an experiment and measured *worse* (v9), then reverted. This is the instrument for the next person: run it, diff against the harness block it prints alongside, and the remaining delta is visible without guessing. 206 engine tests pass; clippy -D warnings and fmt clean. * fix(moa): refine over all drafts, not the bare minimum Eighth eval-vs-production divergence, found by reading the gather policy rather than guessing at prompts. `min_grace_answers` was set to MIN_DRAFTS (2) — the minimum for the refinement round to *run*. On a 4-model pool that let grace stop collecting at 2 of 4 drafts, so production refined over half the perspectives while the study that measured the gain always refined over all of them. Refinement quality scales with how many drafts it sees, so this directly undercuts the mechanism. Now waits for all but one draft (floored at MIN_DRAFTS), so a single straggler still cannot hold the turn and `worker_timeout` still bounds the wait. 206 engine tests pass; clippy -D warnings and fmt clean. * fix(moa): always synthesize after a refinement round Ninth divergence, and the first one found from outcome data rather than by reading code. Pooling the e2e runs showed the shipped path wins at roughly the harness rate (22/240 vs 11/80) but loses 32 times where the harness loses once. Splitting those turns by whether the reducer actually ran located the asymmetry: reducer ran n=66-74 win 8-9 loss 6-10 loss rate 9-14% reducer skipped n= 6-14 win 0 loss 2- 4 loss rate 29-33% MoA won 0 of 20 turns where synthesis was skipped, at triple the loss rate. That is a pure loss mode. Cause: `arbitrate` returns `Answer(payload)` when drafts agree, shipping one worker's text verbatim. That is the right cheap path for raw round-1 answers, but after refinement it discards the round we just paid for and returns whichever single small model represented the cluster. Refined drafts agreeing is not a reason to skip synthesis — it is the best possible input to it. Text turns that refined now always route to the reducer. Three sim assertions updated: the client now sees the reducer's synthesis of refined drafts rather than raw refined text. 206 engine tests pass; clippy -D warnings and fmt clean. * docs(moa): N=2 is not enough — pool-size comparison Direct test of "do 2 different models beat 1 of the same strength?", same 40 prompts / same judge / same harness, only pool size differs: N=2 (qwen3-8b + llama-3.1-8b) 2W 75T 3L p=1.00 5 decided N=4 (+ granite-4.1-8b, ministral) 11W 68T 1L p=0.0063 12 decided Fisher exact on decided win/loss, N=2 vs N=4: p=0.053. At 8B scale two peers produced no measurable gain — 2 wins against 3 losses. The same code with four models across four families wins 11-1. Two quantities move together as peers are added: decided trials rise (5 -> 12; more peers produce differentiated output instead of near-identical answers the judge ties) and the win share among them rises (40% -> 92%). Caveats recorded: 5 decided trials is weak evidence of absence, not evidence of no effect; and this is a claim about small models — Hermes reports a two-model frontier preset beating its stronger member by ~6 points, which is untested here. * docs(moa): N=2 null was scale, not count — mid-scale N=2 wins cleanly Mid-scale re-run refutes the "two peers isn't enough" reading. Same prompts, judge and shipped path, layered arm: N=2 8B (qwen3-8b + llama-3.1-8b) 2W 75T 3L p=1.00 N=4 8B (+ granite, ministral) 11W 68T 1L p=0.006 N=2 mid (qwen3-32b + mistral-24b) 49W 24T 6L p=2e-9 length r=-0.04 N=4 str (32b agg + 14b/24b/minimax) 90W 11T 19L p=3e-12 length r=+0.30 The N=2-mid result is the study's cleanest: no length confound, MoA answers shorter than solo, still won 93% of shorter-MoA trials. So member capability, not raw count, is the active variable — 8B needs ~4 models, 24-32B needs 2. Records the deciding caveat: this shows *ensembling* beats the best single member, not that cross-model *diversity* causes it — the baseline is strength-matched, not compute-matched. A compute-matched Self-MoA arm (two samples of the same model) is running to settle it; if it ties Mixed, the claim becomes "test-time ensembling beats one draw". * docs(moa): Self-MoA settles it — ensembling, not diversity, is the mechanism Compute-matched test (both spend 2 drafts + refine + synthesize; only member identity differs): Mixed (qwen3-32b + mistral-24b) 49W 24T 6L p=1.8e-9 length r=-0.04 Self (qwen3-32b x2) 48W 23T 2L p=2.3e-12 length r=-0.04 Mixed vs Self: Fisher exact p=0.27 — indistinguishable. Both crush the single best member, both confound-free (MoA answers shorter than solo, ~93% winrate on shorter-MoA trials). Different-family membership is not required. The active ingredient is test-time ensembling — several sampled drafts (temp 0.8), a refinement round, synthesis — and it works from repeated samples of one model as well as from different ones. Matches the Self-MoA paper (arXiv:2502.00674). Practical: the mesh does not need a curated diverse pool; any >=2 sufficiently capable participants beat picking one, distinct models or repeated instances. * docs(moa): the 8B ladder — need ~4, and same-vs-different doesn't matter Same prompts/judge/harness, layered arm: 2x 8B different (Qwen+Meta) 2W 75T 3L p=1.00 2x 8B same (Qwen x2) 2W 78T 0L p=0.50 3x 8B different 3W 76T 1L p=0.63 4x 8B different 11W 68T 1L p=0.006 Two 8B instances of the SAME model do exactly what two different ones do (nothing) — confirming count/strength drive the result, not family identity. Four 8B models is where it starts winning. Open gap: 4x same-model 8B not run, so whether the 8B N=4 win needs distinct models or just four drafts is extrapolated (mid-scale Self-MoA says four drafts alone should suffice). * fix(moa): refine on correlated drafts (homogeneous pools), not just small ones The refinement gate keyed on tier: `Auto` refined only when every model was small, and skipped whenever a big-tier model was present. That is wrong for the case the mesh goal cares about — a pool of one model repeated across nodes. Measured (evals/moa-openrouter/RESULTS.md), refinement's value tracks draft *correlation*, not size: same-model 32B x2: 48/2 with refinement vs 35/10 without (decisive) diverse mid 32B+24B: 49/6 layered vs 47/4 single (~no gain) all-small 8B x4: needs the round to beat its best member So `Auto` now refines when the pool is homogeneous (same canonical base — the repeated-instance / quant case) OR all small-tier, and skips the extra fan-out only for a diverse pool that already has a big-tier synthesizer. Adds `worker::canonical_base_name` (mirrors the host's dedup) and `pool_is_homogeneous`, with unit tests; refinement tests updated to pin the new homogeneous-big-tier behaviour. 212 engine tests pass; clippy -D warnings and fmt clean. * feat(moa): admission control — weak workers don't join a strong pool A modest node must not drag down a committee that already has a stronger member. Aggregation quality tracks proposal quality (Self-MoA, arXiv:2502.00674), and an 8B draft added to a 24-32B pool is expected noise-to-harm. `build_moa_config` now applies admission control before the >=2 check: when the pool mixes tiers, small-tier workers are dropped and only the big-tier ones kept. Consequences: * mixed pool (32B + 8B) -> 8B excluded, 32B committee (or solo if alone) * all-small (8B x4) -> untouched (the case that wins) * all-big / homogeneous -> untouched A lone strong model after exclusion falls through the >=2 check and simply serves solo — the safe outcome. `backends`/`models` are parallel vecs linked by backend_index, so both are rebuilt and reindexed together. Four unit tests pin the matrix (drops small when big present, keeps all-small, keeps all-big, keeps homogeneous-big). 68 moa_gateway tests pass; clippy -D warnings and fmt clean. * feat(moa): same-model mesh forms a pool (self-fill from extra instances) A mesh where every node serves the same model got no MoA at all: build_moa_config resolves one worker per canonical base, so N same-model nodes collapsed to 1 worker and failed the >=2 check. Self-MoA (arXiv:2502.00674) shows repeated sampling of one model ensembles as well as different models, so this is exactly the "add a modest node and it helps" case that was missing. When distinct-model resolution leaves a single model, `self_fill_from_extra_ instances` now adds extra reachable *nodes* serving that same model as workers (distinct remote endpoints only, never the same node twice, capped at 2). So: 1 node, 1 model -> 1 worker -> solo (unchanged, no surprise cost) 2 nodes, same model -> 2 workers -> MoA committee Combined with the refinement-gate fix (homogeneous pools now refine), a same-model pool runs the layered path that measured 48/2 for 32B x2. Extracted pool assembly (resolve loop + admission + self-fill) into `assemble_worker_pool` to keep build_moa_config under the cognitive-complexity limit. 68 moa_gateway tests pass; clippy -D warnings and fmt clean. * chore(moa): drop diag_prompt_diff scratch test It was an assertion-free diagnostic that dumps prompts to stdout — useful during the harness-vs-shipped investigation, but it always passes and does not belong in the merge set. The prompts it printed are now understood and the findings are recorded in evals/moa-openrouter/RESULTS.md. * docs(moa): admission control validated — weak node in strong pool is no-upside Arm C (32B x2 + one 8B, weak node admitted) vs arm B (32B x2): B 48W 23T 2L p=2e-12 decided winrate 96% C 50W 25T 5L p=2e-10 decided winrate 91% Fisher B-vs-C p=0.44 (not separable at n=80) but one-way: admitting the weak node never helped and raised losses 2 -> 5. Both still beat solo, so a weak node doesn't collapse the pool, but it adds cost for no upside and a small tail risk. This is the measured basis for tier-based apply_admission_control. * fix(moa): admission control must not collapse a lone-strong pool to solo You asked the right question: we showed admitting a weak node HURTS when the pool already has two strong (arm C), but never tested where admitting HELPS. The missing cell — one strong + one weak — is decisive and overturns the first-cut rule: 32B + 8B layered vs solo 32B: 47W 27T 5L p=1.3e-9 length-clean (r=-0.01) Admission control rejected the 8B here, collapsing the pool to a solo 32B. But the mixed committee beats solo decisively, so rejecting it threw away MoA in exactly the core "modest node joins a strong node and helps" case. Rule corrected: drop small-tier workers only when >=2 big-tier remain (a real committee survives). If dropping would collapse to solo, keep the mix. 32B x2 + 8B -> drop 8B (committee survives; 8B adds nothing, arm C) 32B + 8B -> keep both (dropping = solo, and the mix wins 47/5) Two admission tests updated/added to pin both cases. 69 moa_gateway tests pass; clippy -D warnings and fmt clean. * feat(moa): model=mesh gracefully degrades to single-model instead of 503 Before: a `model=mesh` request on a lone node (or a mesh with <2 workers) returned 503 "MoA requires ≥2 models". That makes `mesh` unusable as a default model — it only worked once a committee could form. Now: when `build_moa_config` can't form a committee but a real model is available, `try_handle_moa` rewrites `model=mesh` to that model and hands the stream back to normal single-model routing. `mesh` works everywhere — passthrough on one node, committee once a second worker joins. Only a node with no model at all still 503s. Both call sites handle the new fall-through: - passive/transport path already routes a returned `Some(stream)` normally; - host/ingress path: `try_handle_moa_intercept` gained a `Degraded { stream, model }` result. The pre-computed `decision.effective_model` is stale ("mesh") after degradation, so the degraded model name is threaded into `route_request`, and the pipeline classifier (computed against "mesh") is skipped for degraded turns. Unchanged from main was the hard 503; this is the missing piece for `mesh` as a superior always-on drop-in. 1859 host tests pass; clippy -D warnings and fmt clean. * feat(moa): cap the committee at 4 workers on large meshes Fan-out cost is ~2N+1 model calls per turn (N drafts + N refines + 1 synthesis), and measured quality is flat past ~4 workers while latency and spend keep climbing. On a big shared mesh — say 20 nodes — an uncapped pool fanned out to all of them: 41 calls for no quality gain. `cap_committee` now trims the assembled pool to the best MAX_COMMITTEE_WORKERS (4) by the same capability ranking used for actor selection (gossiped tool_use, then size, then stable index). Nodes beyond the cap are standbys — they still serve direct traffic, just not this committee. Runs after admission control and self-fill, so the cap applies to the final worker set. Extracted as a helper to keep assemble_worker_pool under the cognitive-complexity limit. 69 moa_gateway tests pass; clippy -D warnings and fmt clean. * test(moa): pin partial-worker-survival robustness sim_all_workers_fail covers total failure (clean structured error). This covers the common mesh reality: nodes flicker, so a subset of the committee dies mid-turn while the rest answer. MoA must degrade to survivors, not the error path. Three cases pinned: - half the committee dies, rest answer -> turn completes, all 4 accounted (2 succeeded, 2 recorded failed, none silently dropped) - lone survivor (3 of 4 die) -> turn still answers - slow-failing worker -> does not stall the turn past the survivors Locks the pre-existing best-effort fanout behaviour against regression now that assembly (admission, self-fill, cap) is more complex. 214 moa tests pass; clippy -D warnings and fmt clean. * refactor(moa): split pool assembly out of workers.rs into pool.rs workers.rs had grown to 1215 lines (over the 1k guideline) mixing two responsibilities: worker-pool assembly/selection and the model backends + config orchestration. Per CodeRabbit review, extract the assembly half. pool.rs (658 lines) now owns: assemble_worker_pool, resolve_one_worker_from_ aliases, add_worker_backend, group_aliases_by_canonical_base, is_locally_served, apply_admission_control, self_fill_from_extra_instances, cap_committee, compute_actor_candidates, canonical_base_name, WorkerBackendResolution, the two size caps, and their tests. workers.rs (578 lines) keeps: build_moa_config (orchestrator), the thinking-override helpers, patience_profile, the Local/Remote model backends + parse_quic_http_response, and their tests. Pure mechanical move, no behavior change. Backends and canonical_base_name made pub(super) for cross-module use. 69 moa_gateway tests pass (unchanged); clippy -D warnings and fmt clean. * test(moa): loosen partial-survival assertion that raced under early-exit `turn_completes_when_some_workers_die` asserted exactly succeeded==2/failed==2, but early-exit consensus can abort a live worker once a usable answer is in hand, so the succeeded/aborted split is timing-dependent. It passed locally but raced red in CI (Rust crate tests shard 0). The robustness contract is unchanged and still pinned: all four dispatched workers appear in summaries (none silently dropped) and the two dead ones are recorded as failed (>=2). Dropped the exact success-count assertion. Stable across 5 consecutive local runs. * fix(moa): model=mesh degrades to single-model serving, verified live `model=mesh` now works as a universal virtual model from one node upward: 1 node / 1 model -> serves that model (single-model passthrough) >=2 workers -> real MoA committee (fan-out + synthesis) 0 models -> 503 (correct: nothing to serve) Two bugs found and fixed against a live 2-node mesh (MacBook 3B + mini 1B): 1. request.raw not rewritten. degrade patched model_name/body_json but the forwarded request rides on request.raw bytes, which still said "mesh", so the embedded openai-frontend 404'd. Now uses rewrite_model_field (patches raw + body + Content-Length together; already unit-tested). 2. wrong model source. degrade read models_being_served(), empty on a fresh serve node. /v1/models draws from three sources; degrade now tries them in order: callable_models(targets) -> models_being_served() -> serving_models(). Live-verified: single node model=mesh returns HTTP 200 with a real answer (was 503/404); 2-node mesh model=mesh returns x-moa-turn=fanout workers=2. Known cold-start transient: the very first mesh request on a fresh serve node can 404 for ~1-2s until the routing table populates; solid once warm. * fix(moa): tier from verified GGUF/gossiped size, not name guessing Addresses two i386 P1s: destructive admission and committee-cap decisions were made from a name heuristic that treats every unparseable name as big-tier, so a small fine-tune/alias (`my-assistant`) could bypass the weak-worker filter or become the reducer — contradicting the branch's core "weak can't soil strong" claim. Producer (each serving node, has its own GGUF): scan_gguf_total_parameters sums tensor element counts for the exact stored-parameter count and publishes it via the existing ServedModelMetadata.parameter_count_b. profile.rs prefers this over name parsing; missing => None (unknown), never a guess. Consumer (MoA orchestrator, has NO peer GGUFs): reads the gossiped parameter_count_b off served descriptors. New SizeTier {Small,Big,Unknown}: * admission excludes only VERIFIED small, counts only VERIFIED big toward the ">=2 big remain" gate; Unknown is never strong and never filtered. * cap_committee ranks by verified size, NOT compute_actor_candidates (tool_use-first), which could evict a big model for tool-advertising small ones on an answer turn. Name parsing (model_param_size_b) remains a lower-confidence fallback only. 71 moa_gateway tests (2 new: unknown-never-filtered, name-parse fallback); model-artifact + host clippy -D warnings and fmt clean. * fix(moa): drop name-based size fallback — no GGUF size ⇒ weakest Per i386: parsing NNb from a served name is brittle and a destructive admission decision must not rest on an unverified label. Removed the name fallback entirely. Producer (profile.rs): parameter_count_b comes ONLY from the GGUF tensor sum. No local GGUF to sum ⇒ None (no guessed count). Deleted the now-dead parameter_count_b_from_text + its test. Consumer (pool.rs): tier_for uses only the gossiped verified size. Dropped SizeTier::Unknown — a model with no verified size is Small (weakest), so an unverifiable label can never masquerade as big and displace a real strong worker. cap_committee ranking updated (no-size ranks last). Admission excludes only verified-small; with no verified big there is nothing to protect so an all-unsized pool is untouched. Tests updated: unsized-worker-is-weakest (excluded next to verified big), no-verified-sizes keeps all. 71 moa_gateway + 28 profile tests pass; clippy -D warnings and fmt clean. * fix(moa): iron law — single physical endpoint never fakes a committee Self-fill could re-add the same physical node (or an endpoint already backing the sole worker), turning ONE box into a fake 2-worker committee that hits it twice for near-identical drafts. Catastrophic for mesh mode: a lone node must degrade to single-model serving, never pretend to be a committee. Rewrote self_fill_from_extra_instances to build the pool from DISTINCT physical endpoints only: the local skippy port (if this node serves the model and context fits) plus each distinct remote peer from hosts_for_model. If fewer than two distinct endpoints serve the model, the pool stays the single worker and MoA degrades to single-model. A single endpoint can no longer appear twice. Also fixes the CodeRabbit self-fill bugs (re-adds same peer; skipped context eligibility) — context fit is now checked on the local endpoint, and endpoints are distinct by construction. 71 moa_gateway tests pass; clippy -D warnings and fmt clean. * fix(moa): reject missing/malformed messages with 400 (homelab API-1) A `model=mesh` request with no `messages`, a non-array `messages`, or an empty array fell through to the workers and fabricated a 200 answer from nothing (homelab validation API-1). try_handle_moa now requires a present, non-empty `messages` array and returns 400 otherwise, before any model call. Verified live: {"model":"mesh"}, string messages, and [] all now 400; a valid request still returns 200. 71 moa_gateway tests pass; clippy -D warnings + fmt clean. * feat(moa): tier-aware committee width — small pools fan out to 6 Width sprint (evals/moa-openrouter, aggregator qwen3-8b, 8B peers, shipped committee path) shows the fan-out cap was throttling the pools that benefit most: 6× diverse 8B vs best member: 12W/65T/2L, p=0.013 (wins) 4× diverse 8B: 5W/73T/0L, p=0.06 (marginal) 2× diverse 8B: 2W/77T/1L, p=1.0 (null) Small, weak drafts need WIDTH — more independent proposals — before aggregation clears the best member. A verified big model, by contrast, wins at 2 and gains nothing past ~4. - committee_cap is now tier-aware: COMMITTEE_CAP_SMALL=6 for all-small pools, COMMITTEE_CAP_BIG=4 when a verified big model is present (replaces the flat MAX_COMMITTEE_WORKERS=4). - eval harness: the three per-trial judge comparisons now run concurrently (tokio::join!) instead of three serial awaits — ~3x faster judging phase, no behaviour change. Also measured (recorded in RESULTS.md): the refine round never beats single aggregation at 8B (refine-vs-single null in every cell) — Hermes' cheaper single-synth cadence matches Together's layered shape here; and at 8B diversity matters (6 diverse >> 6 same), unlike mid-scale. 72 moa_gateway tests (new: committee_cap_is_wide_for_small_pools_tight_for_big); clippy -D warnings and fmt clean. * fix(moa): don't refine small pools — width + single-agg wins, refine is dead cost The width sprint measured refine-vs-single-aggregation as null in every 8B cell (2/4/6 workers, diverse and same). Small pools win by WIDTH under a single aggregation, not by the extra serial refine pass. Previously RefinementPolicy:: Auto ran the round for ALL-small pools — paying two synthesis passes for no measured gain. Auto now refines only for a homogeneous pool that is NOT all-small (i.e. same-model at real scale, where correlated drafts + the round measured 48/2 vs 35/10). All-small and diverse pools skip it — matching Hermes' cheaper single-synth cadence, where refine buys nothing. Effect: an all-small mesh turn drops from 2 serial synthesis passes to 1, roughly halving added latency, with no measured quality loss. Tests: auto_skips_an_all_small_pool (was auto_refines_...); the 5 all-small mechanics sim tests (straggler/grace/degradation) pinned with Always so they still exercise the round; big-pool gate tests keep Auto. 178 + sim tests pass; clippy -D warnings and fmt clean. * fix(moa): make the shipped path deliver the measured win Three shipped-path bugs meant `handle_turn` never ran the committee it was supposed to. Found by measuring the shipped entrypoint at capable scale. 1. Grace pre-empted synthesis. `first_answer_grace` armed on the FIRST answer and finalized the turn, so on fast backends it fired ~every turn and shipped one worker's text. Measured 80/80 EarlyExit at 32B scale. Now grace finalizes on TOOL turns only; on answer turns it is a collection deadline — stop waiting for the tail, then synthesize what arrived. Private-mesh window widened 3s -> 10s so a normal committee completes before it arms. 2. Role-tiered draft budgets (Fast 256 / Specialist 512) truncated the drafts that synthesis consumes. Those tiers existed only to make the grace fast-path cheap; grace no longer finalizes answer turns, so every answer-turn worker now gets the full budget. 3. Answer turns shipped one worker verbatim when drafts agreed. Agreeing drafts are the best input to synthesis, not a reason to skip it. Answer turns with >=2 workers always synthesize now. Also drops the worker preamble from the reducer prompt — the synthesizer is not one of the parallel answerers, and the contradictory framing cost a weak aggregator. Measured through `moa::handle_turn` at production defaults, 40 prompts x 2 draws, out-of-family judge, length-controlled: 32B/24B/35B-MoE/14B pool vs best member: 71W / 8T / 1L p<0.0001 (was 26W/14T/40L before these fixes; wins 28/29 even when MoA is SHORTER, so it is not a length artifact) Robustness contracts unchanged and still verified: slow worker cannot stall a turn, lone survivor answers, hanging refiner cannot hold the turn, patience expiry releases held consensus. * fix(moa): all-small pools serve the best member instead of a committee Measured through the shipped path (OpenRouter, 8B-class peers with an 8B reducer, 40 preregistered prompts x 2 draws, out-of-family position-swapped judge, vs the pool's best member alone): 2x 8B: 0W/43T/37L p<0.0001 6x 8B: 5W/52T/23L p=0.0009 The committee never won and lost about a third of decided trials, with consistently shorter answers (3236-3372 chars vs ~4070 solo). A capable pool is the opposite (71W/8T/1L, p<0.0001), so this is a statement about *this* configuration -- a weak reducer synthesizing weak drafts -- not about small-model MoA in general. The untested cell is small peers with a strong reducer; if a mesh gains a big-tier model the pool is no longer all-small and MoA engages again. Rather than ship a measured regression, an all-small pool now collapses to its single strongest member, so the caller degrades to serving that model directly. Also corrects the private-mesh grace assertion to the widened 10s default. The earlier "6x 8B beats its best member (12W/2L, p=0.013)" harness result did not replicate: the same rig on the same 40 tasks now gives 3W/76T/1L (p=0.63), with 11 of 40 tasks flipping verdict. It rested on ~14 decided trials out of 80 (the rest ties) and was a single unreplicated run. Withdrawn. * docs(moa): withdraw the unreplicated 6x8B win, record all-small shipped results The width-sprint headline (12W/2L, p=0.013) did not replicate: same rig, same tasks, same pool now gives 3W/76T/1L (p=0.63), 11 of 40 tasks flipping. It rested on ~14 decided trials from a single unreplicated run. Records the shipped-path all-small measurements that motivated the best-member gate (2x: 0W/37L, 4x: 5W/12L, 6x: 5W/23L), scopes the claim to weak-reducer configurations, and documents known methodology limitations (tie bucket conflates failures, per-draw vs per-prompt significance, unverified best member, single judge). * fix(moa): min_grace_answers back to 1 — N-1 stalled turns to worker_timeout Public-mesh comparison against released v0.74.0 (same 3B model, same prompts, `model=mesh`) exposed a liveness regression I introduced earlier today: released v0.74.0: 4.0s / 3.2s / 11.3s (reducer never ran) branch (N-1 gate): 61.0s / 6.7s / 61.7s (reducer ran, SHORTER answers) `min_grace_answers` gates whether grace can ARM at all, so any value above 1 is a liveness hazard. With N-1 on a 6-worker public-mesh pool where two peers never returned, only 4 answers ever arrived, grace could never arm, and the turn rode `worker_timeout` (60s) instead. The two 61s turns had 6 workers/4 ok and 6/3; the fast 6.7s turn had 4 workers/3 ok, where N-1=3 was satisfiable. 3/3 consistent. Width comes from the grace WINDOW (10s), not a count gate: healthy peers land inside it and are all synthesized, while a dead peer costs 10s rather than 60s. This also restores the exact configuration the capable-pool win was measured under (71W/8T/1L, p<0.0001) — that run predates the N-1 gate — and the gate bought nothing on small pools either (8W/17L with it, 9W/17L without). * docs(moa): correct the gateway doc and flag the withdrawn small-pool claim MOA_GATEWAY.md described behaviour that no longer holds: - "requires >=2 distinct models, returns 503 if fewer" -> a single model (or an all-small pool collapsed to its best member) now degrades: the virtual `mesh` name is rewritten to a real served model and routed normally. Only a node serving nothing returns 503. - name-derived tiering -> tiering now comes from verified GGUF tensor sums gossiped as `parameter_count_b`, split at 10B, with unverified sizes ranking small so an unparseable alias cannot pose as big-tier. Name parsing mis-tiered real models (gemma-4-E4B stores 7.5B, not 4B). - documents pool shaping in order (admission, all-small best-member fallback, committee cap) with the measured numbers behind each. - records that answer turns pack every worker at the full budget. - stale peer-timeout row (15s) -> 60s worker timeout with the 10s grace window. RESULTS.md: the width-sprint section now carries an explicit WITHDRAWN banner pointing at the replication failure, so the 6x8B number cannot be cited from the middle of the file. * test(moa): pin the tool-turn contract against a live multi-model pool The tool path is deliberately asymmetric — route to the single best tool-caller rather than fan out and vote, because voting on tool calls measured null-to-harmful. That is only safe if structured `tool_calls` still come back intact when several models are on the wire, so this asserts the contract through the shipped `handle_turn` with a real pool: a tool prompt yields exactly one well-formed call whose arguments parse as JSON and whose name is an offered tool, and a no-tool prompt does not invent one. Measured live (OpenRouter, both cells pass): 4x 8B pool -> dispatched=4 ok=3 reducer=true list_dir({"path":"src"}) search({"path":".","pattern":"MeshError::Timeout\\(.*?\\)"}) run_command({"cmd":"pytest --verbose"}) no_tool_concept: no call invented 4x 24-35B pool -> dispatched=1 ok=1 reducer=true same shapes, single actor The dispatch difference is `ReferencePolicy::Auto` working as measured: a small-tier actor gets tool-free advisors (+0.017 uplift), a big-tier actor acts alone (advisors measured -0.037 there). The 4-worker cell also survived a worker flaking (ok=3 of 4) and still emitted a valid call, which is the ensemble-active case that was previously only covered by replay fixtures. Asserts the contract, not a quality delta — the tool chosen may legitimately differ from a single model's first move, so only a non-offered tool or malformed arguments fail. --------- Co-authored-by: Michael Neale <14976+michaelneale@users.noreply.github.com>	2026-08-06 18:41:34 +10:00
James Dumay	6c9e8e0a7f	feat(mesh): add pinned model-only OpenAI serving (#1148) ... * sanitize nested reasoning around tools * clarify local-only serving constraints * wait for owned OpenAI listener	2026-08-04 15:49:49 -04:00
James Dumay	934b8f992e	refactor(openai): remove text-form tool-call rescue (#1145) ... * refactor(openai): remove tool-call rescue paths * fix(openai): preserve truncated tool responses * style: format stacked OpenAI checks	2026-08-04 19:59:06 +10:00
Michael Neale	6adbd5b0f6	skippy: land Laguna Q4 and experimental Inkling split serving (#1118) ... * feat(skippy): add experimental Inkling text split serving * feat(skippy): add Laguna staged runtime candidate * docs(skippy): record Laguna package certification * fix(skippy): integrate hybrid verify recovery and lane cleanup * docs(skippy): record Laguna M5 parity * test(skippy): certify Laguna three-stage parity * docs(skippy): record Laguna distributed serving smoke * docs(skippy): record real Laguna mesh evidence * fix(skippy): make stage memory truly layer-local * fix(packaging): preserve source revisions * feat(skippy): honor package verification depth * test(skippy): assert Laguna cache policy * llama: linearize Laguna Inkling and recovery patches * fix(skippy): reconcile combined family recovery state * Harden combined Skippy family support * fix(skippy): harden combined family runtime * fix(skippy): retire final committed verify span * fix(skippy): retire failed lane before replacement * fix(skippy): retire partial exact replay trials * fix(skippy): close combined review gaps * docs(skippy): promote pinned Laguna Q4 package * refactor(skippy): isolate prediction return startup * fix: address consolidated model review * fix(skippy): surface orphan cleanup failures * skippy: complete Inkling tool path and operator notes * llama: refresh Inkling patch for updated upstream --------- Co-authored-by: Michael Neale <14976+michaelneale@users.noreply.github.com>	2026-08-03 17:03:05 +10:00
Nick DiZazzo	62074f5a5f	fix: speed up dependency-heavy container rebuilds (#1154) ... * fix(llama): support Bookworm Git patching * ci: cache container build dependencies * ci: bound sccache download time	2026-08-02 16:50:20 -04:00
Nick DiZazzo	265afa90bb	fix(ci): harden cache writes and readiness cleanup (#1123) ... * fix(ci): isolate high-fanout compiler cache writes * fix(ci): use service signal for readiness cleanup	2026-07-30 11:57:18 -04:00
Nick DiZazzo	3e30937ada	ci: compose reusable products and add Depot routing (#1113) ... * ci: compose reusable products and add Depot routing * ci: configure job-local sccache storage * fix: harden Windows artifact composition * test: assert pinned nightly artifact action * ci: allow superseded SDK smokes to cancel * docs: document cancellable CI fan-in gates * ci: harden composable build graph and metrics * fix(ci): install actionlint from verified release * fix(installer): normalize runtime digest paths * fix(ci): align installer contract output * docs(ci): ground runner image migration plan * ci: route trusted ARM lanes through Depot selector * ci: enable remote sccache for fast lanes * fix(ci): await remote sccache writes * fix(ci): align sccache policy contract * refactor(ci): reuse typed SDK and static ABI inputs * fix(ci): reuse configured sccache server * fix(ci): harden exact native cache reuse * fix(ci): make PR compiler cache read-only * fix(ci): isolate pull request compiler writes * feat(ci): produce immutable Node addon artifacts * fix(ci): restrict Depot canary to main * fix(ci): isolate Depot canary cache keys	2026-07-30 04:02:53 -04:00
Michael Neale	2d36978117	fix: resolve Qwen3.6 to the qwen35 recurrent family (#1109)	2026-07-30 04:06:09 +10:00
Nick DiZazzo	ed41f366dc	feat: unify release hosts and native runtimes ... - build one backend-neutral host per platform and package native runtimes separately - compose immutable product-v2 bundles from verified host and runtime artifacts - enforce host import policy and runtime provenance across release and SDK lanes - align debug, release, Windows, Kotlin, and Swift validation with the composed product model - update release documentation, CI topology, and agent guidance for the unified path - require no-device client readiness and bounded clean shutdown for packaged runtimes	2026-07-29 10:16:23 -04:00
Nick DiZazzo	09797ad243	Document canonical Homebrew tap (#1102)	2026-07-28 14:26:26 -04:00
Nick DiZazzo	c631070ed6	Rename the Node SDK npm package scope (#1101)	2026-07-28 13:31:59 -04:00
Nick DiZazzo	d91d62b72f	Hand npm publishing to mesh-packaging	2026-07-27 19:39:09 -04:00
Nick DiZazzo	ed286b909d	feat(runtime): add daemon model lifecycle reconciliation (#1082) ... * Add daemon-managed runtime model lifecycle Introduce persistent runtime lifecycle reconciliation with authenticated owner controls, profile-aware load, unload, ensure, and drain semantics, activity-based admission and priority policy, and additive gossip/protocol support. Expose the lifecycle through config, CLI, UI, and management APIs; consolidate shared owner-control protocol handling; and add cross-platform build and QA coverage for CUDA setup, mixed versions, process teardown, and SDK/platform paths.	2026-07-27 19:37:50 -04:00
Nick DiZazzo	cce1b0b417	fix: read-only model download caches (#1042)	2026-07-26 17:43:25 -04:00
Daniel Winter-Wijntjes	88f8b95b74	Add Ngram Suffix Proposer (#1037)	2026-07-22 22:05:39 +10:00
James Dumay	3e87c38060	🚬 Add bounded MTP + N-gram pipelining for latency-limited split inference (#1026) ... * Enable adaptive verify window for ngram/draft speculation The adaptive verify window was never enabled on the split-serving path: to_embedded_openai_args hardcoded adaptive_speculative_window = false. With a fixed window, an early reject never shrank the window, so a sustained reject storm kept proposing at full depth and paying the full 2-round-trip recovery cost per token. On a WAN split this measured as ~40% throughput loss with N-gram speculation ON versus OFF, despite high per-token acceptance. Enable the adaptive window whenever speculation actually proposes a window (ngram or draft mode). The existing shrink_adaptive_window logic then narrows the window toward the observed accept depth after an early reject, cutting recovery frequency. Adds a regression test asserting ngram speculation turns the adaptive window on. * Replace speculative rollback with positional MTP n-gram pipelining * Pipeline speculative verify windows across latency * Fix positional correction and adaptive pipeline depth * Continuously refill the speculative horizon * Productionize pipelined MTP n-gram speculation * Fix speculative docs and UI formatting * Remove stale speculative projections and fix CI * Handle fragmented direct-return fallback replies * Replace speculative repair with fixed-depth positional pipeline * Expose split-stage compute overlap telemetry * Lock split topology placement * Document locked split topology * Address locked topology review feedback * Fix SPEED-Bench timing JSONL output * Bound benchmark telemetry finalization * Hash SPEED-Bench request and response pairs * mesh: stop re-applying formation-time RTT gate to operational stage streams Ported from the WAN lab branch (wip/wan-direct-prediction-return, c340f741), where it was validated live on a ~26ms WAN split. open_stage_transport_stream re-applied the formation-time MAX_SPLIT_RTT_MS ceiling to every fresh operational stream, so per-request direct-return sinks were rejected under normal WAN RTT jitter while pooled forward lanes stayed healthy - surfacing as ready-handshake timeouts and 502s on an already-admitted split. Split admission still gates eligibility via gossiped, hysteresis-smoothed RTT plus re-election; operational streams now warn and proceed. * skippy: raise return-sink ready timeout 5s->20s for cold WAN bridge setup Ported from the WAN lab branch (46108cfc). Over a WAN mesh the return sink connects to a local bridge alias, but the remote ready byte only arrives after the bridge cold-establishes a fresh stage QUIC connection (~10s budget) and the remote handler dials its local server. 5s timed out during that cold setup on a healthy ~26ms split; forward lanes already use a 20s budget. Match it. * runtime: relaunch withdrawn splits when peers return instead of ending the model task Observed live on a real WAN split (Sydney M5 <-> AU 4090): one transient direct-return 502 led periodic_check to mark the remote stage unavailable; after the 75s grace the coordinator withdrew the topology. The Withdraw event returned StartupLoopControl::Break, so startup_local_model_loop tore down and the task ended permanently - while the remote worker sat healthy, logging 'standing by for stage assignment' forever. Only recovery was manually restarting both nodes with a fresh token. Make withdraw non-terminal: a new RelaunchSplit control/outcome runs the full existing teardown, then loops back to the launch phase and re-enters wait_for_split_participants, relaunching the split when an eligible peer returns. The stop channel is checked before relaunch so explicit shutdown still wins. LocalFallback (model fits locally) is unchanged. The participant-wait loop's 30s cadence and stable-participant gating act as the natural retry throttle; no extra backoff added. --------- Co-authored-by: Michael Neale <14976+michaelneale@users.noreply.github.com> Co-authored-by: Michael Neale <michael.neale@gmail.com>	2026-07-22 18:32:34 +10:00
James Dumay	57ffd599c3	Add GLM DSA package contract validation (#1032) ... * Add GLM DSA package contract validation * Address GLM-DSA package review feedback	2026-07-22 14:58:55 +10:00
James Dumay	c67c519909	Lock split topology placement (#1050) ... * Lock split topology placement * Document locked split topology * Address locked topology review feedback	2026-07-22 14:04:25 +10:00
James Dumay	2c5dacf212	Pipeline MTP-anchored n-gram verify windows (#938) ... * Replace Skippy verify span with verify windows * Add verify window reply metadata * Pipeline direct-return n-gram verify windows * Pipeline MTP-anchored n-gram verify windows * Support static release builds without features * Fix split MTP activation-frame serving * Replace native MTP batched verifier with verify windows * Restore native MTP verify window batching * Replace native MTP anchor extension with composite proposals * Keep composite decode branch on development version * Expose decode timings for all generation modes * Retry transient staged lane readiness * Bound persistent lane readiness handshake * Keep pure N-gram decode free of MTP drafts * Report composite proposal totals in decode timings * Gate composite decode pipeline by candidate depth * Account direct GGUF MTP weights in split planning * Avoid MTP cooldown after N-gram tail rejection * Improve hybrid MTP verification telemetry * Pipeline native MTP verification replies * Require useful N-gram tails for hybrid MTP * Adapt N-gram MTP extensions to tail acceptance * Fix direct GGUF planning fallback * Gate N-gram tails on MTP prefix agreement * Widen initial async verify windows * Restore anchored N-gram MTP extensions * Retain ready stages across transient refresh failures * Document pipelined VerifyWindow decode * Use llama.cpp N-gram proposer for Skippy * Add cache-based N-gram proposer * Add declarative speculative proposer package schema * Productize Skippy speculative decode plans * Productize Skippy speculative decode plans * Support direct N-gram Skippy plans * Validate speculative package strategy plans * Add coding agent loop benchmark corpus * Expose Skippy speculative benchmark counters * Validate cache N-gram proposer limits * Document speculative decode configuration * Fix native MTP proposals and fused restore routing * Honor configured N-gram extension width * Document speculative runtime overrides * Refresh speculative config schema contracts * Keep N-gram tail rejects from penalizing MTP * Preserve MTP state after serial tail rejects * Report adaptive verify width changes accurately * Fix short simple N-gram extension budgets * Make VerifyWindow pipelining cost-aware * Profile prospective VerifyWindow widths * docs: WAN split performance model + measured latency/compute decomposition Adds docs/skippy/WAN_SPLIT_PERF.md: the single-stream per-token cost model (TPOT ~= C_total + (S-1)*2*RTT + (S-1)*P), compute-bound vs latency-bound criteria, when adding a stage helps (memory, concurrency/pipeline overlap, dense compute-bound models), and speculation as the WAN amortization lever. Backed by 2026-07-18 Sydney<->Melbourne 2-node measurements: solo 12.9 ms/tok compute, split 57.8 ms/tok, decomposing to 12.9 compute + 40 (2xRTT) + 4.9 protocol. Workload was latency-bound (~78% network). * docs: plan for fast-fail on new requests routed to a dead split stage Documents the measured ~30s hang when a new request routes to a killed split stage, the confirmed root cause (60s heartbeat / lenient failure threshold + slow lane-open timeouts), and a two-layer fix (short steady-state lane-open deadline; feed lane failures into target_health cooldown) plus an explicit validation gate. Mesh-timing changes are out of scope pending live multi-node validation. * Fast-fail lane reconnects so new requests don't hang on a dead split stage When a downstream split stage dies, a new request would open a fresh lane and wait the full ~20s warmup ready-deadline before erroring (observed as a ~30s hang in the Sydney<->Melbourne kill test). The 20s deadline is only needed during pool warmup, when the downstream may still be loading its model. Split the deadline: pool warmup keeps LANE_READY_READ_TIMEOUT (20s); mid-life reconnects from checkout()/replace_lane() on an already-serving mesh use a new LANE_STEADY_CONNECT_TIMEOUT (3s). A healthy peer answers in milliseconds, so a dead stage now fails in ~3s instead of ~30s. Restores receive_persistent_lane_ready as the shared bounded-handshake helper (dropped during the main merge) and removes a now-obsolete retry test that covered pre-#1011 retry behavior. Adds tests asserting the steady-state deadline stays well under the warmup deadline and that the handshake read fails fast on a silent downstream. * docs: latency-aware placement — current behaviour and many-node gaps Records verified planner behaviour (skippy-coordinator/topology.rs, skippy-topology, host-runtime call site): - latency is a placement cost (rtt_ms penalty), not just relay-only exclusion - planner selects a node subset; does not have to use every eligible node - stage count is gated on a decode-TPOT target (shallower-that-meets beats deeper-that-does-not) And the gaps that matter at many-node scale: - no peer-to-peer RTT matrix in production (edge_signals never wired; only coordinator-RTT is used) -> co-located nodes cannot be exploited - network estimate is max(coordinator RTT) x node_count, a worst-case proxy - no first-class prefer-fewer/never-place-above-Y policy beyond the TPOT gate * docs: measured speculative recovery cost over WAN (why ngram hurts a latency-bound split) * Discard dead pooled stage lanes before reuse (fast-fail improvement) A pooled downstream lane whose stage died while checked in was a dead TCP stream; reusing it blocked the next generation read forever (handshake read-timeout is cleared for pooled lanes so long generations don't truncate). checkout() now probes lane liveness with a nonblocking peek and discards a dead lane so it reconnects with the short steady-state deadline instead of hanging. Validated on a loopback 2-node split with a mid-flight worker kill: new request now fails faster than main (60s vs main's 90s baseline). Does not fully solve the recovered-local routing path, tracked as follow-up. --------- Co-authored-by: Michael Neale <14976+michaelneale@users.noreply.github.com>	2026-07-19 20:14:59 +10:00
Nick DiZazzo	5104495551	feat: add extensible owned-node command system (#1002) ... * split node transport responsibilities * harden lifecycle and add inventory result * expose coalesced scan outcomes * dispatch typed scan refresh commands * expose typed owner-control scan refresh * document scan refresh compatibility * enforce command deadlines	2026-07-18 15:01:16 -04:00
Michael Neale	d982c3f194	docs: add agent-guided mesh installation runbook (#1003) ... * publish install runbook at meshllm.cloud/setup-mesh * add setup-mesh self-reference and absolute doc links in install runbook * add goal routing, filter installed-model noise, trim issue links - Add a goal->section routing table so single-machine goals (e.g. local coding agent) take a short path instead of the full multi-node mesh runbook. - Instruct assistants to filter `models installed` output down to runnable models, excluding layer-package internals and split shards, and show at most three candidates. Workaround for Mesh-LLM/mesh-llm#1004. - Collapse the GitHub issue reference list into inline pointers; the operational lessons are already folded into the relevant sections.	2026-07-17 11:30:22 -04:00
Nick DiZazzo	f13c9dc38b	feat: add plugin web UI extensions (#991) ... * Complete plugin web UI host surface * Align plugin web UI contract and docs * Harden plugin web UI release path * Fix plugin navigation and settings UX * Align local and CI repo consistency gates * Run publish consistency in test-all * Simplify native log test module wiring	2026-07-16 16:05:03 -04:00
James Dumay	4bfb2345cf	Refresh general documentation (#976)	2026-07-14 07:50:07 +10:00
Nick DiZazzo	36a44b9188	feature: Refactor setup-first installer flow (#933) ... * add uninstall command * parse checksums without awk intervals	2026-07-10 07:13:20 -04:00
Nick DiZazzo	aebfb02df4	feature: add benchmark 'tune' option (#948) ... * Add measured gpu tune benchmark trials * Record gpu tune benchmark lifecycle timings * Tune mmap and mlock load controls * Fix Linux mlock limit detection * Select benchmark tune settings with throughput tolerance * Key benchmark trial configs by canonical model * Wire speculative decoding benchmark tuning * Fix MTP tune detection for resolved model paths * Wire ngram speculative decoding for staged tune runs * Support layer package benchmark tune targets * Wire configurable native MTP tuning * Add non-frame native MTP decode ABI * add benchmark tune persistence and launch args flags to docs and runner * add support for draft acceptance fields * Resolve HF-style draft refs for speculative MTP config	2026-07-07 02:33:19 -04:00
Michael Neale	9547ac2a0d	Server-side tool-call emulation for models without native tool support (#946) ... * Add server-side tool-call emulation for non-tool-capable models Small / non-tool-trained models served through the mesh /v1 endpoint handle the OpenAI tools field poorly: the chat template ignores the schemas or the model loops re-issuing the same call. goose solved this in its local-inference provider, but that path is bypassed when a client talks to a mesh, and every other OpenAI client hits the same wall. The serving node is the only party that knows the loaded model's actual chat-template capability, so it emulates tool calling when the template cannot do it natively: - Detect capability from the runtime chat-template metadata (parse_tool_calls + non-empty chat_parser), not model size. Tool-capable templates are unchanged. - Adapt the request: strip tools/tool_choice, inject a compact instruction (name + description + compact parameter schema) teaching the TOOL_CALL {json} convention, and rewrite history so the template never sees tool roles. - Parse the response: scan for TOOL_CALL lines into OpenAI tool_calls with finish_reason tool_calls, tolerant of prose and <think> blocks; hold back partial markers while streaming. Ported from goose local-inference (tool_emulation.rs, tool_parsing.rs, tiny_model_system.md). Adds 19 unit + 5 integration tests. * Assert emulated tool_call shape in CI smoke The skippy CI smoke already sends a tools request to SmolLM2-135M, whose chat template does not support native tool calling, so it exercises the new server-side tool-call emulation path. It previously only asserted HTTP 200 and role==assistant. Strengthen it: use a tool-forcing prompt and a real token budget, and assert that any tool_calls returned by the emulation path are well-formed (non-empty function name with arguments that parse as a JSON object). Requiring a 135M model to reliably emit a tool call would be flaky, so a call is not forced, but malformed emulated calls now fail the smoke. * Fix native tool-call detection to use grammar_triggers The ported goose heuristic (parse_tool_calls && non-empty chat_parser) does not work against mesh-llm's patched llama.cpp: parse_tool_calls is true for every tools request and chat_parser is always a non-empty PEG structure, so the check was always true and emulation never fired. Detect native support from grammar_triggers instead: a tool-capable jinja template yields a tool-call grammar trigger (e.g. <tool_call>) when applied with tools, while a template with no native tool support (e.g. SmolLM2-135M) yields an empty grammar_triggers list. Verified live: SmolLM2-135M routes to emulation (native_supported=false), Qwen2.5-0.5B and Qwen3.5-0.8B keep native tool calling (grammar trigger present). * Add force-emulation override and revert CI smoke tool assertion - Add MESH_FORCE_TOOL_EMULATION override (goose's ToolCallingMode:: ForceEmulated analogue) so emulation can be exercised against strong models and used as an escape hatch when a native template misbehaves. Routed through should_emulate_tool_calls(). - Revert scripts/skippy-ci-smoke.sh to main: the two-node/binary smoke runs on tiny models (SmolLM2-135M) that cannot reliably emit a tool call, so probing for emulated tool_calls there would be flaky. The emulation path is covered by unit/integration tests and verified live. Verified end-to-end: with MESH_FORCE_TOOL_EMULATION=1, Qwen2.5-3B emitted the raw text 'TOOL_CALL {"name": "get_weather", "arguments": {"city": "Paris"}}', parsed into a real OpenAI tool_call with finish_reason tool_calls. * Make emulation frame dominant, parse markers anywhere, early-stop on tool call Three changes that make server-side emulation actually work for the weak / non-tool-trained models it targets, proven live on gemma-4-E4B: - Prompt dominance (goose-style, server-safe): the emulation instruction now leads the system message, with the client's original system content preserved below it under '# Task context'. This gives the tool-calling frame the dominant position goose gets by replacing the system prompt, without discarding the client's authoritative prompt (a serving node must not do that). Verified: gemma kept a pirate persona AND emitted the tool call. - Robust parsing: scan for the TOOL_CALL marker anywhere (with balanced-JSON extraction) instead of only at line start, so a call emitted right after a reasoning marker (e.g. gemma's <|channel>thought...channel|>TOOL_CALL {...}) is still parsed. Handles multiple calls and trailing prose. - Early stop (Jasper's tool_call_emitted -> Stop): stop generation once a complete emulated tool call is produced, so the model does not ramble past the call. Only active on emulated tools requests; native paths unaffected. Verified end-to-end with MESH_FORCE_TOOL_EMULATION=1: gemma-4-E4B emitted a parseable get_weather call (78 tokens, early-stopped) under a competing system prompt; native path (no force) unchanged. * Apply emulation early-stop on the split multimodal path Thread emulation_active through SplitMultimodalGeneration so the split multimodal generation path (embedded stage-0 with downstream lanes + media + tools for a non-tool-capable model) also stops generating once a complete emulated TOOL_CALL is produced, matching the local and non-split multimodal paths. Computed from hook_request/prompt at the construction site, which already has both in scope.	2026-07-03 11:50:56 +10:00
Nick DiZazzo	7a80952294	chore: synchronize release version management (#934) ... * chore: correct version strings and streamline release process * add 'just release x.y.z` support	2026-06-30 22:42:09 -04:00
Nick DiZazzo	179ee7f068	feature(config): Revamp configuration settings (#904) ... * add support for multi-model configurations * make configuration schema based * allow plugins to "register" settings * add agent guidance for adding / updating settings / schema	2026-06-26 15:20:46 -04:00
Nick DiZazzo	6d323c6b4e	fix(vram-display): clarify VRAM UX (#892)	2026-06-23 12:57:38 -04:00
Michael Neale	e30ab68da6	update guides for dev loop (#895)	2026-06-22 14:37:34 +10:00
James Dumay	2b8c41fdf2	Add native MTP generation metadata to layer packages (#888)	2026-06-22 10:38:06 +10:00
James Dumay	e65586668e	docs: correct metrics-server storage references (#885)	2026-06-20 11:33:16 +10:00
James Dumay	1324e61e84	Require explicit native runtime version selection (#873) ... * enforce native runtime version selection * Address native runtime selector review feedback * Improve native runtime CLI formatting	2026-06-20 11:31:20 +10:00
James Dumay	9b961c0835	Improve GLM MTP parity and batched verify sampling (#858) ... * Add Skippy cross-request token decode batching * Batch Skippy split decode frames across requests * Avoid fixed decode batch rendezvous waits * Document micstudio-first lab startup * Keep lab startup note in skill only * Carry split GGUF tensor spooling patch * Document native MTP Skippy architecture * Add native MTP n1 verification scaffold * Add GLM DSA native MTP graph patch * Add native MTP n1 decode ABI * Wire native MTP n1 sidecar drafts * Teach correctness harness native MTP sideband * Gate correctness on native MTP drafts * Document GLM 5.1 native MTP proof gate * Pivot native MTP proof gate to GLM 4.7 * Record GLM 4.7 MTP artifact gap * Point GLM 4.7 MTP gate at meshllm artifact * Preflight native MTP correctness artifacts * Preserve fused native MTP drafts * Verify native MTP n1 correctness * Cover native MTP direct return replies * Support DeepSeek2 GLM MTP n1 split drafts * Add native MTP batched verification path * Fix GLM native MTP batched verification * Gate native MTP batched verification * Add native MTP OpenAI A/B correctness check * Compare native MTP against greedy baseline * Prepare native MTP correctness for lab endpoints * Add remote stage1 native MTP correctness launch * Trace lab split stage readiness * Return split predictions over stage lane * Checkpoint native MTP batched rejections * Restore native MTP hybrid trim rejection * Normalize GLM native MTP patch queue * Include native MTP tensors in final slices * Expose GLM MTP hidden state after output norm * Improve GLM Skippy single-stage parity * Sample batched GLM MTP verify frames * Gate serial stage0 MTP verification experiment * Return MTP drafts from verify spans * Improve GLM MTP parity instrumentation * Instrument GLM stage0 MTP verify comparison * Add row-level GLM stage0 compare telemetry * Add serial VerifySpan diagnostic path * Add reject cooldown for batched MTP * Add serial recovery gate for batched MTP * Add MTP verify-next margin sideband * Add MTP verify-next margin summary telemetry * Add deferred MTP reject trim diagnostic * Add MTP cooldown draft suppression diagnostic * Add MTP cooldown suppression limit diagnostic * Add native MTP draft origin telemetry * Add gap-origin MTP recovery override * Add decode-sidecar MTP margin telemetry * Gate decode-sidecar MTP margin computation * Add gap-origin MTP skip probe diagnostic * Target serial stage0 verify to MTP gap reentry * Add native MTP recovery scheduling telemetry * Add native MTP verify-next recovery telemetry * Add VerifySpan summary timing telemetry * Add serial-after-gap direct MTP verify diagnostic * Add HF GGUF quant jobs skill * Remove unused skippy ABI exports * Squash GLM MTP llama patch tail * Split native MTP modules * Fix skippy stage-lane smoke drivers * Fix MTP sidecar review gaps * Instrument Skippy batched MTP verifier overhead * Optimize skippy reply stats transport * Avoid summary-path MTP debug bookkeeping * Reduce summary eviction telemetry on MTP path * Add VerifySpan compute summary telemetry * Add Skippy auto-align summary telemetry * Add Skippy MTP margin outcome telemetry * Fix native MTP margin threshold fallback * Expose VerifySpan width telemetry * Add local VerifySpan microbench * Profile in-process VerifySpan split overhead * Compare split VerifySpan batched and serial paths * Add split VerifySpan timing diagnostics * Expose VerifySpan overhead breakdown * Avoid copying VerifySpan activation inputs * Expose batched MTP proposal timing * Add VerifySpan native timing diagnostics * Break down VerifySpan MTP sync timing * Split VerifySpan MTP sync setup timing * Add Skippy greedy sampling fast path * Trim losing MTP verifier diagnostics * Trim native MTP review path * Restore direct return for split predictions * Remove unsafe deferred MTP reject trim * Fix cached replay token accounting * Split native MTP decode counters * Wire prediction return defaults for embedded skippy * Fail open direct prediction returns * Restore split MTP fallback sideband * Reverse direct prediction return setup * fix embedded split prediction return listener * fix split decode direct return wait	2026-06-18 15:40:12 +10:00
James Dumay	17452d19db	clarify skippy prompt cache reuse (#856)	2026-06-15 21:43:37 +10:00
Nick DiZazzo	247dd7e42a	feat(skippy): Add native runtime event visibility (#842) ... * repair runtime events patch metadata * align Windows ABI cache consistency checks	2026-06-14 02:57:14 -04:00
Michael Neale	226d1c6bae	Consolidate agent skills and fix stale docs (Windows deploy, repo map, design docs) (#836)	2026-06-12 18:44:36 +10:00
Nick DiZazzo	ef406008a1	Add meshllm.cloud website, catalog viewer, and onboarding docs (#806) ... * Add homepage PR screenshots * Add site content PR screenshots * Add install guide PR screenshot * Remove homepage PR screenshots * switch CI to actions-based pages workflow * give site a dedicated workflow * add checksum instructions / switches * add final verification step * Update CLI ref for blackboard and plugins * Remove generated pages from git	2026-06-10 05:18:25 -04:00
James Dumay	2e51730400	Reduce Skippy decode return latency (#794) ... * Make skippy prediction returns direct-only * Fix skippy-bench direct return flow * Fix direct return smoke wiring * Fix correctness direct return smoke * Wire prompt direct prediction returns * Fix split return endpoint address * Always configure split generation returns * Fix split direct return transport bridge * Avoid cloning F32 activation payloads (#795)	2026-06-05 15:18:33 +10:00
James Dumay	c91ec5e83c	Implement latency-aware Skippy stage count (#789)	2026-06-05 15:13:11 +10:00
James Dumay	7872f870f9	Break skippy stage protocol for direct returns (#787)	2026-06-04 14:31:28 +10:00
Michael Neale	97c0cad991	feat: native rust SDK for rust consumers. (#736) ... * a run at SDK, tested with a client * Use in-process shutdown for embedded SDK * Satisfy clippy for embedded shutdown plumbing * Document embedded Rust SDK usage * Add public Rust SDK crate * Tighten embedded SDK lifecycle * Address embedded SDK review feedback * Document native runtime packaging direction * Document runtime CLI namespace * Document runtime CLI UX expectations * Document runtime diagnostics under doctor * Add Windows PowerShell installer * Document recommended runtime install flow * Add native runtime resolver foundation * Wire native runtime release installs * Document native runtime crate * Load versioned native runtimes dynamically * Fix embedded SDK output manager reset * Expose SDK mesh admission controls * Split SDK runtime mapping assertions * Tighten SDK docs and config module * Clarify native runtime SDK TODOs * Stabilize native log note test * Expose native runtime install SDK * Re-export native runtime APIs from SDK crate * Add embedded SDK knobs for Sprout relay mesh (#782) Two opt-in seams the Sprout v1 mesh integration needs: - disable_iroh_relays(bool): when true, embedded runtime selects an explicitly disabled relay policy, which uses RelayMode::Disabled, skips public relay URL fallback, skips raw STUN, and avoids the 5s endpoint.online() wait that cannot succeed without a home relay. Default false preserves existing behavior. - EmbeddedNodeHandle::join_token(token): forwards an invite token over the runtime control channel to node.join_with_retry so an already-running embedded node can dial a new EndpointAddr without restart. Handled in both auto and passive/client runtime loops. Purely additive; existing defaults and startup join_tokens behavior remain unchanged. Co-authored-by: npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@sprout-oss.stage.blox.sqprod.co> Co-authored-by: Perci <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@sprout-oss.stage.blox.sqprod.co> * Fix relay policy test visibility * Make SDK publishable and align language bindings (#771) * Split SDK native runtime publish surface * Split CLI and TUI support crates * Move CLI parser surface into mesh-llm-cli * Extract shared mesh event surface * Move standalone command handlers out of host runtime * Move benchmark and plugin commands out of host runtime * Finish plugin command extraction * Extract auth identity ownership * Move remaining standalone commands out of host runtime * Move model store into model-hf * Move CLI commands out of host runtime * Decouple host runtime from CLI and TUI crates * Expose embedded node SDK facade * Keep client identity dependencies pure * Simplify Rust SDK feature surface * Keep SDK client feature runtime-free * Expose SDK client API base override * Use direct mesh SDK client transport * Remove API base URL client builder shim * Align language SDKs with Rust SDK facade * Document SDK client and serving modes * Fix SDK smoke runtime setup * Package SDK console assets * Fix dynamic runtime CI setup * Restructure SDK docs by language * Fix SDK smoke package loading * Fix native runtime bundle resolution * Add structured native runtime backend metadata * Harden Kotlin native runtime smoke resolution * Fix mesh-llm-sdk clippy imports * Retry smoke model downloads --------- Co-authored-by: James Dumay <jameswdumay@gmail.com> Co-authored-by: tlongwell-block <109685178+tlongwell-block@users.noreply.github.com> Co-authored-by: npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@sprout-oss.stage.blox.sqprod.co> Co-authored-by: Perci <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@sprout-oss.stage.blox.sqprod.co>	2026-06-03 12:39:19 +10:00
Ivan Golovach	84da8b6d52	Harden mDNS LAN discovery contract (#766) ... Harden LAN-only mDNS discovery Validation * Validation tier: Tier 2R - post-review correction for PR #766. mDNS LAN detail advertisement is gated by management API reachability, and LAN detail responses use the same publication metadata as the mDNS publish loop. No mesh wire/protobuf, Skippy ABI, plugin protocol, or release metadata changed. * git fetch --no-tags origin main:refs/remotes/origin/main: PASS, origin/main at f9bd75a973. * git rebase origin/main: PASS, no conflicts. * git diff --check origin/main...HEAD: PASS, no output. * git diff --check: PASS, no output. * git diff --cached --check: PASS, no output. * cargo fmt --all -- --check: PASS. * LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal cargo test -p mesh-llm-host-runtime lan_advertisement --lib -- --test-threads=1: PASS, 4 passed. * LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal cargo test -p mesh-llm-host-runtime mdns --lib -- --test-threads=1: PASS, 8 passed. * LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal cargo test -p mesh-llm-host-runtime lan_details --lib -- --test-threads=1: PASS, 4 passed. * LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal cargo check -p mesh-llm: PASS. * LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal /opt/homebrew/bin/cargo-clippy clippy -p mesh-llm-host-runtime --all-targets -- -D warnings: PASS. * Ledger: not applicable - not required for selected validation tier/change family. * Version: not applicable - no release/version sync required for this non-release LAN discovery hardening correction. * Not run: live two-node LAN mDNS smoke - no second LAN node was available in this isolated worktree; advertisement gating, route metadata, token-proof, mDNS mode, relay policy, shipped-binary check, and clippy cover the changed branches. * Not run: full workspace suite locally - mandatory PR CI is the final full-suite proof for the pushed SHA. Rollback * git revert HEAD	2026-06-02 09:17:05 -07:00
Ivan Golovach	a9c1862e93	Add local target reputation routing (#767) ... Validation * Validation tier: Tier 3 - shared OpenAI/mesh routing health behavior plus post-review design/test hardening for PR #767; retryable target failures leave a bounded local reputation penalty, route selection avoids cooling targets when alternatives exist, and /api/status exposes local target-reputation counters without protocol, gossip, or trust-score changes. * git fetch --no-tags origin main:refs/remotes/origin/main: PASS, origin/main at f9bd75a973. * git rebase origin/main: PASS, no conflicts. * git diff --check origin/main...HEAD: PASS, no output. * git diff --check: PASS, no output. * git diff --cached --check: PASS, no output. * cargo fmt --all: PASS. * cargo fmt --all -- --check: PASS. * LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal cargo test -p mesh-llm-host-runtime affinity --lib -- --test-threads=1: PASS, 18 passed. * LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal cargo test -p mesh-llm-host-runtime target_health --lib -- --test-threads=1: PASS, 13 passed. * LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal cargo check -p mesh-llm: PASS. * LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal /opt/homebrew/bin/cargo-clippy clippy -p mesh-llm-host-runtime --all-targets -- -D warnings: PASS. * Ledger: not applicable - not required for selected validation tier/change family. * Version: not applicable - no release/version sync required for this non-release local routing-health change. * Not run: live multi-node reputation/routing smoke - no local multi-node model runtime endpoint was available; deterministic target-health, affinity-routing simulation, and status-payload coverage from the base PR cover the changed local behavior. * Not run: full workspace suite locally - mandatory PR CI is the final full-suite proof for the pushed SHA. Rollback * git revert HEAD	2026-06-01 17:08:39 +10:00
Ivan Golovach	d9d9096955	Add plugin startup resilience diagnostics (#759) ... Add plugin startup resilience diagnostics Validation * Validation tier: Tier 3 - shared plugin startup/config/runtime behavior plus doctor diagnostic capture, refreshed onto current main for PR #759; no plugin protocol/schema or release metadata change. * git fetch --no-tags origin main:refs/remotes/origin/main codex/plugin-startup-resilience-doctor:refs/remotes/origin/codex/plugin-startup-resilience-doctor: PASS, origin/main at 4f02a65c. * git rebase origin/main: PASS, no conflicts. * git diff --check origin/main...HEAD: PASS, no output. * git diff --check: PASS, no output. * git diff --cached --check: PASS, no output. * cargo fmt --all -- --check: PASS. * cargo test -p mesh-llm-config plugin_startup --lib: PASS, 2 passed. * LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo test -p mesh-llm-host-runtime plugin::tests --lib -- --test-threads=1: PASS, 25 passed. * LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo test -p mesh-llm-host-runtime split_doctor_captures_plugin_startup_surfaces --lib -- --test-threads=1: PASS, 1 passed. * LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo test -p mesh-llm-host-runtime runtime_data --lib -- --test-threads=1: PASS, 27 passed. * LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo check -p mesh-llm: PASS. * LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo clippy -p mesh-llm-config -p mesh-llm-host-runtime --all-targets -- -D warnings: PASS. * Remote CI: PASS on refreshed head 06f9411e; PR Builds and PR Quality Checks completed successfully. * Ledger: not applicable - not required for selected validation tier/change family. * Version: not applicable - no release/version sync required for this non-release plugin runtime diagnostic change. * Not run: just build - not required for selected validation tier; no UI assets or release bundle changed. * Not run: live legacy CPU/intelSDE plugin startup smoke - no local legacy/emulated plugin host was available; targeted config, runtime, API, and doctor tests cover the changed branches. Rollback * git revert <merge-commit-sha>	2026-05-31 13:52:48 -07:00
James Dumay	283462001a	Drop deprecated mesh-bundle release asset (#755)	2026-05-31 11:18:07 +10:00
Ivan Golovach	06cad32feb	Upgrade demanded local models during reconciliation (#754) ... Upgrade demanded local models during reconciliation Validation * Validation tier: Tier 3 - opt-in runtime reconciliation can replace a lower-demand local model with a locally present, fresh active-demand target; config admission, planner policy, runtime unload/load sequencing, API target signals, and docs are refreshed on current main after PR #753 landed. * git fetch --no-tags origin main:refs/remotes/origin/main: PASS, origin/main at c0f8990bb1. * git rebase origin/main: PASS after resolving one test-module conflict in crates/mesh-llm-host-runtime/src/runtime/mod.rs by preserving both the newly landed mDNS relay-policy tests and this PR's model-target replacement sequencing test. * git diff --check origin/main...HEAD: PASS, no output. * git diff --check: PASS, no output. * git diff --cached --check: PASS, no output. * cargo fmt --all -- --check: PASS. * cargo test -p mesh-llm-config --lib -- --test-threads=1: PASS, 10 passed. * LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo test -p mesh-llm-host-runtime model_target_reconciliation --lib -- --test-threads=1: PASS, 22 passed. * LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo test -p mesh-llm-host-runtime api::model_targets --lib -- --test-threads=1: PASS, 3 passed. * LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo test -p mesh-llm-host-runtime mdns_discovery --lib -- --test-threads=1: PASS, 3 passed. * LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo check -p mesh-llm: PASS. * LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo clippy -p mesh-llm-config -p mesh-llm-host-runtime --all-targets -- -D warnings: PASS. * Remote CI: PASS on final PR #754 SHA d48861a2, including PR Quality Checks, Linux/macOS builds, fmt, clippy, unit/protocol/skippy/sdk smoke, client-auto boot, two-node client-serving smoke, and two-node split smoke. * Ledger: not applicable - not required for selected validation tier/change family. * Version: not applicable - no release/version sync required for this non-release opt-in runtime reconciliation change. * Not run: live multi-node demand/reconciliation smoke - no local multi-node runtime endpoint and local GGUF set were available; config, model-target signal, planner, runtime-control, conflict-adjacent mDNS tests, and remote CI cover the changed branches. Rollback * git revert HEAD	2026-05-30 15:00:35 -07:00
Ivan Golovach	fef0f4dedd	Harden LAN-only mDNS mesh startup (#746) ... Make mDNS mesh startup LAN-only Validation * Validation tier: Tier 2R - post-review/rebase correction for PR #746; LAN/mDNS relay policy remains runtime-significant, but the final update is a narrow same-PR correction that gates relay-health monitoring by discovery mode and refreshes proof on current main. * git fetch --no-tags origin main:refs/remotes/origin/main: PASS, origin/main at 95695f9f7d. * git rebase origin/main: PASS, no conflicts. * git rev-list --left-right --count origin/main...HEAD: PASS, 0 behind / 1 ahead. * git diff --check origin/main...HEAD: PASS, no output. * git diff --check: PASS, no output. * git diff --cached --check: PASS, no output. * cargo fmt --all -- --check: PASS. * LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo test -p mesh-llm-host-runtime relay_policy --lib -- --test-threads=1: PASS, 4 passed. * LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo test -p mesh-llm-host-runtime relay_health_monitor --lib -- --test-threads=1: PASS, 2 passed. * LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo test -p mesh-llm-host-runtime mdns --lib -- --test-threads=1: PASS, 6 passed. * LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo test -p mesh-llm-host-runtime relay_health --lib -- --test-threads=1: PASS, 6 passed. * LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo test -p mesh-llm-host-runtime runtime_data --lib -- --test-threads=1: PASS, 27 passed. * LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo test -p mesh-llm-host-runtime status_payload --lib -- --test-threads=1: PASS, 12 passed. * LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo check -p mesh-llm: PASS. * LLAMA_STAGE_BUILD_DIR=<stage-build-dir> /opt/homebrew/bin/cargo-clippy clippy -p mesh-llm-host-runtime --all-targets -- -D warnings: PASS. * Remote PR Quality Checks on c060d1dc7a: PASS. * Remote PR Builds on c060d1dc7a: PASS. * Ledger: not applicable - not required for selected validation tier/change family. * Version: not applicable - no release/version sync required for this non-release runtime discovery hardening PR. * Not run: live two-node LAN mDNS smoke - no second LAN node was available in this isolated worktree; relay policy, monitor gating, CLI rejection, status propagation, and host-runtime targeted tests cover the changed paths. Rollback * git revert HEAD	2026-05-29 23:49:41 -07:00