mirror of
https://github.com/Mesh-LLM/mesh-llm.git
synced 2026-08-08 22:23:19 -04:00
137 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
James Dumay
|
6c9e8e0a7f
|
feat(mesh): add pinned model-only OpenAI serving (#1148)
* sanitize nested reasoning around tools * clarify local-only serving constraints * wait for owned OpenAI listener |
||
|
hola-there
|
3ff69b9753
|
Enhance README with Windows & WSL2 troubleshooting info (#1132) | ||
|
Nick DiZazzo
|
3e30937ada
|
ci: compose reusable products and add Depot routing (#1113)
* ci: compose reusable products and add Depot routing * ci: configure job-local sccache storage * fix: harden Windows artifact composition * test: assert pinned nightly artifact action * ci: allow superseded SDK smokes to cancel * docs: document cancellable CI fan-in gates * ci: harden composable build graph and metrics * fix(ci): install actionlint from verified release * fix(installer): normalize runtime digest paths * fix(ci): align installer contract output * docs(ci): ground runner image migration plan * ci: route trusted ARM lanes through Depot selector * ci: enable remote sccache for fast lanes * fix(ci): await remote sccache writes * fix(ci): align sccache policy contract * refactor(ci): reuse typed SDK and static ABI inputs * fix(ci): reuse configured sccache server * fix(ci): harden exact native cache reuse * fix(ci): make PR compiler cache read-only * fix(ci): isolate pull request compiler writes * feat(ci): produce immutable Node addon artifacts * fix(ci): restrict Depot canary to main * fix(ci): isolate Depot canary cache keys |
||
|
Nick DiZazzo
|
ed41f366dc
|
feat: unify release hosts and native runtimes
- build one backend-neutral host per platform and package native runtimes separately - compose immutable product-v2 bundles from verified host and runtime artifacts - enforce host import policy and runtime provenance across release and SDK lanes - align debug, release, Windows, Kotlin, and Swift validation with the composed product model - update release documentation, CI topology, and agent guidance for the unified path - require no-device client readiness and bounded clean shutdown for packaged runtimes |
||
|
Nick DiZazzo
|
09797ad243
|
Document canonical Homebrew tap (#1102) | ||
|
Nick DiZazzo
|
a6c7ddb7fd
|
Document public packaging installation channels
Closes #971. Replace the nonexistent Homebrew tap instructions, document public packages and images, and route release dispatches to Mesh-LLM/mesh-packaging. |
||
|
Nick DiZazzo
|
36a44b9188
|
feature: Refactor setup-first installer flow (#933)
* add uninstall command * parse checksums without awk intervals |
||
|
Michael Neale
|
97c0cad991
|
feat: native rust SDK for rust consumers. (#736)
* a run at SDK, tested with a client * Use in-process shutdown for embedded SDK * Satisfy clippy for embedded shutdown plumbing * Document embedded Rust SDK usage * Add public Rust SDK crate * Tighten embedded SDK lifecycle * Address embedded SDK review feedback * Document native runtime packaging direction * Document runtime CLI namespace * Document runtime CLI UX expectations * Document runtime diagnostics under doctor * Add Windows PowerShell installer * Document recommended runtime install flow * Add native runtime resolver foundation * Wire native runtime release installs * Document native runtime crate * Load versioned native runtimes dynamically * Fix embedded SDK output manager reset * Expose SDK mesh admission controls * Split SDK runtime mapping assertions * Tighten SDK docs and config module * Clarify native runtime SDK TODOs * Stabilize native log note test * Expose native runtime install SDK * Re-export native runtime APIs from SDK crate * Add embedded SDK knobs for Sprout relay mesh (#782) Two opt-in seams the Sprout v1 mesh integration needs: - disable_iroh_relays(bool): when true, embedded runtime selects an explicitly disabled relay policy, which uses RelayMode::Disabled, skips public relay URL fallback, skips raw STUN, and avoids the 5s endpoint.online() wait that cannot succeed without a home relay. Default false preserves existing behavior. - EmbeddedNodeHandle::join_token(token): forwards an invite token over the runtime control channel to node.join_with_retry so an already-running embedded node can dial a new EndpointAddr without restart. Handled in both auto and passive/client runtime loops. Purely additive; existing defaults and startup join_tokens behavior remain unchanged. Co-authored-by: npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@sprout-oss.stage.blox.sqprod.co> Co-authored-by: Perci <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@sprout-oss.stage.blox.sqprod.co> * Fix relay policy test visibility * Make SDK publishable and align language bindings (#771) * Split SDK native runtime publish surface * Split CLI and TUI support crates * Move CLI parser surface into mesh-llm-cli * Extract shared mesh event surface * Move standalone command handlers out of host runtime * Move benchmark and plugin commands out of host runtime * Finish plugin command extraction * Extract auth identity ownership * Move remaining standalone commands out of host runtime * Move model store into model-hf * Move CLI commands out of host runtime * Decouple host runtime from CLI and TUI crates * Expose embedded node SDK facade * Keep client identity dependencies pure * Simplify Rust SDK feature surface * Keep SDK client feature runtime-free * Expose SDK client API base override * Use direct mesh SDK client transport * Remove API base URL client builder shim * Align language SDKs with Rust SDK facade * Document SDK client and serving modes * Fix SDK smoke runtime setup * Package SDK console assets * Fix dynamic runtime CI setup * Restructure SDK docs by language * Fix SDK smoke package loading * Fix native runtime bundle resolution * Add structured native runtime backend metadata * Harden Kotlin native runtime smoke resolution * Fix mesh-llm-sdk clippy imports * Retry smoke model downloads --------- Co-authored-by: James Dumay <jameswdumay@gmail.com> Co-authored-by: tlongwell-block <109685178+tlongwell-block@users.noreply.github.com> Co-authored-by: npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@sprout-oss.stage.blox.sqprod.co> Co-authored-by: Perci <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@sprout-oss.stage.blox.sqprod.co> |
||
|
Nick DiZazzo
|
9ead3281a5
|
fix(cuda-gpus): correct gpu detection, and split CI CUDA lanes by version (#721)
* add probe for Tegra-based SoC * Add Linux ARM64 CUDA release flavour * split CUDA build lanes into matrix by toolkit version * Remove cuda-blackwell as separate BinaryFlavor; consolidate into cuda matrix lane * install: guard detect_cuda_major to only return published versions (12, 13) * set CUDA build recipes to use MESH_CUDA_VERSION env var |
||
|
Nick DiZazzo
|
17157f92a6
|
feature(mesh-genesis): add mesh genesis rules (#589)
* Start a mesh with immutable properties - all nodes conform * Implement min/max version, signed binary attestation, min/max protocol rules |
||
|
Michael Neale
|
d7c6910fba
|
moa: harden gateway from PR #566 review (panic surface, error propagation, dedup race, dead code) (#612)
* docs(README): mark `model: \"mesh\"` MoA as experimental The MoA gateway is new and still being tuned (routing heuristics, error shapes, tuning knobs). Flag it explicitly in the README so readers do not assume the surface is stable. * fix(moa): close panic surface in worker output normalization PR #566 review (Copilot): the MoA crate had several latent panics around untrusted parsed responses. This commit closes them in one pass and adds regression tests for each. Changes ------- normalize.rs * Single sanitize pass in `normalize_worker_output` runs on the result of *every* parse strategy (JSON, KV, heuristic), not just the heuristic path. The previous shape returned early on JSON/KV success and let non-finite confidences leak into the arbiter where `partial_cmp/total_cmp` could panic on `.unwrap()`. The new helper `sanitize_worker_output` clamps NaN/Inf confidence to 0.5 and collapses non-object `tool_arguments` (Null, primitives, arrays) to `Some({})`. * New `extract_tool_arguments` helper replaces two dead `obj.get("arguments").cloned().or_else(\u2026)` chains in `try_json_parse`. The `or_else` branch was unreachable because `.cloned()` on `Some(Value::String(\u2026))` is already `Some`, so string-encoded JSON arguments leaked through unparsed. `extract_tool_arguments` now explicitly branches on String vs Object vs Null and parses the inner JSON when needed. backend.rs * `parse_retry_after` switched from `to_lowercase()` (Unicode-aware, can change UTF-8 byte length) to `to_ascii_lowercase()` (1:1 byte mapping). The earlier shape sliced the original string using the offset from the lowercased one, which could land mid-codepoint and panic for non-ASCII inputs before the marker. * `extract_text_from_response` switched from direct-indexing `resp["choices"][0]["message"]` to `.pointer()` chaining, returning a structured `Err("malformed response: \u2026")` on missing fields instead of silently producing empty content or panicking downstream. lib.rs * `best_answer` now uses `total_cmp` instead of `partial_cmp(\u2026).unwrap()`. `total_cmp` is total over all f32 (NaN/ Inf included), so even if a future caller bypasses `normalize_worker_output`, this site is panic-free. * `tool_call_response` now explicitly handles every input shape callers can construct: object \u2192 serialize, Null \u2192 `"{}"`, primitive / array \u2192 `"{}"`, validated JSON string \u2192 pass through, invalid string \u2192 `"{}"`. Previously `Value::Null` serialized to the literal four-char string "null", which downstream OpenAI tool-call consumers reject. Tests ----- 13 new tests (now 83/83 pass for the crate): * normalize.rs: kv_path_clamps_nan_confidence, kv_path_clamps_inf_confidence, json_string_encoded_arguments_are_parsed_to_object, null_tool_arguments_become_none, primitive_tool_arguments_collapse_to_empty_object. * backend.rs: parse_retry_after_handles_non_ascii_prefix_without_panic, extract_text_returns_err_on_missing_choices, extract_text_returns_err_on_empty_choices. * lib.rs (new `response_builder_tests` mod): best_answer_does_not_panic_on_nan_confidence, tool_call_response_emits_object_args_for_null, tool_call_response_emits_object_args_for_primitive, tool_call_response_passes_through_string_form_when_valid, tool_call_response_rejects_invalid_string_form. Validation ---------- * cargo test -p mesh-mixture-of-agents --lib: 83/83 pass * cargo check -p mesh-llm: clean * cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings: clean * cargo fmt --all -- --check: clean * fix(moa): propagate failure signals to HTTP status, SSE, and tool-result reducer PR #566 review (Copilot) \u2014 three related fixes that all touch how MoA failures reach the caller. 1. HTTP status follows the body's failure signal, not TurnKind ------------------------------------------------------------- `write_moa_response` previously only used HTTP 502 when `TurnKind == Failed`. The tool-result reducer path (`handle_tool_result`) can return `error_response(\u2026)` with `TurnKind::ToolResult` when every reducer candidate fails \u2014 that returned HTTP 200 with an in-band error body, so dumb clients that only check the status code saw a "success". New helper `is_moa_failure_body(body)` recognises the two canonical failure signals MoA emits: a top-level `error` field, and / or `choices[0].finish_reason == "error"`. Status decision now uses this helper, so *every* error-shaped MoA response surfaces as 502 regardless of which sub-flow produced it. 2. SSE adapter propagates the original finish_reason ---------------------------------------------------- `send_moa_as_sse` used to hard-code `finish_reason: "stop"` (or `"tool_calls"` if any tool_calls were present). SSE clients keyed on `finish_reason` (Goose, OpenAI SDKs) therefore saw MoA failures as successful completions. Now the adapter reads `choices[0].finish_reason` from the response body and propagates it ("error" wins over the tool_calls heuristic), and when `is_moa_failure_body` is true it emits an explicit error chunk (`{ object: chat.completion.chunk, choices: [], error: \u2026 }`) before the final finish_reason chunk so SSE clients that scan deltas for an `error` field see the failure too. 3. Tool-result reducer emits tool_calls whenever tool_name is set ----------------------------------------------------------------- Both the tool-result path (`handle_tool_result`) and the fanout/arbiter path (`resolve_decision`) had the same bug: a `ToolProposal` from the reducer only became a real `tool_calls` reply when *both* `tool_name` AND `tool_arguments` were present. With `tool_arguments` missing, both paths silently fell back to a `chat_response` carrying the reducer's prose \u2014 which agent harnesses (Goose, OpenCode) ignore because they only act on `tool_calls`. Now both paths emit `tool_calls` whenever `tool_name` is set, and `tool_call_response` already collapses missing / non-object arguments to `"{}"`. Behaviour is consistent across paths and agent harnesses no longer lose tool calls to reducer prose. Tests ----- Four new `is_moa_failure_body` unit tests in moa_gateway.rs: top-level error, finish_reason=error, success body, tool_calls body. Validation ---------- * cargo test -p mesh-mixture-of-agents --lib: 83/83 pass * cargo test -p mesh-llm-host-runtime --lib: 1446/1446 pass * cargo clippy -p mesh-mixture-of-agents -p mesh-llm-host-runtime --all-targets -- -D warnings: clean * cargo fmt --all -- --check: clean * fix(moa): group aliases by canonical base before resolving backend PR #566 review (Copilot, item #10): the worker pool builder committed to a single alias per canonical model *before* trying to resolve a backend. Two real failure modes: 1. Stale-peer drop. The shortest alias is advertised only by a peer that drops between gossip refresh and orchestration. The peer is gone, `hosts_for_model(alias)` returns empty, the model is silently removed from the worker pool, and longer-form aliases for the same canonical model from still-reachable peers are rejected as duplicates. 2. Forced QUIC hop. The local node advertises a longer convention (e.g. `unsloth/Qwen3-8B-GGUF:Q4_K_M`) while a peer advertises a shorter variant (e.g. `Qwen3-8B-Q4_K_M`). The shortest-name rule picks the peer alias, `add_worker_backend` looks for a local port under that specific string in `targets`, finds nothing, and forces a QUIC tunnel even though the model is locally served. Fix: group all advertised aliases by canonical base first, then within each group sort so the most likely optimization wins first try (locally-served alias before remote, then shortest first as a tiebreaker). The resolver walks the group in order and takes the first alias that produces a backend, so an unreachable preferred alias falls back to a reachable longer one instead of dropping the model. Refactor extracts a small `resolve_one_worker_from_aliases` helper to keep `build_moa_config` under the cognitive-complexity limit. Tests (4 new) in moa_gateway.rs: * group_aliases_keeps_all_aliases_per_canonical_base * group_aliases_prefers_locally_served_alias_even_when_longer * group_aliases_falls_back_to_shortest_when_no_local * group_aliases_distinct_models_stay_in_separate_groups Validation ---------- * cargo test -p mesh-llm-host-runtime --lib moa_gateway: 13/13 pass * cargo test -p mesh-llm-host-runtime --lib: 1450/1450 pass * cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings: clean * cargo fmt --all -- --check: clean * fix(moa): strip dead Session continuation/running-summary machinery PR #566 review (Copilot, item #11): `Session::classify_turn` used a `self.turns` counter to decide Fresh vs Continuation, but the gateway constructs a fresh `Session` per inbound request and never invokes `record_assistant_response` / `record_turn_outcome` in production. As a result `turns` was always 1, `Continuation` never fired, and the entire deterministic running-summary feature (`accepted_facts`, `AcceptedFact`, `record_turn_outcome`, `rebuild_summary`, `running_summary` accessor) silently never executed. `pack_fast` had a `turn_count() > 1` branch that appended the summary to the worker's system prompt; that branch was dead too. This commit reflects the gateway's actual design: MoA is request-scoped, and the caller (Goose, OpenCode, an SDK) owns the multi-turn loop and sends the full history each request. Continuation context comes from `session.messages()`, not from a gateway-owned summary. Changes ------- session.rs * Drop `TurnType::Continuation` (only `Fresh` and `ToolResult` remain). The header comment documents the request-scoped lifetime. * Drop fields: `turns`, `last_was_tool_call`, `accepted_facts`, `running_summary`. * Drop methods: `record_assistant_response`, `record_turn_outcome`, `rebuild_summary`, `running_summary`, `accepted_facts`, `turn_count`, `has_unprocessed_tool_results`. * Drop struct: `AcceptedFact`. * Rewrite `ingest` to rebuild `pending_tools` from the caller-provided history each call (delta tracking only worked when Sessions were persisted across requests). Both assistant-emitted tool_calls and tool-result messages are picked up; agent harnesses unchanged. * Simplify `classify_turn`: walk history backwards, return `ToolResult` if a `role: "tool"` message appears before any `role: "assistant"`, else `Fresh`. lib.rs * Drop the `Continuation` match arm. context.rs * Drop the `turn_count() > 1` running-summary injection from `pack_fast`. Context comes from `session.messages()`. Tests ----- * tool_result_turn test rewritten to not rely on the removed `record_assistant_response`; verifies the same behaviour with caller-provided history. * No other test changes needed. Net: 132 LoC of dead machinery removed. Validation ---------- * cargo test -p mesh-mixture-of-agents --lib: 83/83 pass * cargo test -p mesh-mixture-of-agents (incl. integration): all pass * cargo test -p mesh-llm-host-runtime --lib: 1450/1450 pass * cargo check -p mesh-llm: clean * cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings: clean * cargo fmt --all -- --check: clean * fix(moa): tighten error code, linearize strip_thinking, sanitize header names, drop dead branch PR #566 review (Copilot) \u2014 batch of MEDIUM cleanups. mesh-mixture-of-agents ---------------------- * `error_response` now takes a `code: &str` parameter and the two call sites pass distinct constants: - `MOA_ERR_ALL_WORKERS_FAILED` for the fanout/arbiter failure path. - `MOA_ERR_ALL_REDUCERS_FAILED` for the tool-result reducer path. Previously both paths emitted `code = "all_workers_failed"`, which was misleading for clients branching on `error.code`. * `strip_thinking` rewritten as a single linear pass over the input. The previous shape rebuilt the entire string on every think block (`format!` + `replace` in a loop), which is O(n*k) for long worker outputs with many `<think>` blocks. Three new tests cover the new shape: orphan close-tag handling, fifty-block linear behaviour, and UTF-8 preservation through multibyte content. mesh-llm-host-runtime --------------------- * `network::openai::transport`: header NAMES are now validated against the RFC 7230 tchar grammar via a new `is_valid_header_name` helper. Names that fail the grammar are dropped with a tracing warning rather than written verbatim. CR/LF in header values is still stripped.\n Both `send_json_ok_with_headers` and\n `send_json_with_status_and_headers` route through a shared\n `append_safe_header(\u2026)` so the validation can't be bypassed by a\n future caller. Four new tests cover the validator and the safe\n header writer.\n\n* `network::openai::moa_gateway::send_moa_as_sse` now reuses\n `append_safe_header` so the SSE adapter gets the same name validation\n as the JSON writers.\n\n* `network::openai::ingress::try_intercept_moa`: dropped the\n unreachable `if let Some(_unused_stream) = \u2026 { tracing::error!(\u2026) }`\n branch. `try_handle_moa` self-gates on the model name and the outer\n gate guarantees it matches, so the inner call always returns\n `None` here. Replaced with `let _ = \u2026.await;` and a comment\n explaining the invariant.\n\nValidation\n----------\n* cargo test -p mesh-mixture-of-agents --lib: 86/86 pass (3 new)\n* cargo test -p mesh-llm-host-runtime --lib: 1454/1454 pass (4 new)\n* cargo clippy -p mesh-mixture-of-agents -p mesh-llm-host-runtime\n --all-targets -- -D warnings: clean\n* cargo fmt --all -- --check: clean * docs(moa): align stale comments and doc claims with current code PR #566 review (Copilot) \u2014 batch of LOW comment/doc corrections so future readers don't trust outdated narrative. * mesh-llm-host-runtime/src/network/openai/moa_gateway.rs: `try_handle_moa` doc said "returns `true` if handled" but the signature is `Option<TcpStream>`. Rewrote the doc to describe the actual contract: `Some(stream)` means "not MoA, fall through", `None` means "MoA consumed the stream and responded; do not respond again". * model-ref/src/lib.rs: comment claimed the function "emits a lowercase selector", but the implementation slices from the\n original (case-preserving) stem and lowercasing is only used for\n marker-position lookup. Reworded so the comment matches the code.\n\n* skippy-cache/src/resident/prefix.rs: regression-test comment\n referenced a "4*min_tokens floor" \u2014 that derivation was replaced by\n a hard `MIN_CTX_FOR_CELL_CAP = 8192` threshold in a follow-up\n commit. Updated to describe the floor that actually ships.\n\n* mesh-llm-host-runtime/src/inference/skippy/family_policy.rs:\n inline comment said the 16-entry cap "does not fully eliminate"\n unified-KV starvation and that the "proper fix is out of scope".\n The proper fix (token-based budget via `max_resident_tokens` in\n `ResidentCacheConfig::from_stage`) shipped in PR #566. Updated to\n describe entry-count as the coarse lever and `max_resident_tokens`\n as the complementary fine-grained budget.\n\n* docs/design/MOA_GATEWAY.md:\n - Role assignment section claimed "the largest of the big tier gets\n Strong". The impl only partitions small vs big, no\n within-tier sort by parameter count. Reworded to describe the\n actual heuristic and call out the future option.\n - Test plan said "29 unit tests" with a stale per-area breakdown.\n Updated to 86 with current coverage areas, and added a note to\n bump the number when adding tests.\n\nValidation\n----------\n* cargo test -p mesh-mixture-of-agents --lib: 86/86 pass\n* cargo test -p mesh-llm-host-runtime --lib: 1454/1454 pass\n* cargo test -p skippy-cache --lib: 13/13 pass\n* cargo test -p model-ref --lib: 10/10 pass\n* cargo clippy -p mesh-mixture-of-agents -p mesh-llm-host-runtime\n -p skippy-cache -p model-ref --all-targets -- -D warnings: clean\n* cargo fmt --all -- --check: clean * fix(moa): drop malformed tool_calls instead of inserting empty-id placeholders PR #612 review (Copilot) \u2014 two follow-ups. 1. session.rs: malformed tool_calls in caller history ----------------------------------------------------- The earlier shape defaulted missing/empty `id` and `function.name` to `""` and still pushed a `PendingToolCall`. With adversarial or buggy caller history this had two real consequences: * Two malformed calls share `call_id == ""` and become indistinguishable, so any later `role: "tool"` whose `tool_call_id` is also missing matches the first one rather than the intended call. * The `tool_call_response` wire-shape invariant (non-empty function name) is violated downstream. Now both `assistant`-side ingestion and `tool`-side matching skip entries with missing or empty `id`/`tool_call_id` (and `assistant` ingestion also skips entries with empty `function.name`), emitting a `tracing::warn!` so the issue is visible in logs. Well-formed history is unaffected. 2. normalize.rs: doc/code mismatch on tool_arguments ---------------------------------------------------- The `extract_tool_arguments` doc claimed missing/null `arguments` collapses to `Some({})`. The code returns `None` and relies on the downstream `tool_call_response` to substitute `"{}"` when serializing the wire shape. The wire output is correct either way, but the doc misled future maintainers about the invariant. Rewrote the comment to describe the actual invariant ("`None` or an object; `None` means emit `{}` at wire time") so a future refactor doesn't reintroduce the literal-"null" bug. Regression test --------------- `malformed_tool_calls_are_dropped_not_collapsed_to_empty_id` covers four malformed shapes in one history (missing id, missing function.name, empty id, plus an orphaned tool result) and asserts only the one well-formed call survives and the orphaned result doesn't attach. Validation ---------- * cargo test -p mesh-mixture-of-agents --lib: 87/87 pass (+1 new) * cargo test -p mesh-llm-host-runtime --lib: 1454/1454 pass * cargo clippy -p mesh-mixture-of-agents -p mesh-llm-host-runtime --all-targets -- -D warnings: clean * cargo fmt --all -- --check: clean Live end-to-end validation -------------------------- Built locally and deployed to the 2-node private mesh (M4 Qwen3-8B + Studio MiniMax). All four agent harness shapes from PR #566/#612 exercised the changed code paths: * mini-agent.py model=auto \u2014 2 turns, correct. * mini-agent.py model=mesh \u2014 2 turns, correct. * mini-agent2.py model=mesh \u2014 4 turns multi-file, correct. * Goose model=mesh \u2014 1 tool call, correct. No regressions in MoA fan-out / tool-result reducer routing. * fix(moa): route streaming failures to HTTP 502 (drop in-band SSE error) PR #612 review feedback (Nick) \u2014 two related findings on the streaming failure path. 1. Streaming MoA failures returned HTTP 200 ------------------------------------------- `write_moa_response` previously routed all streaming responses through `send_moa_as_sse`, which always emits `HTTP/1.1 200 OK` because SSE clients expect 200 before they start parsing the event stream. That meant `stream: true` MoA failures (`all_workers_failed`, `all_reducers_failed`, reducer hedge exhaustion) arrived as a 200 SSE that *happened* to carry `finish_reason: "error"` \u2014 dumb HTTP clients saw a "successful" stream and didn't realise inference had failed. The body is fully available before we decide how to write it, so we can collapse failure-shaped streaming responses to a non-streaming HTTP 502 JSON response with the structured error body. This matches the OpenAI API shape (failures on streaming endpoints come back as a single non-streaming JSON error response with the right HTTP status) and means streaming and non-streaming MoA failures are now consistent at the HTTP layer. 2. Error-only SSE chunk could break OpenAI-shape clients -------------------------------------------------------- The previous error chunk emitted on failure was `{ object: chat.completion.chunk, choices: [], error: \u2026 }`. Many OpenAI client SDKs assume every `chat.completion.chunk` has `choices[0]` and index blindly into it, so an empty `choices: []` array crashes those clients with an index error. With finding #1's routing change, failure-shaped bodies never reach `send_moa_as_sse` anymore, so the error-chunk emission is dead code. Drop it. `send_moa_as_sse` now does one clear thing: emit a single delta chunk + a `finish_reason: "stop" | "tool_calls"` stop chunk, both with a real `choices[0]`. A `debug_assert!` pins the invariant in tests. Tests ----- Four new unit tests covering the four corners of the routing decision: * `streaming_success_routes_to_sse` * `streaming_failure_routes_to_json_502_not_sse` * `non_streaming_success_routes_to_json_200` * `non_streaming_failure_routes_to_json_502` Validation ---------- * cargo test -p mesh-llm-host-runtime --lib: 1458/1458 pass (4 new) * cargo test -p mesh-mixture-of-agents --lib: 87/87 pass * cargo clippy ... --all-targets -- -D warnings: clean * cargo fmt --all -- --check: clean Live end-to-end validation on 2-node mesh (M4 + Studio MiniMax): * Non-streaming MoA happy path: HTTP 200 + structured response. * Streaming MoA happy path: HTTP 200 + `text/event-stream` + `[DONE]`. * mini-agent.py model=mesh: 2 turns, correct. * mini-agent2.py model=mesh: 4 turns multi-file, correct. * Goose model=mesh: 1 tool call, correct. The failure path is exercised by unit tests; production-triggering streaming failures would require fault injection beyond what the harness covers, but the routing logic and the SSE invariant are locked down at the code level. |
||
|
Michael Neale
|
1b1aaf432b
|
MoA: mesh mode and many inference critical fixes, and quic keep alive (#566)
* feat: MoA gateway — stateful mixture-of-agents with tool arbitration
New standalone crate (moa-gateway) that fans out to N heterogeneous LLM
endpoints in parallel, normalizes dirty worker outputs, arbitrates with
deterministic logic, and manages the full tool call lifecycle across turns.
Tested live against 3 ollama models (llama3.2:3b, qwen3:4b, qwen3.6:27b):
- Knowledge/reasoning: picks highest-confidence answer across models
- Tool calling: correctly produces tool_calls when workers propose tools
- Tool lifecycle: full cycle query → tool_call → result → final answer
- Tool results bypass fan-out, go to reducer only (one transcript)
The gateway is transport-agnostic — works against any OpenAI-compatible
endpoint (ollama, mesh-llm, remote APIs). Integration with mesh model
discovery is the next step.
* feat: multi-turn context efficiency + mesh endpoint discovery
Progressive running summary: workers get compact deterministic summaries
instead of raw message history. Tested across 3-turn conversations —
workers retain context (Melbourne → restaurant recommendation) through
the summary, not through replaying 20k tokens of history.
New moa-mesh binary discovers models from any OpenAI-compatible endpoint
(mesh-llm proxy, ollama, vLLM) and runs the full MoA test suite against
it. Designed to work with 'mesh-llm client --auto' out of the box.
Multi-turn tool lifecycle proven end-to-end:
Turn 1: weather query → fan-out → tool_call (get_weather)
Turn 2: tool result → reducer only → text answer
Turn 3: follow-up 'bring jacket?' → fan-out with running summary → contextual answer
* fix: increase running summary budget to ~2k tokens
Per-fact truncation: 200 → 500 chars
Recent fact window: 5 → 15 facts
Tool result truncation: 80 → 300 chars
Turn outcome capture: first sentence → first 400 chars
200 tokens was too aggressive for real agent sessions where system
prompts alone can be 500+ tokens. 2k tokens gives enough room for
15 turns of meaningful context while still being much cheaper than
replaying raw history.
* feat: agentic workload test + improved prose tool detection
New moa-agent binary simulates a multi-step coding agent: read file,
analyze bug, edit fix, run tests, diagnose failure, iterate. Exercises
7+ turns with accumulating context, repeated tool use, and loop detection.
Improved normalizer: small models that describe tool usage in prose
("I'll use the edit_file tool") are now correctly classified as
tool proposals via known-tool-name + action-verb heuristic.
Key findings from agentic testing:
- Gateway routing, context management, and tool lifecycle are solid
through 7 turns / 15 messages / 6 reducer calls
- Loop detection catches repeated identical tool calls
- Small models (3b/4b) produce correct tool calls for read/search
but describe edits in prose instead of calling edit_file
- Multi-step plan execution (tool→analyze→tool→fix) needs a stronger
model in the reducer role — the arbiter and routing aren't the
bottleneck, model capability is
* fix: mesh-tested agentic flow — local reducer, model dedup, KV parse fix
Tested against live mesh: GLM-4.7-Flash (local) + Qwen3-8B (remote peer).
Three fixes from live mesh testing:
1. Local reducer: prefer first endpoint (local model) as reducer instead
of last. Remote models over QUIC relay are fine as parallel workers
but timeout as the sequential reducer. Result: reducer response
times dropped from 180s (remote timeout) to 2-18s (local).
2. Model dedup: mesh-llm exposes the same model under multiple aliases
(e.g. unsloth/GLM-4.7-Flash-GGUF and @main:Q4_K_M variant).
discover_endpoints() now deduplicates by normalized display name.
Extracted as shared lib function used by all three test binaries.
3. KV parse fix: models that say 'kind: answer' but also include
'tool: read_file' are now correctly classified as ToolProposal.
This was the #1 cause of missed tool calls in the agentic flow.
* moa: integrate into mesh proxy, SSE streaming, Goose support
MoA is now available as model="moa" through the mesh proxy on :9337.
When ≥2 models are available (local + mesh peers), the MoA virtual model
appears in /v1/models automatically.
Integration:
- mesh-llm-host-runtime depends on moa-gateway
- ingress.rs intercepts model="moa" requests, builds endpoints from
callable models, calls Gateway::turn(), returns the result
- SSE framing: converts the non-streaming MoA response into SSE chunks
for streaming clients (Goose, pi, etc.)
- Tool calls passed through in SSE format with finish_reason="tool_calls"
Passthrough mode (tools present):
- When the request includes tools (agentic use via Goose/pi), workers
receive the original messages+tools unmodified — no MoA envelope
- This avoids conflicting system prompts confusing small models
- First successful worker response is returned, providing redundancy
Content cleanup:
- Think tags (<think>...</think>) stripped from all responses
- Orphan </think> tags cleaned up
- KV envelope lines (kind:/confidence:/payload:) stripped when they
leak into heuristic-classified output
- Normalizer pre-cleans think tags before trying JSON/KV parse
Tested with:
- Direct curl (non-streaming + streaming)
- Goose CLI: factual questions, tool execution (shell, execute_typescript)
- 22 unit tests passing
* fix: add moa-gateway to Docker builds
The docker-client CI job failed because crates/moa-gateway/ was missing
from both Dockerfile.client and fly/Dockerfile. cargo metadata couldn't
resolve the workspace member, breaking the cargo-chef prepare step.
* moa: early-exit on worker consensus
Instead of waiting for all workers before arbitrating, check for
consensus after each worker returns. When 2+ workers agree on an
answer or tool call, return immediately and abort remaining workers.
This eliminates the 'slowest worker' bottleneck. In testing:
- Average latency dropped from 21.0s to 5.8s (single model: 7.1s)
- MoA is now 19% faster than querying a single model
- Worst case (code-debug) went from 120s timeout to 4.2s
The key insight: with parallel fan-out, we only need to wait for the
fastest N workers that agree, not all of them. Slow/dead remote
workers no longer block the response.
Also adds 5 new arbiter tests for early decision logic (27 total).
* moa: real context slices, not synthetic envelopes
Major architectural change to how MoA packs context for workers.
Before: workers got a synthetic system prompt ('You are a fast analysis
worker...') that replaced the agent's real system prompt, tool schemas,
and conversation history. Workers were asked to respond in a KV envelope
format (kind:/confidence:/payload:) that small models followed unreliably.
When tools were present, the entire MoA pipeline was bypassed via a
'passthrough mode' that raced identical requests to all workers.
After: workers get slices of the REAL context — the agent's actual system
prompt and messages — with depth varying by role:
- Fast: system prompt + last user msg + tool names only
- Specialist: system prompt + last 4 msgs + tool summaries
- Strong: system prompt + full recent history + native tool schemas
- Reducer: system prompt + worker outputs + full tool schemas
The gateway augments with a one-line preamble, not a replacement. The
passthrough mode is removed — tool-use goes through the full normalize →
arbitrate pipeline. Strong workers get native tool schemas forwarded so
they can produce real tool_calls.
Also:
- Dedup model aliases in ingress (GLM and GLM@main:Q4_K_M are the same)
- Early exit handles failed workers (sole survivor returns immediately)
- Worker timeout reduced from 120s to 30s
- 28 unit tests (up from 27)
* moa: rename to mesh-mixture-of-agents, mesh-native transport, model='mesh'
Renamed crate from moa-gateway to mesh-mixture-of-agents. Keeps the
crate isolated (own tests, own compilation unit) while connecting it
to mesh transport via a ModelBackend trait.
Transport is now mesh-native instead of HTTP loopback:
- LocalModelBackend: direct HTTP to skippy port (bypasses proxy)
- RemoteModelBackend: QUIC tunnel to peer (bypasses proxy + tunnel layer)
- Both set mesh_hooks: false to prevent recursive consultation
The ModelBackend trait keeps the crate testable in isolation — the
default HttpBackend works against any OpenAI-compatible endpoint.
The mesh backends are implemented in ingress.rs where Node and
InferenceTarget are available.
Virtual model renamed from 'moa' to 'mesh'. Appears in /v1/models
when ≥2 distinct models are available.
Test bins removed (used old Gateway API). 29 unit tests remain.
* fix: update Dockerfiles for moa-gateway → mesh-mixture-of-agents rename
* moa: remove 'mesh' from /v1/models list
The 'mesh' virtual model is a routing directive like 'auto', not a
real model. It should not appear in the models list. Clients that
want MoA fan-out use model: "mesh" explicitly.
* fix: clippy warnings in mesh-mixture-of-agents
* fix: clippy unnecessary_lazy_evaluations in ingress build_moa_config
* docs: update MoA design doc with current architecture and test plan
Reflects: mesh-mixture-of-agents crate rename, ModelBackend trait,
handle_turn() stateless API, mesh-native transport, model='mesh'
virtual routing, early-exit consensus, and eval plan.
* moa: fix tool call arguments lost in arbitration
Two fixes:
- Arbiter now prefers tool proposals with actual arguments over
proposals that only have the tool name (from fast workers that
don't get native tool schemas).
- Specialist workers now receive native tool schemas so they can
produce structured tool_calls with arguments, not just mention
tool names in text.
Before: read_file({})
After: read_file({"path":"/tmp/test.txt"})
* moa: worker diversity sampling, 429 retry, faster timeouts, sole-survivor early exit
Three improvements to MoA reliability and response quality:
- Workers get high temperature (0.8) + top_p (0.95) for diverse
exploration; reducer gets low temperature (0.3) for precise synthesis.
SamplingParams flows through the ModelBackend trait.
- 429 rate-limit errors trigger one automatic retry after the server's
retry-after delay (default 1s).
- Worker timeout 30s → 15s, reducer 45s → 30s.
- Sole survivor returns immediately when majority of other workers have
already failed, instead of waiting for remaining stragglers.
39 unit tests (up from 29).
* Fix MoA tool result handling and NaN confidence (PR review feedback)
Three issues from Copilot review on PR #534:
1. Tool result turns now include actual tool output content.
pack_for_tool_result_turn was reading from pending_tools which
is always empty on a fresh session (stateless per request).
Now forwards the raw message sequence including assistant
tool_call + tool result messages so the reducer sees the full
context. Added regression test.
2. NaN confidence no longer panics arbiter comparisons.
Replaced partial_cmp().unwrap() with total_cmp() in arbiter,
and added a sanitizer in normalize that clamps non-finite
confidence to 0.5. Added test.
3. Exclude spec-prefill-poc from workspace members (Docker fix).
Moved to Cargo.toml exclude list so Docker builds don't fail
on the missing experimental crate.
* ci: align WORKSPACE_MEMBERS with current workspace
- Add mesh-mixture-of-agents (new crate on this branch)
- Rename mesh-llm-client → mesh-client (renamed on main)
Fixes the scripts/affected-crates.sh consistency check that gates the
Linux CPU CI build.
* moa: size-aware role assignment + hallucinated tool name filtering
Two surgical fixes from real-world goose testing:
1. Role assignment by capacity tier, not list-order.
assign_roles previously used list order: first=fast, last=strong.
When a small local model (e.g. Qwen2.5-3B) was loaded last, it got
tagged Strong and used as the reducer — exactly when goose needs a
capable model for tool arbitration.
Now we sort by size tier using the same is_single_digit_b_name
heuristic as the main router's pick_model_classified, so MoA's
strong worker matches what auto would pick. MiniMax-M2.5 and
Qwen3-32B (big tier) become Strong; Qwen3-8B and Qwen2.5-3B (small
tier) become Fast.
2. Filter hallucinated tool names from worker proposals.
A worker proposing a tool not declared in the request (e.g. local
3B hallucinating 'execute_typescript' when only 'shell' was offered)
would bypass arbitration and reach the client, causing silent
failures when goose tried to dispatch the unknown tool.
gather_workers_incremental and the reducer output paths now demote
such proposals to Uncertainty with a tracing warning. The arbiter
sees a clean set of valid proposals and resolves correctly.
Threading: handle_query, handle_tool_result, gather_workers_incremental,
and resolve_decision all now take &[String] allowed_tools derived from
session.tool_names(). When allowed_tools is empty (no tools on request)
the filter is a no-op.
* moa: reducer candidate fallback on 5xx / timeout
When the chosen reducer peer is broken (e.g. stale binary returning 502
on tool grammars) or unreachable, the tool-result turn or NeedsReducer
arbitration would fail with that single error, even though other strong
peers were available.
Replace single-pick pick_reducer with reducer_candidates returning all
big-tier models (multi-digit B or no size in name) followed by small-tier
as last-resort fallback. Both call sites — handle_tool_result and
resolve_decision NeedsReducer — now iterate candidates and break on the
first success.
This rescues the common goose-on-public-mesh case where one strong peer
(e.g. Qwen3-32B host) is running a stale binary that 502s on tool calls,
while another (e.g. MiniMax) is healthy. Without this, MoA's tool-result
turn was as fragile as auto routing.
* ci: fix workspace member drift — keep mesh-llm-client package name, add mesh-mixture-of-agents to clippy script
Same fix as the prefill-draft branch:
- The mesh-client directory rename did not change the package name —
the crate is still published as 'mesh-llm-client'. Revert the
scripts/affected-crates.sh edit that broke the CI consistency check.
- Add mesh-mixture-of-agents to plan-clippy-batches.sh which carries
its own WORKSPACE_MEMBERS list with the same drift constraint.
* moa: support model:"mesh" on client/standby nodes via forward-to-host
Pure --client nodes and standby GPU nodes accept inbound HTTP via
handle_mesh_request (in transport.rs) instead of the model-aware api_proxy
in ingress.rs. The MoA fan-out intercept lives in api_proxy, so when a
client received "model": "mesh" it fell through to the "no host serves
this model" branch and 429d.
* moa: fix clippy lints surfaced by CI
Two pre-existing lint violations in mesh-mixture-of-agents that CI didn't
see before because the crate wasn't in the affected-crates / clippy
workspace lists. Now that the WORKSPACE_MEMBERS drift is fixed they show
up on every PR clippy run.
- worker.rs:67 `x == false` -> `!x` (clippy::bool_comparison)
- lib.rs:523 `&name` -> `name` (clippy::needless_borrow)
No behavior change — tool_call_response takes &str either way, and the
sort key inverts identically.
* moa: hedge reducer candidates instead of sequential fallback
Cut worst-case reducer latency from N×timeout to roughly
reducer_timeout + (N-1)·hedge_delay. Big win when a peer is slow or
broken; zero cost on the happy path.
Before:
for candidate in candidates:
call(candidate, timeout=30s) # wait up to 30s per stale peer
if ok: return
# 3 stale big-tier peers ⇒ 90s before falling through to small-tier
After:
spawn candidate[0]
loop:
select:
a candidate finished:
ok → cancel rest, return
err → spawn next candidate immediately (no hedge wait)
hedge_delay elapsed and more candidates remain:
spawn next alongside in-flight ones (race)
Cost shape:
- Happy path (cand 0 OK in <hedge_delay): exactly 1 backend call. Free.
- Slow first (cand 0 takes hedge_delay..reducer_timeout): up to 2
overlapping calls, accept whichever wins, cancel loser.
- Fast-fail (cand 0 errors quickly): next candidate immediately, 1 call.
- All fail: ≤N calls, capped at reducer_timeout + (N-1)·hedge_delay.
Wall-clock improvement for 3 stale big-tier peers (worker_timeout=15s,
reducer_timeout=15s, hedge_delay=5s):
- Before: 3 × 30s = 90s before reaching small-tier fallback.
- After: 15s + 2 × 5s = 25s. Plus reducer_timeout itself drops 30s → 15s
now that the hedged ladder makes a single per-attempt cap safe to
shorten.
Changes:
- Add hedged_reducer_call() in mesh-mixture-of-agents/src/lib.rs.
- Replace the for-loop in handle_tool_result() with it.
- Replace the for-loop in resolve_decision()'s NeedsReducer arm with it.
- Add hedge_delay field to GatewayConfig (defaults set at the single
construction site, build_moa_config in ingress.rs).
- Lower reducer_timeout 30s → 15s in build_moa_config.
- 4 new unit tests cover happy path, hedge-on-slow, fast-fail, all-fail.
- Refresh stale numbers in docs/design/MOA_GATEWAY.md and replace the
"first model wins" reducer paragraph with the hedged-ladder description.
Verified:
cargo fmt --all -- --check # clean
cargo check -p mesh-llm-host-runtime # clean
cargo clippy -p mesh-llm-host-runtime --lib # clean
cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings # clean
cargo test -p mesh-llm-host-runtime --lib # 1381 passed
cargo test -p mesh-mixture-of-agents --lib # 47 passed (4 new hedge tests)
* evals: add bench-moa.sh — quick wall-clock benchmark for model:"mesh"
POSTs N chat completion requests to a running mesh-llm endpoint with
model="mesh" and reports p50/p95/p99 wall-clock latency. Probes /v1/models
first and warns if fewer than 2 models are present (MoA returns 503).
Usage:
./evals/bench-moa.sh # 20 requests, localhost:9337
N=50 ./evals/bench-moa.sh
BASE_URL=http://host:9337 ./evals/bench-moa.sh
PROMPT="why is the sky blue?" ./evals/bench-moa.sh
Output is per-request lines plus a summary block (min/p50/p95/p99/max/
mean/stdev). Failed requests are reported but excluded from percentiles.
Raw TSV results are kept in a tmpdir so before/after comparisons are easy.
Dependencies: curl, jq, python3 — nothing exotic. Run on the same machine
as a serving host so wall-clock is dominated by inference + arbitration.
* moa: split lib.rs into backend / reducer / fanout modules
lib.rs was past the 1k LoC refactoring threshold and had three separable responsibilities. Extract them into named modules and keep lib.rs as the orchestration entrypoint (handle_turn, GatewayConfig, TurnResult, response builders).
- backend.rs: ModelBackend trait, HttpBackend, SamplingParams, ModelEntry, call_backend + retry-after parsing
- reducer.rs: reducer_candidates ordering, hedged_reducer_call ladder
- fanout.rs: gather_workers_incremental
ModelEntry, HttpBackend, ModelBackend, and SamplingParams are re-exported from lib.rs so existing callers (worker.rs, host-runtime ingress) keep working.
Tests move with their owner: backend.rs gets the sampling/retry-after tests, reducer.rs gets the 4 hedged-reducer tests plus the FakeBackend helper. No behavior change.
lib.rs: 1267 -> 545 LoC
47 existing tests still pass.
* moa: cover role-shaped context packing with tests
context.rs owned the role-shaped packing logic (fast/specialist/strong/reducer depth contract) without any tests. Pin the design claims:
- per-role token budgets (256 / 512 / 1024)
- fast: system + last user only, tool names only, no tools field
- specialist: tool summaries in system, native tools populated
- strong: deep history (>=6 msgs), native tools populated
- generalist/reducer roles alias the strong shape
- MoA preamble augments rather than replaces the agent's system prompt
- reducer context includes the conflict reason and labeled worker payloads
- long worker payloads are truncated with an ellipsis to bound context
8 tests, all green.
* docs: surface model:"mesh" (MoA) in README
Add a workflow-table row pointing at MOA_GATEWAY.md and a short 'Mixture-of-Agents' section with a curl example and the two-models-required gate, so the feature is discoverable from the project entrypoint.
* moa: extract tool_guard module, drop dead Endpoint/discover_endpoints
Two small cleanups on lib.rs:
- Move enforce_allowed_tools into its own tool_guard.rs module. It's a
content-policy concern (demote hallucinated tool names to Uncertainty
before arbitration), not orchestration, so it doesn't belong in the
handle_turn entrypoint file. Comes with 4 unit tests covering allowed
pass-through, unknown-tool demotion (incl. confidence drop),
empty-allowed-list noop, and non-proposal outputs untouched.
- Delete the Endpoint struct and discover_endpoints helper from lib.rs.
Their doc comments described them as 'convenience for test harnesses'
but they have zero call sites in-tree and no out-of-tree consumers we
know of. Dead code from the standalone phase before mesh-native
backends landed.
lib.rs: 545 -> 454 LoC.
Tests: 55 -> 59 (4 new in tool_guard).
* docs(moa): drop the speculative hook-integration line
MoA and hooks are intentionally independent — worker requests set
mesh_hooks: false so the hook pipeline can't re-enter a worker call.
The old design doc closed the relationship section with 'they could
integrate later (hook signals as arbiter weights)', which makes it
look like roadmap. It isn't — keeping them separate is the design.
Replace that line with one that states the separation as intentional
and points at the mesh_hooks: false invariant that enforces it.
* router: weight 'auto' selection by locally observed tok/s
Before, 'auto' picked uniformly at random within the multi-digit-B
tier. On the public mesh this meant a fast MiniMax on a 4090 and a
slow 35B-A3B on an M2 Air were equally likely to be chosen, even
though we'd already measured the throughput gap in routing_metrics
and were just not reading it.
Now: each big-tier candidate is weighted by its locally observed
avg_tokens_per_second (clamped to [5, 100] tok/s so nothing fully
starves and no outlier monopolizes). Models without enough samples
(< 3) get a neutral weight so they compete fairly until data
accumulates. A 15% exploration probability ignores weights and
picks uniformly, which keeps the system from locking onto stale
rankings and guarantees cold peers see traffic.
Plumbing:
- RoutingMetrics::tps_for_model(name) -> Option<(f64, u64)>: cheap
per-model lookup that locks only the relevant shard, avoiding the
per-call HashMap allocation model_snapshots() does in the hot path.
- Node::routing_metrics() public accessor (Arc-backed, cheap).
- RoutingCandidate { name, caps, tps_hint, throughput_samples }
replaces the anonymous (&str, f64, ModelCapabilities) tuple whose
middle slot was literally always 0.0 at every populated call site.
The struct makes the tps hint a real, typed concept rather than a
dangling hook.
Behaviour preserved:
- Single-digit-B partition (smalls stay last-resort) unchanged.
- All-cold candidate pool falls back to ~uniform pick (regression
test confirms no model is starved when there's no data yet).
- Capability filtering for tools / reasoning / vision unchanged.
Plumbing per call site:
- ingress.rs + transport.rs: live routing path, look up tps_hint
from the local RoutingMetrics handle for each candidate.
- discovery.rs + integrations.rs: pre-startup paths with no live
metrics; build candidates with RoutingCandidate::unscored() so
they get the cold-neutral weight.
Tests:
- weighted_pick_all_cold_is_roughly_uniform — regression safety.
- weighted_pick_fast_wins_majority_but_slow_still_gets_some —
fast wins by >=1.5x but slow still gets >30/600 picks.
- weighted_pick_cold_model_competes_with_hot_fast — newcomer gets
>100/600 picks against an established fast peer (so it can
actually accumulate samples and earn its score).
- weighted_pick_low_sample_count_treated_as_cold — 1-sample
measurements don't dominate routing.
- candidate_weight_clamps_extremes — weight stays in [5, 100],
cold = 25.
Removed:
- shuffle_in_place (replaced by SplitMix64 + pick_weighted).
- The dishonest 0.0 f64 slot in the candidate tuple, everywhere.
Validation:
cargo fmt --all -- --check # clean
cargo check -p mesh-llm-host-runtime # clean
cargo clippy -p mesh-llm-host-runtime --lib # clean
cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings # clean
cargo test -p mesh-llm-host-runtime --lib # 1398 passed (17 in router)
cargo test -p mesh-mixture-of-agents --lib # 59 passed (no regression)
* docs(moa): clarify topology — N workers + serial 2-call shape
- Replace topology diagram with one that shows N workers fanned out in
parallel and the serial fan-out → arbiter → reducer path.
- Add explicit "how many models" table (2..N) and the worker → role
mapping so readers don't have to infer it from worker.rs.
- Spell out that a worst-case MoA turn is 2 LLM round-trips serially
(fan-out wall-clock = slowest worker, then optional reducer), and
that happy paths collapse to 1 (consensus or tool-result turn).
- Refresh stale crate-structure table: post-split LoC + test counts
for backend / reducer / fanout / tool_guard / arbiter / context /
worker / session / normalize / lib.
* moa: emit x-moa-* observability headers from gateway
Extend TurnResult with turn_kind (Fanout / EarlyExit / ToolResult / Failed)
and reducer_attempts (candidates actually spawned). hedged_reducer_call
now returns a named HedgedReducerOk struct carrying winner, text, and
spawn count so the caller can attribute hedge cost.
The ingress MoA intercept reads these and emits:
x-moa-elapsed-ms
x-moa-turn fanout | early-exit | tool-result | failed
x-moa-workers total workers dispatched
x-moa-workers-ok workers that returned a usable answer
x-moa-reducer true | false
x-moa-reducer-attempts 0 on no-reducer path, 1 happy, >=2 hedged
Headers are emitted on both the JSON and SSE response paths via a new
send_json_ok_with_headers helper and an extra_headers arg to
send_moa_as_sse. Normal OpenAI clients ignore unknown headers; benches
and ops tooling can read them without parsing the body.
Side fix: handle_tool_result previously reported attempts = total
candidate pool size rather than candidates actually spawned. Now
correctly reports the spawn count from HedgedReducerOk.
* bench-moa: aggregate gateway path, reducer, and hedge stats
Read x-moa-* response headers per request and roll them up in the
summary. New aggregates:
Gateway paths histogram of fanout / early-exit / tool-result / failed
Reducer invocation rate + hedge rate + avg/max attempts
Worker fan-out average width + histogram by N
Latency p50 split by gateway path (so 'reducer turns are 4x slower'
is visible at a glance)
Per-request log line now shows the turn kind, worker count, and reducer
status alongside the latency, making it easier to eyeball individual
outliers.
Older mesh-llm binaries that don't emit x-moa-* headers degrade to the
old summary (latency-only) with a note that headers were not seen, so
this works against any mesh-llm version.
* evals: remove bench-moa.sh — never actually run
The script was written but never executed against a live mesh. MoA verification is the manual live-mesh testing called out in the PR body. Real aggregates, if we want them, should come from passive counters fed by real traffic, not a synthetic curl loop.
* moa: orchestrate from any node, build worker pool from mesh-wide gossip
Before this change MoA only ran when the request hit `api_proxy` on a
serving host. A pure `--client` node received `model: "mesh"` in
`handle_mesh_request`, fell through to the "forward to any host"
fallback, and the receiving host either ran an older binary that
ignored the "mesh" name or built a single-model config because its
local `ModelTargets` only had its own model. End result: no fan-out
happened, MoA was effectively dead from any client node, and the
worker pool depended on which node received the request rather than
on what was actually in the mesh.
Two structural fixes:
1. New `moa_gateway` module owns the intercept. Both `api_proxy` and
`handle_mesh_request` now call `try_handle_moa` — the request is
handled wherever it lands, whether the node serves models locally
or not.
2. `build_moa_config` enumerates `Node::models_being_served()` (the
mesh-wide union of local + gossip) instead of the local routing
table. Locally-served models are wired directly to the skippy port
via the routing table when one is present (host mode); everything
else opens a QUIC tunnel to the hash-preferred peer that advertises
the model. On a pure client every worker is remote.
Side effects:
- Canonical-base dedup now strips an `@branch` segment without losing
the trailing quant tag, so `unsloth/Qwen3-8B-GGUF@main:Q4_K_M` and
`Qwen3-8B-Q4_K_M` collapse to one worker instead of being treated
as two distinct models.
- The MoA-related backends (`LocalModelBackend`, `RemoteModelBackend`)
and the SSE wrapper moved out of `ingress.rs` into the shared
module; `ingress.rs` shrinks by ~400 lines.
Verified live against the public mesh from a `--client --auto` node:
- Chat completion: `x-moa-workers: 4` (all 4 mesh-wide models),
early-exit path, correct answer.
- Tool-equipped request: 3 tool-capable workers, early-exit path,
correct shape.
* moa: carry attempts on reducer-failure path; copilot review fixes
Two real bugs surfaced by live goose testing against the public mesh from a
--client --auto node.
Bug 1 — attempts accounting on the failure path.
hedged_reducer_call returned Result<HedgedReducerOk, String> where the Ok
arm carried 'attempts: u32' but the Err arm dropped it. Both call sites in
lib.rs (handle_tool_result, resolve_decision) reported attempts=0 on the
all-fail path, producing nonsense like 'Reducer failed (tried 0): remote
timeout after 15s'. Replace with Result<HedgedReducerOk, HedgedReducerErr>
where Err carries attempts too. Surface the real spawn count to logs and
to the user-visible error string.
Bug 2 — copilot review issues.
- UTF-8-safe truncation: 2 panicking '&text[..len.min(N)]' sites in
moa_gateway replaced with new moa::truncate_chars helper that walks back
to a char boundary.
- Remote-read cap 256 KiB → 4 MiB. Long reasoning + tool synthesis answers
can exceed 256 KiB.
- CR/LF sanitization on x-moa-* header values. Cheap insurance.
- Removed dead ('unknown', 0) fallback in reducer_candidates. Let
hedged_reducer_call's empty-input path surface real errors instead of
silently dispatching to backend_index=0 with a bogus name.
- Consolidated three byte-identical strip_thinking implementations
(worker.rs, normalize.rs, moa_gateway.rs) onto one canonical
moa::worker::strip_thinking with re-export from moa crate root.
- Warn on response-write failure rather than swallowing the error.
Tests: 4 new (truncate_chars on UTF-8 boundary, all-fail-reports-attempts),
all 63 moa + 1403 host pass. Clippy + fmt clean.
Live verified from --client --auto on this Mac, joined to public mesh:
- Plain chat: x-moa-workers: 2, early-exit, 1.3s, correct answer.
- Goose end-to-end (tool propose → shell exec → tool-result turn → final):
full loop completed, server log shows fanout + early-exit on both turns,
goose printed DONE and exited 0.
* moa: address remaining Copilot review items
- context.rs / session.rs / backend.rs: replace byte-index truncation
with crate::worker::truncate_chars (UTF-8 safe). Worker payloads,
tool outputs, and HTTP error bodies all come from external sources
that can contain multi-byte characters.
- reducer.rs hedge loop: once `remaining` is exhausted, stop arming
the hedge timer and just await join_next() directly. Previously the
select! kept rebuilding a fresh hedge_sleep every iteration and
firing every hedge_delay just to no-op. Untidy, not a correctness
bug — but easier to reason about now.
Closes inline review feedback on PR #566.
* moa: tighten early-exit content check with subset+negation rule
Early-exit previously claimed "workers agree" whenever 2+ outputs were
Answer-kind, without comparing payload content. Two workers replying
"Paris" and "Berlin" both with confidence ~0.5 (the default for plain
prose) would early-exit on whichever was returned first.
New rule: two answers agree iff
- the smaller content-token set is a subset of the larger, AND
- their symmetric difference contains no negation tokens.
Tokenization: lowercase, strip punctuation, drop stopwords and tokens
<3 chars (digits and negation words always kept).
This is biased toward false-negatives: terse-vs-verbose paraphrases
like "Paris" / "Paris is the capital of France" cluster correctly,
while same-shape disagreements like "...is Paris" / "...is Berlin"
do not. When the rule declines to cluster, we just wait for more
workers and fall through to arbitrate() — no extra reducer call.
Also:
- session.rs:341: replace one remaining &first_line[..77] byte-slice
with worker::truncate_chars (multi-byte panic risk on tool names
containing emoji).
- ingress.rs MoA intercept: replace let _ = try_handle_moa(...) with
if let Some(...) and a tracing::error! so the impossible "returned
unused stream" case is loudly logged instead of silently leaked.
Tests:
- 4 reworked early-exit tests (terse-vs-verbose, normalized-equivalent,
majority cluster, shared-scaffolding-still-blocks)
- 3 new negation guard tests (not, don't, "use grep" vs "do not use
grep")
- 1 numeric agreement test ("42" vs "the answer is 42")
73 moa tests pass, 1426 host-runtime tests pass, both clippies clean.
* docs(moa): add pressure-test research plan to MOA_GATEWAY
Replace the earlier 'A/B plan' sketch with a research plan that is
designed to falsify the mixture hypothesis, not confirm it.
- Sharpened hypothesis with three falsifiable corollaries
- Pre-committed falsification conditions (so we cannot move goalposts)
- Step 1: variance floor measurement as prerequisite for any A/B claim
- Adversarial scenarios including failure-mode-amplification cases
- Pareto curve as the headline deliverable, not win/tie/loss
- Ablations to separate 'mixture' from 'variance reduction'
- Composition sweep to test the 'modest models' framing directly
- Grader robustness checks (position swap, dual grader, hand spot-check)
- Real-task replay as the strongest defense against cherry-picking
- Reporting discipline: what must be in a result before calling it a win
Documentation only. No crate changes. Worker-set knob noted as a
harness-side concern, not a crate change.
* docs(moa): reframe pressure test around equal-VRAM split-vs-mix on mesh
The earlier pressure-test plan was "is mixture smarter than single best,"
which is the wrong load-bearing question. The honest question for a mesh
is: given fixed aggregate (V)RAM, when does running multiple diverse
mid-size models locally beat sharding one large model across the network?
Reframes the eval around the equal-VRAM trade between Skippy split-large
and MoA mix-diverse, with network conditions (RTT, loss) as the primary
axis. Existing scenario/ablation content becomes the quality measurement
implementation, not the headline. Adds pre-committed falsification
conditions specific to the network-tolerance and scalability claims.
Docs-only.
* docs(moa): reframe as operating-envelope, not benchmark fight
The earlier draft framed MoA vs split-large as a quality competition. The
real claim is that split-large has a hard practical ceiling on a real
mesh — every cross-node hop is on every token's critical path — and MoA
has a much higher ceiling because workers run fully local and the
network is only touched at fan-out/collect/reducer.
Reframe accordingly:
- Headline is *operating-envelope analysis*, not Pareto fight
- Define what 'acceptable' means (TTFT, total turn, failure rate, quality
floor) before any measurement, so we cannot retrofit it
- Deliverable is a *viability map* (config x network condition), not a
win/tie/loss table
- Quality is demoted to a tertiary axis inside the viable region; its job
is to confirm MoA's MoA-only-region answers clear the single-mid floor
- Pre-committed falsification conditions are specific to the new claims
(envelope shrinkage with mesh size/network, MoA's envelope extending
past split's, MoA quality above single-mid floor, mixture vs variance
reduction)
- single-mid baseline added explicitly so we cannot accidentally ship
'MoA = single-best + overhead'
Complementary positioning, not competitive: use split when the network
allows; use mix when it doesn't.
* docs(moa): promote 'why MoA exists' to top of design doc
The opening of MOA_GATEWAY.md described mechanism (fan out, arbitrate)
but not purpose. The motivation \u2014 'use the mesh anyway when split-large
isn't viable for the current network conditions' \u2014 was buried ~450
lines down inside the operating-envelope section.
Add a brief 'Why MoA exists' section at the top that states:
* The intended operating region (where split-large stops being viable).
* That MoA is not trying to beat split-large on quality.
* The complementary, network-conditions-decide-which framing.
* A link down to the experimental envelope discussion that already exists.
No design or behavior change. Pure framing of existing content.
* fix(moa): signal all-workers-fail as a proper error response
PR #566 review feedback (Apr 2026):
> One concurrency request returned HTTP 200 even though the response
> body said all MoA workers failed. That's a bad client contract.
> If all workers fail, the API should probably return a proper error,
> not a successful-looking response with failure text inside it.
The MoA gateway was returning a body shaped identically to a
successful `chat.completion` with the error string smuggled into
`choices[0].message.content` and `finish_reason: "stop"`. The
ingress wrapped that body in an HTTP 200. A client checking either
the HTTP status, the top-level `error` field, or `finish_reason`
saw "success."
## Test (added first, observed failing)
`crates/mesh-mixture-of-agents/tests/sim_all_workers_fail.rs` drives
`moa::handle_turn` with three `AlwaysErrBackend`s, asserts the
result body is distinguishable from a successful `chat.completion`
\u2014 either the top-level `object` is not `chat.completion`, or there
is a top-level `error`, or `finish_reason` is one of `error` /
`moa_failed`.
The test fails against the pre-fix gateway with output
> object=Some("chat.completion"), finish_reason=Some("stop"),
> has top-level error=false
## Fix
* `mesh-mixture-of-agents/src/lib.rs` \u2014 `error_response()` now
attaches a top-level OpenAI-shape `error` object and emits
`finish_reason: "error"`. The error text stays in `content`
for unstructured clients.
* `mesh-llm-host-runtime/src/network/openai/transport.rs` \u2014 new
`send_json_with_status_and_headers()` helper for sending a custom
status code with a full structured body and observability headers.
* `mesh-llm-host-runtime/src/network/openai/moa_gateway.rs` \u2014
`write_moa_response` now takes the full `TurnResult` and sends
HTTP 502 (Bad Gateway) when `turn_kind == Failed` for non-streaming
responses. Streaming SSE stays 200 because we can't change the
status after the headers are sent; the failure rides in the chunked
body (which now carries the structured error).
## Validation
`cargo test -p mesh-mixture-of-agents` \u2014 70 unit + 1 new
integration test pass.
`cargo test -p mesh-llm-host-runtime --lib` \u2014 1434/1434 pass.
`cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings`
\u2014 clean.
`cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings`
\u2014 clean.
`cargo fmt --all -- --check` \u2014 clean.
* fix(moa): account for aborted workers in worker_summaries
PR #566 review feedback (Apr 2026):
> Worker accounting was inconsistent:
> - Similar requests reported different x-moa-workers values.
> - Similar requests reported different x-moa-workers-ok values.
> - Some successful responses used fewer workers than expected.
> - Some churn responses still appeared to report stale worker counts.
`worker_summaries.len()` is what the `x-moa-workers` header reports.
When the arbiter early-exits on consensus, the gateway called
`JoinSet::abort_all` and then drained `join_next()` with
`if let Ok(...)`. `JoinSet::abort_all` causes aborted tasks to
return `Err(JoinError::cancelled)`, with no `(model, role)`
payload \u2014 those tasks were silently dropped from `summaries`. A
4-worker fan-out that early-exited from 2 fast workers reported
`x-moa-workers: 2`, hiding the fact that 2 workers were cancelled
mid-flight. Panicked tasks had the same problem.
## Test (added first, observed failing)
`crates/mesh-mixture-of-agents/tests/sim_worker_accounting.rs` sets
up 4 mock backends, 2 fast/agreeing and 2 slow, and asserts that
`worker_summaries.len() == 4` after early-exit \u2014 i.e. that the
header faithfully reflects the dispatched count.
The test fails against the pre-fix gateway with output
> Got 2 summaries: ["fast-a-3b", "fast-b-3b"]; expected 4.
## Fix
* `fanout.rs` \u2014 `gather_workers_incremental` now takes the
dispatched-worker list (`&[DispatchedWorker]`) instead of just a
count. After fan-out finishes (whether via normal completion or
early-exit drain), `reconcile_dispatched` walks the dispatched
list and synthesizes a `succeeded: false` summary for any worker
whose name does not appear in `summaries`. Aborted tasks and
panicked tasks are now both attributed.
* `lib.rs` \u2014 builds a `Vec<DispatchedWorker>` alongside the
`JoinSet` and threads it through to `gather_workers_incremental`.
The header `x-moa-workers` now always equals the worker count we
actually dispatched. `x-moa-workers-ok` continues to reflect
genuinely-succeeded workers only.
## Validation
`cargo test -p mesh-mixture-of-agents` \u2014 70 unit + 2 integration
tests pass (new test plus the existing all-workers-fail one).
`cargo test -p mesh-llm-host-runtime --lib` \u2014 1434/1434 pass.
`cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings`
\u2014 clean.
`cargo fmt --all -- --check` \u2014 clean.
* fix(moa): route tool-result follow-ups to reducer, not fan-out
PR #566 review feedback (Apr 2026):
> The tool-result path isn't ready for agent loops:
> - A tool-result follow-up was treated like another fanout turn.
> - It wasn't handled like a controlled reducer/synthesis turn.
> - Tool results should be handled carefully and predictably, not
> sprayed back through the whole fanout path.
`Session::classify_turn` only routed to `TurnType::ToolResult` when
the very last message had `role: "tool"`. Many agent harnesses send
the tool result followed by a short `user` nudge ("continue", "what
did you find?"). That landed at the very-last-message check as
`user`, so the gateway classified the turn as Continuation, fanned
out to all workers, and invited a worker to re-propose the same tool
call whose result was already in context.
The session-state fallback at `last_was_tool_call &&
has_unprocessed_tool_results` was dead code in production: the
gateway never invokes `record_assistant_response` between turns, so
`last_was_tool_call` is always false.
## Test (added first, observed failing)
`crates/mesh-mixture-of-agents/tests/sim_tool_result_routes_to_reducer.rs`
\u2014 three scenarios, all using mock backends that count calls:
* OpenAI canonical shape (last msg role=tool) \u2014 must classify as
`ToolResult`, exactly one backend call (reducer only). Already
passed pre-fix; pinned to prevent regression.
* Trailing-user-after-unsynthesised-tool-result \u2014 must also classify
as `ToolResult`, exactly one backend call. **Failed pre-fix with
`TurnKind::EarlyExit`** (fanned out, multiple worker calls).
* Plain fresh user question \u2014 must still fan out. Pins that we don't
over-trigger the tool-result path.
## Fix
Scan messages from the end in `Session::classify_turn`:
* First message we hit with `role: "tool"` \u2192 classify as
`ToolResult`. The tool result has not yet been synthesised by an
assistant message after it.
* First message we hit with `role: "assistant"` \u2192 stop. The
assistant has already spoken since the last tool result; the next
turn is a normal continuation.
* Other roles (`user`, `system`) \u2192 keep scanning. A user nudge
after an unsynthesised tool result still belongs in the
reducer-only path.
If the scan reaches the start without hitting either, fall through to
the existing `Fresh`/`Continuation` classification.
## Validation
`cargo test -p mesh-mixture-of-agents` \u2014 70 unit + 5 integration
tests pass (this PR\u2019s 3 new tests + the two earlier sim files).
`cargo test -p mesh-llm-host-runtime --lib` \u2014 1434/1434 pass.
`cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings`
\u2014 clean.
`cargo fmt --all -- --check` \u2014 clean.
* fix(moa): recognise OpenAI-shape inline tool JSON in worker output
PR #566 review feedback (Apr 2026):
> In the read-tool probe, the model wrote text that looked like a
> tool call instead of actually invoking the read tool.
Agent harnesses (Goose, OpenCode, pi) only act on real `tool_calls`.
If a worker emits inline OpenAI-shape tool JSON \u2014
"I'll read the README. {\"function\": \"read_file\",
\"arguments\": {\"path\": \"README.md\"}}"
\u2014 today's normalizer's `try_json_parse` requires a `kind` field in
the JSON, which the OpenAI tool-call shape never has. `try_json_parse`
returns None, the heuristic classifier doesn't have an action verb
that matches ("I'll read" isn't on its list), and the worker output
falls through to `OutputKind::Answer`. Three workers all return the
same text \u2192 arbiter agrees \u2192 `chat_response(text)` \u2014 the agent
gets the JSON-bearing prose as `content`, no `tool_calls` field,
and silently does nothing.
## Test (added first, observed failing)
`tests/sim_tool_call_text_not_passed_as_content.rs` \u2014 two scenarios:
* `workers_with_inline_tool_json_emit_real_tool_call` \u2014 workers
return prose with embedded `{"function": "read_file",
"arguments": {...}}`. The response body must carry a real
`tool_calls` array with the proposed function name. **Failed
pre-fix**: body had `content` with the prose, no `tool_calls`.
* `workers_describing_tool_call_must_emit_structured_tool_call` \u2014
workers describe a tool call in pure prose with no JSON. Today
the heuristic catches this and synthesises a `tool_calls` entry
(with empty arguments). Pinned so the JSON-shape fix below
doesn't regress the pure-prose path.
## Fix
`normalize::try_json_parse` now also recognises the OpenAI tool-call
shape when no `kind` field is present:
{"function": "read_file", "arguments": {...}}
{"name": "read_file", "arguments": {...}}
{"tool": "read_file", "arguments": {...}}
A structurally well-formed inline tool proposal scores confidence
0.75 (above the heuristic's 0.6) so the arbiter prefers it on ties.
The rest of the original `kind`-driven envelope path is unchanged.
## Validation
`cargo test -p mesh-mixture-of-agents` \u2014 70 unit + 7 integration
tests pass (this PR\u2019s 2 new tests + earlier sim files).
`cargo test -p mesh-llm-host-runtime --lib` \u2014 1434/1434 pass.
`cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings`
\u2014 clean.
`cargo fmt --all -- --check` \u2014 clean.
* fix(moa): /v1/models advertises quant-suffix IDs that route back
PR #566 review feedback (Apr 2026):
> Some IDs in /v1/models dropped quant suffixes. Other endpoints
> used the full model refs.
> [...]
> Direct calls from Carrack to worker-hosted models didn't work.
> Carrack to Lemony 35B returned HTTP 404.
Reproduced on a 2-node mesh (Mac M4 Max + Mac Studio M3 Ultra):
* M4 served `Qwen/Qwen2.5-3B-Instruct-GGUF:qwen2.5-3b-instruct-q4_k_m`.
* Studio served `unsloth/Qwen3-0.6B-GGUF:BF16`.
* M4 `/v1/models` listed:
Qwen/Qwen2.5-3B-Instruct-GGUF (quant suffix lost)
unsloth/Qwen3-0.6B-GGUF:BF16 (full id)
* Calling either listed id directly:
local short id \u2192 200 (rewritten via internal alias table)
remote short id \u2192 404 (no alias for remote models)
The natural client flow \u2014 read `/v1/models`, take an id, call
`/v1/chat/completions` with it \u2014 was broken for remote-hosted
models, and inconsistent for local ones.
## Two root causes
1. **`quant_selector_from_gguf_file` was uppercase-only** when
matching markers like `-Q`, `-BF16`. Real GGUF filenames mix
cases (`...-Q4_K_M.gguf` vs `...-q4_k_m.gguf`). The matcher
on the read side (`gguf_matches_quant_selector`) was already
case-insensitive, so emitting a lowercase selector is safe and
keeps the public id round-trippable.
2. **`public_huggingface_model_ref` only handled artifact-as-filename.**
Locally-built `ServedModelDescriptor`s set
`artifact = model_ref.selector` \u2014 e.g.
`"qwen2.5-3b-instruct-q4_k_m"`, a quant selector, not a GGUF
filename. `quant_selector_from_gguf_file` returned None for
anything not ending `.gguf`, so the public id collapsed to just
the repo name.
The `public_model_id` selection logic also needed tightening so we
fall back to local disk only when the descriptor cannot produce a
lossless id (e.g. catalog or local-gguf identities without enough
metadata).
## Test (added first, observed failing)
`models_list_id_preserves_quant_suffix_when_descriptor_has_no_artifact`
in `transport.rs` builds a HuggingFace descriptor with no `artifact`
field and asserts the resulting public id either matches the internal
model_name verbatim or carries a non-empty quant tag. Pre-fix the
public id collapsed to bare repo and the test failed.
## Fix
* `model-ref/src/lib.rs::quant_selector_from_gguf_file` \u2014 lowercases
the stem before matching markers so lowercase-quant filenames like
`qwen2.5-3b-instruct-q4_k_m.gguf` extract `q4_k_m` instead of None.
Returns the slice from the original stem so display casing is
preserved.
* `transport.rs::public_huggingface_model_ref` \u2014 accepts artifact
values that are already a quant selector (no `.gguf` suffix) in
addition to GGUF filenames. The selector now round-trips through
the resolver.
* `transport.rs::public_model_id` \u2014 prefers the descriptor only
when its identity carries enough information to produce a lossless
id (HuggingFace needs an artifact; Catalog needs a canonical_ref).
Otherwise falls back to the on-disk file, then the model_name
itself \u2014 never silently drops information.
## Validation
`cargo test -p mesh-llm-host-runtime --lib` \u2014 1435/1435 pass.
`cargo test -p model-ref` \u2014 10/10 pass.
Live 2-node mesh (M4 gateway + Studio peer):
`/v1/models` now reports both models with their full ids:
Qwen/Qwen2.5-3B-Instruct-GGUF:q4_k_m
unsloth/Qwen3-0.6B-GGUF:BF16
Direct `/v1/chat/completions` calls with either listed id return
200 and real inference for both local and remote models.
`model: "mesh"` (MoA) still works end-to-end on the same setup,
returning a real fanout response (`Tokyo` from the capital-of-japan
prompt).
`cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings`
\u2014 clean. `cargo fmt --all -- --check` \u2014 clean.
* fix(moa): bump worker/reducer timeouts to 60s for agent-scale prompts
PR #566 review feedback (Apr 2026) flagged that MoA worker accounting
sometimes reported zero successful workers, especially under churn or
load. Investigating from a real 2-node mesh (Mac M4 Max + Mac Studio
M3 Ultra) with an OpenCode agent driving `model: "mesh"` showed
that the root cause was the 15s worker_timeout being too tight for
agent-scale prompts:
* OpenCode's default system prompt is ~13.7k tokens.
* Large strong-tier models (MiniMax-M2.5 Q4_K_M, Qwen3-32B+) at 13k+
prompts + tool schemas take 20\u201340s for a first useful response \u2014
the reasoning preamble alone often eats the 15s budget.
* The MoA gateway killed the strong worker at exactly 15s every turn:
moa: worker unsloth/MiniMax-M2.5-GGUF:Q4_K_M (strong) failed
after 15001ms: remote timeout after 15s
* The arbiter then early-exited on the surviving small worker, never
giving the strong worker a chance to land. The strong worker was
effectively unreachable for OpenCode/Goose-style flows.
Bump both `worker_timeout` and `reducer_timeout` from 15s \u2192 60s
in `build_moa_config`. Live verification on the same 2-node mesh:
* With 15s: `model: mesh` from OpenCode finished 0 of 3 turns
successfully. Every turn returned 1/2 workers, strong worker
timeout, no useful response.
* With 60s: `model: mesh` from OpenCode finished 2 of 3 turns
successfully \u2014 strong worker landed, MoA produced the structured
`tool_calls` field, OpenCode invoked the file-read tool correctly.
(The 3rd turn hit a separate llama_decode / connection-lost issue
in the local stage runtime that is unrelated to MoA timing.)
The trade-off is that a single hung remote worker can stall a turn
for 60s instead of 15s. That is acceptable for an interactive agent
loop where the alternative is consistent failure to land the strong
worker at all. The hedged-reducer ladder (`hedge_delay` = 5s)
still keeps end-to-end latency bounded when only the *reducer* is
slow.
`cargo test -p mesh-llm-host-runtime --lib` \u2014 1435/1435 pass.
`cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings`
\u2014 clean. `cargo fmt --all -- --check` \u2014 clean.
* fix(mesh): enable QUIC keep-alive on mesh transport (was: connections dropping mid-inference)
PR #566 review feedback flagged MoA returning early with 0/N workers
under load. Live debug on a 2-node mesh (M4 Max + Mac Studio M3 Ultra)
running an OpenCode agent against `model: "mesh"` showed repeated:
WARN noq_proto::connection: failed closing path err=LastOpenPath
INFO mesh: Connection to <peer> closed: timed out
WARN moa: reducer ... failed: recv: read error: connection lost
happening 30-60s into otherwise healthy inference calls, including
plain non-MoA `stream: false` requests through `model: "auto"`.
Root cause: noq-proto's default `max_idle_timeout` is 30s and
`keep_alive_interval` is `None` (the spec, RFC 9000 §10.1.2, makes
keep-alive opt-in; quinn / noq follow that). Non-streaming inference
requests send no application bytes while the remote model is
generating tokens, so the wire is idle. Under concurrent load
(parallel MoA workers + reducer + gossip + heartbeats), noq's
multipath bookkeeping closes the idle path, and when it is the last
open path the entire connection drops mid-stream. The in-flight HTTP
tunnel errors with `connection lost` and the caller must retry.
This only became visible recently because:
* Streaming OpenAI clients (Goose, Claude Code, pi, the web UI) all
set `stream: true` by default. SSE chunks flow continuously and
reset the idle timer, so the bug never manifests for them.
* MoA `RemoteModelBackend` is the first significant non-streaming
long-running RPC in the codebase (`stream: false` hardcoded in
`crates/mesh-mixture-of-agents/src/backend.rs`).
* Reasoning models with big agent prompts (MiniMax-M2.5 on a 13k
OpenCode system prompt, Qwen3-32B class reducers) routinely take
30-90s for a first useful response. That is the combination that
exceeds the default 30s idle window.
Fix: set `keep_alive_interval = 10s` and `max_idle_timeout = 5m`
on the mesh QUIC transport config, plus the matching multipath
`default_path_keep_alive_interval` and
`default_path_max_idle_timeout` so individual paths don't get torn
down while the connection-level idle timer is fine.
Cost: one QUIC PING (~30-60 bytes) every 10s per connection only
when no other application data has been sent for that long. In a
typical mesh with periodic gossip and heartbeats this fires rarely.
`keep_alive` is opportunistic, not unconditional.
Live verification on the same 2-node mesh:
* 60s idle test, before fix: 2x `Connection to <peer> closed:
timed out`. After fix: 0x. Connection stays healthy.
* 75s of mixed non-streaming inference (53s `auto` to MiniMax +
21s `mesh` 2-worker fanout), before fix: multiple `LastOpenPath`
+ `connection lost` errors. After fix: 0x. Both completed
successfully with finish_reason=stop and full content.
* OpenCode `model: mesh` agent loop, before fix: 0 of 2 turns
landed. After fix: 2 of 3 turns landed (the 3rd hit a separate
KV cache exhaustion in the local stage runtime, tracked
independently).
Validation: `cargo fmt --all -- --check` clean,
`cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings`
clean, `cargo test -p mesh-llm-host-runtime --lib` 1435/1435 pass.
* fix(planner): cap auto lane count to llama-server's 4-lane unified-KV default
PR #566 review feedback uncovered a hard 502 from the embedded skippy
stage runtime under concurrent agent-style workloads. On a Mac M4 Max
serving Qwen3-8B at the model's native 32k context, the auto planner
was picking `slots = 16` (MAX_AUTO_PARALLEL_SLOTS). Three concurrent
~14k-token requests \u2014 the exact shape an OpenCode agent loop produces
when MoA fans a worker call out next to a reducer call \u2014 fail in the
embedded llama with:
decode: failed to find a memory slot for batch of size 2048
surfacing as HTTP 502:
skippy ABI call failed: RuntimeError: llama_decode failed
Root cause: skippy's stage runtime sets `kv_unified = true` whenever
`lane_count > 1` (`third_party/llama.cpp/patches/0034-Add-shared-execution-lanes-to-skippy-ABI.patch`).
In unified mode llama allocates exactly `n_ctx` cells total, shared
across all `n_seq_max` sequences. The previous planner derived
`slots` from VRAM as if each lane carved off its own
`n_ctx \u00d7 bytes_per_token` allocation \u2014 which is the
`kv_unified = false` semantics, not what skippy actually does. On a
node with comfortable VRAM the math happily returned the snapped
maximum of 16 lanes, even though all 16 raced for the *same* fixed
pool of `n_ctx` cells.
Fix: drop `MAX_AUTO_PARALLEL_SLOTS` from 16 to 4, matching upstream
llama-server's own auto default for the same reason. From
`.deps/llama.cpp/tools/server/server.cpp`:
LOG_INF("n_parallel is set to auto, using n_parallel = 4 and
kv_unified = true");
params.n_parallel = 4;
params.kv_unified = true;
Lane count is purely a concurrency-policy knob under `kv_unified =
true`; it does not change the KV cache allocation. Going from 16 to
4 frees zero RAM; it just gates admission control to a sane number
of concurrent in-flight requests for the shared cell pool.
Operators who know their workload (short chat turns, low-concurrency
hosts, etc.) can still pick a higher value via the existing
`parallel_override` plumbing, including `[models.throughput]
parallel = N` in the TOML config from PR #564.
Live verification on the same 2-node mesh used to find the bug:
* M4 + Qwen3-8B at 32k `n_ctx`: planner now picks `slots = 4`,
llama logs `n_seq_max = 4`, KV cache stays at 2448 MiB (one
shared buffer; no RAM cost change).
* Studio + MiniMax-M2.5 at 128k `n_ctx`: planner now picks
`slots = 4`, llama logs `n_seq_max = 4`, KV cache stays at 8928
MiB. 4 \u00d7 32k cells per lane on average is plenty of headroom for
agent prompts.
* Repro that previously 502'd \u2014 3 parallel ~15k-prompt tool-result
follow-ups on the M4 \u2014 now all succeed with `finish_reason=stop`,
full content, ~20s wall time. Zero `find_slot` failures, zero
`llama_decode` errors, zero skippy ABI errors.
* Burst test \u2014 5 parallel at the same prompt shape \u2014 the 5th
request correctly hits the admission-control queue and returns a
clean
`{"type":"rate_limit_error","code":"rate_limit_exceeded"}`
after the admission timeout, instead of an opaque mid-flight 502.
Adds two regression tests in `context_planning::tests`:
* `auto_slots_capped_at_llama_server_default` covers the
high-VRAM small-model case that used to plan 16.
* `explicit_parallel_can_exceed_auto_ceiling` covers the
override path so operators retain control.
Validation: `cargo fmt --all -- --check` clean,
`cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings`
clean, `cargo test -p mesh-llm-host-runtime --lib` 1437/1437 pass
(includes the two new regression tests).
* docs(moa): report on micn/moa branch's critical fixes beyond MoA itself
While iterating on PR #566 review feedback, this branch surfaced and
fixed several pre-existing host-runtime bugs that were either hard to
hit before or silently masked by KV-leakage bugs that have since been
fixed. Capture the full picture in one place so PR reviewers and
future readers don't have to spelunk through 60+ commits to see what
landed.
Headline fixes documented:
1. Mesh QUIC keep-alive (
|
||
|
IvGolovach
|
2ffdc482ae |
Add Flash-MoE SSD backend plugin
Validation * Validation tier: Tier 2R - post-review conflict/base refresh of an existing shared runtime integration PR; the manual conflict scope was README.md, with targeted Flash-MoE/runtime checks rerun on the final rebased diff. * git fetch --no-tags origin main:refs/remotes/origin/main: PASS * git rebase origin/main: PASS, resolved conflict in README.md. * git diff --check origin/main...HEAD: PASS * git diff --cached --check: PASS * cargo fmt --all -- --check: PASS * LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal rustup run stable cargo test -p mesh-llm-host-runtime flash_moe --lib: PASS, 13 passed, 0 failed * LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal rustup run stable cargo check -p mesh-llm: PASS * Ledger: not applicable - not required for selected validation tier/change family. * Version: not applicable - not required for selected validation tier/change family. * Not run: just build - not required for this conflict/base-refresh tier; targeted Rust checks covered the affected runtime paths and GitHub CI will rerun the final PR SHA. * Not run: full cargo test -p mesh-llm-host-runtime --lib - not required for selected tier; targeted Flash-MoE tests covered the changed plugin/runtime path. Rollback * git revert HEAD |
||
|
Nick DiZazzo
|
53f4e6c596
|
feature(split-control): split control protocol from primary mesh-llm ALPN (#539)
* guard runtime control API routes to localhost callers * strip old mesh/1 config control from protocol |
||
|
Nick DiZazzo
|
c2796ead80 |
chore(docs): update docs to match planned updates
* Add new doc revision based on notes in Canva |
||
|
Nick DiZazzo
|
7d03f4a444 |
chore(windows-ci): restore Windows release and CI lanes
* Restore Windows release build lanes alongside Linux/macOS * Add sccache-based platform ABI caching with warm-up and retry hardening * Set up dedicated Windows GPU SDK jobs for CUDA and ROCm toolchains * Link OpenMP runtime, disable GGML OpenMP, fix ROCm HIP compiler configuration * Use Clang define syntax for Windows ROCm builds * Fix Windows llama prepare cleanup and nvcc/sccache incompatibility * Narrow GPU smoke test architectures and skip launches without drivers * Shorten cache-hit runs and avoid redundant build launches on Windows * Add fast CPU check path and skip unused Node setup in CI * Run Windows checks on pull requests for early feedback * Update thread leak tests to only count growth rather than absolute count |
||
|
James Dumay
|
e6b723d199
|
mesh-llm crate decomposition (#459) | ||
|
James Dumay
|
7ea662e0e5
|
Certify additional split-serving families (#439)
* Prepare workspace for skippy integration * Copy skippy model crates * Use skippy model crates in resolver * Add mesh compatibility tests for model crates * docs: plan skippy serving replacement * Expose skippy generation signal ABI * Integrate skippy staged serving and cleanup runtime state * Surface backend-neutral stage status and split lifecycle * Add hook parity fixtures and debug forcing * Support selected backend device in skippy runtime * Add direct GGUF package provenance to skippy status * Add skippy stage materialization cleanup * Add backend-neutral runtime status and Skippy stage payloads * Propagate projector paths through stage status * Add skippy multimodal skills * Add multimodal prefill support * Finish multimodal split serving * Add multimodal status and smoke tests * Split skippy and mesh API modules * Fix skippy ABI setup in CI * Import skippy model slicing tooling * Import skippy correctness diagnostics * Import skippy operator tooling * Import skippy benchmark workflows * Fix skippy bench test fixtures * Use model refs in OpenAI model listing * Rename llama-stage patch queue and clean stale runtime refs * Remove legacy llama runtime lane * Serialize startup model loads and honor pinned Vulkan flavor * fix vulkan pinned device resolution * remove legacy moe smoke hooks * Fix client Docker image build * Remove legacy runtime lane and clean up mesh-llm * Split skippy stage protocol and prune runtime leftovers * Prune experimental llama cache patches * Mark skippy test deploy version * Fix local GGUF model ids in OpenAI model list * Speed up Linux llama ABI builds * Refactor staged runtime and telemetry plumbing * Preserve model refs in console labels * Rebase llama.cpp stage patches onto main * Fix CI native llama linking * Refactor documentation across crates for mesh-driven architecture, OpenAI API integration, and staged runtime updates. * Rename skippy model package tooling * Update SKILL.md for prompt-owned staged workflows and backend-specific deployment rules * Add skippy execution lanes and KV cache policy * feature(config-fields): wire configuration fields to kv-quant, batch settings (#429) * feature(config-fields): wire configuration fields to kv-quant, batch settings * address PR comments * Add skippy execution lanes and KV cache policy (#428) * Add skippy execution lanes and KV cache policy * feature(config-fields): wire configuration fields to kv-quant, batch settings (#429) * feature(config-fields): wire configuration fields to kv-quant, batch settings * address PR comments * Support OpenAI tool calls in skippy * Support grammar-constrained tool calls * Trim stale skippy correctness state handoff * Fix stale model config test fixtures --------- Co-authored-by: Nick DiZazzo <728690+ndizazzo@users.noreply.github.com> * Add skippy exact prefix cache * Wire exact prefix cache into OpenAI serving * Optimize skippy cache serving * Add DeepSeek3 package cache support * Normalize cache benchmark evidence * Certify DeepSeek3 package cache strategy * Add HF use-case cache benchmark matrix * Include skippy cache in docker build * Group cache benchmark families * Add reproducible cache family benchmarks * Explain Skippy cache benchmark wins * Rename cache use-case benchmark heading * Derive skippy cache caps from stage KV settings * Wire skippy prompt cache defaults * Add split family certification support * Certify LFM2 split execution * Certify Mamba split execution * Certify Jamba split execution * Certify Gemma f32 stage wire policy * Certify RWKV6 split execution * Certify Qwen MoE split execution * Clarify Llama4 parity artifact status * Document cache correctness exit gate * Add cache correctness remap gate * Support RWKV7 staged activation sideband * Certify recurrent cache restore families * Guard recurrent families from resident KV cache * Classify Qwen3 active-parameter coder as MoE * Add dense cache smoke parity evidence * Add MoE expert cache smoke evidence * Record multimodal family smoke status * Certify Phi2 runtime slices * Certify Granite runtime variants * Certify Hunyuan dense runtime slices * Certify additional decoder families * Certify Hunyuan MoE family * Promote Qwen MoE family evidence * Certify InternLM2 family * Certify more dense decoder families * Certify OLMo and decoder families * Promote recurrent family policies * Certify PhiMoE split support * Promote remaining dense family evidence * Certify GLM4 MoE family * Certify Qwen-VL text split cache lane * Add package manifest support for mmproj artifacts * Update layer package repository spec * Add broad llama family stage parity coverage * Certify additional skippy families * Cover certified family cache policies * Fix materialized stage cache test isolation * Restore CI workflows from main * Fix clippy warnings * Address split-family review feedback * Certify Kimi Linear split support * Certify split multimodal families * Certify Gemma3n and EXAONE-MoE split support * Fix split-family CI regressions * Fix skippy runtime clippy warnings --------- Co-authored-by: Nick DiZazzo <728690+ndizazzo@users.noreply.github.com> |
||
|
Michael Neale
|
549f480487
|
fix: wire --discover into serve path, add --name to discover subcommand (#453)
* fix: wire --discover into serve path, add --name to discover subcommand
--discover <name> was completely ignored on the serve/client startup
path — only worked for the MCP path. Users running
'mesh-llm --discover mic22' would silently create a new standalone
mesh instead of joining mic22.
Changes:
- --discover triggers Nostr discovery on serve path (same as --auto)
- --discover name is used as target mesh name filter
- discover subcommand gains --name flag for filtering by mesh name
- Suppress misleading 'private by default' warning when joining
- Fix discover subcommand help text (was suggesting nonexistent
'discover --join')
- MeshFilter gains name field for case-insensitive exact mesh name
matching
Tested: --join token, --discover name (serve), client --discover name,
discover --name, relay-only join
* fix: retry join once after 5s if first attempt fails
Relay-only connections (no reachable direct IPs) can fail on the
first attempt because the local relay websocket may not be fully
established when the join fires. The 5s relay startup wait at
Node::start isn't always enough, especially on slower networks
or when connecting to a relay in a different region.
Add a single retry with 5s delay in both the main join loop and
the MCP join path. This gives the relay connection time to
establish without slowing down the common case (direct IP joins
succeed immediately on the first try).
* fix: add ..Default::default() to filter_combined test for new MeshFilter.name field
* docs: add --discover and --name to CLI, usage, and website docs
* docs: mention --discover in named mesh tutorial section
* refactor: move join retry into Node::join_with_retry, skip retry on decode errors; add MeshFilter.name tests
* fix: bare --discover behaves like --auto; --discover gets lifecycle recovery, watchdog, and retry
Addresses review feedback from @ndizazzo:
- Bare --discover (no name) parsed as Some("") which filtered out all meshes
in smart_auto. Now normalised to None so it falls through to --auto behavior.
- --discover now gates rediscovery, originator re-discovery, watchdog takeover,
and auto model assignment — previously only --auto triggered these lifecycle
paths.
- MCP discovery join candidates now use join_with_retry for consistent retry
behavior across all join paths.
|
||
|
Ivan Golovach
|
e2a0b5770a
|
future(telemetry-plugin): add opt-in survey OTLP metrics exporter | ||
|
Michael Neale
|
2f84b16073
|
ci: dual-lane CUDA releases (cuda on 12.6.3, cuda-blackwell on 12.8) (#355)
* ci: dual-lane CUDA releases (cuda on 12.6.3, cuda-blackwell on 12.8) Published v0.60.3 CUDA bundle crashes at the first matmul on A30/A100 (SM 8.0) with `CUDA error: device kernel image is invalid` when run against R535-series drivers. Root cause confirmed by A/B build on 29 GPUs (14x A30, 15x A100, driver 535.288.01): nvcc 12.8 emits cubins whose minor-version metadata the R535 driver (CUDA 12.2 native) rejects at kernel load, even though sm_80 cubins are physically present. The same source tree built under CUDA 12.6.3 produces a working bundle on identical hardware. A toolkit downgrade alone would drop Blackwell (sm_100/103/120, which CUDA 12.6 cannot emit). Instead, split the CUDA release into two lanes: primary -cuda CUDA 12.6.3 sm_75..sm_90 R535+ driver Blackwell lane -cuda-blackwell CUDA 12.8 adds sm_100/103/120 R550+ driver Both lanes ship inner binaries named -cuda (the mesh-llm runtime BinaryFlavor enum has one cuda variant); only the outer archive name distinguishes them. Users pick the archive matching their driver; both bundles invoke as `--llama-flavor cuda`. Changes: - .github/workflows/release.yml: keep build_linux_cuda on 12.6.3; add a sibling build_linux_cuda_blackwell job on 12.8, wired into the publish job's needs list. New Actions variable vars.CUDA_BLACKWELL_VERSION (fallback 12.8.0) mirrors vars.CUDA_VERSION (now fallback 12.6.3). - .github/workflows/ci.yml, .github/workflows/llama-cache-keys.yml: pin fallback to 12.6.3 for the primary lane; add a second cache key cuda_blackwell_fat_cache_key so the two lanes do not collide. - .github/workflows/docker.yml: narrow CUDA_ARCH build-arg to sm_75-90 to match the primary-lane Dockerfile base. - Justfile: release-build-cuda default arch narrowed to sm_75-90; add release-build-cuda-blackwell, release-bundle-cuda-blackwell (plus Windows mirrors); docker-build-cuda gains a cuda_version parameter. - docker/Dockerfile.cuda: parameterize the three nvidia/cuda FROM lines via ARG CUDA_VERSION (default 12.6.3) so Blackwell docker builds can override; ARG CUDA_ARCH default narrowed to sm_75-90. docker image itself tracks primary lane only for now; Blackwell docker tag is a planned follow-up. - scripts/package-release.sh: add cuda-blackwell to linux/x86_64 supported_release_flavors; introduce binary_flavor_for_release_flavor so cuda-blackwell bundles still name inner binaries llama-server-cuda etc. - install.sh: expose cuda-blackwell as an installable flavor and map to the new archive name. - mesh-llm/tests/fixtures/release-target-matrix.json: fixture rows for cuda-blackwell on linux/x86_64 (supported), windows/x86_64 (supported), linux/aarch64 + macos/aarch64 (unknown) so the xtask release-target consistency check passes. - RELEASE.md, README.md, docker/SPEC_COMPLIANCE.md: document both lanes and the driver-compatibility matrix. Add docs/cuda-release-lanes.md with the end-to-end rationale. Verified locally via manual parity tests against install.sh and scripts/package-release.sh (supported_flavors, asset_name / versioned_asset_name, release_target_support, bundle_bin_name) for cpu/cuda/cuda-blackwell on linux/x86_64 and linux/aarch64. A full xtask repo-consistency run should be re-executed in CI. Fixes issue #304. * docker: hoist CUDA_VERSION ARG above first FROM Without this, the second and third nvidia/cuda FROM lines resolve ${CUDA_VERSION} to empty — Docker drops pre-FROM ARG scope once the first FROM is parsed (the ui-builder FROM at line 23). BuildKit fails with: invalid reference format nvidia/cuda:-runtime-ubuntu22.04 Caught by a docker build against docker/Dockerfile.cuda on a CUDA 12.6.3 host. Hoisting the ARG above the ui-builder FROM makes it a true global in scope for every FROM directive. The intermediate re-declarations are removed — they zero-valued the default and actively broke resolution. End-to-end verification (ops-grid02, nvidia/cuda:12.6.3-devel): llama.cpp compile: 435/435 kernels OK cubin audit: 177 sm_75, 177 sm_80, 177 sm_86, 177 sm_87, 177 sm_89, 177 sm_90 no sm_100/103/120 (primary lane is R535-compatible as designed) GGML_CUDA_FA_ALL_QUANTS=ON verified in CMakeCache * ci: drop sm_103 from blackwell arch list — nvcc 12.8.0 does not support it nvcc 12.8.0 supports sm_100, sm_101, sm_120 but NOT sm_103 — that compute capability was introduced in a later CUDA release. Caught by a docker build against docker/Dockerfile.cuda with CUDA_VERSION=12.8.0 on a CUDA-12.8 host: nvcc fatal : Unsupported gpu architecture 'compute_103' Updated Blackwell arch list from 75;80;86;87;89;90;100;103;120 to 75;80;86;87;89;90;100;120. Covers all currently-shipping Blackwell hardware nvcc 12.8.0 can emit for (B100/B200 via sm_100, RTX 50-series via sm_120). Files touched: Justfile, .github/workflows/release.yml, .github/workflows/llama-cache-keys.yml, docs/cuda-release-lanes.md, README.md. * Add BinaryFlavor::CudaBlackwell so release-target tests recognize the new lane The dual-lane CUDA release introduces a cuda-blackwell tarball, but the BinaryFlavor enum and supported-combos list did not know about it. With main's new release_target fixture tests, that surfaced as three panics for 'unknown fixture flavor: cuda-blackwell'. Teach the enum, the preferred-devices table, the supported-combos list, the asset suffix, and the fixture-test flavor parser about the new variant. * ci: address Copilot review feedback on dual-lane CUDA Four fixes from Copilot's PR review: 1. release.yml build_linux_cuda_blackwell: add skip_gpu_bundles guard, needs: prepare_release, and ref: release_sha on checkout so the job matches the other GPU bundle jobs. Use RELEASE_TAG from prepare_release.outputs.tag instead of GITHUB_REF_NAME so the bundle version matches the release commit/tag. 2. scripts/package-release.ps1: separate release flavor (outer archive name) from binary flavor (inner executable suffix). cuda-blackwell archives now contain -cuda binaries on Windows, matching the bash packager so the runtime's BinaryFlavor::Cuda lookup still finds rpc-server-cuda.exe / llama-server-cuda.exe inside a mesh-llm-*-cuda-blackwell.zip. 3. docker/SPEC_COMPLIANCE.md Q3: correct the misleading claim that compute_120 is in release-build-cuda. Describe the lane split so readers know compute_120 only ships in release-build-cuda-blackwell. * Address CUDA lane review comments --------- Co-authored-by: Ventz Petkov <ventz@vpetkov.net> Co-authored-by: James Dumay <jameswdumay@gmail.com> |
||
|
James Dumay
|
5454baac0b
|
Replace llama-server with skippy runtime (#422) | ||
|
Michael Neale
|
48079fde10
|
feat: add built-in pi launcher (mesh-llm pi) (#389)
* Add mesh-llm pi command that writes a mesh provider into ~/.pi/agent/models.json and launches pi. Uses the auto model ID so mesh-llm picks the strongest available model automatically. |
||
|
James Dumay
|
330a3381ae
|
Prepare repo layout for skippy merges (#420)
* Prepare workspace for skippy integration * Copy skippy model crates * Use skippy model crates in resolver * Add mesh compatibility tests for model crates |
||
|
Nick DiZazzo
|
6e045b5dec
|
feature(console-output): completely revise mesh-llm console output for JSONL output + pretty printing (#388) | ||
|
Michael Neale
|
41e296bac0
|
openai-endpoint plugin: vLLM, TGI, Ollama passthrough (#395)
* openai-endpoint plugin: route inference to vLLM, TGI, Ollama, etc. Built-in plugin that registers any OpenAI-compatible server as an inference endpoint. Same pattern as the existing lemonade plugin. Enable in config.toml, set the URL via env var: [[plugin]] name = "openai-endpoint" MESH_LLM_OPENAI_ENDPOINT_URL=http://gpu-box:8000/v1 mesh-llm serve The plugin health-checks via GET /v1/models and discovers model names automatically. Default URL: http://localhost:8000/v1. No struct changes, no new dependencies, no cli changes. 53 lines of plugin code, rest is config wiring and docs. * add external backend hint to --help output * add tests for openai-endpoint plugin config resolution - openai_endpoint_can_be_enabled_explicitly: verifies enable adds spec with correct args (--plugin openai-endpoint) - openai_endpoint_rejects_custom_command: verifies built-in guard * remove lemonade plugin, add url config support - Delete lemonade plugin (documented as openai-endpoint with url = "http://localhost:8000/api/v1" in README instead) - Add `url` field to plugin config for setting backend URL: [[plugin]] name = "openai-endpoint" url = "http://gpu-box:8000/v1" - Config url takes precedence over MESH_LLM_OPENAI_ENDPOINT_URL env var - Update --help to show config file url example - mesh-llm client with plugin enabled = no llama.cpp loaded - Net -64 lines (lemonade removal > additions) * fix review nits: remove stale lemonade README section, fix endpoint_id in proxy test * fix site feature card to reference config url not env var * pass url via ExternalPluginSpec instead of process env var - URL carried on ExternalPluginSpec, set as child env var in spawner (no process-global set_var, safe for parallel tests) - Test asserts spec.url matches configured value - blackboard/blobstore reject url field in config (not just command/args) * update docs/USAGE.md lemonade section to use openai-endpoint plugin, fix README to show config url |
||
|
James Dumay
|
e2bd6fe26b | Move llama.cpp customizations to patch queue | ||
|
Nick DiZazzo
|
62df7a919c
|
Add persistent OpenCode config writing with --write flag (#373)
|
||
|
Nick DiZazzo
|
87ad3b8df4
|
Add support to explicitly publish mesh via command-line (#371) | ||
|
Nick DiZazzo
|
34625cd663
|
Add --headless flag to disable the web UI while keeping the management API alive (#349)
* ci: accept COPY . . as valid workspace manifest copy in docker precheck |
||
|
Nick DiZazzo
|
10f49fb658
|
feature: revise node states to be simpler (#298) | ||
|
James Dumay
|
a3cf86ad31 | Skip GPU bundles for prerelease releases | ||
|
James Dumay
|
71aafe10b9 | Update prerelease release workflow to skip CUDA/ROCm builds | ||
|
IvGolovach
|
e21dc45857 |
models: add managed cache cleanup
Validation: - cd mesh-llm/ui && npm ci - cd mesh-llm/ui && npm run build - CARGO_HOME=/tmp/mesh-llm-cargo-home CARGO_TARGET_DIR=/tmp/mesh-llm-model-cleanup-target cargo check -p mesh-llm -> success - CARGO_HOME=/tmp/mesh-llm-cargo-home CARGO_TARGET_DIR=/tmp/mesh-llm-model-cleanup-target cargo test -p mesh-llm models::usage -- --nocapture -> 3 passed; 0 failed - CARGO_HOME=/tmp/mesh-llm-cargo-home CARGO_TARGET_DIR=/tmp/mesh-llm-model-cleanup-target cargo test -p mesh-llm cleanup_age_parser -- --nocapture -> 2 passed; 0 failed - CARGO_HOME=/tmp/mesh-llm-cargo-home CARGO_TARGET_DIR=/tmp/mesh-llm-model-cleanup-target cargo test -p mesh-llm models::catalog::tests -- --nocapture -> 21 passed; 0 failed - CARGO_HOME=/tmp/mesh-llm-cargo-home CARGO_TARGET_DIR=/tmp/mesh-llm-model-cleanup-target cargo test -p mesh-llm models::resolve::tests -- --nocapture -> 37 passed; 0 failed - CARGO_HOME=/tmp/mesh-llm-cargo-home CARGO_TARGET_DIR=/tmp/mesh-llm-model-cleanup-target cargo test -p mesh-llm -- --skip network::nostr::integration_tests::publish_discover_round_trip -> unrelated existing api/auth failures outside this diff; targeted model/cache tests passed - cargo fmt --all -- mesh-llm/src/cli/commands/models/formatters.rs mesh-llm/src/cli/commands/models/formatters_console.rs mesh-llm/src/cli/commands/models/formatters_json.rs mesh-llm/src/cli/commands/models/mod.rs mesh-llm/src/cli/models.rs mesh-llm/src/models/catalog.rs mesh-llm/src/models/mod.rs mesh-llm/src/models/resolve/mod.rs mesh-llm/src/models/usage.rs - git diff --check -> clean Rollback: - git revert HEAD Ledger-Id: models-managed-cache |
||
|
Michael Neale
|
b33123e02f
|
Virtual LLM engine — callback hooks from llama-server into mesh (#225)
* docs: virtual LLM engine design — in-engine mesh hooks
Design for adding hooks inside llama-server's C++ token generation loop
so it can consult other models in the mesh during inference.
Hook points: pre-inference, per-token, pre-response, slot pause/resume.
Communication via HTTP callback to mesh-llm which routes to any model
in the mesh. Token injection into live KV cache without restarting.
Relates to #183, #165
* docs: uncertainty signals and callback design for virtual LLM hooks
Four signal types: per-token entropy, top-token margin, sequence variance
(confidence trajectory), and self-consistency via multi-completion.
Hooks are simple: measure signals in C++, call back to mesh-llm with the
data, mesh-llm decides what to do and responds with an action (inject,
continue, stop, none). All decision logic lives in the Rust side.
Single callback endpoint: POST /mesh/hook with hook type, signal data,
and generation context. Three callback modes: async fire, sync call,
poll check.
* docs: clean callback protocol and signal design for virtual LLM
Rewrite to separate concerns clearly:
- C++ side: compute signals (entropy, margin, window stats), call mesh-llm
- Rust side: all decision logic, model routing, consultation
JSON over localhost HTTP chosen over FFI/shared-memory because:
- cpp-httplib already linked, axum already running
- callback latency (~0.1ms) irrelevant vs consultation time (seconds)
- debuggable with curl, independently versionable
Five hooks, one callback shape, three actions (none/inject/stop).
Async consultation via fire-at-hook-1, poll-at-hook-3 pattern.
Per-request threshold config alongside existing task_params.
* docs: simplify to four hooks, drop per-token callback
Per-token callback was wrong — 30-100 tok/s makes it wasteful.
Signal computation stays per-token in C++ (cheap arithmetic),
but callbacks only fire at: pre-inference, post-prefill,
pre-response, and complete (telemetry).
Each hook now shows exactly what data it sends to mesh-llm:
- pre_inference: original messages array, images, model capabilities
- post_prefill: first-token entropy/margin, top-5 candidates
- pre_response: full generated text, signal summary stats
- complete: telemetry (fire-and-forget)
Mid-generation intervention deferred as optional threshold breakout.
* docs: rewrite VIRTUAL_LLM.md — hooks, callbacks, examples only
Cut all proxy discussion, architecture justification, and research notes.
Doc now covers: callback protocol, four hooks with exact JSON payloads
and worked examples, signal computation, token injection mechanics,
and the specific C++ and Rust changes needed.
* docs: add async hooks — pending action + poll for non-blocking consultation
Hook 1 can return 'pending' with an async_id instead of blocking.
llama-server stores the id and polls GET /mesh/hook/poll/{id} every
16 tokens during generation. When the result is ready (200), tokens
are injected into the live KV cache. When not ready (202), generation
continues uninterrupted.
Each hook now annotated with blocking behavior:
- pre_inference: sync or async (mesh-llm decides)
- post_prefill: always sync (just reads numbers, <1ms)
- pre_response: always sync (need verdict before sending)
- complete: fire-and-forget
* docs: drop complete hook, three hooks is enough
Telemetry can be added later if needed. The core mechanism is
pre_inference, post_prefill, pre_response + async polling.
* docs: clarify hook triggers — always fire, mesh-llm decides
* docs: Hook 1 configures hooks 2 and 3 via entropy_threshold and verify
Hook 1 always fires — it's the setup hook. Its response tells C++
what to watch for: entropy_threshold enables Hook 2, verify enables
Hook 3. Most requests: only Hook 1 fires, mesh-llm returns none.
* docs: trigger-based hooks — only fire when structural criteria met
All hooks are now conditional:
- Hook 1: images+text-only, context pressure (>75% of ctx),
long session (>10 turns), large user paste
- Hook 2: first-token entropy > threshold (set by Hook 1)
- Hook 3: max_tokens cutoff, very short response, high uncertainty,
tail entropy spike, mid-sentence cutoff
- Polling: only when async work pending
Added examples: context pressure → conversation summarization,
max_tokens cutoff → summarize + continue, very short response →
retry with encouragement, tail entropy spike → verify ending.
* docs: complete rewrite — triggers, examples, implementation details
Three sections:
1. Triggers and hooks — what fires when, one table per hook
2. Examples — concrete JSON for each scenario (captioning, summarization,
context pressure, refusal, truncation, hallucination)
3. Implementation — exact C++ code at each hook point in server-context.cpp,
signal window struct, inject helper, mesh-llm endpoints and module
* docs: full callback payloads under each hook
Each hook now shows the exact JSON mesh-llm receives:
- Hook 1: full messages array (images, audio, all turns), token counts,
context size, trigger name, model capabilities
- Hook 2: first-token signals only (entropy, margin, top-5), request_id
links back to Hook 1 data
- Hook 3: full generated text, stop reason, signal summary, request_id
links back to original request
* docs: all hooks send full messages — can't assume Hook 1 fired
Hook 2 and 3 now include the messages array. Hook 1 only fires
on specific triggers, so mesh-llm may never have seen the request.
Each hook is self-contained — sends everything mesh-llm needs.
* api: add /mesh/hook endpoint for llama-server callbacks
New route handler for mesh hook callbacks. Three hooks handled:
- pre_inference: logs trigger, returns none + entropy_threshold
- post_prefill: logs entropy signal, returns none
- pre_response: logs n_decoded + stop_reason, returns none
Poll endpoint GET /mesh/hook/poll/{id} returns 202 (stub).
All hooks return 'none' for now — plumbing first, decision logic next.
* launch: pass --mesh-port to llama-server for hook callbacks
Sets MESH_API_PORT env var at runtime startup, launch.rs reads it
and passes --mesh-port {port} to llama-server. This tells llama-server
where to POST hook callbacks on localhost.
* llama-patches: snapshot C++ mesh hook changes for co-iteration
Contains the git format-patch from llama.cpp mesh-hooks branch plus
the standalone header file. This lets us iterate on C++ and Rust
together on micn/virtual-llm. When stable, apply the patch to the
llama.cpp fork's mesh-hooks branch and remove this directory.
* llama-patches: update with trigger fixes and e2e test results
* llama-patches: update with working token injection
* TEMPORARY: inline C++ mesh hook files for single-repo iteration
llama-patches/ contains the 7 modified/new C++ files that implement
mesh hooks in llama-server. sync.sh copies them into llama.cpp/ and
build-mac.sh runs it automatically after pulling upstream-latest.
This is a temporary setup for the micn/virtual-llm branch so C++ and
Rust changes live in one repo / one PR. When stable, the C++ moves to
the mesh-hooks branch on the llama.cpp fork and this directory is deleted.
* inference: add virtual_llm decision module with documented stubs
New module inference/virtual_llm.rs — the brain behind mesh hooks.
Three handlers, one per hook type, each with:
- Documented trigger table
- Match on trigger name
- Logging with context (model, token counts, entropy, etc.)
- TODO comments describing the planned consultation
Also: HookAction enum, AsyncConsultations struct stub.
Route handler in mesh_hook.rs now delegates to virtual_llm instead
of inlining the decision logic.
* virtual_llm: add HookContext for model-aware decisions
HookContext enriches the model filename from the C++ hook payload with:
- Tier, strengths, tools from ModelProfile
- Vision/multimodal from ModelCapabilities
- Available peers by capability (vision, stronger, all)
This lets the decision engine pick complementary models: vision when
text-only, stronger when uncertain, different specialty when needed.
Struct is defined but not yet wired — handlers still take raw payload.
* simplify: remove polling, all hooks are synchronous
Delete the async poll mechanism (pending_async_ids, poll_async,
should_poll, GET /mesh/hook/poll/{id}). All hooks are now plain
synchronous POST calls.
Background work uses a simpler pattern: Hook 1 spawns a tokio task
and stores it in a DashMap keyed by request_id. Hook 3 checks the
map — if the result is ready, use it; if not, let the response go.
No polling needed.
C++ side: removed poll loop from generation, cleaned up mesh_hook_ctx.
Rust side: removed poll endpoint, removed Pending variant, updated docs.
* add mesh_request_id for request correlation, rewrite design doc
mesh-llm generates a mesh_request_id and includes it in the request
body. llama-server passes it back in every hook payload. mesh-llm
stores the original request (messages, images, etc.) keyed by this
ID so hooks can access the full conversation without C++ sending it.
Design doc rewritten to match current implementation:
- Removed polling/pending (all hooks sync now)
- Documented background work pattern (Hook 1 spawns, Hook 3 collects)
- Documented request correlation via mesh_request_id
- Updated all example payloads and scenario descriptions
- Updated implementation tables to match actual files
* implement Hook 2 KV cache injection
After prefill, if the model is uncertain (high entropy / low margin),
mesh-llm can return inject text. This text is now tokenized and
decoded into the KV cache via a temporary batch — the model 'sees'
the injected context before generating its first token.
The injection uses the same chunked decode pattern as normal prefill:
tokens are added to a temp batch, decoded in n_batch-sized chunks,
and positions are tracked via slot.prompt.tokens. Only the last
inject token requests logits, and slot.i_batch is set so sampling
reads from the correct position.
This is the key mechanism that makes the virtual LLM interesting —
a small model that's uncertain can receive context from a stronger
model mid-inference, changing the trajectory of generation.
* update docs for Hook 2 KV injection
- Design doc: Hook 2 description updated from 'informational' to
documenting inject action, how KV cache injection works, and new
scenario showing uncertain model receiving live help
- virtual_llm.rs: lifecycle diagram updated (Hook 2 now returns
inject, shows KV decode step), function doc describes injection
mechanism, removed stale 'pending' action from Hook 1
* implement real consultation logic: image captioning, summarization, second opinion, verification
New module: inference/consult.rs — peer consultation over QUIC mesh.
Provides four patterns:
- caption_image: send image to vision peer, get text description
- summarize_conversation: condense early turns to free context
- second_opinion: ask a different model the same question
- verify_response: check if a response's ending is accurate
Peer discovery finds peers by capability (vision) or by difference
(different model architecture). Prefers low-RTT peers. The value of
Hook 2 and Hook 3 verification is diversity — a different model's
perspective, not necessarily a stronger one.
Hook handlers are now async, accepting the mesh Node to find and
consult peers. All four scenarios wired:
- Hook 1 images_no_multimodal → caption via vision peer → inject
- Hook 1 context_pressure → summarize via any peer → inject
- Hook 2 high_entropy → second opinion from different model → inject into KV
- Hook 3 tail_entropy_spike/verify → verify via different model → append correction
Added MeshApi::node() helper for route handlers.
* prefer similar-tier peers, send only last message, keep injection concise
find_different_model_peer now scores by tier distance (±1 preferred)
then RTT. A 4B model asks another small model, not a 70B.
second_opinion sends only the last user message with 'answer briefly
in 2-3 sentences' — max 192 tokens. The peer returns a concise
answer, not a full essay. Keeps the blocking time and KV injection
short.
Injection text capped at 512 chars. Every injected token is decoded
into KV cache and adds latency — brevity matters.
* only annotate truncation when response looks cut off mid-sentence
max_tokens fires on every request that hits the token limit, including
intentional truncation (user set max_tokens: 100). Now we check if
the response actually ends mid-thought: no sentence-ending punctuation
means genuinely cut off, annotate it. Clean ending (period, question
mark, etc.) means the model finished naturally, skip the note.
* remove max_tokens from Hook 3 triggers
max_tokens is normal operation — the user or server set a limit and
the model hit it. finish_reason: 'length' already signals this.
Firing a hook round-trip to append a redundant note adds latency
for no value.
Hook 3 now only fires on signals that indicate the model struggled:
very_short, high_uncertainty, tail_entropy_spike, verify.
Removed from C++ trigger evaluation, Rust handler, and design doc.
* fix: add trailing newline to lib.rs for CI rustfmt
* fix: use console port (3131) for mesh hooks, not proxy port (9337)
MESH_API_PORT was set to cli.port (proxy, 9337) instead of
cli.console (management API, 3131). Hooks calling the proxy
would loop back to llama-server.
* Hook 2 always armed — default entropy_threshold 5.0
Hook 2 (post-prefill uncertainty check) no longer depends on Hook 1
firing first. It stands on its own: if the model is uncertain after
reading the prompt, ask a peer. Hook 1 can still override the
threshold for specific request types.
* Hook 1: only media triggers — drop context_pressure, long_session, large_user_message
Hook 1 is for when the model can't handle the input modality:
images without vision, audio without audio support. These are
clear problems with clear solutions (caption, transcribe).
context_pressure/long_session/large_user_message were speculative
triggers that didn't have good actions — appending a summary to
an already-long prompt makes it longer, not shorter.
The interesting hooks during inference are Hook 2 (uncertainty)
and Hook 3 (bad output signals). Those stand on their own now.
Also removes unused find_any_peer and summarize_conversation.
* mid-generation Hook 2b, Hook 3 replace, --mesh-hook-debug
Hook 2b: fires during generation when sustained entropy spike
detected (75% of rolling window has entropy > 4.0). Same KV cache
injection as Hook 2 — model continues generating but now informed
by peer context. 32-token cooldown between fires.
Hook 3: 'replace' action replaces generated_text entirely instead
of appending. When verification says the response is bad, the
peer's corrected answer replaces the model's output.
--mesh-hook-debug / MESH_HOOK_DEBUG: lowers all thresholds so hooks
fire on almost any request. entropy_threshold 0.5, mid-gen spike
ratio 0.25, cooldown 8 tokens. For testing.
* pass messages in all hook payloads, handle /mesh/hook in proxy
Messages stored in mesh_messages on task params, included in all
hook payloads (Hook 1, 2, 2b, 3). Without this, Hook 2 had no
messages to send to a peer for second opinion.
/mesh/hook handled directly in the proxy dispatcher since the
management API and proxy share port 3131 after main consolidated
ports. Proxy intercepts the route before forwarding to llama-server.
Tested: hooks fire, peer consultation initiates over QUIC mesh.
Hook 2 finds Qwen3-8B on the public mesh when local model is gemma.
Consultation latency is high (~60s) — needs investigation.
* fan-out consultation: race 2 peers, 10s timeout
Hook 2 now finds the top 2 different-model peers and races them
via JoinSet. First successful response wins, loser gets aborted.
If only 1 peer exists, falls back to single call.
10s timeout on all consultation calls — hooks block the local
model's slot, can't wait forever for a remote peer.
Verify (Hook 3) trims generated_text to last 500 chars and caps
at 256 tokens — the tail is where the spike happened.
Tested on public mesh: raced Hermes-7B vs Qwen3-8B, Qwen won
in 7s, total request time 12s (was 65s before timeout).
* fix port split: proxy on 9337, management API on 3131
Reverted api_port back to cli.port (9337) so the proxy binds there.
MESH_API_PORT stays as cli.console (3131) so llama-server hooks
hit the management API which has the /mesh/hook route.
Removed the /mesh/hook intercept from the proxy — it was a
workaround for both services fighting over port 3131.
* clean up virtual_llm: separate handlers per hook, shared helper
- handle_post_prefill (Hook 2) and handle_mid_generation (Hook 2b)
are now separate functions, both using race_second_opinion helper
- Removed dead DashMap TODO — everything is synchronous
- Removed stale doc comments about background work pattern
- 152 insertions, 218 deletions — net -66 lines
* typed handlers: handle_image, handle_uncertain, handle_drift, handle_verify
Route handler parses payload once and passes typed args to each
function — no more stringly-typed Value digging in the handlers.
Dropped very_short trigger from C++ and Rust — a short response
to a long prompt isn't necessarily wrong.
handle_uncertain: model stuck at start (high entropy first token)
handle_drift: model losing coherence mid-generation (sustained spike)
handle_verify: check output before sending (tail spike / high uncertainty)
get_peer_hint: shared helper, races 2 peers for a second opinion
* doc: crisp handler docs with args, return values, and behavior
* docs: move racing detail to get_peer_hint, simplify caller docs
* typed handle_image args, find_audio_peer, video placeholder
handle_image now takes (image_url, user_text) instead of raw payload.
extract_image moved to pub for route handler to call.
Added find_audio_peer (same pattern as find_vision_peer).
transcribe_audio finds a capable peer but audio extraction not yet wired.
video_no_support placeholder.
* lower entropy threshold from 5.0 to 3.0
5.0 was unreachable — even uncertain models rarely hit it because
it requires ~32 equally-likely tokens. 3.0 (~8 equally-likely tokens)
fires on genuine uncertainty.
Tested with Qwen3-0.6B: hooks fire on hard questions (Swahili
translation, obscure facts). Hook 3 caught high_uncertainty
(mean_entropy=3.46), Hook 2b caught mid-gen drift (tail_entropy=6.28).
Gemma-4B still doesn't trigger — it's well-calibrated.
* peer selection: exclude smaller models, prefer larger; fix context leak; eval script
Peer selection now filters out models with a lower tier than the
current model. No point asking a weaker model for help. Larger
models are slightly preferred (tier 4 > tier 3 > same tier).
Changed injection format from '[Context: ...]' to a natural
instruction ('Here is relevant information...') so models don't
echo the wrapper text in their output.
Added evals/virtual_llm_eval.py — runs a set of questions with
and without hooks, captures timing and responses to JSONL.
* deduplicate peers by model name, eval results
Two nodes running the same model don't provide diversity — they
give the same answer. Keep only the best-scored node per model name.
First eval results with Qwen3-0.6B:
- Easy questions: no hooks fire, no overhead (0.2s same)
- Swahili translation: hooks fire but model too weak to use hint
- Nauru population: improved from 200k to 24k (real ~12k)
- 2 questions got 10s slower (MiniMax timeout on verification)
* drop Hook 3 (pre_response verify)
Hook 3 mostly timed out — by the time generation is done, a 10s
sync verification adds latency for marginal value. The useful work
happens in Hook 2 (uncertain start) and Hook 2b (mid-gen drift),
both of which inject context before/during generation.
Removed from C++: entire pre_response block in send_final_response,
verify flag from mesh_hook_ctx.
Removed from Rust: handle_verify, verify_response, pre_response route.
Removed find_different_model_peer (single-peer wrapper, unused).
-197 lines across 5 files.
* injection framing test + switch to 'reference' framing
Integration test (virtual_llm_injection.rs) that spins up a mock
hook server + llama-server, sends factual/translation/reasoning
questions, and verifies the model incorporates injected hints.
Tested 4 framings against Qwen3-0.6B baseline (1/5 correct):
- 'reference': 5/5, cleanest output, no artifacts
- 'assistant_draft': 5/5, clean
- 'current': 5/5, leaks </think> into content
- 'rag': 5/5, sometimes echoes question
Switched production framing to 'Reference answer: ...' — produces
the cleanest model output with no artifacts.
Also: debug mode now always fires Hook 2 (post_prefill), regardless
of entropy. The first token is always confident for thinking models
(<think>), so entropy gating in debug mode was preventing testing.
* TODO: virtual LLM remaining work items
Track slow peer consultation investigation (MiniMax timeout issue),
peer responsiveness tracking, audio extraction, and non-thinking
model testing.
* fix TODO: MiniMax was slow, not failing
* TODO: use TTFT perf tracker for consultation peer selection (PR #271)
* prevent recursive consultation loops
Outgoing consultation requests now include mesh_hooks: false,
which tells the peer's llama-server to skip hook callbacks for
that request. Without this, peer A could consult peer B, which
could consult peer C, etc.
Flagged by GPT-5.4 code review as a critical production risk.
* per-hook timeouts and compact mid-gen injection
Hook 2 (pre-generation): 15s timeout — user is waiting for first
token anyway, longer timeout lets slower but better peers respond.
Hook 2b (mid-generation): 5s timeout — user sees a stall if we
block too long. Also uses compact 'Key fact: ...' framing (256
char cap) instead of full 'Reference answer' (512 chars) to avoid
derailing the model's continuation style.
Hook 1 (media captioning): 15s timeout — captioning is a one-time
pre-generation cost.
* drop compact mid-gen framing — tested, worse than reference
Tested 'Key fact: X' (compact) vs 'Reference answer: X. Use the
reference above...' (reference). Compact got 4/5 — Swahili
translation failed because the model echoed 'Key fact:' back
instead of using the hint. The instruction to act on it matters.
Use 'reference' framing for both Hook 2 and 2b. Simpler, tested.
* two new mid-gen triggers: repetition loop + surprise break
GPT-5.4 research: entropy alone doesn't catch 'confidently wrong' —
Gemma-4B first-token entropy is 0.03-1.33 even on hallucinated
answers. Need different signals.
New trigger 1: REPETITION LOOP
Track last 32 token IDs, compute 3-gram repeat ratio. Fire when
ratio >= 0.18. Catches degenerate loops (Gemma stuck in 'I need
to access my knowledge base' patterns). Tested: fired correctly
on Nauru query at token 43, and on p-values explanation at tokens
101 and 153.
New trigger 2: SURPRISE BREAK
EWMA of -log(p_chosen) with z-score spike detection. Fires when
2+ tokens spike >2.5 sigma after a calm run of 4+ low-z tokens.
Catches 'was flowing confidently, then suddenly broke'.
Both signals added to mesh_signal_window struct and wired into
should_fire_midgen(). Any of the three triggers (entropy spike,
repetition, surprise) can independently fire Hook 2b, all sharing
the same cooldown.
Tested on Gemma-4B with real mesh peers:
- Repetition: 3 fires across 12 queries (all genuine loops)
- Entropy spike: 0 fires (Gemma too confident, as expected)
- Hook 2 (first-token): 2 fires on creative prompts (haiku)
where Gemma genuinely didn't know how to start
- Zero false positives on factual/math queries
* 20s consultation timeout across all hooks
Triggers are rare — when they fire, it's worth waiting. The 5s
mid-gen timeout was causing all repetition-loop consultations to
fail (peers need 6-10s). Unified to a single 20s constant.
* add rank instability tracking, disable as trigger
Implemented top-8 Jaccard similarity tracking between consecutive
generation steps. Computes overlap of top-8 candidate tokens —
low Jaccard means the model is considering completely different
continuations each step.
DISABLED as a trigger: Gemma-4B has inherently unstable top-8
distributions — fires on ~100% of queries even with strict
thresholds (5/6 window, 24-token warmup). The signal needs
per-model baseline calibration or combination with another signal.
Tracking code is kept (cheap, data in signals JSON) for research.
Repetition loop remains the most reliable mid-gen trigger.
* remove rank instability signal
100% false positive rate on Gemma-4B — top-8 candidates are
inherently unstable on modern models. Not a useful signal without
per-model calibration. -94 lines.
* Hook 1 working: image captioning on text-only models
When a text-only model (no mmproj) receives an image request:
1. server-common.cpp strips image to '[image attached]' text
instead of rejecting with 500 error (when mesh hooks enabled)
2. Original image URL preserved as mesh_image_url in messages
3. server-context.cpp detects mesh_image_url, fires Hook 1
4. Rust handler extracts image URL, consults vision peer
5. Caption injected into prompt before generation
Tested end-to-end with mock hook server + Qwen3-0.6B:
- Image request accepted (was: 500 error)
- Hook 1 fires with images_no_multimodal trigger
- Caption injected (13 tokens)
- Model output matches caption content exactly
Also adds server-common.cpp to llama-patches/ (was only
server-common.h before) for the image stripping logic.
* update docs: VIRTUAL_LLM.md rewrite + TODO refresh
VIRTUAL_LLM.md:
- Removed Hook 3 (dropped earlier), background work pattern,
DashMap, mesh_request_id correlation (simplified away)
- Added Hook 2b with 3 triggers (repetition, entropy, surprise)
- Added signal detection research section with results
- Added injection framing test results
- Updated C++ file list (added server-common.cpp)
- Updated consultation section (fan-out, recursion guard, 20s timeout)
- Corrected entropy threshold (3.0 not 5.0)
TODO.md:
- Removed non-thinking model testing item (done — Gemma-4B tested)
- Added: test Hook 1 with real vision peer
- Added: push C++ to fork when stable
- Added: image caption caching
- Updated description to reflect current state
* docs: highlight inter-model collaboration across README, docs site, roadmap
README: add paragraph in 'How it works' section describing the feature,
link to VIRTUAL_LLM.md in 'More docs'.
docs/index.html: add feature card with brain emoji, add tag pill,
update research section to note collaboration is live.
ROADMAP.md: add 'Inter-model collaboration (Virtual LLM)' section
with working features, key insight, paper reference, and next steps.
* move inter-model collaboration from roadmap to README feature list
This is a shipped feature, not a roadmap item. Remove the ROADMAP
section and add a bullet to the top-level feature list instead.
* move llama.cpp to Mesh-LLM org fork, pin by SHA
Fork: github.com/Mesh-LLM/llama.cpp (master branch)
Pinned SHA in LLAMA_CPP_SHA (single source of truth)
Changes:
- All build scripts, CI workflows, Dockerfiles now use Mesh-LLM/llama.cpp
- CI reads SHA from LLAMA_CPP_SHA file, falls back to ls-remote
- Delete mesh-llm/llama-patches/ (-15k lines) — fork is the source of truth
- Add mesh-llm/docs/LLAMA_CPP_FORK.md with maintenance instructions:
how to update from upstream, how to tell an agent to sync, files we touch
- Justfile diff recipe updated for new fork layout
The fork carries 8 commits on master:
3x RPC (zero-transfer, alloc cache, B2B transfers)
4x MoE (expert mask, split tool, ranking, shared-expert fix)
1x mesh hooks (virtual LLM engine)
* AGENTS.md: document llama.cpp fork, warn agents not to update it unprompted
* build-mac: pin llama.cpp to SHA from LLAMA_CPP_SHA
Same behaviour as build-linux.sh: reads the pinned SHA, fetches it,
checks out detached. If LLAMA_CPP_SHA is missing, falls back to
pulling master HEAD.
* test: find CI model at ~/.models/ so hook test runs in CI
Was silently skipping because it only checked HuggingFace cache.
CI already downloads SmolLM2-135M to ~/.models/.
* Revert "test: find CI model at ~/.models/ so hook test runs in CI"
This reverts commit
|
||
|
Nick DiZazzo
|
5497f01460
|
feature(opencode): add OpenCode support to mesh-llm as a provider (#281) | ||
|
Nick DiZazzo
|
02bc816097
|
chore(architecture-normalization): make architecture naming consistent in the project, add checks (#274) | ||
|
Nick DiZazzo
|
70e33e475a
|
feat: add fail-closed pinned GPU assignment for local config (#260)
- add per-model gpu_id support to local config and validate pinned vs auto shape rules - preserve gpu_id through config snapshot protobuf sync and hashing - add stable-ID GPU resolver that rejects fallback IDs like index:* and backend device names - preflight config-owned startup models and fail before launch when pinned IDs are invalid or stale - enforce pinned-device selection in config push and local launch paths without silent fallback to auto or multi-GPU aggregation - block pinned config sync to older peers that do not support the feature - document pinned gpu_id setup, valid ID sources, and fail-closed behavior - add config, proto, runtime, and mixed-version compatibility tests - Harden pinned GPU launch selection - Fix Tegra pinned GPU detection |
||
|
Michael Neale
|
8b385ebce4 |
Update all michaelneale/mesh-llm references to Mesh-LLM/mesh-llm
Repo was transferred to the Mesh-LLM org. Update all GitHub URLs, install script defaults, auto-update endpoints, CLI defaults, UI links, docs site links, and container image references. Intentionally left alone: - michaelneale/llama.cpp (fork not transferred) - relay.michaelneale.mesh-llm.iroh.link (iroh relay domain, not a GitHub ref) |
||
|
James Dumay
|
d2436508da | Support exact release updates | ||
|
Nick DiZazzo
|
1bd6238999
|
feat(hardware): add hardware information enrichment
* Propagate GPU TFLOPS through benchmarks, status, and gossip * Add a vendor-neutral reserved_bytes contract for GPU facts * Propagate both through hardware discovery, mesh gossip, protobuf transport, and /api/status. Use best-effort collection for NVIDIA, AMD, and Intel * Add CI for benchmark smoke tests * add `mesh-llm gpus benchmark` command * rocm and intel dont have proper reserved byte outputs, unset those for now * move benchmarks to mesh-llm dir, update refs |
||
|
Nick DiZazzo
|
0129df7615
|
fix: strengthen cache-warming job to fail on resource exhaustion (#230)
* normalize AMD -> ROCm + optimize * add reusable workflow to avoid drift between cache warm jobs |
||
|
James Dumay
|
94fb5243b2 | Document Lemonade integration | ||
|
copilot-swe-agent[bot]
|
100a649ddf
|
Address review comments: dynamic config path in warning, fix --ctx-size docs, remove dead service-args code
Agent-Logs-Url: https://github.com/michaelneale/mesh-llm/sessions/2edc2366-a8ef-4876-b950-5c4074df8a67 Co-authored-by: i386 <50156+i386@users.noreply.github.com> |
||
|
James Dumay
|
16b916b35e | Load startup models from config.toml | ||
|
James Dumay
|
ac2170b18b | Add serve/client commands and local GPU inspection | ||
|
James Dumay
|
49c6231227 |
Merge remote-tracking branch 'origin/main' into multimodal-support
# Conflicts: # README.md |
||
|
James Dumay
|
2ccf1322af | Document multimodal support matrix | ||
|
copilot-swe-agent[bot]
|
6068d16a36 |
Merge main: fix check_for_update to verify specific bundle flavor asset
Co-authored-by: i386 <50156+i386@users.noreply.github.com> |
||
|
James Dumay
|
af5cfd733c | tighten README and split reference docs |