mirror of
https://github.com/Mesh-LLM/mesh-llm.git
synced 2026-08-08 22:23:19 -04:00
39 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
James Dumay
|
4bd1453fad
|
feat(skippy): expose authoritative generation lifecycle events (#1149)
* feat(skippy): expose authoritative generation lifecycle * refactor(mesh): remove injected local model launcher * fix(skippy): make serving hooks independent * feat(mesh): load deadline-safe native serving plugins (#1155) * feat(openai): propagate stable agent session identity * feat(mesh): load deadline-safe native serving plugins * docs: document native serving integrations * fix(skippy): preserve native proposal query contract * fix(skippy): compile native proposal plugin tests * fix(plugin): use public proposal query constructor * fix(skippy): make lifecycle delivery nonblocking * refactor(skippy): add lifecycle ingress adapter * fix: address generation lifecycle review --------- Co-authored-by: Nick DiZazzo <nick.dizazzo@gmail.com> |
||
|
Nick DiZazzo
|
f1953738b7
|
fix: keep Windows release verifier native-ABI-free (#1114)
* refactor: decouple release verifier from runtime ABI * ci: route release footer changes through Windows |
||
|
James Dumay
|
a878919175
|
fix: use published mesh-llm-hf-hub crate (#1096)
* fix: use published mesh-llm-hf-hub crate * fix: consume mesh-llm-hf-hub 1.0.2 |
||
|
Daniel Winter-Wijntjes
|
88f8b95b74
|
Add Ngram Suffix Proposer (#1037) | ||
|
Nick DiZazzo
|
aebfb02df4
|
feature: add benchmark 'tune' option (#948)
* Add measured gpu tune benchmark trials * Record gpu tune benchmark lifecycle timings * Tune mmap and mlock load controls * Fix Linux mlock limit detection * Select benchmark tune settings with throughput tolerance * Key benchmark trial configs by canonical model * Wire speculative decoding benchmark tuning * Fix MTP tune detection for resolved model paths * Wire ngram speculative decoding for staged tune runs * Support layer package benchmark tune targets * Wire configurable native MTP tuning * Add non-frame native MTP decode ABI * add benchmark tune persistence and launch args flags to docs and runner * add support for draft acceptance fields * Resolve HF-style draft refs for speculative MTP config |
||
|
Nick DiZazzo
|
7a80952294
|
chore: synchronize release version management (#934)
* chore: correct version strings and streamline release process * add 'just release x.y.z` support |
||
|
James Dumay
|
8952341a80
|
Add native skippy-quantize conversion and quantization CLI (#898) | ||
|
Nick DiZazzo
|
f36d79ae69
|
feature(version): normalize version markers for different build types (#831) | ||
|
Michael Neale
|
97c0cad991
|
feat: native rust SDK for rust consumers. (#736)
* a run at SDK, tested with a client * Use in-process shutdown for embedded SDK * Satisfy clippy for embedded shutdown plumbing * Document embedded Rust SDK usage * Add public Rust SDK crate * Tighten embedded SDK lifecycle * Address embedded SDK review feedback * Document native runtime packaging direction * Document runtime CLI namespace * Document runtime CLI UX expectations * Document runtime diagnostics under doctor * Add Windows PowerShell installer * Document recommended runtime install flow * Add native runtime resolver foundation * Wire native runtime release installs * Document native runtime crate * Load versioned native runtimes dynamically * Fix embedded SDK output manager reset * Expose SDK mesh admission controls * Split SDK runtime mapping assertions * Tighten SDK docs and config module * Clarify native runtime SDK TODOs * Stabilize native log note test * Expose native runtime install SDK * Re-export native runtime APIs from SDK crate * Add embedded SDK knobs for Sprout relay mesh (#782) Two opt-in seams the Sprout v1 mesh integration needs: - disable_iroh_relays(bool): when true, embedded runtime selects an explicitly disabled relay policy, which uses RelayMode::Disabled, skips public relay URL fallback, skips raw STUN, and avoids the 5s endpoint.online() wait that cannot succeed without a home relay. Default false preserves existing behavior. - EmbeddedNodeHandle::join_token(token): forwards an invite token over the runtime control channel to node.join_with_retry so an already-running embedded node can dial a new EndpointAddr without restart. Handled in both auto and passive/client runtime loops. Purely additive; existing defaults and startup join_tokens behavior remain unchanged. Co-authored-by: npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@sprout-oss.stage.blox.sqprod.co> Co-authored-by: Perci <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@sprout-oss.stage.blox.sqprod.co> * Fix relay policy test visibility * Make SDK publishable and align language bindings (#771) * Split SDK native runtime publish surface * Split CLI and TUI support crates * Move CLI parser surface into mesh-llm-cli * Extract shared mesh event surface * Move standalone command handlers out of host runtime * Move benchmark and plugin commands out of host runtime * Finish plugin command extraction * Extract auth identity ownership * Move remaining standalone commands out of host runtime * Move model store into model-hf * Move CLI commands out of host runtime * Decouple host runtime from CLI and TUI crates * Expose embedded node SDK facade * Keep client identity dependencies pure * Simplify Rust SDK feature surface * Keep SDK client feature runtime-free * Expose SDK client API base override * Use direct mesh SDK client transport * Remove API base URL client builder shim * Align language SDKs with Rust SDK facade * Document SDK client and serving modes * Fix SDK smoke runtime setup * Package SDK console assets * Fix dynamic runtime CI setup * Restructure SDK docs by language * Fix SDK smoke package loading * Fix native runtime bundle resolution * Add structured native runtime backend metadata * Harden Kotlin native runtime smoke resolution * Fix mesh-llm-sdk clippy imports * Retry smoke model downloads --------- Co-authored-by: James Dumay <jameswdumay@gmail.com> Co-authored-by: tlongwell-block <109685178+tlongwell-block@users.noreply.github.com> Co-authored-by: npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@sprout-oss.stage.blox.sqprod.co> Co-authored-by: Perci <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@sprout-oss.stage.blox.sqprod.co> |
||
|
James Dumay
|
b335a8162b |
Revert "Add metadata-only GGUF model fit scoring"
This reverts commit
|
||
|
James Dumay
|
541fb5a353 | Add metadata-only GGUF model fit scoring | ||
|
James Dumay
|
5b10e3785b
|
Let plugins provide agent skills (#725)
* Let plugins provide agent skills * Fix skills CLI clippy warnings * Address plugin skill review feedback |
||
|
James Dumay
|
a3fd23db09
|
Plugin management (#701) | ||
|
James Dumay
|
49ed50a706
|
Use combined hf-hub fork for model downloads (#709)
* Use resumable hf-hub fork * Use combined hf-hub branch |
||
|
James Dumay
|
fa490cf558
|
bump version to 0.68.0 (#711) | ||
|
James Dumay
|
05df6785a3
|
Add shared config.toml crate for SDKs (#675)
* Add optional SDK console asset server * extract shared mesh config crate * Include console server crate in client Docker build * Address config crate review feedback |
||
|
James Dumay
|
1500ea95be | Upgrade workspace to Rust 2024 edition | ||
|
Nick DiZazzo
|
6631cc0dde
|
feature(tool-calling): harden tool calling for agent harnesses (#645) | ||
|
James Dumay
|
c36132fd3d
|
Ship real client and serving SDKs across Swift, Kotlin, and Node.js (#634) | ||
|
Michael Neale
|
1b1aaf432b
|
MoA: mesh mode and many inference critical fixes, and quic keep alive (#566)
* feat: MoA gateway — stateful mixture-of-agents with tool arbitration
New standalone crate (moa-gateway) that fans out to N heterogeneous LLM
endpoints in parallel, normalizes dirty worker outputs, arbitrates with
deterministic logic, and manages the full tool call lifecycle across turns.
Tested live against 3 ollama models (llama3.2:3b, qwen3:4b, qwen3.6:27b):
- Knowledge/reasoning: picks highest-confidence answer across models
- Tool calling: correctly produces tool_calls when workers propose tools
- Tool lifecycle: full cycle query → tool_call → result → final answer
- Tool results bypass fan-out, go to reducer only (one transcript)
The gateway is transport-agnostic — works against any OpenAI-compatible
endpoint (ollama, mesh-llm, remote APIs). Integration with mesh model
discovery is the next step.
* feat: multi-turn context efficiency + mesh endpoint discovery
Progressive running summary: workers get compact deterministic summaries
instead of raw message history. Tested across 3-turn conversations —
workers retain context (Melbourne → restaurant recommendation) through
the summary, not through replaying 20k tokens of history.
New moa-mesh binary discovers models from any OpenAI-compatible endpoint
(mesh-llm proxy, ollama, vLLM) and runs the full MoA test suite against
it. Designed to work with 'mesh-llm client --auto' out of the box.
Multi-turn tool lifecycle proven end-to-end:
Turn 1: weather query → fan-out → tool_call (get_weather)
Turn 2: tool result → reducer only → text answer
Turn 3: follow-up 'bring jacket?' → fan-out with running summary → contextual answer
* fix: increase running summary budget to ~2k tokens
Per-fact truncation: 200 → 500 chars
Recent fact window: 5 → 15 facts
Tool result truncation: 80 → 300 chars
Turn outcome capture: first sentence → first 400 chars
200 tokens was too aggressive for real agent sessions where system
prompts alone can be 500+ tokens. 2k tokens gives enough room for
15 turns of meaningful context while still being much cheaper than
replaying raw history.
* feat: agentic workload test + improved prose tool detection
New moa-agent binary simulates a multi-step coding agent: read file,
analyze bug, edit fix, run tests, diagnose failure, iterate. Exercises
7+ turns with accumulating context, repeated tool use, and loop detection.
Improved normalizer: small models that describe tool usage in prose
("I'll use the edit_file tool") are now correctly classified as
tool proposals via known-tool-name + action-verb heuristic.
Key findings from agentic testing:
- Gateway routing, context management, and tool lifecycle are solid
through 7 turns / 15 messages / 6 reducer calls
- Loop detection catches repeated identical tool calls
- Small models (3b/4b) produce correct tool calls for read/search
but describe edits in prose instead of calling edit_file
- Multi-step plan execution (tool→analyze→tool→fix) needs a stronger
model in the reducer role — the arbiter and routing aren't the
bottleneck, model capability is
* fix: mesh-tested agentic flow — local reducer, model dedup, KV parse fix
Tested against live mesh: GLM-4.7-Flash (local) + Qwen3-8B (remote peer).
Three fixes from live mesh testing:
1. Local reducer: prefer first endpoint (local model) as reducer instead
of last. Remote models over QUIC relay are fine as parallel workers
but timeout as the sequential reducer. Result: reducer response
times dropped from 180s (remote timeout) to 2-18s (local).
2. Model dedup: mesh-llm exposes the same model under multiple aliases
(e.g. unsloth/GLM-4.7-Flash-GGUF and @main:Q4_K_M variant).
discover_endpoints() now deduplicates by normalized display name.
Extracted as shared lib function used by all three test binaries.
3. KV parse fix: models that say 'kind: answer' but also include
'tool: read_file' are now correctly classified as ToolProposal.
This was the #1 cause of missed tool calls in the agentic flow.
* moa: integrate into mesh proxy, SSE streaming, Goose support
MoA is now available as model="moa" through the mesh proxy on :9337.
When ≥2 models are available (local + mesh peers), the MoA virtual model
appears in /v1/models automatically.
Integration:
- mesh-llm-host-runtime depends on moa-gateway
- ingress.rs intercepts model="moa" requests, builds endpoints from
callable models, calls Gateway::turn(), returns the result
- SSE framing: converts the non-streaming MoA response into SSE chunks
for streaming clients (Goose, pi, etc.)
- Tool calls passed through in SSE format with finish_reason="tool_calls"
Passthrough mode (tools present):
- When the request includes tools (agentic use via Goose/pi), workers
receive the original messages+tools unmodified — no MoA envelope
- This avoids conflicting system prompts confusing small models
- First successful worker response is returned, providing redundancy
Content cleanup:
- Think tags (<think>...</think>) stripped from all responses
- Orphan </think> tags cleaned up
- KV envelope lines (kind:/confidence:/payload:) stripped when they
leak into heuristic-classified output
- Normalizer pre-cleans think tags before trying JSON/KV parse
Tested with:
- Direct curl (non-streaming + streaming)
- Goose CLI: factual questions, tool execution (shell, execute_typescript)
- 22 unit tests passing
* fix: add moa-gateway to Docker builds
The docker-client CI job failed because crates/moa-gateway/ was missing
from both Dockerfile.client and fly/Dockerfile. cargo metadata couldn't
resolve the workspace member, breaking the cargo-chef prepare step.
* moa: early-exit on worker consensus
Instead of waiting for all workers before arbitrating, check for
consensus after each worker returns. When 2+ workers agree on an
answer or tool call, return immediately and abort remaining workers.
This eliminates the 'slowest worker' bottleneck. In testing:
- Average latency dropped from 21.0s to 5.8s (single model: 7.1s)
- MoA is now 19% faster than querying a single model
- Worst case (code-debug) went from 120s timeout to 4.2s
The key insight: with parallel fan-out, we only need to wait for the
fastest N workers that agree, not all of them. Slow/dead remote
workers no longer block the response.
Also adds 5 new arbiter tests for early decision logic (27 total).
* moa: real context slices, not synthetic envelopes
Major architectural change to how MoA packs context for workers.
Before: workers got a synthetic system prompt ('You are a fast analysis
worker...') that replaced the agent's real system prompt, tool schemas,
and conversation history. Workers were asked to respond in a KV envelope
format (kind:/confidence:/payload:) that small models followed unreliably.
When tools were present, the entire MoA pipeline was bypassed via a
'passthrough mode' that raced identical requests to all workers.
After: workers get slices of the REAL context — the agent's actual system
prompt and messages — with depth varying by role:
- Fast: system prompt + last user msg + tool names only
- Specialist: system prompt + last 4 msgs + tool summaries
- Strong: system prompt + full recent history + native tool schemas
- Reducer: system prompt + worker outputs + full tool schemas
The gateway augments with a one-line preamble, not a replacement. The
passthrough mode is removed — tool-use goes through the full normalize →
arbitrate pipeline. Strong workers get native tool schemas forwarded so
they can produce real tool_calls.
Also:
- Dedup model aliases in ingress (GLM and GLM@main:Q4_K_M are the same)
- Early exit handles failed workers (sole survivor returns immediately)
- Worker timeout reduced from 120s to 30s
- 28 unit tests (up from 27)
* moa: rename to mesh-mixture-of-agents, mesh-native transport, model='mesh'
Renamed crate from moa-gateway to mesh-mixture-of-agents. Keeps the
crate isolated (own tests, own compilation unit) while connecting it
to mesh transport via a ModelBackend trait.
Transport is now mesh-native instead of HTTP loopback:
- LocalModelBackend: direct HTTP to skippy port (bypasses proxy)
- RemoteModelBackend: QUIC tunnel to peer (bypasses proxy + tunnel layer)
- Both set mesh_hooks: false to prevent recursive consultation
The ModelBackend trait keeps the crate testable in isolation — the
default HttpBackend works against any OpenAI-compatible endpoint.
The mesh backends are implemented in ingress.rs where Node and
InferenceTarget are available.
Virtual model renamed from 'moa' to 'mesh'. Appears in /v1/models
when ≥2 distinct models are available.
Test bins removed (used old Gateway API). 29 unit tests remain.
* fix: update Dockerfiles for moa-gateway → mesh-mixture-of-agents rename
* moa: remove 'mesh' from /v1/models list
The 'mesh' virtual model is a routing directive like 'auto', not a
real model. It should not appear in the models list. Clients that
want MoA fan-out use model: "mesh" explicitly.
* fix: clippy warnings in mesh-mixture-of-agents
* fix: clippy unnecessary_lazy_evaluations in ingress build_moa_config
* docs: update MoA design doc with current architecture and test plan
Reflects: mesh-mixture-of-agents crate rename, ModelBackend trait,
handle_turn() stateless API, mesh-native transport, model='mesh'
virtual routing, early-exit consensus, and eval plan.
* moa: fix tool call arguments lost in arbitration
Two fixes:
- Arbiter now prefers tool proposals with actual arguments over
proposals that only have the tool name (from fast workers that
don't get native tool schemas).
- Specialist workers now receive native tool schemas so they can
produce structured tool_calls with arguments, not just mention
tool names in text.
Before: read_file({})
After: read_file({"path":"/tmp/test.txt"})
* moa: worker diversity sampling, 429 retry, faster timeouts, sole-survivor early exit
Three improvements to MoA reliability and response quality:
- Workers get high temperature (0.8) + top_p (0.95) for diverse
exploration; reducer gets low temperature (0.3) for precise synthesis.
SamplingParams flows through the ModelBackend trait.
- 429 rate-limit errors trigger one automatic retry after the server's
retry-after delay (default 1s).
- Worker timeout 30s → 15s, reducer 45s → 30s.
- Sole survivor returns immediately when majority of other workers have
already failed, instead of waiting for remaining stragglers.
39 unit tests (up from 29).
* Fix MoA tool result handling and NaN confidence (PR review feedback)
Three issues from Copilot review on PR #534:
1. Tool result turns now include actual tool output content.
pack_for_tool_result_turn was reading from pending_tools which
is always empty on a fresh session (stateless per request).
Now forwards the raw message sequence including assistant
tool_call + tool result messages so the reducer sees the full
context. Added regression test.
2. NaN confidence no longer panics arbiter comparisons.
Replaced partial_cmp().unwrap() with total_cmp() in arbiter,
and added a sanitizer in normalize that clamps non-finite
confidence to 0.5. Added test.
3. Exclude spec-prefill-poc from workspace members (Docker fix).
Moved to Cargo.toml exclude list so Docker builds don't fail
on the missing experimental crate.
* ci: align WORKSPACE_MEMBERS with current workspace
- Add mesh-mixture-of-agents (new crate on this branch)
- Rename mesh-llm-client → mesh-client (renamed on main)
Fixes the scripts/affected-crates.sh consistency check that gates the
Linux CPU CI build.
* moa: size-aware role assignment + hallucinated tool name filtering
Two surgical fixes from real-world goose testing:
1. Role assignment by capacity tier, not list-order.
assign_roles previously used list order: first=fast, last=strong.
When a small local model (e.g. Qwen2.5-3B) was loaded last, it got
tagged Strong and used as the reducer — exactly when goose needs a
capable model for tool arbitration.
Now we sort by size tier using the same is_single_digit_b_name
heuristic as the main router's pick_model_classified, so MoA's
strong worker matches what auto would pick. MiniMax-M2.5 and
Qwen3-32B (big tier) become Strong; Qwen3-8B and Qwen2.5-3B (small
tier) become Fast.
2. Filter hallucinated tool names from worker proposals.
A worker proposing a tool not declared in the request (e.g. local
3B hallucinating 'execute_typescript' when only 'shell' was offered)
would bypass arbitration and reach the client, causing silent
failures when goose tried to dispatch the unknown tool.
gather_workers_incremental and the reducer output paths now demote
such proposals to Uncertainty with a tracing warning. The arbiter
sees a clean set of valid proposals and resolves correctly.
Threading: handle_query, handle_tool_result, gather_workers_incremental,
and resolve_decision all now take &[String] allowed_tools derived from
session.tool_names(). When allowed_tools is empty (no tools on request)
the filter is a no-op.
* moa: reducer candidate fallback on 5xx / timeout
When the chosen reducer peer is broken (e.g. stale binary returning 502
on tool grammars) or unreachable, the tool-result turn or NeedsReducer
arbitration would fail with that single error, even though other strong
peers were available.
Replace single-pick pick_reducer with reducer_candidates returning all
big-tier models (multi-digit B or no size in name) followed by small-tier
as last-resort fallback. Both call sites — handle_tool_result and
resolve_decision NeedsReducer — now iterate candidates and break on the
first success.
This rescues the common goose-on-public-mesh case where one strong peer
(e.g. Qwen3-32B host) is running a stale binary that 502s on tool calls,
while another (e.g. MiniMax) is healthy. Without this, MoA's tool-result
turn was as fragile as auto routing.
* ci: fix workspace member drift — keep mesh-llm-client package name, add mesh-mixture-of-agents to clippy script
Same fix as the prefill-draft branch:
- The mesh-client directory rename did not change the package name —
the crate is still published as 'mesh-llm-client'. Revert the
scripts/affected-crates.sh edit that broke the CI consistency check.
- Add mesh-mixture-of-agents to plan-clippy-batches.sh which carries
its own WORKSPACE_MEMBERS list with the same drift constraint.
* moa: support model:"mesh" on client/standby nodes via forward-to-host
Pure --client nodes and standby GPU nodes accept inbound HTTP via
handle_mesh_request (in transport.rs) instead of the model-aware api_proxy
in ingress.rs. The MoA fan-out intercept lives in api_proxy, so when a
client received "model": "mesh" it fell through to the "no host serves
this model" branch and 429d.
* moa: fix clippy lints surfaced by CI
Two pre-existing lint violations in mesh-mixture-of-agents that CI didn't
see before because the crate wasn't in the affected-crates / clippy
workspace lists. Now that the WORKSPACE_MEMBERS drift is fixed they show
up on every PR clippy run.
- worker.rs:67 `x == false` -> `!x` (clippy::bool_comparison)
- lib.rs:523 `&name` -> `name` (clippy::needless_borrow)
No behavior change — tool_call_response takes &str either way, and the
sort key inverts identically.
* moa: hedge reducer candidates instead of sequential fallback
Cut worst-case reducer latency from N×timeout to roughly
reducer_timeout + (N-1)·hedge_delay. Big win when a peer is slow or
broken; zero cost on the happy path.
Before:
for candidate in candidates:
call(candidate, timeout=30s) # wait up to 30s per stale peer
if ok: return
# 3 stale big-tier peers ⇒ 90s before falling through to small-tier
After:
spawn candidate[0]
loop:
select:
a candidate finished:
ok → cancel rest, return
err → spawn next candidate immediately (no hedge wait)
hedge_delay elapsed and more candidates remain:
spawn next alongside in-flight ones (race)
Cost shape:
- Happy path (cand 0 OK in <hedge_delay): exactly 1 backend call. Free.
- Slow first (cand 0 takes hedge_delay..reducer_timeout): up to 2
overlapping calls, accept whichever wins, cancel loser.
- Fast-fail (cand 0 errors quickly): next candidate immediately, 1 call.
- All fail: ≤N calls, capped at reducer_timeout + (N-1)·hedge_delay.
Wall-clock improvement for 3 stale big-tier peers (worker_timeout=15s,
reducer_timeout=15s, hedge_delay=5s):
- Before: 3 × 30s = 90s before reaching small-tier fallback.
- After: 15s + 2 × 5s = 25s. Plus reducer_timeout itself drops 30s → 15s
now that the hedged ladder makes a single per-attempt cap safe to
shorten.
Changes:
- Add hedged_reducer_call() in mesh-mixture-of-agents/src/lib.rs.
- Replace the for-loop in handle_tool_result() with it.
- Replace the for-loop in resolve_decision()'s NeedsReducer arm with it.
- Add hedge_delay field to GatewayConfig (defaults set at the single
construction site, build_moa_config in ingress.rs).
- Lower reducer_timeout 30s → 15s in build_moa_config.
- 4 new unit tests cover happy path, hedge-on-slow, fast-fail, all-fail.
- Refresh stale numbers in docs/design/MOA_GATEWAY.md and replace the
"first model wins" reducer paragraph with the hedged-ladder description.
Verified:
cargo fmt --all -- --check # clean
cargo check -p mesh-llm-host-runtime # clean
cargo clippy -p mesh-llm-host-runtime --lib # clean
cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings # clean
cargo test -p mesh-llm-host-runtime --lib # 1381 passed
cargo test -p mesh-mixture-of-agents --lib # 47 passed (4 new hedge tests)
* evals: add bench-moa.sh — quick wall-clock benchmark for model:"mesh"
POSTs N chat completion requests to a running mesh-llm endpoint with
model="mesh" and reports p50/p95/p99 wall-clock latency. Probes /v1/models
first and warns if fewer than 2 models are present (MoA returns 503).
Usage:
./evals/bench-moa.sh # 20 requests, localhost:9337
N=50 ./evals/bench-moa.sh
BASE_URL=http://host:9337 ./evals/bench-moa.sh
PROMPT="why is the sky blue?" ./evals/bench-moa.sh
Output is per-request lines plus a summary block (min/p50/p95/p99/max/
mean/stdev). Failed requests are reported but excluded from percentiles.
Raw TSV results are kept in a tmpdir so before/after comparisons are easy.
Dependencies: curl, jq, python3 — nothing exotic. Run on the same machine
as a serving host so wall-clock is dominated by inference + arbitration.
* moa: split lib.rs into backend / reducer / fanout modules
lib.rs was past the 1k LoC refactoring threshold and had three separable responsibilities. Extract them into named modules and keep lib.rs as the orchestration entrypoint (handle_turn, GatewayConfig, TurnResult, response builders).
- backend.rs: ModelBackend trait, HttpBackend, SamplingParams, ModelEntry, call_backend + retry-after parsing
- reducer.rs: reducer_candidates ordering, hedged_reducer_call ladder
- fanout.rs: gather_workers_incremental
ModelEntry, HttpBackend, ModelBackend, and SamplingParams are re-exported from lib.rs so existing callers (worker.rs, host-runtime ingress) keep working.
Tests move with their owner: backend.rs gets the sampling/retry-after tests, reducer.rs gets the 4 hedged-reducer tests plus the FakeBackend helper. No behavior change.
lib.rs: 1267 -> 545 LoC
47 existing tests still pass.
* moa: cover role-shaped context packing with tests
context.rs owned the role-shaped packing logic (fast/specialist/strong/reducer depth contract) without any tests. Pin the design claims:
- per-role token budgets (256 / 512 / 1024)
- fast: system + last user only, tool names only, no tools field
- specialist: tool summaries in system, native tools populated
- strong: deep history (>=6 msgs), native tools populated
- generalist/reducer roles alias the strong shape
- MoA preamble augments rather than replaces the agent's system prompt
- reducer context includes the conflict reason and labeled worker payloads
- long worker payloads are truncated with an ellipsis to bound context
8 tests, all green.
* docs: surface model:"mesh" (MoA) in README
Add a workflow-table row pointing at MOA_GATEWAY.md and a short 'Mixture-of-Agents' section with a curl example and the two-models-required gate, so the feature is discoverable from the project entrypoint.
* moa: extract tool_guard module, drop dead Endpoint/discover_endpoints
Two small cleanups on lib.rs:
- Move enforce_allowed_tools into its own tool_guard.rs module. It's a
content-policy concern (demote hallucinated tool names to Uncertainty
before arbitration), not orchestration, so it doesn't belong in the
handle_turn entrypoint file. Comes with 4 unit tests covering allowed
pass-through, unknown-tool demotion (incl. confidence drop),
empty-allowed-list noop, and non-proposal outputs untouched.
- Delete the Endpoint struct and discover_endpoints helper from lib.rs.
Their doc comments described them as 'convenience for test harnesses'
but they have zero call sites in-tree and no out-of-tree consumers we
know of. Dead code from the standalone phase before mesh-native
backends landed.
lib.rs: 545 -> 454 LoC.
Tests: 55 -> 59 (4 new in tool_guard).
* docs(moa): drop the speculative hook-integration line
MoA and hooks are intentionally independent — worker requests set
mesh_hooks: false so the hook pipeline can't re-enter a worker call.
The old design doc closed the relationship section with 'they could
integrate later (hook signals as arbiter weights)', which makes it
look like roadmap. It isn't — keeping them separate is the design.
Replace that line with one that states the separation as intentional
and points at the mesh_hooks: false invariant that enforces it.
* router: weight 'auto' selection by locally observed tok/s
Before, 'auto' picked uniformly at random within the multi-digit-B
tier. On the public mesh this meant a fast MiniMax on a 4090 and a
slow 35B-A3B on an M2 Air were equally likely to be chosen, even
though we'd already measured the throughput gap in routing_metrics
and were just not reading it.
Now: each big-tier candidate is weighted by its locally observed
avg_tokens_per_second (clamped to [5, 100] tok/s so nothing fully
starves and no outlier monopolizes). Models without enough samples
(< 3) get a neutral weight so they compete fairly until data
accumulates. A 15% exploration probability ignores weights and
picks uniformly, which keeps the system from locking onto stale
rankings and guarantees cold peers see traffic.
Plumbing:
- RoutingMetrics::tps_for_model(name) -> Option<(f64, u64)>: cheap
per-model lookup that locks only the relevant shard, avoiding the
per-call HashMap allocation model_snapshots() does in the hot path.
- Node::routing_metrics() public accessor (Arc-backed, cheap).
- RoutingCandidate { name, caps, tps_hint, throughput_samples }
replaces the anonymous (&str, f64, ModelCapabilities) tuple whose
middle slot was literally always 0.0 at every populated call site.
The struct makes the tps hint a real, typed concept rather than a
dangling hook.
Behaviour preserved:
- Single-digit-B partition (smalls stay last-resort) unchanged.
- All-cold candidate pool falls back to ~uniform pick (regression
test confirms no model is starved when there's no data yet).
- Capability filtering for tools / reasoning / vision unchanged.
Plumbing per call site:
- ingress.rs + transport.rs: live routing path, look up tps_hint
from the local RoutingMetrics handle for each candidate.
- discovery.rs + integrations.rs: pre-startup paths with no live
metrics; build candidates with RoutingCandidate::unscored() so
they get the cold-neutral weight.
Tests:
- weighted_pick_all_cold_is_roughly_uniform — regression safety.
- weighted_pick_fast_wins_majority_but_slow_still_gets_some —
fast wins by >=1.5x but slow still gets >30/600 picks.
- weighted_pick_cold_model_competes_with_hot_fast — newcomer gets
>100/600 picks against an established fast peer (so it can
actually accumulate samples and earn its score).
- weighted_pick_low_sample_count_treated_as_cold — 1-sample
measurements don't dominate routing.
- candidate_weight_clamps_extremes — weight stays in [5, 100],
cold = 25.
Removed:
- shuffle_in_place (replaced by SplitMix64 + pick_weighted).
- The dishonest 0.0 f64 slot in the candidate tuple, everywhere.
Validation:
cargo fmt --all -- --check # clean
cargo check -p mesh-llm-host-runtime # clean
cargo clippy -p mesh-llm-host-runtime --lib # clean
cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings # clean
cargo test -p mesh-llm-host-runtime --lib # 1398 passed (17 in router)
cargo test -p mesh-mixture-of-agents --lib # 59 passed (no regression)
* docs(moa): clarify topology — N workers + serial 2-call shape
- Replace topology diagram with one that shows N workers fanned out in
parallel and the serial fan-out → arbiter → reducer path.
- Add explicit "how many models" table (2..N) and the worker → role
mapping so readers don't have to infer it from worker.rs.
- Spell out that a worst-case MoA turn is 2 LLM round-trips serially
(fan-out wall-clock = slowest worker, then optional reducer), and
that happy paths collapse to 1 (consensus or tool-result turn).
- Refresh stale crate-structure table: post-split LoC + test counts
for backend / reducer / fanout / tool_guard / arbiter / context /
worker / session / normalize / lib.
* moa: emit x-moa-* observability headers from gateway
Extend TurnResult with turn_kind (Fanout / EarlyExit / ToolResult / Failed)
and reducer_attempts (candidates actually spawned). hedged_reducer_call
now returns a named HedgedReducerOk struct carrying winner, text, and
spawn count so the caller can attribute hedge cost.
The ingress MoA intercept reads these and emits:
x-moa-elapsed-ms
x-moa-turn fanout | early-exit | tool-result | failed
x-moa-workers total workers dispatched
x-moa-workers-ok workers that returned a usable answer
x-moa-reducer true | false
x-moa-reducer-attempts 0 on no-reducer path, 1 happy, >=2 hedged
Headers are emitted on both the JSON and SSE response paths via a new
send_json_ok_with_headers helper and an extra_headers arg to
send_moa_as_sse. Normal OpenAI clients ignore unknown headers; benches
and ops tooling can read them without parsing the body.
Side fix: handle_tool_result previously reported attempts = total
candidate pool size rather than candidates actually spawned. Now
correctly reports the spawn count from HedgedReducerOk.
* bench-moa: aggregate gateway path, reducer, and hedge stats
Read x-moa-* response headers per request and roll them up in the
summary. New aggregates:
Gateway paths histogram of fanout / early-exit / tool-result / failed
Reducer invocation rate + hedge rate + avg/max attempts
Worker fan-out average width + histogram by N
Latency p50 split by gateway path (so 'reducer turns are 4x slower'
is visible at a glance)
Per-request log line now shows the turn kind, worker count, and reducer
status alongside the latency, making it easier to eyeball individual
outliers.
Older mesh-llm binaries that don't emit x-moa-* headers degrade to the
old summary (latency-only) with a note that headers were not seen, so
this works against any mesh-llm version.
* evals: remove bench-moa.sh — never actually run
The script was written but never executed against a live mesh. MoA verification is the manual live-mesh testing called out in the PR body. Real aggregates, if we want them, should come from passive counters fed by real traffic, not a synthetic curl loop.
* moa: orchestrate from any node, build worker pool from mesh-wide gossip
Before this change MoA only ran when the request hit `api_proxy` on a
serving host. A pure `--client` node received `model: "mesh"` in
`handle_mesh_request`, fell through to the "forward to any host"
fallback, and the receiving host either ran an older binary that
ignored the "mesh" name or built a single-model config because its
local `ModelTargets` only had its own model. End result: no fan-out
happened, MoA was effectively dead from any client node, and the
worker pool depended on which node received the request rather than
on what was actually in the mesh.
Two structural fixes:
1. New `moa_gateway` module owns the intercept. Both `api_proxy` and
`handle_mesh_request` now call `try_handle_moa` — the request is
handled wherever it lands, whether the node serves models locally
or not.
2. `build_moa_config` enumerates `Node::models_being_served()` (the
mesh-wide union of local + gossip) instead of the local routing
table. Locally-served models are wired directly to the skippy port
via the routing table when one is present (host mode); everything
else opens a QUIC tunnel to the hash-preferred peer that advertises
the model. On a pure client every worker is remote.
Side effects:
- Canonical-base dedup now strips an `@branch` segment without losing
the trailing quant tag, so `unsloth/Qwen3-8B-GGUF@main:Q4_K_M` and
`Qwen3-8B-Q4_K_M` collapse to one worker instead of being treated
as two distinct models.
- The MoA-related backends (`LocalModelBackend`, `RemoteModelBackend`)
and the SSE wrapper moved out of `ingress.rs` into the shared
module; `ingress.rs` shrinks by ~400 lines.
Verified live against the public mesh from a `--client --auto` node:
- Chat completion: `x-moa-workers: 4` (all 4 mesh-wide models),
early-exit path, correct answer.
- Tool-equipped request: 3 tool-capable workers, early-exit path,
correct shape.
* moa: carry attempts on reducer-failure path; copilot review fixes
Two real bugs surfaced by live goose testing against the public mesh from a
--client --auto node.
Bug 1 — attempts accounting on the failure path.
hedged_reducer_call returned Result<HedgedReducerOk, String> where the Ok
arm carried 'attempts: u32' but the Err arm dropped it. Both call sites in
lib.rs (handle_tool_result, resolve_decision) reported attempts=0 on the
all-fail path, producing nonsense like 'Reducer failed (tried 0): remote
timeout after 15s'. Replace with Result<HedgedReducerOk, HedgedReducerErr>
where Err carries attempts too. Surface the real spawn count to logs and
to the user-visible error string.
Bug 2 — copilot review issues.
- UTF-8-safe truncation: 2 panicking '&text[..len.min(N)]' sites in
moa_gateway replaced with new moa::truncate_chars helper that walks back
to a char boundary.
- Remote-read cap 256 KiB → 4 MiB. Long reasoning + tool synthesis answers
can exceed 256 KiB.
- CR/LF sanitization on x-moa-* header values. Cheap insurance.
- Removed dead ('unknown', 0) fallback in reducer_candidates. Let
hedged_reducer_call's empty-input path surface real errors instead of
silently dispatching to backend_index=0 with a bogus name.
- Consolidated three byte-identical strip_thinking implementations
(worker.rs, normalize.rs, moa_gateway.rs) onto one canonical
moa::worker::strip_thinking with re-export from moa crate root.
- Warn on response-write failure rather than swallowing the error.
Tests: 4 new (truncate_chars on UTF-8 boundary, all-fail-reports-attempts),
all 63 moa + 1403 host pass. Clippy + fmt clean.
Live verified from --client --auto on this Mac, joined to public mesh:
- Plain chat: x-moa-workers: 2, early-exit, 1.3s, correct answer.
- Goose end-to-end (tool propose → shell exec → tool-result turn → final):
full loop completed, server log shows fanout + early-exit on both turns,
goose printed DONE and exited 0.
* moa: address remaining Copilot review items
- context.rs / session.rs / backend.rs: replace byte-index truncation
with crate::worker::truncate_chars (UTF-8 safe). Worker payloads,
tool outputs, and HTTP error bodies all come from external sources
that can contain multi-byte characters.
- reducer.rs hedge loop: once `remaining` is exhausted, stop arming
the hedge timer and just await join_next() directly. Previously the
select! kept rebuilding a fresh hedge_sleep every iteration and
firing every hedge_delay just to no-op. Untidy, not a correctness
bug — but easier to reason about now.
Closes inline review feedback on PR #566.
* moa: tighten early-exit content check with subset+negation rule
Early-exit previously claimed "workers agree" whenever 2+ outputs were
Answer-kind, without comparing payload content. Two workers replying
"Paris" and "Berlin" both with confidence ~0.5 (the default for plain
prose) would early-exit on whichever was returned first.
New rule: two answers agree iff
- the smaller content-token set is a subset of the larger, AND
- their symmetric difference contains no negation tokens.
Tokenization: lowercase, strip punctuation, drop stopwords and tokens
<3 chars (digits and negation words always kept).
This is biased toward false-negatives: terse-vs-verbose paraphrases
like "Paris" / "Paris is the capital of France" cluster correctly,
while same-shape disagreements like "...is Paris" / "...is Berlin"
do not. When the rule declines to cluster, we just wait for more
workers and fall through to arbitrate() — no extra reducer call.
Also:
- session.rs:341: replace one remaining &first_line[..77] byte-slice
with worker::truncate_chars (multi-byte panic risk on tool names
containing emoji).
- ingress.rs MoA intercept: replace let _ = try_handle_moa(...) with
if let Some(...) and a tracing::error! so the impossible "returned
unused stream" case is loudly logged instead of silently leaked.
Tests:
- 4 reworked early-exit tests (terse-vs-verbose, normalized-equivalent,
majority cluster, shared-scaffolding-still-blocks)
- 3 new negation guard tests (not, don't, "use grep" vs "do not use
grep")
- 1 numeric agreement test ("42" vs "the answer is 42")
73 moa tests pass, 1426 host-runtime tests pass, both clippies clean.
* docs(moa): add pressure-test research plan to MOA_GATEWAY
Replace the earlier 'A/B plan' sketch with a research plan that is
designed to falsify the mixture hypothesis, not confirm it.
- Sharpened hypothesis with three falsifiable corollaries
- Pre-committed falsification conditions (so we cannot move goalposts)
- Step 1: variance floor measurement as prerequisite for any A/B claim
- Adversarial scenarios including failure-mode-amplification cases
- Pareto curve as the headline deliverable, not win/tie/loss
- Ablations to separate 'mixture' from 'variance reduction'
- Composition sweep to test the 'modest models' framing directly
- Grader robustness checks (position swap, dual grader, hand spot-check)
- Real-task replay as the strongest defense against cherry-picking
- Reporting discipline: what must be in a result before calling it a win
Documentation only. No crate changes. Worker-set knob noted as a
harness-side concern, not a crate change.
* docs(moa): reframe pressure test around equal-VRAM split-vs-mix on mesh
The earlier pressure-test plan was "is mixture smarter than single best,"
which is the wrong load-bearing question. The honest question for a mesh
is: given fixed aggregate (V)RAM, when does running multiple diverse
mid-size models locally beat sharding one large model across the network?
Reframes the eval around the equal-VRAM trade between Skippy split-large
and MoA mix-diverse, with network conditions (RTT, loss) as the primary
axis. Existing scenario/ablation content becomes the quality measurement
implementation, not the headline. Adds pre-committed falsification
conditions specific to the network-tolerance and scalability claims.
Docs-only.
* docs(moa): reframe as operating-envelope, not benchmark fight
The earlier draft framed MoA vs split-large as a quality competition. The
real claim is that split-large has a hard practical ceiling on a real
mesh — every cross-node hop is on every token's critical path — and MoA
has a much higher ceiling because workers run fully local and the
network is only touched at fan-out/collect/reducer.
Reframe accordingly:
- Headline is *operating-envelope analysis*, not Pareto fight
- Define what 'acceptable' means (TTFT, total turn, failure rate, quality
floor) before any measurement, so we cannot retrofit it
- Deliverable is a *viability map* (config x network condition), not a
win/tie/loss table
- Quality is demoted to a tertiary axis inside the viable region; its job
is to confirm MoA's MoA-only-region answers clear the single-mid floor
- Pre-committed falsification conditions are specific to the new claims
(envelope shrinkage with mesh size/network, MoA's envelope extending
past split's, MoA quality above single-mid floor, mixture vs variance
reduction)
- single-mid baseline added explicitly so we cannot accidentally ship
'MoA = single-best + overhead'
Complementary positioning, not competitive: use split when the network
allows; use mix when it doesn't.
* docs(moa): promote 'why MoA exists' to top of design doc
The opening of MOA_GATEWAY.md described mechanism (fan out, arbitrate)
but not purpose. The motivation \u2014 'use the mesh anyway when split-large
isn't viable for the current network conditions' \u2014 was buried ~450
lines down inside the operating-envelope section.
Add a brief 'Why MoA exists' section at the top that states:
* The intended operating region (where split-large stops being viable).
* That MoA is not trying to beat split-large on quality.
* The complementary, network-conditions-decide-which framing.
* A link down to the experimental envelope discussion that already exists.
No design or behavior change. Pure framing of existing content.
* fix(moa): signal all-workers-fail as a proper error response
PR #566 review feedback (Apr 2026):
> One concurrency request returned HTTP 200 even though the response
> body said all MoA workers failed. That's a bad client contract.
> If all workers fail, the API should probably return a proper error,
> not a successful-looking response with failure text inside it.
The MoA gateway was returning a body shaped identically to a
successful `chat.completion` with the error string smuggled into
`choices[0].message.content` and `finish_reason: "stop"`. The
ingress wrapped that body in an HTTP 200. A client checking either
the HTTP status, the top-level `error` field, or `finish_reason`
saw "success."
## Test (added first, observed failing)
`crates/mesh-mixture-of-agents/tests/sim_all_workers_fail.rs` drives
`moa::handle_turn` with three `AlwaysErrBackend`s, asserts the
result body is distinguishable from a successful `chat.completion`
\u2014 either the top-level `object` is not `chat.completion`, or there
is a top-level `error`, or `finish_reason` is one of `error` /
`moa_failed`.
The test fails against the pre-fix gateway with output
> object=Some("chat.completion"), finish_reason=Some("stop"),
> has top-level error=false
## Fix
* `mesh-mixture-of-agents/src/lib.rs` \u2014 `error_response()` now
attaches a top-level OpenAI-shape `error` object and emits
`finish_reason: "error"`. The error text stays in `content`
for unstructured clients.
* `mesh-llm-host-runtime/src/network/openai/transport.rs` \u2014 new
`send_json_with_status_and_headers()` helper for sending a custom
status code with a full structured body and observability headers.
* `mesh-llm-host-runtime/src/network/openai/moa_gateway.rs` \u2014
`write_moa_response` now takes the full `TurnResult` and sends
HTTP 502 (Bad Gateway) when `turn_kind == Failed` for non-streaming
responses. Streaming SSE stays 200 because we can't change the
status after the headers are sent; the failure rides in the chunked
body (which now carries the structured error).
## Validation
`cargo test -p mesh-mixture-of-agents` \u2014 70 unit + 1 new
integration test pass.
`cargo test -p mesh-llm-host-runtime --lib` \u2014 1434/1434 pass.
`cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings`
\u2014 clean.
`cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings`
\u2014 clean.
`cargo fmt --all -- --check` \u2014 clean.
* fix(moa): account for aborted workers in worker_summaries
PR #566 review feedback (Apr 2026):
> Worker accounting was inconsistent:
> - Similar requests reported different x-moa-workers values.
> - Similar requests reported different x-moa-workers-ok values.
> - Some successful responses used fewer workers than expected.
> - Some churn responses still appeared to report stale worker counts.
`worker_summaries.len()` is what the `x-moa-workers` header reports.
When the arbiter early-exits on consensus, the gateway called
`JoinSet::abort_all` and then drained `join_next()` with
`if let Ok(...)`. `JoinSet::abort_all` causes aborted tasks to
return `Err(JoinError::cancelled)`, with no `(model, role)`
payload \u2014 those tasks were silently dropped from `summaries`. A
4-worker fan-out that early-exited from 2 fast workers reported
`x-moa-workers: 2`, hiding the fact that 2 workers were cancelled
mid-flight. Panicked tasks had the same problem.
## Test (added first, observed failing)
`crates/mesh-mixture-of-agents/tests/sim_worker_accounting.rs` sets
up 4 mock backends, 2 fast/agreeing and 2 slow, and asserts that
`worker_summaries.len() == 4` after early-exit \u2014 i.e. that the
header faithfully reflects the dispatched count.
The test fails against the pre-fix gateway with output
> Got 2 summaries: ["fast-a-3b", "fast-b-3b"]; expected 4.
## Fix
* `fanout.rs` \u2014 `gather_workers_incremental` now takes the
dispatched-worker list (`&[DispatchedWorker]`) instead of just a
count. After fan-out finishes (whether via normal completion or
early-exit drain), `reconcile_dispatched` walks the dispatched
list and synthesizes a `succeeded: false` summary for any worker
whose name does not appear in `summaries`. Aborted tasks and
panicked tasks are now both attributed.
* `lib.rs` \u2014 builds a `Vec<DispatchedWorker>` alongside the
`JoinSet` and threads it through to `gather_workers_incremental`.
The header `x-moa-workers` now always equals the worker count we
actually dispatched. `x-moa-workers-ok` continues to reflect
genuinely-succeeded workers only.
## Validation
`cargo test -p mesh-mixture-of-agents` \u2014 70 unit + 2 integration
tests pass (new test plus the existing all-workers-fail one).
`cargo test -p mesh-llm-host-runtime --lib` \u2014 1434/1434 pass.
`cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings`
\u2014 clean.
`cargo fmt --all -- --check` \u2014 clean.
* fix(moa): route tool-result follow-ups to reducer, not fan-out
PR #566 review feedback (Apr 2026):
> The tool-result path isn't ready for agent loops:
> - A tool-result follow-up was treated like another fanout turn.
> - It wasn't handled like a controlled reducer/synthesis turn.
> - Tool results should be handled carefully and predictably, not
> sprayed back through the whole fanout path.
`Session::classify_turn` only routed to `TurnType::ToolResult` when
the very last message had `role: "tool"`. Many agent harnesses send
the tool result followed by a short `user` nudge ("continue", "what
did you find?"). That landed at the very-last-message check as
`user`, so the gateway classified the turn as Continuation, fanned
out to all workers, and invited a worker to re-propose the same tool
call whose result was already in context.
The session-state fallback at `last_was_tool_call &&
has_unprocessed_tool_results` was dead code in production: the
gateway never invokes `record_assistant_response` between turns, so
`last_was_tool_call` is always false.
## Test (added first, observed failing)
`crates/mesh-mixture-of-agents/tests/sim_tool_result_routes_to_reducer.rs`
\u2014 three scenarios, all using mock backends that count calls:
* OpenAI canonical shape (last msg role=tool) \u2014 must classify as
`ToolResult`, exactly one backend call (reducer only). Already
passed pre-fix; pinned to prevent regression.
* Trailing-user-after-unsynthesised-tool-result \u2014 must also classify
as `ToolResult`, exactly one backend call. **Failed pre-fix with
`TurnKind::EarlyExit`** (fanned out, multiple worker calls).
* Plain fresh user question \u2014 must still fan out. Pins that we don't
over-trigger the tool-result path.
## Fix
Scan messages from the end in `Session::classify_turn`:
* First message we hit with `role: "tool"` \u2192 classify as
`ToolResult`. The tool result has not yet been synthesised by an
assistant message after it.
* First message we hit with `role: "assistant"` \u2192 stop. The
assistant has already spoken since the last tool result; the next
turn is a normal continuation.
* Other roles (`user`, `system`) \u2192 keep scanning. A user nudge
after an unsynthesised tool result still belongs in the
reducer-only path.
If the scan reaches the start without hitting either, fall through to
the existing `Fresh`/`Continuation` classification.
## Validation
`cargo test -p mesh-mixture-of-agents` \u2014 70 unit + 5 integration
tests pass (this PR\u2019s 3 new tests + the two earlier sim files).
`cargo test -p mesh-llm-host-runtime --lib` \u2014 1434/1434 pass.
`cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings`
\u2014 clean.
`cargo fmt --all -- --check` \u2014 clean.
* fix(moa): recognise OpenAI-shape inline tool JSON in worker output
PR #566 review feedback (Apr 2026):
> In the read-tool probe, the model wrote text that looked like a
> tool call instead of actually invoking the read tool.
Agent harnesses (Goose, OpenCode, pi) only act on real `tool_calls`.
If a worker emits inline OpenAI-shape tool JSON \u2014
"I'll read the README. {\"function\": \"read_file\",
\"arguments\": {\"path\": \"README.md\"}}"
\u2014 today's normalizer's `try_json_parse` requires a `kind` field in
the JSON, which the OpenAI tool-call shape never has. `try_json_parse`
returns None, the heuristic classifier doesn't have an action verb
that matches ("I'll read" isn't on its list), and the worker output
falls through to `OutputKind::Answer`. Three workers all return the
same text \u2192 arbiter agrees \u2192 `chat_response(text)` \u2014 the agent
gets the JSON-bearing prose as `content`, no `tool_calls` field,
and silently does nothing.
## Test (added first, observed failing)
`tests/sim_tool_call_text_not_passed_as_content.rs` \u2014 two scenarios:
* `workers_with_inline_tool_json_emit_real_tool_call` \u2014 workers
return prose with embedded `{"function": "read_file",
"arguments": {...}}`. The response body must carry a real
`tool_calls` array with the proposed function name. **Failed
pre-fix**: body had `content` with the prose, no `tool_calls`.
* `workers_describing_tool_call_must_emit_structured_tool_call` \u2014
workers describe a tool call in pure prose with no JSON. Today
the heuristic catches this and synthesises a `tool_calls` entry
(with empty arguments). Pinned so the JSON-shape fix below
doesn't regress the pure-prose path.
## Fix
`normalize::try_json_parse` now also recognises the OpenAI tool-call
shape when no `kind` field is present:
{"function": "read_file", "arguments": {...}}
{"name": "read_file", "arguments": {...}}
{"tool": "read_file", "arguments": {...}}
A structurally well-formed inline tool proposal scores confidence
0.75 (above the heuristic's 0.6) so the arbiter prefers it on ties.
The rest of the original `kind`-driven envelope path is unchanged.
## Validation
`cargo test -p mesh-mixture-of-agents` \u2014 70 unit + 7 integration
tests pass (this PR\u2019s 2 new tests + earlier sim files).
`cargo test -p mesh-llm-host-runtime --lib` \u2014 1434/1434 pass.
`cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings`
\u2014 clean.
`cargo fmt --all -- --check` \u2014 clean.
* fix(moa): /v1/models advertises quant-suffix IDs that route back
PR #566 review feedback (Apr 2026):
> Some IDs in /v1/models dropped quant suffixes. Other endpoints
> used the full model refs.
> [...]
> Direct calls from Carrack to worker-hosted models didn't work.
> Carrack to Lemony 35B returned HTTP 404.
Reproduced on a 2-node mesh (Mac M4 Max + Mac Studio M3 Ultra):
* M4 served `Qwen/Qwen2.5-3B-Instruct-GGUF:qwen2.5-3b-instruct-q4_k_m`.
* Studio served `unsloth/Qwen3-0.6B-GGUF:BF16`.
* M4 `/v1/models` listed:
Qwen/Qwen2.5-3B-Instruct-GGUF (quant suffix lost)
unsloth/Qwen3-0.6B-GGUF:BF16 (full id)
* Calling either listed id directly:
local short id \u2192 200 (rewritten via internal alias table)
remote short id \u2192 404 (no alias for remote models)
The natural client flow \u2014 read `/v1/models`, take an id, call
`/v1/chat/completions` with it \u2014 was broken for remote-hosted
models, and inconsistent for local ones.
## Two root causes
1. **`quant_selector_from_gguf_file` was uppercase-only** when
matching markers like `-Q`, `-BF16`. Real GGUF filenames mix
cases (`...-Q4_K_M.gguf` vs `...-q4_k_m.gguf`). The matcher
on the read side (`gguf_matches_quant_selector`) was already
case-insensitive, so emitting a lowercase selector is safe and
keeps the public id round-trippable.
2. **`public_huggingface_model_ref` only handled artifact-as-filename.**
Locally-built `ServedModelDescriptor`s set
`artifact = model_ref.selector` \u2014 e.g.
`"qwen2.5-3b-instruct-q4_k_m"`, a quant selector, not a GGUF
filename. `quant_selector_from_gguf_file` returned None for
anything not ending `.gguf`, so the public id collapsed to just
the repo name.
The `public_model_id` selection logic also needed tightening so we
fall back to local disk only when the descriptor cannot produce a
lossless id (e.g. catalog or local-gguf identities without enough
metadata).
## Test (added first, observed failing)
`models_list_id_preserves_quant_suffix_when_descriptor_has_no_artifact`
in `transport.rs` builds a HuggingFace descriptor with no `artifact`
field and asserts the resulting public id either matches the internal
model_name verbatim or carries a non-empty quant tag. Pre-fix the
public id collapsed to bare repo and the test failed.
## Fix
* `model-ref/src/lib.rs::quant_selector_from_gguf_file` \u2014 lowercases
the stem before matching markers so lowercase-quant filenames like
`qwen2.5-3b-instruct-q4_k_m.gguf` extract `q4_k_m` instead of None.
Returns the slice from the original stem so display casing is
preserved.
* `transport.rs::public_huggingface_model_ref` \u2014 accepts artifact
values that are already a quant selector (no `.gguf` suffix) in
addition to GGUF filenames. The selector now round-trips through
the resolver.
* `transport.rs::public_model_id` \u2014 prefers the descriptor only
when its identity carries enough information to produce a lossless
id (HuggingFace needs an artifact; Catalog needs a canonical_ref).
Otherwise falls back to the on-disk file, then the model_name
itself \u2014 never silently drops information.
## Validation
`cargo test -p mesh-llm-host-runtime --lib` \u2014 1435/1435 pass.
`cargo test -p model-ref` \u2014 10/10 pass.
Live 2-node mesh (M4 gateway + Studio peer):
`/v1/models` now reports both models with their full ids:
Qwen/Qwen2.5-3B-Instruct-GGUF:q4_k_m
unsloth/Qwen3-0.6B-GGUF:BF16
Direct `/v1/chat/completions` calls with either listed id return
200 and real inference for both local and remote models.
`model: "mesh"` (MoA) still works end-to-end on the same setup,
returning a real fanout response (`Tokyo` from the capital-of-japan
prompt).
`cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings`
\u2014 clean. `cargo fmt --all -- --check` \u2014 clean.
* fix(moa): bump worker/reducer timeouts to 60s for agent-scale prompts
PR #566 review feedback (Apr 2026) flagged that MoA worker accounting
sometimes reported zero successful workers, especially under churn or
load. Investigating from a real 2-node mesh (Mac M4 Max + Mac Studio
M3 Ultra) with an OpenCode agent driving `model: "mesh"` showed
that the root cause was the 15s worker_timeout being too tight for
agent-scale prompts:
* OpenCode's default system prompt is ~13.7k tokens.
* Large strong-tier models (MiniMax-M2.5 Q4_K_M, Qwen3-32B+) at 13k+
prompts + tool schemas take 20\u201340s for a first useful response \u2014
the reasoning preamble alone often eats the 15s budget.
* The MoA gateway killed the strong worker at exactly 15s every turn:
moa: worker unsloth/MiniMax-M2.5-GGUF:Q4_K_M (strong) failed
after 15001ms: remote timeout after 15s
* The arbiter then early-exited on the surviving small worker, never
giving the strong worker a chance to land. The strong worker was
effectively unreachable for OpenCode/Goose-style flows.
Bump both `worker_timeout` and `reducer_timeout` from 15s \u2192 60s
in `build_moa_config`. Live verification on the same 2-node mesh:
* With 15s: `model: mesh` from OpenCode finished 0 of 3 turns
successfully. Every turn returned 1/2 workers, strong worker
timeout, no useful response.
* With 60s: `model: mesh` from OpenCode finished 2 of 3 turns
successfully \u2014 strong worker landed, MoA produced the structured
`tool_calls` field, OpenCode invoked the file-read tool correctly.
(The 3rd turn hit a separate llama_decode / connection-lost issue
in the local stage runtime that is unrelated to MoA timing.)
The trade-off is that a single hung remote worker can stall a turn
for 60s instead of 15s. That is acceptable for an interactive agent
loop where the alternative is consistent failure to land the strong
worker at all. The hedged-reducer ladder (`hedge_delay` = 5s)
still keeps end-to-end latency bounded when only the *reducer* is
slow.
`cargo test -p mesh-llm-host-runtime --lib` \u2014 1435/1435 pass.
`cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings`
\u2014 clean. `cargo fmt --all -- --check` \u2014 clean.
* fix(mesh): enable QUIC keep-alive on mesh transport (was: connections dropping mid-inference)
PR #566 review feedback flagged MoA returning early with 0/N workers
under load. Live debug on a 2-node mesh (M4 Max + Mac Studio M3 Ultra)
running an OpenCode agent against `model: "mesh"` showed repeated:
WARN noq_proto::connection: failed closing path err=LastOpenPath
INFO mesh: Connection to <peer> closed: timed out
WARN moa: reducer ... failed: recv: read error: connection lost
happening 30-60s into otherwise healthy inference calls, including
plain non-MoA `stream: false` requests through `model: "auto"`.
Root cause: noq-proto's default `max_idle_timeout` is 30s and
`keep_alive_interval` is `None` (the spec, RFC 9000 §10.1.2, makes
keep-alive opt-in; quinn / noq follow that). Non-streaming inference
requests send no application bytes while the remote model is
generating tokens, so the wire is idle. Under concurrent load
(parallel MoA workers + reducer + gossip + heartbeats), noq's
multipath bookkeeping closes the idle path, and when it is the last
open path the entire connection drops mid-stream. The in-flight HTTP
tunnel errors with `connection lost` and the caller must retry.
This only became visible recently because:
* Streaming OpenAI clients (Goose, Claude Code, pi, the web UI) all
set `stream: true` by default. SSE chunks flow continuously and
reset the idle timer, so the bug never manifests for them.
* MoA `RemoteModelBackend` is the first significant non-streaming
long-running RPC in the codebase (`stream: false` hardcoded in
`crates/mesh-mixture-of-agents/src/backend.rs`).
* Reasoning models with big agent prompts (MiniMax-M2.5 on a 13k
OpenCode system prompt, Qwen3-32B class reducers) routinely take
30-90s for a first useful response. That is the combination that
exceeds the default 30s idle window.
Fix: set `keep_alive_interval = 10s` and `max_idle_timeout = 5m`
on the mesh QUIC transport config, plus the matching multipath
`default_path_keep_alive_interval` and
`default_path_max_idle_timeout` so individual paths don't get torn
down while the connection-level idle timer is fine.
Cost: one QUIC PING (~30-60 bytes) every 10s per connection only
when no other application data has been sent for that long. In a
typical mesh with periodic gossip and heartbeats this fires rarely.
`keep_alive` is opportunistic, not unconditional.
Live verification on the same 2-node mesh:
* 60s idle test, before fix: 2x `Connection to <peer> closed:
timed out`. After fix: 0x. Connection stays healthy.
* 75s of mixed non-streaming inference (53s `auto` to MiniMax +
21s `mesh` 2-worker fanout), before fix: multiple `LastOpenPath`
+ `connection lost` errors. After fix: 0x. Both completed
successfully with finish_reason=stop and full content.
* OpenCode `model: mesh` agent loop, before fix: 0 of 2 turns
landed. After fix: 2 of 3 turns landed (the 3rd hit a separate
KV cache exhaustion in the local stage runtime, tracked
independently).
Validation: `cargo fmt --all -- --check` clean,
`cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings`
clean, `cargo test -p mesh-llm-host-runtime --lib` 1435/1435 pass.
* fix(planner): cap auto lane count to llama-server's 4-lane unified-KV default
PR #566 review feedback uncovered a hard 502 from the embedded skippy
stage runtime under concurrent agent-style workloads. On a Mac M4 Max
serving Qwen3-8B at the model's native 32k context, the auto planner
was picking `slots = 16` (MAX_AUTO_PARALLEL_SLOTS). Three concurrent
~14k-token requests \u2014 the exact shape an OpenCode agent loop produces
when MoA fans a worker call out next to a reducer call \u2014 fail in the
embedded llama with:
decode: failed to find a memory slot for batch of size 2048
surfacing as HTTP 502:
skippy ABI call failed: RuntimeError: llama_decode failed
Root cause: skippy's stage runtime sets `kv_unified = true` whenever
`lane_count > 1` (`third_party/llama.cpp/patches/0034-Add-shared-execution-lanes-to-skippy-ABI.patch`).
In unified mode llama allocates exactly `n_ctx` cells total, shared
across all `n_seq_max` sequences. The previous planner derived
`slots` from VRAM as if each lane carved off its own
`n_ctx \u00d7 bytes_per_token` allocation \u2014 which is the
`kv_unified = false` semantics, not what skippy actually does. On a
node with comfortable VRAM the math happily returned the snapped
maximum of 16 lanes, even though all 16 raced for the *same* fixed
pool of `n_ctx` cells.
Fix: drop `MAX_AUTO_PARALLEL_SLOTS` from 16 to 4, matching upstream
llama-server's own auto default for the same reason. From
`.deps/llama.cpp/tools/server/server.cpp`:
LOG_INF("n_parallel is set to auto, using n_parallel = 4 and
kv_unified = true");
params.n_parallel = 4;
params.kv_unified = true;
Lane count is purely a concurrency-policy knob under `kv_unified =
true`; it does not change the KV cache allocation. Going from 16 to
4 frees zero RAM; it just gates admission control to a sane number
of concurrent in-flight requests for the shared cell pool.
Operators who know their workload (short chat turns, low-concurrency
hosts, etc.) can still pick a higher value via the existing
`parallel_override` plumbing, including `[models.throughput]
parallel = N` in the TOML config from PR #564.
Live verification on the same 2-node mesh used to find the bug:
* M4 + Qwen3-8B at 32k `n_ctx`: planner now picks `slots = 4`,
llama logs `n_seq_max = 4`, KV cache stays at 2448 MiB (one
shared buffer; no RAM cost change).
* Studio + MiniMax-M2.5 at 128k `n_ctx`: planner now picks
`slots = 4`, llama logs `n_seq_max = 4`, KV cache stays at 8928
MiB. 4 \u00d7 32k cells per lane on average is plenty of headroom for
agent prompts.
* Repro that previously 502'd \u2014 3 parallel ~15k-prompt tool-result
follow-ups on the M4 \u2014 now all succeed with `finish_reason=stop`,
full content, ~20s wall time. Zero `find_slot` failures, zero
`llama_decode` errors, zero skippy ABI errors.
* Burst test \u2014 5 parallel at the same prompt shape \u2014 the 5th
request correctly hits the admission-control queue and returns a
clean
`{"type":"rate_limit_error","code":"rate_limit_exceeded"}`
after the admission timeout, instead of an opaque mid-flight 502.
Adds two regression tests in `context_planning::tests`:
* `auto_slots_capped_at_llama_server_default` covers the
high-VRAM small-model case that used to plan 16.
* `explicit_parallel_can_exceed_auto_ceiling` covers the
override path so operators retain control.
Validation: `cargo fmt --all -- --check` clean,
`cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings`
clean, `cargo test -p mesh-llm-host-runtime --lib` 1437/1437 pass
(includes the two new regression tests).
* docs(moa): report on micn/moa branch's critical fixes beyond MoA itself
While iterating on PR #566 review feedback, this branch surfaced and
fixed several pre-existing host-runtime bugs that were either hard to
hit before or silently masked by KV-leakage bugs that have since been
fixed. Capture the full picture in one place so PR reviewers and
future readers don't have to spelunk through 60+ commits to see what
landed.
Headline fixes documented:
1. Mesh QUIC keep-alive (
|
||
|
Nick DiZazzo
|
34a63446e0
|
chore(version): synchronize version bump everywhere (#562)
* chore(version): synchronize version bump everywhere * make version a shared thing so it updates in 1 spot --------- Co-authored-by: Michael Neale <michael.neale@gmail.com> |
||
|
Michael Neale
|
cb9472ee26
|
build: ozempic — slim binary -42 MB / -47 MB (#592)
* ui: strip bundled ONNX WASM, lazy-load via CDN
The @huggingface/transformers image-describe path was pulling ~21 MB of
ONNX Runtime WASM into dist/ via onnxruntime-web's
new URL('ort-wasm-simd-threaded.jsep.wasm', import.meta.url)
literals. At runtime Transformers.js already defaults wasmPaths to
jsDelivr's CDN, so the bundled copy was pure dead weight inside the
embedded UI blob that ships in the mesh-llm binary.
Changes:
- Vite plugin stripBundledOnnxWasm: transforms the offending literals
in onnxruntime-web modules so Rollup never emits the .wasm asset.
- image-describe.ts: pin the CDN URL to the @huggingface/transformers
version in package.json so the runtime fetches the matching
wasm/jsep artifacts when a user attaches an image.
- The Transformers.js JS bundle (~830 KB) stays dynamic-imported as
before, so it is still a separate chunk only loaded on first use.
dist/ size: 26 MB → 5.7 MB (-20.3 MB).
UI test sweep: 637 passed, 3 skipped (no regressions).
* build: tighten release profile (thin LTO, cg=1, strip debuginfo)
Cargo defaults for [profile.release] are conservative: no LTO, 16
codegen units, no symbol stripping. For a shipped binary that's
several MB of dead weight.
- lto = "thin": parallel link-time optimization. ~80% of fat-LTO's
benefit at ~30% of the link cost. Fat LTO would push release link
from ~30 s to several minutes.
- codegen-units = 1: pairs with thin LTO so the optimizer can inline
across the whole crate without re-link boundaries.
- strip = "debuginfo": drops DWARF from the Mach-O / ELF output.
Function symbols stay intact so backtraces from production nodes
remain readable (addr2line / nm on the symbol table still works).
panic stays at the default 'unwind' per AGENTS.md — plugin / MCP
error recovery relies on unwinding.
macOS arm64 binary: 106 MB → 64 MB (combined with the UI WASM strip
in the previous commit; -42 MB total).
Release build time: ~30 s → ~4 min on M4 Max. CI release builds
already take many minutes for codesigning/bundling so this is well
within budget. Dev builds (which 'just build' uses by default) are
unchanged.
* build: add web-ui feature flag (default on) for lib-friendly builds
Adds a default-on `web-ui` Cargo feature that propagates through:
mesh-llm (binary)
-> mesh-llm-host-runtime
-> mesh-llm-ui (`embed-assets` feature)
When on (default), the React console is embedded into the binary via
`include_dir!` exactly as before. When off, `mesh-llm-ui::asset()` and
`::index()` return `None`, the `include_dir!` invocation is elided,
and the crate no longer depends on `include_dir`. The asset HTTP
responders in mesh-llm-host-runtime route 404s for asset paths but
every other management-API surface keeps working.
The default-features = false path is intended for future lib-style
consumers of `mesh-llm-host-runtime` who want the runtime/networking
without the embedded web payload.
Binary size on macOS arm64:
- web-ui on (default): 64 MB
- web-ui off: 59 MB
Combined with the UI WASM strip and tightened release profile from
the previous two commits:
- main: 106 MB
- micn/ozempic with web-ui on (default): 64 MB (-42 MB)
- micn/ozempic with --no-default-features: 59 MB (-47 MB)
Tests pass in both feature configurations:
- mesh-llm-ui: 2 tests (embed-assets), 1 test (stub)
- mesh-llm-host-runtime: --no-default-features --lib checks clean
The fmt + clippy / unused warning surface stays clean because the
content-type / cache-control helpers and the tests are gated to the
embed-assets feature.
|
||
|
Nick DiZazzo
|
5e556cfa6b
|
chore(code-quality): Set max 200 line limit for long methods (#595) | ||
|
Nick DiZazzo
|
6e94ab8483
|
chore(cleanup): lots of fixes for cyclomatic complexity, test warnings, bugs (#584) | ||
|
James Dumay
|
57cdac9881
|
Report backend GPUs and bundle GPU benchmarks (#509)
* Expose skippy backend device survey * Bundle GPU benchmark backends * Fix CI benchmark wiring * Allow manual Docker CI runs * Restore CPU-only runtime capacity * Fix CUDA loading issue on Linux * add debug information to help trace issues on CUDA devices with two GPUs * fix linters / tests * fixups for remaining tests via mutex * fix ROCm build * coupel more fixes for windows * debugging on ci is cool * fix remaining red tests after rebase --------- Co-authored-by: Nick DiZazzo <nick.dizazzo@gmail.com> |
||
|
Michael Neale
|
d136df1d18
|
Improve context planning, admission, and coordinator fencing (#513)
* feat: split-aware context planning with KV quant negotiation Context planning now produces useful context windows for split models instead of falling back to 4096. Split-aware budget: the planner now accepts a local_layer_fraction so it can compute the KV cache cost for just this node's layers, not the whole model. For layer packages, the fraction is estimated from the VRAM ratio (local / total mesh VRAM). KV quant negotiation: when the requested KV quantisation (e.g. f16) cannot reach the model's native context length, the planner walks a quant ladder (f16 → q8_0 → q4_0) and picks the least aggressive quant that fits. The negotiated quant is applied to the stage load request automatically. Layer package metadata: for split models, the planner now reads GGUF architecture metadata from the layer package's shared/metadata.gguf instead of returning None (which caused a fallback to 4096 default). Also fixes 13 pre-existing compile errors in mesh-llm-host-runtime test code (missing latency fields from #491, wrong function names and stale struct fields from #485). * fix: repair broken host-runtime tests and add to CI Fix 13 compile errors and 3 test failures in mesh-llm-host-runtime that were silently broken on main (CI only ran mesh-llm --lib which has zero tests). Compile fixes: - Add missing latency fields (latency_ms, latency_source, latency_age_ms, latency_observer_id) to PeerAnnouncement test constructions in protocol/mod.rs (#491 missed these sites) - Fix test_endpoint_id → make_test_endpoint_id in mesh/tests.rs (#485 used wrong function name) - Update ServedModelDescriptor to current struct shape (capabilities + topology instead of format + quantization + size_bytes) - Add missing available_model_sizes field Test fixes: - gossip_frame_roundtrip_preserves_scanned_model_metadata: add ModelRuntimeDescriptor with context_length to served_model_runtime (was empty vec, then asserted on first element) - initial_pretty_session_mode: update expectation to match current implementation (Client surface now allows dashboard) - Remove broken timing-dependent streaming proxy test (covered by two other streaming tests that pass) - Mark HF download test as #[ignore] (downloads 800MB, needs auth) CI: - Add cargo test -p mesh-llm-host-runtime --lib to both Linux and macOS CI jobs - Add cargo test -p model-artifact --lib to both jobs * fix: respect user KV quant override, avoid blocking async, tighten test assertions Address Copilot review feedback: - Skip KV quant negotiation when the user explicitly set --cache-type-k or --cache-type-v. Previously the planner would negotiate to q4_0 for a larger context, but the downstream load honoured the user's f16 override — producing a context/memory mismatch. New kv_quant_user_locked flag prevents this. - Wrap scan_layer_package_metadata in spawn_blocking to avoid filesystem I/O on the async executor (GGUF header reads, stat calls). - Tighten negotiate_kv_quant_upgrades_to_reach_native_context assertion to check exact expected value (16K) instead of just > 8K. - Add user_locked_kv_quant_skips_negotiation test proving the lock prevents negotiation and produces a smaller context than unlocked. * Add split coordinator fencing * simplify: remove KV quant negotiation, universal Q8_0 default Drop the tiered KvCachePolicy (f16/q8_0/q4_0 by model size) and the negotiation ladder in context_planning. KV cache is now Q8_0 everywhere unless the user explicitly sets --cache-type-k/v. The planner just does: VRAM budget ÷ per-token KV cost → context length. No tiers, no negotiation, no negotiated_kv_quant, no kv_quant_user_locked. Split path now runs the same planner (was hardcoded to 4096). -216 lines net. * fix: rename test to match actual behavior (copilot review) * fix: solo load path no longer scales by peer VRAM The local load path (start_runtime_local_model) incorrectly computed a fractional layer share based on mesh VRAM ratio when loading layer-package models. Since this path loads the entire model on one node, the fraction should always be 1.0 — fractional scaling only applies in the split path. This could overestimate free VRAM and plan a context window larger than actually fits. Also: fallback on invalid --cache-type-k/v now defaults to Q8_0 (was f16), remove dead total_peer_vram_bytes(), fix stale doc comment. * Bound skippy OpenAI generation admission * Co-plan context and skippy concurrency * Revert "Co-plan context and skippy concurrency" This reverts commit |
||
|
Nick DiZazzo
|
59275d2929 | fix(debug-model-loading): fix extremely long model loading times in debug builds | ||
|
Michael Neale
|
52267b3268
|
feat: huggingface split job for skippy individually downloadable layers (#455)
* the plan
* first cut and building in a model splitter tool
* fix: add model-prepare crate to Docker COPY stages
* fly: add missing skippy-cache COPY to Dockerfile
skippy-cache is a workspace member depended on transitively via
skippy-server. Without it, cargo metadata fails on clean builds.
* model-prepare: address copilot review feedback
- Defer HfJobsClient creation so quant listing works without HF_TOKEN
- Fix stale docstring on from_env()
- Use per-PID temp dirs to avoid race conditions
- Update script header comment to reflect job spec entrypoint
* address copilot feedback: buffer SSE lines, fix step counters, clarify ref help
* fix clippy: refactor dispatch_model_prepare to use args struct
* fix catalog schema and variant keying (i386 P1 feedback)
- New catalog entries now include source_repo and use dict-style variants
- Variant key derived from source file stem, not MODEL_ID
- Strip shard suffix from variant name (-00001-of-00002)
- --logs warns when job is still running (won't silently block)
- Fixed broken Qwen3-8B catalog entry in meshllm/catalog
* fix: skip NCCL linking on driverless CI (LLAMA_STAGE_SKIP_NCCL)
* Revert "fix: skip NCCL linking on driverless CI (LLAMA_STAGE_SKIP_NCCL)"
This reverts commit
|
||
|
James Dumay
|
e6b723d199
|
mesh-llm crate decomposition (#459) | ||
|
Nick DiZazzo
|
4b73d57b5c
|
feat: auto-resolve layer packages from meshllm/catalog rebased (#452)
* feat: auto-resolve layer packages from meshllm/catalog When --split is active, the runtime now checks the remote meshllm/catalog HF dataset for a pre-split layer package before requiring the full GGUF on disk. If a layer package exists (e.g. meshllm/Qwen3-Coder-480B-...-layers), each node downloads only its assigned layers instead of the entire model. Flow: 1. start_runtime_split_model checks resolve_layer_package_ref() 2. If found, calls identity_from_layer_package() which downloads only the manifest JSON (~1KB) to get layer count, activation width, etc. 3. Topology planning proceeds as normal 4. Each node's stage materialization fetches only its layer range New module: models/remote_catalog.rs - Fetches meshllm/catalog dataset from HF (cached 24h) - Fuzzy matches model names to layer-package entries - Falls back gracefully if catalog unavailable Catalog entries added to meshllm/catalog HF dataset: - meshllm/Qwen3-Coder-480B-A35B-Instruct-UD-Q4_K_XL-layers (62 layers) - meshllm/DeepSeek-V3.2-UD-Q4_K_XL-layers (61 layers) - meshllm/Qwen3-235B-A22B-UD-Q4_K_XL-layers (94 layers) * add model resolver from @i386 suggest implementation notes * address PR comments and fixes for tests * How about now? * last minor fixup --------- Co-authored-by: Michael Neale <michael.neale@gmail.com> Co-authored-by: James Dumay <jameswdumay@gmail.com> |
||
|
James Dumay
|
44f7da867c
|
Add Skippy cache, benchmark evidence, and prompt defaults (#430) | ||
|
James Dumay
|
5454baac0b
|
Replace llama-server with skippy runtime (#422) | ||
|
James Dumay
|
330a3381ae
|
Prepare repo layout for skippy merges (#420)
* Prepare workspace for skippy integration * Copy skippy model crates * Use skippy model crates in resolver * Add mesh compatibility tests for model crates |
||
|
James Dumay
|
31e9ae78c3 |
Merge remote-tracking branch 'origin/main' into HEAD
# Conflicts: # Cargo.toml # mesh-llm/src/models/capabilities.rs |
||
|
Nick DiZazzo
|
02bc816097
|
chore(architecture-normalization): make architecture naming consistent in the project, add checks (#274) | ||
|
James Dumay
|
553c811463 | Add live SDK smoke coverage and disable mac CI runners | ||
|
Nick DiZazzo
|
835276979a
|
feature: add embedded SDK support
- extract a native embedded client SDK with Rust core, FFI bindings, and test harness support - add Swift and Kotlin wrappers, packaging, smoke tests, and protocol compatibility coverage - target iOS, macOS, and Android embedded-client flows without localhost sidecars |
||
|
James Dumay
|
14d3856da8 | Refine plugin DSL and internal RPC plugins | ||
|
James Dumay
|
7e2d4855bf | Move Cargo workspace root to the repository root |