legion-ref/mesh-llm
1
0
Fork 0
mirror of https://github.com/Mesh-LLM/mesh-llm.git synced 2026-08-08 22:23:19 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
mesh-llm/ROADMAP.md
Ivan Golovach 06cad32feb
Upgrade demanded local models during reconciliation (#754) 
Upgrade demanded local models during reconciliation

Validation
* Validation tier: Tier 3 - opt-in runtime reconciliation can replace a lower-demand local model with a locally present, fresh active-demand target; config admission, planner policy, runtime unload/load sequencing, API target signals, and docs are refreshed on current main after PR #753 landed.
* git fetch --no-tags origin main:refs/remotes/origin/main: PASS, origin/main at c0f8990bb1.
* git rebase origin/main: PASS after resolving one test-module conflict in crates/mesh-llm-host-runtime/src/runtime/mod.rs by preserving both the newly landed mDNS relay-policy tests and this PR's model-target replacement sequencing test.
* git diff --check origin/main...HEAD: PASS, no output.
* git diff --check: PASS, no output.
* git diff --cached --check: PASS, no output.
* cargo fmt --all -- --check: PASS.
* cargo test -p mesh-llm-config --lib -- --test-threads=1: PASS, 10 passed.
* LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo test -p mesh-llm-host-runtime model_target_reconciliation --lib -- --test-threads=1: PASS, 22 passed.
* LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo test -p mesh-llm-host-runtime api::model_targets --lib -- --test-threads=1: PASS, 3 passed.
* LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo test -p mesh-llm-host-runtime mdns_discovery --lib -- --test-threads=1: PASS, 3 passed.
* LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo check -p mesh-llm: PASS.
* LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo clippy -p mesh-llm-config -p mesh-llm-host-runtime --all-targets -- -D warnings: PASS.
* Remote CI: PASS on final PR #754 SHA d48861a2, including PR Quality Checks, Linux/macOS builds, fmt, clippy, unit/protocol/skippy/sdk smoke, client-auto boot, two-node client-serving smoke, and two-node split smoke.
* Ledger: not applicable - not required for selected validation tier/change family.
* Version: not applicable - no release/version sync required for this non-release opt-in runtime reconciliation change.
* Not run: live multi-node demand/reconciliation smoke - no local multi-node runtime endpoint and local GGUF set were available; config, model-target signal, planner, runtime-control, conflict-adjacent mDNS tests, and remote CI cover the changed branches.

Rollback
* git revert HEAD
2026-05-30 15:00:35 -07:00

4.3 KiB
Raw Permalink Blame History

Roadmap

High-level directions for mesh-llm. Not promises — just things we're thinking about, or have thought about.

Smart model router ✅

Implemented. Heuristic classifier detects Code/Reasoning/Chat/Creative/ToolCall with Quick/Moderate/Deep complexity. Task-dominant scoring ensures the right model handles each request. Tool capability is a hard filter. Multi-model per node with auto packs by VRAM tier. Auto-fallback ladders walk to the next-best model when the top pick's peers are unhealthy.

Mixture of Agents (MoA) ✅

Implemented as the mesh virtual model. Fan-out across multiple worker models on the mesh, reducer synthesizes the result. Streaming output, tool-call passthrough, opinionated no-think default, configurable first-answer grace. See docs/design/MOA_GATEWAY.md.

This could do with ongoing development and benchmarking to improve.

Mobile chat app (exemplar)

A native mobile app that joins a mesh by scanning a QR code. Client-only — no GPU, no model serving. Just a beautiful chat interface backed by the mesh's GPU pool.

  • Scan QR code → join mesh → chat with any model the mesh serves
  • Uses iroh relay for connectivity (works through NAT, cellular, WiFi)
  • OpenAI-compatible API underneath (same as any mesh client)
  • iOS first (Swift + iroh-ffi), Android follow-up
  • "AirDrop for AI" — one scan and you're talking to a 235B parameter model

This is the best way to show what mesh-llm does: zero setup, zero config, just scan and chat.

Multimodal

Vision, audio, and image generation/editing routed across the mesh. Capability advertisement gossiped so requests find compatible peers automatically. See docs/design/MULTI_MODAL.md.

Done:

  • Vision input on capable models (Qwen3-VL, MiniMax-M2.5, etc.)
  • Audio input (transcription, multimodal audio understanding)
  • Capability-aware routing — image/audio requests only go to peers that advertise the capability
  • Blob plugin for request-scoped media storage

Wanted:

  • Image generation models (SDXL, FLUX, etc.) as first-class mesh peers — same gossip + capability + routing story, just emits PNG bytes instead of tokens
  • Image editing / inpainting — accept an input image + mask + prompt, return edited image
  • Audio generation (TTS) as a peer role
  • Video generation as a future peer role

The goal is "every modality is just another model behind the mesh's OpenAI-compatible facade." Same QR-code-to-join story works for image-gen as for chat.

Speculative decoding

Verify draft tokens against the target model to accelerate generation. Experimental, opt-in. See PR #567. More work around using ngrams, drafting models is in progress. Some experimental work around predictive prompt completion into prefile has been done (yet to be proven, prefil is parallel so latency tolerant) MTP work with llama.cpp ongoing, but should be part of this to accelerate inference and especially reduce vulnerability to latency between layers.

Demand-based rebalancing

Partially done. Unified demand map via gossip, standby nodes promote to serve, and large-VRAM hosts can opt into fresh active-demand upgrades for local artifacts. Next: download-backed upgrades, split-aware upgrades, and replica-count balancing.

Blackboard ✅

Blackboard is moving out to its own plugin repository. The mesh-llm host keeps the generic plugin transport and CLI dispatch; blackboard installs through the plugin manager and owns its own CLI/MCP surface there.

MoE expert sharding ✅

Implemented. Auto-detects MoE, computes overlapping expert assignments, splits locally, and uses session-sticky routing with zero cross-node expert traffic. Best thought of as experimental, most results show this doesn't perform as well as one would hope, more research is needed to see if expert sharding this way is actually practical.

Platform targetting, Desktop apps, embedding of mesh SDK, distribution

Mesh is packaged for many platforms, and can be run as a background process, but it would make sense to have desktop/GUI apps which host it in way that can offer utility to end consumer utility as well as being able to yield compute when needed locally.

Mesh also needs to be packaged as an SDK which can be used from various client languages to launch as a client/serve node as seamlessly as possible.