mirror of
https://github.com/Mesh-LLM/mesh-llm.git
synced 2026-08-08 22:23:19 -04:00
2c5dacf212
* Replace Skippy verify span with verify windows * Add verify window reply metadata * Pipeline direct-return n-gram verify windows * Pipeline MTP-anchored n-gram verify windows * Support static release builds without features * Fix split MTP activation-frame serving * Replace native MTP batched verifier with verify windows * Restore native MTP verify window batching * Replace native MTP anchor extension with composite proposals * Keep composite decode branch on development version * Expose decode timings for all generation modes * Retry transient staged lane readiness * Bound persistent lane readiness handshake * Keep pure N-gram decode free of MTP drafts * Report composite proposal totals in decode timings * Gate composite decode pipeline by candidate depth * Account direct GGUF MTP weights in split planning * Avoid MTP cooldown after N-gram tail rejection * Improve hybrid MTP verification telemetry * Pipeline native MTP verification replies * Require useful N-gram tails for hybrid MTP * Adapt N-gram MTP extensions to tail acceptance * Fix direct GGUF planning fallback * Gate N-gram tails on MTP prefix agreement * Widen initial async verify windows * Restore anchored N-gram MTP extensions * Retain ready stages across transient refresh failures * Document pipelined VerifyWindow decode * Use llama.cpp N-gram proposer for Skippy * Add cache-based N-gram proposer * Add declarative speculative proposer package schema * Productize Skippy speculative decode plans * Productize Skippy speculative decode plans * Support direct N-gram Skippy plans * Validate speculative package strategy plans * Add coding agent loop benchmark corpus * Expose Skippy speculative benchmark counters * Validate cache N-gram proposer limits * Document speculative decode configuration * Fix native MTP proposals and fused restore routing * Honor configured N-gram extension width * Document speculative runtime overrides * Refresh speculative config schema contracts * Keep N-gram tail rejects from penalizing MTP * Preserve MTP state after serial tail rejects * Report adaptive verify width changes accurately * Fix short simple N-gram extension budgets * Make VerifyWindow pipelining cost-aware * Profile prospective VerifyWindow widths * docs: WAN split performance model + measured latency/compute decomposition Adds docs/skippy/WAN_SPLIT_PERF.md: the single-stream per-token cost model (TPOT ~= C_total + (S-1)*2*RTT + (S-1)*P), compute-bound vs latency-bound criteria, when adding a stage helps (memory, concurrency/pipeline overlap, dense compute-bound models), and speculation as the WAN amortization lever. Backed by 2026-07-18 Sydney<->Melbourne 2-node measurements: solo 12.9 ms/tok compute, split 57.8 ms/tok, decomposing to 12.9 compute + 40 (2xRTT) + 4.9 protocol. Workload was latency-bound (~78% network). * docs: plan for fast-fail on new requests routed to a dead split stage Documents the measured ~30s hang when a new request routes to a killed split stage, the confirmed root cause (60s heartbeat / lenient failure threshold + slow lane-open timeouts), and a two-layer fix (short steady-state lane-open deadline; feed lane failures into target_health cooldown) plus an explicit validation gate. Mesh-timing changes are out of scope pending live multi-node validation. * Fast-fail lane reconnects so new requests don't hang on a dead split stage When a downstream split stage dies, a new request would open a fresh lane and wait the full ~20s warmup ready-deadline before erroring (observed as a ~30s hang in the Sydney<->Melbourne kill test). The 20s deadline is only needed during pool warmup, when the downstream may still be loading its model. Split the deadline: pool warmup keeps LANE_READY_READ_TIMEOUT (20s); mid-life reconnects from checkout()/replace_lane() on an already-serving mesh use a new LANE_STEADY_CONNECT_TIMEOUT (3s). A healthy peer answers in milliseconds, so a dead stage now fails in ~3s instead of ~30s. Restores receive_persistent_lane_ready as the shared bounded-handshake helper (dropped during the main merge) and removes a now-obsolete retry test that covered pre-#1011 retry behavior. Adds tests asserting the steady-state deadline stays well under the warmup deadline and that the handshake read fails fast on a silent downstream. * docs: latency-aware placement — current behaviour and many-node gaps Records verified planner behaviour (skippy-coordinator/topology.rs, skippy-topology, host-runtime call site): - latency is a placement cost (rtt_ms penalty), not just relay-only exclusion - planner selects a node subset; does not have to use every eligible node - stage count is gated on a decode-TPOT target (shallower-that-meets beats deeper-that-does-not) And the gaps that matter at many-node scale: - no peer-to-peer RTT matrix in production (edge_signals never wired; only coordinator-RTT is used) -> co-located nodes cannot be exploited - network estimate is max(coordinator RTT) x node_count, a worst-case proxy - no first-class prefer-fewer/never-place-above-Y policy beyond the TPOT gate * docs: measured speculative recovery cost over WAN (why ngram hurts a latency-bound split) * Discard dead pooled stage lanes before reuse (fast-fail improvement) A pooled downstream lane whose stage died while checked in was a dead TCP stream; reusing it blocked the next generation read forever (handshake read-timeout is cleared for pooled lanes so long generations don't truncate). checkout() now probes lane liveness with a nonblocking peek and discards a dead lane so it reconnects with the short steady-state deadline instead of hanging. Validated on a loopback 2-node split with a mid-flight worker kill: new request now fails faster than main (60s vs main's 90s baseline). Does not fully solve the recovered-local routing path, tracked as follow-up. --------- Co-authored-by: Michael Neale <14976+michaelneale@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| design | ||
| plans | ||
| plugins | ||
| sdk | ||
| skippy | ||
| specs | ||
| AGENTS.md | ||
| BENCHMARKS.md | ||
| CLI.md | ||
| CODESIGNING.md | ||
| cuda-release-lanes.md | ||
| EXO_COMPARISON.md | ||
| LAYER_PACKAGE_REPOS.md | ||
| mesh-llm-wordmark.png | ||
| mesh.png | ||
| MESHES.md | ||
| NODE_REP.md | ||
| README.md | ||
| SDK.md | ||
| setup-mesh | ||
| SKIPPY.md | ||
| SKIPPY_SPLITS.md | ||
| SWARM_CAPTURE.md | ||
| USAGE.md | ||
Documentation
Use this hub to find project guides that are not owned by a single Rust crate.
Start here
| Need | Doc |
|---|---|
| Install, run, service mode, model storage | USAGE.md |
| Private meshes, published meshes, public joining | MESHES.md |
| Local routing reputation and target cooldowns | NODE_REP.md |
| SDK usage, examples, errors, lifecycle, platform support | SDK.md |
| Language-specific SDK examples | Rust, Node.js, Swift, Kotlin/Android |
| Run big models with Skippy layer splits | SKIPPY_SPLITS.md |
| Contribute or publish layer package repositories | LAYER_PACKAGE_REPOS.md |
| Goose, Claude Code, OpenCode, Pi, curl, blackboard | AGENTS.md |
| Command-by-command CLI reference | CLI.md |
| Exo comparison | EXO_COMPARISON.md |
Skippy and model-package docs
| Doc | What it covers |
|---|---|
| skippy/FAMILY_STATUS.md | Certified family/split/wire-dtype status |
| skippy/NEW_MODEL_ONBOARDING.md | New-model split/certification intake checklist |
| skippy/FAMILY_CERTIFY.md | Certification workflow for new families |
| skippy/TOPOLOGY_PLANNER.md | Stage topology planning behavior |
| skippy/CONFIGURATION.md | Authoritative operator matrix for Skippy config keys and rejection boundaries |
| skippy/PROMPT_CACHE.md | OpenAI prompt-prefix cache behavior, defaults, telemetry, and benchmark flow |
| skippy/PIPELINED_VERIFY_WINDOW.md | Native MTP, anchored N-gram extension, VerifyWindow protocol, pipeline behavior, and telemetry |
| skippy/DATA_FLOW.md | Stage data flow and transport details |
| skippy/LLAMA_PARITY.md | Remaining llama.cpp parity queue |
| specs/layer-package-repos.md | Manifest schema and package artifact rules |
| specs/mesh-setup-installer.md | Bootstrap installer and mesh-llm setup ownership boundary |
| SKIPPY.md | Skippy integration readiness and parity notes |
Use SKIPPY_SPLITS.md for Skippy split-serving workflows.
Other references
| Doc or directory | What belongs there |
|---|---|
| BENCHMARKS.md | Current benchmark numbers and performance context |
| SWARM_CAPTURE.md | Opt-in local debug capture for mesh membership and connection diagnostics |
| design/ | Architecture notes, protocol design, testing playbooks, carried llama.cpp patch documentation |
| design/NATIVE_RUNTIMES.md | Native runtime artifact packaging, exact version matching, resolver behavior, and SDK/autoupdater ownership |
| design/NODE_OWNER_IDENTITY.md | Owner identity, trust policy, and how owner trust stays separate from release attestation |
| design/EMITTER_HOOKS.md | Inventory of hook, callback, and emitter surfaces plus readiness ownership. |
| plugins/ | Plugin architecture, web UI projection contract, exemplars, and implementation planning |
| plans/ | Narrow implementation plans that are not yet general design docs |
| specs/ | Focused behavior specs for individual features |
| design/OPENAI_GUARDRAILS.md | OpenAI guardrail rollout defaults, v1 limits, telemetry privacy, and evidence scaffolding |
Per-crate docs stay with their crates. The main binary crate overview lives at ../crates/mesh-llm/README.md, and the web console/embedded asset crate overview lives at ../crates/mesh-llm-ui/README.md. Shared protocol-facing model/type ownership lives at ../crates/mesh-llm-types/README.md. Shared owner identity and envelope crypto lives at ../crates/mesh-llm-identity/README.md. Shared wire protocol ownership lives at ../crates/mesh-llm-protocol/README.md. Shared routing target ownership lives at ../crates/mesh-llm-routing/README.md.