mirror of
https://github.com/Mesh-LLM/mesh-llm.git
synced 2026-08-08 22:23:19 -04:00
2c5dacf212
* Replace Skippy verify span with verify windows * Add verify window reply metadata * Pipeline direct-return n-gram verify windows * Pipeline MTP-anchored n-gram verify windows * Support static release builds without features * Fix split MTP activation-frame serving * Replace native MTP batched verifier with verify windows * Restore native MTP verify window batching * Replace native MTP anchor extension with composite proposals * Keep composite decode branch on development version * Expose decode timings for all generation modes * Retry transient staged lane readiness * Bound persistent lane readiness handshake * Keep pure N-gram decode free of MTP drafts * Report composite proposal totals in decode timings * Gate composite decode pipeline by candidate depth * Account direct GGUF MTP weights in split planning * Avoid MTP cooldown after N-gram tail rejection * Improve hybrid MTP verification telemetry * Pipeline native MTP verification replies * Require useful N-gram tails for hybrid MTP * Adapt N-gram MTP extensions to tail acceptance * Fix direct GGUF planning fallback * Gate N-gram tails on MTP prefix agreement * Widen initial async verify windows * Restore anchored N-gram MTP extensions * Retain ready stages across transient refresh failures * Document pipelined VerifyWindow decode * Use llama.cpp N-gram proposer for Skippy * Add cache-based N-gram proposer * Add declarative speculative proposer package schema * Productize Skippy speculative decode plans * Productize Skippy speculative decode plans * Support direct N-gram Skippy plans * Validate speculative package strategy plans * Add coding agent loop benchmark corpus * Expose Skippy speculative benchmark counters * Validate cache N-gram proposer limits * Document speculative decode configuration * Fix native MTP proposals and fused restore routing * Honor configured N-gram extension width * Document speculative runtime overrides * Refresh speculative config schema contracts * Keep N-gram tail rejects from penalizing MTP * Preserve MTP state after serial tail rejects * Report adaptive verify width changes accurately * Fix short simple N-gram extension budgets * Make VerifyWindow pipelining cost-aware * Profile prospective VerifyWindow widths * docs: WAN split performance model + measured latency/compute decomposition Adds docs/skippy/WAN_SPLIT_PERF.md: the single-stream per-token cost model (TPOT ~= C_total + (S-1)*2*RTT + (S-1)*P), compute-bound vs latency-bound criteria, when adding a stage helps (memory, concurrency/pipeline overlap, dense compute-bound models), and speculation as the WAN amortization lever. Backed by 2026-07-18 Sydney<->Melbourne 2-node measurements: solo 12.9 ms/tok compute, split 57.8 ms/tok, decomposing to 12.9 compute + 40 (2xRTT) + 4.9 protocol. Workload was latency-bound (~78% network). * docs: plan for fast-fail on new requests routed to a dead split stage Documents the measured ~30s hang when a new request routes to a killed split stage, the confirmed root cause (60s heartbeat / lenient failure threshold + slow lane-open timeouts), and a two-layer fix (short steady-state lane-open deadline; feed lane failures into target_health cooldown) plus an explicit validation gate. Mesh-timing changes are out of scope pending live multi-node validation. * Fast-fail lane reconnects so new requests don't hang on a dead split stage When a downstream split stage dies, a new request would open a fresh lane and wait the full ~20s warmup ready-deadline before erroring (observed as a ~30s hang in the Sydney<->Melbourne kill test). The 20s deadline is only needed during pool warmup, when the downstream may still be loading its model. Split the deadline: pool warmup keeps LANE_READY_READ_TIMEOUT (20s); mid-life reconnects from checkout()/replace_lane() on an already-serving mesh use a new LANE_STEADY_CONNECT_TIMEOUT (3s). A healthy peer answers in milliseconds, so a dead stage now fails in ~3s instead of ~30s. Restores receive_persistent_lane_ready as the shared bounded-handshake helper (dropped during the main merge) and removes a now-obsolete retry test that covered pre-#1011 retry behavior. Adds tests asserting the steady-state deadline stays well under the warmup deadline and that the handshake read fails fast on a silent downstream. * docs: latency-aware placement — current behaviour and many-node gaps Records verified planner behaviour (skippy-coordinator/topology.rs, skippy-topology, host-runtime call site): - latency is a placement cost (rtt_ms penalty), not just relay-only exclusion - planner selects a node subset; does not have to use every eligible node - stage count is gated on a decode-TPOT target (shallower-that-meets beats deeper-that-does-not) And the gaps that matter at many-node scale: - no peer-to-peer RTT matrix in production (edge_signals never wired; only coordinator-RTT is used) -> co-located nodes cannot be exploited - network estimate is max(coordinator RTT) x node_count, a worst-case proxy - no first-class prefer-fewer/never-place-above-Y policy beyond the TPOT gate * docs: measured speculative recovery cost over WAN (why ngram hurts a latency-bound split) * Discard dead pooled stage lanes before reuse (fast-fail improvement) A pooled downstream lane whose stage died while checked in was a dead TCP stream; reusing it blocked the next generation read forever (handshake read-timeout is cleared for pooled lanes so long generations don't truncate). checkout() now probes lane liveness with a nonblocking peek and discards a dead lane so it reconnects with the short steady-state deadline instead of hanging. Validated on a loopback 2-node split with a mid-flight worker kill: new request now fails faster than main (60s vs main's 90s baseline). Does not fully solve the recovered-local routing path, tracked as follow-up. --------- Co-authored-by: Michael Neale <14976+michaelneale@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| tests | ||
| affected-crates.sh | ||
| build-linux-rocm.sh | ||
| build-linux.sh | ||
| build-llama.sh | ||
| build-mac.sh | ||
| build-release.sh | ||
| build-ui.sh | ||
| build-windows.ps1 | ||
| certify-split-startup-recovery.sh | ||
| check-sdk-contract.sh | ||
| ci-agent-live-fixture-lib.sh | ||
| ci-client-auto-test.sh | ||
| ci-compat-smoke.sh | ||
| ci-goose-smoke.sh | ||
| ci-hf-download-smoke.sh | ||
| ci-install-native-runtime.sh | ||
| ci-kotlin-sdk-smoke.sh | ||
| ci-langchain-openai-smoke.py | ||
| ci-litellm-smoke.py | ||
| ci-openai-node-smoke.cjs | ||
| ci-openai-python-smoke.py | ||
| ci-opencode-smoke.sh | ||
| ci-pi-smoke.sh | ||
| ci-prepare-native-runtime.sh | ||
| ci-rust-sdk-smoke.sh | ||
| ci-sdk-fixture.sh | ||
| ci-smoke-test.sh | ||
| ci-swift-sdk-smoke.sh | ||
| ci-two-node-client-serving-smoke.sh | ||
| ci-two-node-split-smoke.sh | ||
| console-format.js | ||
| console-format.test.js | ||
| detect-cuda-arch.sh | ||
| detect-llama-device.sh | ||
| detect-rocm-arch.sh | ||
| download-skippy-parity-candidates.sh | ||
| family-certify.sh | ||
| generate-bench-corpus.py | ||
| generate-native-runtime-release-manifest.sh | ||
| install-windows-sdk.ps1 | ||
| package-native-runtime.sh | ||
| package-native-sdk-crate.sh | ||
| package-native-sdk.sh | ||
| package-release.ps1 | ||
| package-release.sh | ||
| package-sdk-console-assets.sh | ||
| plan-clippy-batches.sh | ||
| prepare-llama.sh | ||
| prepare-swift-package-release.sh | ||
| publish-crates.sh | ||
| qa-agent-tool-call-reliability.py | ||
| qa-control-plane-mixed-version.sh | ||
| qa-kv-tool-loop-stability.py | ||
| qa-model-download-stats.sh | ||
| qa-nightly-stability.py | ||
| qwen-lab-preflight.sh | ||
| rc-release-smoke.sh | ||
| release-version.sh | ||
| release.sh | ||
| reset-swift-package-manifest.sh | ||
| run-llama-benchy-openai.sh | ||
| run-openai-guardrail-corpus.py | ||
| skippy-ci-smoke.sh | ||
| skippy-llama-parity.py | ||
| skippy-openai-smoke.sh | ||
| skippy-wan-calibrate.sh | ||
| summarize-llama-upstream.sh | ||
| task-8-real-binary.sh | ||
| update-llama-pin.sh | ||
| update-swift-package-manifest.sh | ||
| validate-release-native-runtime-matrix.py | ||
| verify-native-runtime-package.sh | ||
| verify-native-sdk-package.sh | ||
| verify-sdk-console-assets.sh | ||
| verify-swift-package-manifest.sh | ||
| verify-swift-privacy-manifest.sh | ||
| verify-swift-release-artifact.sh | ||