mirror of
https://github.com/Mesh-LLM/mesh-llm.git
synced 2026-08-08 22:23:19 -04:00
fix(planner): cap auto lane count to llama-server's 4-lane unified-KV default
PR #566 review feedback uncovered a hard 502 from the embedded skippy stage runtime under concurrent agent-style workloads. On a Mac M4 Max serving Qwen3-8B at the model's native 32k context, the auto planner was picking `slots = 16` (MAX_AUTO_PARALLEL_SLOTS). Three concurrent ~14k-token requests \u2014 the exact shape an OpenCode agent loop produces when MoA fans a worker call out next to a reducer call \u2014 fail in the embedded llama with: decode: failed to find a memory slot for batch of size 2048 surfacing as HTTP 502: skippy ABI call failed: RuntimeError: llama_decode failed Root cause: skippy's stage runtime sets `kv_unified = true` whenever `lane_count > 1` (`third_party/llama.cpp/patches/0034-Add-shared-execution-lanes-to-skippy-ABI.patch`). In unified mode llama allocates exactly `n_ctx` cells total, shared across all `n_seq_max` sequences. The previous planner derived `slots` from VRAM as if each lane carved off its own `n_ctx \u00d7 bytes_per_token` allocation \u2014 which is the `kv_unified = false` semantics, not what skippy actually does. On a node with comfortable VRAM the math happily returned the snapped maximum of 16 lanes, even though all 16 raced for the *same* fixed pool of `n_ctx` cells. Fix: drop `MAX_AUTO_PARALLEL_SLOTS` from 16 to 4, matching upstream llama-server's own auto default for the same reason. From `.deps/llama.cpp/tools/server/server.cpp`: LOG_INF("n_parallel is set to auto, using n_parallel = 4 and kv_unified = true"); params.n_parallel = 4; params.kv_unified = true; Lane count is purely a concurrency-policy knob under `kv_unified = true`; it does not change the KV cache allocation. Going from 16 to 4 frees zero RAM; it just gates admission control to a sane number of concurrent in-flight requests for the shared cell pool. Operators who know their workload (short chat turns, low-concurrency hosts, etc.) can still pick a higher value via the existing `parallel_override` plumbing, including `[models.throughput] parallel = N` in the TOML config from PR #564. Live verification on the same 2-node mesh used to find the bug: * M4 + Qwen3-8B at 32k `n_ctx`: planner now picks `slots = 4`, llama logs `n_seq_max = 4`, KV cache stays at 2448 MiB (one shared buffer; no RAM cost change). * Studio + MiniMax-M2.5 at 128k `n_ctx`: planner now picks `slots = 4`, llama logs `n_seq_max = 4`, KV cache stays at 8928 MiB. 4 \u00d7 32k cells per lane on average is plenty of headroom for agent prompts. * Repro that previously 502'd \u2014 3 parallel ~15k-prompt tool-result follow-ups on the M4 \u2014 now all succeed with `finish_reason=stop`, full content, ~20s wall time. Zero `find_slot` failures, zero `llama_decode` errors, zero skippy ABI errors. * Burst test \u2014 5 parallel at the same prompt shape \u2014 the 5th request correctly hits the admission-control queue and returns a clean `{"type":"rate_limit_error","code":"rate_limit_exceeded"}` after the admission timeout, instead of an opaque mid-flight 502. Adds two regression tests in `context_planning::tests`: * `auto_slots_capped_at_llama_server_default` covers the high-VRAM small-model case that used to plan 16. * `explicit_parallel_can_exceed_auto_ceiling` covers the override path so operators retain control. Validation: `cargo fmt --all -- --check` clean, `cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings` clean, `cargo test -p mesh-llm-host-runtime --lib` 1437/1437 pass (includes the two new regression tests).
This commit is contained in:
parent
f5cf4b8678
commit
1b90121975
1 changed files with 85 additions and 1 deletions
|
|
@ -3,7 +3,38 @@ use crate::models::gguf::{GgufCompactMeta, GgufKvCacheQuant};
|
|||
const DEFAULT_CONTEXT_LENGTH: u32 = 4096;
|
||||
const DEFAULT_PARALLEL_SLOTS: usize = 4;
|
||||
const MIN_AUTO_CONTEXT_LENGTH: u32 = 512;
|
||||
const MAX_AUTO_PARALLEL_SLOTS: usize = 16;
|
||||
/// Auto-planner ceiling on concurrent lanes.
|
||||
///
|
||||
/// Matches upstream llama-server: when `--parallel` is left to auto,
|
||||
/// llama-server picks `n_parallel = 4` and turns on `kv_unified = true`
|
||||
/// (see `tools/server/server.cpp`,
|
||||
/// `"n_parallel is set to auto, using n_parallel = 4 and kv_unified = true"`).
|
||||
///
|
||||
/// Skippy's stage-runtime patches also set `kv_unified = true` whenever
|
||||
/// `lane_count > 1` (`third_party/llama.cpp/patches/0034-*.patch`). In
|
||||
/// unified mode llama allocates exactly `n_ctx` cells total, shared
|
||||
/// across all `n_seq_max` sequences. The previous ceiling of 16 was
|
||||
/// inherited from a VRAM-based slot calculation that pretended each
|
||||
/// lane carved off its own `n_ctx × bytes_per_token` allocation —
|
||||
/// which is the `kv_unified = false` semantics, not what skippy
|
||||
/// actually does. On any node with comfortable VRAM that math
|
||||
/// happily picked 16 lanes even though all 16 raced for the *same*
|
||||
/// pool of `n_ctx` cells.
|
||||
///
|
||||
/// Concrete failure mode that prompted this change: Qwen3-8B on a
|
||||
/// 32k `n_ctx` got `slots = 16`. Three concurrent agent-shape
|
||||
/// requests (~14k tokens each — OpenCode system prompt plus tools
|
||||
/// plus a tool-result follow-up) need ~45k cells in the shared 32k
|
||||
/// pool; llama's `find_slot` fails on the third request and skippy
|
||||
/// surfaces it as an HTTP 502 with body `skippy ABI call failed:
|
||||
/// RuntimeError: llama_decode failed`.
|
||||
///
|
||||
/// 4 is the same conservative ceiling llama-server uses for the
|
||||
/// same `kv_unified = true` reason. Operators who know their
|
||||
/// workload (e.g. all short chat turns, or a single-user MoA host)
|
||||
/// can still go higher via `parallel_override` /
|
||||
/// `[models.throughput] parallel = N` in the TOML config.
|
||||
const MAX_AUTO_PARALLEL_SLOTS: usize = 4;
|
||||
const KV_CACHE_BUDGET_NUMERATOR: u64 = 85;
|
||||
const KV_CACHE_BUDGET_DENOMINATOR: u64 = 100;
|
||||
|
||||
|
|
@ -264,6 +295,59 @@ mod tests {
|
|||
assert_eq!(plan.slots, 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn auto_slots_capped_at_llama_server_default() {
|
||||
// Regression: a small model on a huge-VRAM box used to plan
|
||||
// `slots = 16` because the VRAM-derived per-lane math pretended
|
||||
// each lane carved off its own `n_ctx × bytes/token` allocation.
|
||||
// With `kv_unified = true` (skippy patch 0034) those 16 lanes
|
||||
// race for the same `n_ctx` cell pool, and 3 concurrent agent
|
||||
// requests at ~14k tokens each blow it up with
|
||||
// `find_slot` failures → HTTP 502
|
||||
// `RuntimeError: llama_decode failed`.
|
||||
//
|
||||
// Match llama-server's auto default of 4 (see
|
||||
// `.deps/llama.cpp/tools/server/server.cpp`: "n_parallel is
|
||||
// set to auto, using n_parallel = 4 and kv_unified = true").
|
||||
let metadata = gqa_metadata(32_768);
|
||||
let plan = plan_runtime_resources(RuntimeResourcePlanInput {
|
||||
ctx_size_override: None,
|
||||
parallel_override: None,
|
||||
model_bytes: 5_000_000_000,
|
||||
// 128GB free — plenty for many "per-lane" slots under the
|
||||
// old broken math.
|
||||
vram_bytes: 128_000_000_000,
|
||||
metadata: Some(&metadata),
|
||||
kv_cache_quant: GgufKvCacheQuant::Q8_0,
|
||||
local_layer_fraction: None,
|
||||
});
|
||||
|
||||
assert_eq!(plan.context_length, 32_768);
|
||||
assert!(
|
||||
plan.slots <= 4,
|
||||
"auto-planner should not exceed llama-server's 4-lane unified-KV ceiling; got {}",
|
||||
plan.slots
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn explicit_parallel_can_exceed_auto_ceiling() {
|
||||
// Operators who know their workload can still go higher than
|
||||
// the auto ceiling via `parallel_override`.
|
||||
let metadata = gqa_metadata(131_072);
|
||||
let plan = plan_runtime_resources(RuntimeResourcePlanInput {
|
||||
ctx_size_override: None,
|
||||
parallel_override: Some(8),
|
||||
model_bytes: 5_000_000_000,
|
||||
vram_bytes: 128_000_000_000,
|
||||
metadata: Some(&metadata),
|
||||
kv_cache_quant: GgufKvCacheQuant::Q8_0,
|
||||
local_layer_fraction: None,
|
||||
});
|
||||
|
||||
assert_eq!(plan.slots, 8);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn split_model_uses_local_layer_fraction() {
|
||||
// 480B-class model: 94 layers, 264GB total, host holds 62/94 layers.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue