fix(planner): cap auto lane count to llama-server's 4-lane unified-KV default

PR #566 review feedback uncovered a hard 502 from the embedded skippy
stage runtime under concurrent agent-style workloads. On a Mac M4 Max
serving Qwen3-8B at the model's native 32k context, the auto planner
was picking `slots = 16` (MAX_AUTO_PARALLEL_SLOTS). Three concurrent
~14k-token requests \u2014 the exact shape an OpenCode agent loop produces
when MoA fans a worker call out next to a reducer call \u2014 fail in the
embedded llama with:

  decode: failed to find a memory slot for batch of size 2048

surfacing as HTTP 502:

  skippy ABI call failed: RuntimeError: llama_decode failed

Root cause: skippy's stage runtime sets `kv_unified = true` whenever
`lane_count > 1` (`third_party/llama.cpp/patches/0034-Add-shared-execution-lanes-to-skippy-ABI.patch`).
In unified mode llama allocates exactly `n_ctx` cells total, shared
across all `n_seq_max` sequences. The previous planner derived
`slots` from VRAM as if each lane carved off its own
`n_ctx \u00d7 bytes_per_token` allocation \u2014 which is the
`kv_unified = false` semantics, not what skippy actually does. On a
node with comfortable VRAM the math happily returned the snapped
maximum of 16 lanes, even though all 16 raced for the *same* fixed
pool of `n_ctx` cells.

Fix: drop `MAX_AUTO_PARALLEL_SLOTS` from 16 to 4, matching upstream
llama-server's own auto default for the same reason. From
`.deps/llama.cpp/tools/server/server.cpp`:

  LOG_INF("n_parallel is set to auto, using n_parallel = 4 and
           kv_unified = true");
  params.n_parallel = 4;
  params.kv_unified = true;

Lane count is purely a concurrency-policy knob under `kv_unified =
true`; it does not change the KV cache allocation. Going from 16 to
4 frees zero RAM; it just gates admission control to a sane number
of concurrent in-flight requests for the shared cell pool.

Operators who know their workload (short chat turns, low-concurrency
hosts, etc.) can still pick a higher value via the existing
`parallel_override` plumbing, including `[models.throughput]
parallel = N` in the TOML config from PR #564.

Live verification on the same 2-node mesh used to find the bug:

* M4 + Qwen3-8B at 32k `n_ctx`: planner now picks `slots = 4`,
  llama logs `n_seq_max = 4`, KV cache stays at 2448 MiB (one
  shared buffer; no RAM cost change).
* Studio + MiniMax-M2.5 at 128k `n_ctx`: planner now picks
  `slots = 4`, llama logs `n_seq_max = 4`, KV cache stays at 8928
  MiB. 4 \u00d7 32k cells per lane on average is plenty of headroom for
  agent prompts.
* Repro that previously 502'd \u2014 3 parallel ~15k-prompt tool-result
  follow-ups on the M4 \u2014 now all succeed with `finish_reason=stop`,
  full content, ~20s wall time. Zero `find_slot` failures, zero
  `llama_decode` errors, zero skippy ABI errors.
* Burst test \u2014 5 parallel at the same prompt shape \u2014 the 5th
  request correctly hits the admission-control queue and returns a
  clean
  `{"type":"rate_limit_error","code":"rate_limit_exceeded"}`
  after the admission timeout, instead of an opaque mid-flight 502.

Adds two regression tests in `context_planning::tests`:

* `auto_slots_capped_at_llama_server_default` covers the
  high-VRAM small-model case that used to plan 16.
* `explicit_parallel_can_exceed_auto_ceiling` covers the
  override path so operators retain control.

Validation: `cargo fmt --all -- --check` clean,
`cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings`
clean, `cargo test -p mesh-llm-host-runtime --lib` 1437/1437 pass
(includes the two new regression tests).
This commit is contained in:
Michael Neale 2026-05-20 21:20:39 +10:00
parent f5cf4b8678
commit 1b90121975

View file

@ -3,7 +3,38 @@ use crate::models::gguf::{GgufCompactMeta, GgufKvCacheQuant};
const DEFAULT_CONTEXT_LENGTH: u32 = 4096;
const DEFAULT_PARALLEL_SLOTS: usize = 4;
const MIN_AUTO_CONTEXT_LENGTH: u32 = 512;
const MAX_AUTO_PARALLEL_SLOTS: usize = 16;
/// Auto-planner ceiling on concurrent lanes.
///
/// Matches upstream llama-server: when `--parallel` is left to auto,
/// llama-server picks `n_parallel = 4` and turns on `kv_unified = true`
/// (see `tools/server/server.cpp`,
/// `"n_parallel is set to auto, using n_parallel = 4 and kv_unified = true"`).
///
/// Skippy's stage-runtime patches also set `kv_unified = true` whenever
/// `lane_count > 1` (`third_party/llama.cpp/patches/0034-*.patch`). In
/// unified mode llama allocates exactly `n_ctx` cells total, shared
/// across all `n_seq_max` sequences. The previous ceiling of 16 was
/// inherited from a VRAM-based slot calculation that pretended each
/// lane carved off its own `n_ctx × bytes_per_token` allocation —
/// which is the `kv_unified = false` semantics, not what skippy
/// actually does. On any node with comfortable VRAM that math
/// happily picked 16 lanes even though all 16 raced for the *same*
/// pool of `n_ctx` cells.
///
/// Concrete failure mode that prompted this change: Qwen3-8B on a
/// 32k `n_ctx` got `slots = 16`. Three concurrent agent-shape
/// requests (~14k tokens each — OpenCode system prompt plus tools
/// plus a tool-result follow-up) need ~45k cells in the shared 32k
/// pool; llama's `find_slot` fails on the third request and skippy
/// surfaces it as an HTTP 502 with body `skippy ABI call failed:
/// RuntimeError: llama_decode failed`.
///
/// 4 is the same conservative ceiling llama-server uses for the
/// same `kv_unified = true` reason. Operators who know their
/// workload (e.g. all short chat turns, or a single-user MoA host)
/// can still go higher via `parallel_override` /
/// `[models.throughput] parallel = N` in the TOML config.
const MAX_AUTO_PARALLEL_SLOTS: usize = 4;
const KV_CACHE_BUDGET_NUMERATOR: u64 = 85;
const KV_CACHE_BUDGET_DENOMINATOR: u64 = 100;
@ -264,6 +295,59 @@ mod tests {
assert_eq!(plan.slots, 2);
}
#[test]
fn auto_slots_capped_at_llama_server_default() {
// Regression: a small model on a huge-VRAM box used to plan
// `slots = 16` because the VRAM-derived per-lane math pretended
// each lane carved off its own `n_ctx × bytes/token` allocation.
// With `kv_unified = true` (skippy patch 0034) those 16 lanes
// race for the same `n_ctx` cell pool, and 3 concurrent agent
// requests at ~14k tokens each blow it up with
// `find_slot` failures → HTTP 502
// `RuntimeError: llama_decode failed`.
//
// Match llama-server's auto default of 4 (see
// `.deps/llama.cpp/tools/server/server.cpp`: "n_parallel is
// set to auto, using n_parallel = 4 and kv_unified = true").
let metadata = gqa_metadata(32_768);
let plan = plan_runtime_resources(RuntimeResourcePlanInput {
ctx_size_override: None,
parallel_override: None,
model_bytes: 5_000_000_000,
// 128GB free — plenty for many "per-lane" slots under the
// old broken math.
vram_bytes: 128_000_000_000,
metadata: Some(&metadata),
kv_cache_quant: GgufKvCacheQuant::Q8_0,
local_layer_fraction: None,
});
assert_eq!(plan.context_length, 32_768);
assert!(
plan.slots <= 4,
"auto-planner should not exceed llama-server's 4-lane unified-KV ceiling; got {}",
plan.slots
);
}
#[test]
fn explicit_parallel_can_exceed_auto_ceiling() {
// Operators who know their workload can still go higher than
// the auto ceiling via `parallel_override`.
let metadata = gqa_metadata(131_072);
let plan = plan_runtime_resources(RuntimeResourcePlanInput {
ctx_size_override: None,
parallel_override: Some(8),
model_bytes: 5_000_000_000,
vram_bytes: 128_000_000_000,
metadata: Some(&metadata),
kv_cache_quant: GgufKvCacheQuant::Q8_0,
local_layer_fraction: None,
});
assert_eq!(plan.slots, 8);
}
#[test]
fn split_model_uses_local_layer_fraction() {
// 480B-class model: 94 layers, 264GB total, host holds 62/94 layers.