2026-03-05 08:02:26 -05:00
# Agent Notes
## Repo Overview
2026-03-24 15:16:35 +11:00
This repo (`mesh-llm` ) contains mesh-llm — a Rust binary that pools GPUs over QUIC for distributed LLM inference using llama.cpp.
2026-03-05 08:02:26 -05:00
2026-06-12 18:44:36 +10:00
The workspace is split across many crates under `crates/` . The shipped binary `mesh-llm` (`crates/mesh-llm/` ) is a thin entry point: it builds the Tokio runtime, parses the CLI via `mesh-llm-cli` , dispatches one-shot commands (via its `commands/` module and `mesh-llm-commands` ), and hands the runtime surfaces (`serve` / `client` ) to `mesh-llm-host-runtime` , where the bulk of host-side logic lives. A lighter parallel crate `mesh-client` (`mesh-llm-client` ) carries the same domain shape for client-only usage. Embedded llama.cpp staged-runtime support lives in the `skippy-*` crates.
2026-05-19 17:26:09 +10:00
2026-03-05 08:02:26 -05:00
## Key Docs
| Doc | What it covers |
|---|---|
2026-05-12 16:18:14 -04:00
| `README.md` | Quickstart and documentation hub |
| `docs/MESHES.md` | Public/private meshes, publishing, discovery, join flows |
| `docs/SKIPPY_SPLITS.md` | Running big models with Skippy split serving |
| `docs/LAYER_PACKAGE_REPOS.md` | Contributing and publishing layer package repos |
| `docs/EXO_COMPARISON.md` | mesh-llm vs Exo comparison |
2026-03-05 08:02:26 -05:00
| `CONTRIBUTING.md` | Build from source, dev workflow, UI dev |
| `RELEASE.md` | Release process (build, bundle, tag, GitHub release) |
| `ROADMAP.md` | Future directions |
2026-05-02 09:44:52 +10:00
| `crates/mesh-llm/TODO.md` | Current work items and backlog |
| `crates/mesh-llm/README.md` | Rust crate overview and file map |
| `docs/README.md` | Documentation map and topic directory guide |
| `docs/design/DESIGN.md` | Architecture, protocols, features |
| `docs/design/TESTING.md` | Test playbook, scenarios, remote deploy |
| `docs/design/MULTI_MODAL.md` | Multimodal design: capability model, blob plugin, console, routing |
| `docs/design/VIRTUAL_LLM.md` | Virtual LLM engine (inter-model collaboration) |
2026-06-12 18:44:36 +10:00
| `docs/design/LLAMA_STAGE_INTEGRATION_PLAN.md` | llama.cpp staged-runtime integration and patch-queue background |
| `docs/SKIPPY.md` | Skippy integration readiness and parity notes |
2026-05-02 09:44:52 +10:00
| `docs/plugins/README.md` | Plugin architecture and plugin development |
2026-03-05 08:02:26 -05:00
| `fly/README.md` | Fly.io deployment (console + API apps) |
2026-05-12 16:18:14 -04:00
| `tools/relay-fly-legacy/README.md` | Archived self-hosted iroh relay reference; production uses services.iroh.computer |
2026-03-05 08:02:26 -05:00
2026-06-10 05:18:25 -04:00
## Public Website
The public static website lives in `website/` and is built with Eleventy.
Treat `website/` as the only maintained source for the public marketing/docs
site. The build writes static-hosting output into `docs/` , alongside the repo's
existing Markdown documentation. The root `docs/` tree is therefore mixed
ownership by path: generated website artifacts live at `docs/index.html` ,
2026-07-18 01:30:22 +10:00
`docs/CNAME` , `docs/install.sh` , `docs/install.ps1` , `docs/setup-mesh` ,
`docs/mesh-llm-logo.svg` , `docs/funding.json` , `docs/.well-known/` ,
2026-06-10 05:18:25 -04:00
`docs/catalog/` , `docs/assets/` , `docs/pagefind/` , and `docs/docs/` ; project
documentation Markdown such as `docs/MESHES.md` , `docs/design/**` ,
`docs/plugins/**` , and `docs/specs/**` remains source. Do not hand-edit the
generated website artifact paths; update files under `website/src/` and rebuild
instead.
```bash
just website-build # cd website & & npm run build; writes generated output to docs/
just website-dev # Eleventy dev server on port 8765
just website-clean # remove generated website output while preserving docs/ source
```
The website build runs Tailwind first, then Eleventy, then Pagefind. Eleventy
copies `website/src/CNAME` , `website/src/assets/` , `website/src/mesh-llm-logo.svg` ,
2026-07-18 01:30:22 +10:00
and the repo-root `install.sh` / `install.ps1` (plus `install.md` published as
`docs/setup-mesh` ) into `docs/` for deployment.
2026-06-10 05:18:25 -04:00
`website/src/assets/site.generated.css` is generated by Tailwind and should not
be edited by hand. Use `just website-clean` before rebuilding when you need to
purge generated website output without deleting authored Markdown docs.
2026-03-05 08:02:26 -05:00
## Building
Always use `just` . Never build manually.
```bash
2026-05-20 13:17:03 +10:00
just build # DEBUG build → ./target/debug/mesh-llm (fast, for iteration)
just release-build # RELEASE build → ./target/release/mesh-llm (slow, for serious testing / deploy)
just bundle # portable tarball (uses the release binary)
just stop # stop tracked mesh-llm runtime processes
just test # quick inference test against :9337
just auto # build + stop + start with --auto
just ui-dev # vite dev server with HMR
2026-06-10 05:18:25 -04:00
just website-build # build website/ into docs/ for static hosting
just website-dev # Eleventy dev server on :8765
2026-06-12 18:44:36 +10:00
just ui-clean # nuke node_modules + dist (fixes stale npm state)
2026-03-05 08:02:26 -05:00
```
2026-05-20 13:17:03 +10:00
**Which build to use:**
2026-07-29 10:16:23 -04:00
- `just build` → produces `./target/debug/mesh-llm` plus its adjacent
`target/debug/native-runtimes/` directory. It is the normal fast local
product: a backend-neutral dynamic host and one locally packaged runtime.
Use it for iteration and startup checks; use a release product for serious
behavior/performance testing or deployment.
2026-05-20 13:17:03 +10:00
- `just release-build` → produces `./target/release/mesh-llm` . Use this for any
2026-06-22 14:37:34 +10:00
serious testing, deploying to test machines, bundling, or releases. Release
2026-07-29 10:16:23 -04:00
builds always produce one backend-neutral host plus a packageable native
runtime. When validating branch-local Skippy ABI, llama.cpp patches, MAS
hidden-state, or native tensor changes, use `just release-host-build` and
`just release-runtime-build <backend>` , then point the host at
`dist/native-runtimes` with `MESH_LLM_NATIVE_RUNTIME_BUNDLE_DIR` . Static
backend linkage is not a release or packaging lane.
2026-05-20 13:17:03 +10:00
- `./target/release/mesh-llm` may exist from a *previous* `just release-build`
or `just build-dev` invocation even after you run only `just build` — its
presence is **not** evidence that your latest code is in it. When in doubt,
check `stat ./target/release/mesh-llm` against the time you last ran
`just release-build` , or just re-run `just release-build` .
- `cargo check` / `cargo build` do **not** count as a build for this repo —
they skip llama.cpp ABI prep and the UI, and `cargo check` produces no
binary at all.
2026-07-29 10:16:23 -04:00
When in doubt for testing or shipping changes: use the composed output from
`just release-bundle vX.Y.Z <output>` , which packages the backend-neutral host
with one selected runtime under `native-runtimes/` . For native ABI development,
first decide whether you need the default dynamic release packaging path or an
embedded branch-local native ABI; do not test new ABI symbols against downloaded
release native runtimes.
Release artifacts follow one three-layer graph:
| Layer | Command | Output |
|---|---|---|
| Neutral host | `just release-host-build` | `target/release/mesh-llm` plus an import-policy report during packaging |
| Native runtime | `just release-runtime-build <backend>` | `dist/native-runtimes/<runtime-id>/` plus archive/checksum |
| Product | `just release-bundle vX.Y.Z <output>` | `mesh-bundle/` containing the host, one runtime, and product/host-import manifests |
The host dependency policy is enforced by
`scripts/verify-host-dependencies.py` . Release, installer, SDK, native-package,
and image lanes must not bypass it or copy backend libraries beside the host.
For an isolated local runtime test, set
`MESH_LLM_NATIVE_RUNTIME_BUNDLE_DIR="$PWD/dist/native-runtimes"` and a fresh
`MESH_LLM_NATIVE_RUNTIME_CACHE_DIR` . Discovery never searches the current
working directory. Do not reintroduce an external `llama-server` or
`rpc-server` lane.
2026-05-20 13:17:03 +10:00
2026-04-08 09:37:15 +10:00
### npm "Exit handler never called" error
If `just build` fails on the UI step with `npm error Exit handler never called!` , run:
```bash
2026-06-12 18:44:36 +10:00
just ui-clean
2026-04-08 09:37:15 +10:00
just build
```
This is an npm bug that surfaces when `node_modules` gets into a bad state (e.g. after branch switches that change `package-lock.json` ). Nuking `node_modules` and letting `npm ci` reinstall from scratch fixes it.
2026-03-05 08:03:51 -05:00
See `CONTRIBUTING.md` for full dev workflow.
2026-05-05 11:48:40 +10:00
## llama.cpp ABI Patch Queue
Virtual LLM engine — callback hooks from llama-server into mesh (#225)
* docs: virtual LLM engine design — in-engine mesh hooks
Design for adding hooks inside llama-server's C++ token generation loop
so it can consult other models in the mesh during inference.
Hook points: pre-inference, per-token, pre-response, slot pause/resume.
Communication via HTTP callback to mesh-llm which routes to any model
in the mesh. Token injection into live KV cache without restarting.
Relates to #183, #165
* docs: uncertainty signals and callback design for virtual LLM hooks
Four signal types: per-token entropy, top-token margin, sequence variance
(confidence trajectory), and self-consistency via multi-completion.
Hooks are simple: measure signals in C++, call back to mesh-llm with the
data, mesh-llm decides what to do and responds with an action (inject,
continue, stop, none). All decision logic lives in the Rust side.
Single callback endpoint: POST /mesh/hook with hook type, signal data,
and generation context. Three callback modes: async fire, sync call,
poll check.
* docs: clean callback protocol and signal design for virtual LLM
Rewrite to separate concerns clearly:
- C++ side: compute signals (entropy, margin, window stats), call mesh-llm
- Rust side: all decision logic, model routing, consultation
JSON over localhost HTTP chosen over FFI/shared-memory because:
- cpp-httplib already linked, axum already running
- callback latency (~0.1ms) irrelevant vs consultation time (seconds)
- debuggable with curl, independently versionable
Five hooks, one callback shape, three actions (none/inject/stop).
Async consultation via fire-at-hook-1, poll-at-hook-3 pattern.
Per-request threshold config alongside existing task_params.
* docs: simplify to four hooks, drop per-token callback
Per-token callback was wrong — 30-100 tok/s makes it wasteful.
Signal computation stays per-token in C++ (cheap arithmetic),
but callbacks only fire at: pre-inference, post-prefill,
pre-response, and complete (telemetry).
Each hook now shows exactly what data it sends to mesh-llm:
- pre_inference: original messages array, images, model capabilities
- post_prefill: first-token entropy/margin, top-5 candidates
- pre_response: full generated text, signal summary stats
- complete: telemetry (fire-and-forget)
Mid-generation intervention deferred as optional threshold breakout.
* docs: rewrite VIRTUAL_LLM.md — hooks, callbacks, examples only
Cut all proxy discussion, architecture justification, and research notes.
Doc now covers: callback protocol, four hooks with exact JSON payloads
and worked examples, signal computation, token injection mechanics,
and the specific C++ and Rust changes needed.
* docs: add async hooks — pending action + poll for non-blocking consultation
Hook 1 can return 'pending' with an async_id instead of blocking.
llama-server stores the id and polls GET /mesh/hook/poll/{id} every
16 tokens during generation. When the result is ready (200), tokens
are injected into the live KV cache. When not ready (202), generation
continues uninterrupted.
Each hook now annotated with blocking behavior:
- pre_inference: sync or async (mesh-llm decides)
- post_prefill: always sync (just reads numbers, <1ms)
- pre_response: always sync (need verdict before sending)
- complete: fire-and-forget
* docs: drop complete hook, three hooks is enough
Telemetry can be added later if needed. The core mechanism is
pre_inference, post_prefill, pre_response + async polling.
* docs: clarify hook triggers — always fire, mesh-llm decides
* docs: Hook 1 configures hooks 2 and 3 via entropy_threshold and verify
Hook 1 always fires — it's the setup hook. Its response tells C++
what to watch for: entropy_threshold enables Hook 2, verify enables
Hook 3. Most requests: only Hook 1 fires, mesh-llm returns none.
* docs: trigger-based hooks — only fire when structural criteria met
All hooks are now conditional:
- Hook 1: images+text-only, context pressure (>75% of ctx),
long session (>10 turns), large user paste
- Hook 2: first-token entropy > threshold (set by Hook 1)
- Hook 3: max_tokens cutoff, very short response, high uncertainty,
tail entropy spike, mid-sentence cutoff
- Polling: only when async work pending
Added examples: context pressure → conversation summarization,
max_tokens cutoff → summarize + continue, very short response →
retry with encouragement, tail entropy spike → verify ending.
* docs: complete rewrite — triggers, examples, implementation details
Three sections:
1. Triggers and hooks — what fires when, one table per hook
2. Examples — concrete JSON for each scenario (captioning, summarization,
context pressure, refusal, truncation, hallucination)
3. Implementation — exact C++ code at each hook point in server-context.cpp,
signal window struct, inject helper, mesh-llm endpoints and module
* docs: full callback payloads under each hook
Each hook now shows the exact JSON mesh-llm receives:
- Hook 1: full messages array (images, audio, all turns), token counts,
context size, trigger name, model capabilities
- Hook 2: first-token signals only (entropy, margin, top-5), request_id
links back to Hook 1 data
- Hook 3: full generated text, stop reason, signal summary, request_id
links back to original request
* docs: all hooks send full messages — can't assume Hook 1 fired
Hook 2 and 3 now include the messages array. Hook 1 only fires
on specific triggers, so mesh-llm may never have seen the request.
Each hook is self-contained — sends everything mesh-llm needs.
* api: add /mesh/hook endpoint for llama-server callbacks
New route handler for mesh hook callbacks. Three hooks handled:
- pre_inference: logs trigger, returns none + entropy_threshold
- post_prefill: logs entropy signal, returns none
- pre_response: logs n_decoded + stop_reason, returns none
Poll endpoint GET /mesh/hook/poll/{id} returns 202 (stub).
All hooks return 'none' for now — plumbing first, decision logic next.
* launch: pass --mesh-port to llama-server for hook callbacks
Sets MESH_API_PORT env var at runtime startup, launch.rs reads it
and passes --mesh-port {port} to llama-server. This tells llama-server
where to POST hook callbacks on localhost.
* llama-patches: snapshot C++ mesh hook changes for co-iteration
Contains the git format-patch from llama.cpp mesh-hooks branch plus
the standalone header file. This lets us iterate on C++ and Rust
together on micn/virtual-llm. When stable, apply the patch to the
llama.cpp fork's mesh-hooks branch and remove this directory.
* llama-patches: update with trigger fixes and e2e test results
* llama-patches: update with working token injection
* TEMPORARY: inline C++ mesh hook files for single-repo iteration
llama-patches/ contains the 7 modified/new C++ files that implement
mesh hooks in llama-server. sync.sh copies them into llama.cpp/ and
build-mac.sh runs it automatically after pulling upstream-latest.
This is a temporary setup for the micn/virtual-llm branch so C++ and
Rust changes live in one repo / one PR. When stable, the C++ moves to
the mesh-hooks branch on the llama.cpp fork and this directory is deleted.
* inference: add virtual_llm decision module with documented stubs
New module inference/virtual_llm.rs — the brain behind mesh hooks.
Three handlers, one per hook type, each with:
- Documented trigger table
- Match on trigger name
- Logging with context (model, token counts, entropy, etc.)
- TODO comments describing the planned consultation
Also: HookAction enum, AsyncConsultations struct stub.
Route handler in mesh_hook.rs now delegates to virtual_llm instead
of inlining the decision logic.
* virtual_llm: add HookContext for model-aware decisions
HookContext enriches the model filename from the C++ hook payload with:
- Tier, strengths, tools from ModelProfile
- Vision/multimodal from ModelCapabilities
- Available peers by capability (vision, stronger, all)
This lets the decision engine pick complementary models: vision when
text-only, stronger when uncertain, different specialty when needed.
Struct is defined but not yet wired — handlers still take raw payload.
* simplify: remove polling, all hooks are synchronous
Delete the async poll mechanism (pending_async_ids, poll_async,
should_poll, GET /mesh/hook/poll/{id}). All hooks are now plain
synchronous POST calls.
Background work uses a simpler pattern: Hook 1 spawns a tokio task
and stores it in a DashMap keyed by request_id. Hook 3 checks the
map — if the result is ready, use it; if not, let the response go.
No polling needed.
C++ side: removed poll loop from generation, cleaned up mesh_hook_ctx.
Rust side: removed poll endpoint, removed Pending variant, updated docs.
* add mesh_request_id for request correlation, rewrite design doc
mesh-llm generates a mesh_request_id and includes it in the request
body. llama-server passes it back in every hook payload. mesh-llm
stores the original request (messages, images, etc.) keyed by this
ID so hooks can access the full conversation without C++ sending it.
Design doc rewritten to match current implementation:
- Removed polling/pending (all hooks sync now)
- Documented background work pattern (Hook 1 spawns, Hook 3 collects)
- Documented request correlation via mesh_request_id
- Updated all example payloads and scenario descriptions
- Updated implementation tables to match actual files
* implement Hook 2 KV cache injection
After prefill, if the model is uncertain (high entropy / low margin),
mesh-llm can return inject text. This text is now tokenized and
decoded into the KV cache via a temporary batch — the model 'sees'
the injected context before generating its first token.
The injection uses the same chunked decode pattern as normal prefill:
tokens are added to a temp batch, decoded in n_batch-sized chunks,
and positions are tracked via slot.prompt.tokens. Only the last
inject token requests logits, and slot.i_batch is set so sampling
reads from the correct position.
This is the key mechanism that makes the virtual LLM interesting —
a small model that's uncertain can receive context from a stronger
model mid-inference, changing the trajectory of generation.
* update docs for Hook 2 KV injection
- Design doc: Hook 2 description updated from 'informational' to
documenting inject action, how KV cache injection works, and new
scenario showing uncertain model receiving live help
- virtual_llm.rs: lifecycle diagram updated (Hook 2 now returns
inject, shows KV decode step), function doc describes injection
mechanism, removed stale 'pending' action from Hook 1
* implement real consultation logic: image captioning, summarization, second opinion, verification
New module: inference/consult.rs — peer consultation over QUIC mesh.
Provides four patterns:
- caption_image: send image to vision peer, get text description
- summarize_conversation: condense early turns to free context
- second_opinion: ask a different model the same question
- verify_response: check if a response's ending is accurate
Peer discovery finds peers by capability (vision) or by difference
(different model architecture). Prefers low-RTT peers. The value of
Hook 2 and Hook 3 verification is diversity — a different model's
perspective, not necessarily a stronger one.
Hook handlers are now async, accepting the mesh Node to find and
consult peers. All four scenarios wired:
- Hook 1 images_no_multimodal → caption via vision peer → inject
- Hook 1 context_pressure → summarize via any peer → inject
- Hook 2 high_entropy → second opinion from different model → inject into KV
- Hook 3 tail_entropy_spike/verify → verify via different model → append correction
Added MeshApi::node() helper for route handlers.
* prefer similar-tier peers, send only last message, keep injection concise
find_different_model_peer now scores by tier distance (±1 preferred)
then RTT. A 4B model asks another small model, not a 70B.
second_opinion sends only the last user message with 'answer briefly
in 2-3 sentences' — max 192 tokens. The peer returns a concise
answer, not a full essay. Keeps the blocking time and KV injection
short.
Injection text capped at 512 chars. Every injected token is decoded
into KV cache and adds latency — brevity matters.
* only annotate truncation when response looks cut off mid-sentence
max_tokens fires on every request that hits the token limit, including
intentional truncation (user set max_tokens: 100). Now we check if
the response actually ends mid-thought: no sentence-ending punctuation
means genuinely cut off, annotate it. Clean ending (period, question
mark, etc.) means the model finished naturally, skip the note.
* remove max_tokens from Hook 3 triggers
max_tokens is normal operation — the user or server set a limit and
the model hit it. finish_reason: 'length' already signals this.
Firing a hook round-trip to append a redundant note adds latency
for no value.
Hook 3 now only fires on signals that indicate the model struggled:
very_short, high_uncertainty, tail_entropy_spike, verify.
Removed from C++ trigger evaluation, Rust handler, and design doc.
* fix: add trailing newline to lib.rs for CI rustfmt
* fix: use console port (3131) for mesh hooks, not proxy port (9337)
MESH_API_PORT was set to cli.port (proxy, 9337) instead of
cli.console (management API, 3131). Hooks calling the proxy
would loop back to llama-server.
* Hook 2 always armed — default entropy_threshold 5.0
Hook 2 (post-prefill uncertainty check) no longer depends on Hook 1
firing first. It stands on its own: if the model is uncertain after
reading the prompt, ask a peer. Hook 1 can still override the
threshold for specific request types.
* Hook 1: only media triggers — drop context_pressure, long_session, large_user_message
Hook 1 is for when the model can't handle the input modality:
images without vision, audio without audio support. These are
clear problems with clear solutions (caption, transcribe).
context_pressure/long_session/large_user_message were speculative
triggers that didn't have good actions — appending a summary to
an already-long prompt makes it longer, not shorter.
The interesting hooks during inference are Hook 2 (uncertainty)
and Hook 3 (bad output signals). Those stand on their own now.
Also removes unused find_any_peer and summarize_conversation.
* mid-generation Hook 2b, Hook 3 replace, --mesh-hook-debug
Hook 2b: fires during generation when sustained entropy spike
detected (75% of rolling window has entropy > 4.0). Same KV cache
injection as Hook 2 — model continues generating but now informed
by peer context. 32-token cooldown between fires.
Hook 3: 'replace' action replaces generated_text entirely instead
of appending. When verification says the response is bad, the
peer's corrected answer replaces the model's output.
--mesh-hook-debug / MESH_HOOK_DEBUG: lowers all thresholds so hooks
fire on almost any request. entropy_threshold 0.5, mid-gen spike
ratio 0.25, cooldown 8 tokens. For testing.
* pass messages in all hook payloads, handle /mesh/hook in proxy
Messages stored in mesh_messages on task params, included in all
hook payloads (Hook 1, 2, 2b, 3). Without this, Hook 2 had no
messages to send to a peer for second opinion.
/mesh/hook handled directly in the proxy dispatcher since the
management API and proxy share port 3131 after main consolidated
ports. Proxy intercepts the route before forwarding to llama-server.
Tested: hooks fire, peer consultation initiates over QUIC mesh.
Hook 2 finds Qwen3-8B on the public mesh when local model is gemma.
Consultation latency is high (~60s) — needs investigation.
* fan-out consultation: race 2 peers, 10s timeout
Hook 2 now finds the top 2 different-model peers and races them
via JoinSet. First successful response wins, loser gets aborted.
If only 1 peer exists, falls back to single call.
10s timeout on all consultation calls — hooks block the local
model's slot, can't wait forever for a remote peer.
Verify (Hook 3) trims generated_text to last 500 chars and caps
at 256 tokens — the tail is where the spike happened.
Tested on public mesh: raced Hermes-7B vs Qwen3-8B, Qwen won
in 7s, total request time 12s (was 65s before timeout).
* fix port split: proxy on 9337, management API on 3131
Reverted api_port back to cli.port (9337) so the proxy binds there.
MESH_API_PORT stays as cli.console (3131) so llama-server hooks
hit the management API which has the /mesh/hook route.
Removed the /mesh/hook intercept from the proxy — it was a
workaround for both services fighting over port 3131.
* clean up virtual_llm: separate handlers per hook, shared helper
- handle_post_prefill (Hook 2) and handle_mid_generation (Hook 2b)
are now separate functions, both using race_second_opinion helper
- Removed dead DashMap TODO — everything is synchronous
- Removed stale doc comments about background work pattern
- 152 insertions, 218 deletions — net -66 lines
* typed handlers: handle_image, handle_uncertain, handle_drift, handle_verify
Route handler parses payload once and passes typed args to each
function — no more stringly-typed Value digging in the handlers.
Dropped very_short trigger from C++ and Rust — a short response
to a long prompt isn't necessarily wrong.
handle_uncertain: model stuck at start (high entropy first token)
handle_drift: model losing coherence mid-generation (sustained spike)
handle_verify: check output before sending (tail spike / high uncertainty)
get_peer_hint: shared helper, races 2 peers for a second opinion
* doc: crisp handler docs with args, return values, and behavior
* docs: move racing detail to get_peer_hint, simplify caller docs
* typed handle_image args, find_audio_peer, video placeholder
handle_image now takes (image_url, user_text) instead of raw payload.
extract_image moved to pub for route handler to call.
Added find_audio_peer (same pattern as find_vision_peer).
transcribe_audio finds a capable peer but audio extraction not yet wired.
video_no_support placeholder.
* lower entropy threshold from 5.0 to 3.0
5.0 was unreachable — even uncertain models rarely hit it because
it requires ~32 equally-likely tokens. 3.0 (~8 equally-likely tokens)
fires on genuine uncertainty.
Tested with Qwen3-0.6B: hooks fire on hard questions (Swahili
translation, obscure facts). Hook 3 caught high_uncertainty
(mean_entropy=3.46), Hook 2b caught mid-gen drift (tail_entropy=6.28).
Gemma-4B still doesn't trigger — it's well-calibrated.
* peer selection: exclude smaller models, prefer larger; fix context leak; eval script
Peer selection now filters out models with a lower tier than the
current model. No point asking a weaker model for help. Larger
models are slightly preferred (tier 4 > tier 3 > same tier).
Changed injection format from '[Context: ...]' to a natural
instruction ('Here is relevant information...') so models don't
echo the wrapper text in their output.
Added evals/virtual_llm_eval.py — runs a set of questions with
and without hooks, captures timing and responses to JSONL.
* deduplicate peers by model name, eval results
Two nodes running the same model don't provide diversity — they
give the same answer. Keep only the best-scored node per model name.
First eval results with Qwen3-0.6B:
- Easy questions: no hooks fire, no overhead (0.2s same)
- Swahili translation: hooks fire but model too weak to use hint
- Nauru population: improved from 200k to 24k (real ~12k)
- 2 questions got 10s slower (MiniMax timeout on verification)
* drop Hook 3 (pre_response verify)
Hook 3 mostly timed out — by the time generation is done, a 10s
sync verification adds latency for marginal value. The useful work
happens in Hook 2 (uncertain start) and Hook 2b (mid-gen drift),
both of which inject context before/during generation.
Removed from C++: entire pre_response block in send_final_response,
verify flag from mesh_hook_ctx.
Removed from Rust: handle_verify, verify_response, pre_response route.
Removed find_different_model_peer (single-peer wrapper, unused).
-197 lines across 5 files.
* injection framing test + switch to 'reference' framing
Integration test (virtual_llm_injection.rs) that spins up a mock
hook server + llama-server, sends factual/translation/reasoning
questions, and verifies the model incorporates injected hints.
Tested 4 framings against Qwen3-0.6B baseline (1/5 correct):
- 'reference': 5/5, cleanest output, no artifacts
- 'assistant_draft': 5/5, clean
- 'current': 5/5, leaks </think> into content
- 'rag': 5/5, sometimes echoes question
Switched production framing to 'Reference answer: ...' — produces
the cleanest model output with no artifacts.
Also: debug mode now always fires Hook 2 (post_prefill), regardless
of entropy. The first token is always confident for thinking models
(<think>), so entropy gating in debug mode was preventing testing.
* TODO: virtual LLM remaining work items
Track slow peer consultation investigation (MiniMax timeout issue),
peer responsiveness tracking, audio extraction, and non-thinking
model testing.
* fix TODO: MiniMax was slow, not failing
* TODO: use TTFT perf tracker for consultation peer selection (PR #271)
* prevent recursive consultation loops
Outgoing consultation requests now include mesh_hooks: false,
which tells the peer's llama-server to skip hook callbacks for
that request. Without this, peer A could consult peer B, which
could consult peer C, etc.
Flagged by GPT-5.4 code review as a critical production risk.
* per-hook timeouts and compact mid-gen injection
Hook 2 (pre-generation): 15s timeout — user is waiting for first
token anyway, longer timeout lets slower but better peers respond.
Hook 2b (mid-generation): 5s timeout — user sees a stall if we
block too long. Also uses compact 'Key fact: ...' framing (256
char cap) instead of full 'Reference answer' (512 chars) to avoid
derailing the model's continuation style.
Hook 1 (media captioning): 15s timeout — captioning is a one-time
pre-generation cost.
* drop compact mid-gen framing — tested, worse than reference
Tested 'Key fact: X' (compact) vs 'Reference answer: X. Use the
reference above...' (reference). Compact got 4/5 — Swahili
translation failed because the model echoed 'Key fact:' back
instead of using the hint. The instruction to act on it matters.
Use 'reference' framing for both Hook 2 and 2b. Simpler, tested.
* two new mid-gen triggers: repetition loop + surprise break
GPT-5.4 research: entropy alone doesn't catch 'confidently wrong' —
Gemma-4B first-token entropy is 0.03-1.33 even on hallucinated
answers. Need different signals.
New trigger 1: REPETITION LOOP
Track last 32 token IDs, compute 3-gram repeat ratio. Fire when
ratio >= 0.18. Catches degenerate loops (Gemma stuck in 'I need
to access my knowledge base' patterns). Tested: fired correctly
on Nauru query at token 43, and on p-values explanation at tokens
101 and 153.
New trigger 2: SURPRISE BREAK
EWMA of -log(p_chosen) with z-score spike detection. Fires when
2+ tokens spike >2.5 sigma after a calm run of 4+ low-z tokens.
Catches 'was flowing confidently, then suddenly broke'.
Both signals added to mesh_signal_window struct and wired into
should_fire_midgen(). Any of the three triggers (entropy spike,
repetition, surprise) can independently fire Hook 2b, all sharing
the same cooldown.
Tested on Gemma-4B with real mesh peers:
- Repetition: 3 fires across 12 queries (all genuine loops)
- Entropy spike: 0 fires (Gemma too confident, as expected)
- Hook 2 (first-token): 2 fires on creative prompts (haiku)
where Gemma genuinely didn't know how to start
- Zero false positives on factual/math queries
* 20s consultation timeout across all hooks
Triggers are rare — when they fire, it's worth waiting. The 5s
mid-gen timeout was causing all repetition-loop consultations to
fail (peers need 6-10s). Unified to a single 20s constant.
* add rank instability tracking, disable as trigger
Implemented top-8 Jaccard similarity tracking between consecutive
generation steps. Computes overlap of top-8 candidate tokens —
low Jaccard means the model is considering completely different
continuations each step.
DISABLED as a trigger: Gemma-4B has inherently unstable top-8
distributions — fires on ~100% of queries even with strict
thresholds (5/6 window, 24-token warmup). The signal needs
per-model baseline calibration or combination with another signal.
Tracking code is kept (cheap, data in signals JSON) for research.
Repetition loop remains the most reliable mid-gen trigger.
* remove rank instability signal
100% false positive rate on Gemma-4B — top-8 candidates are
inherently unstable on modern models. Not a useful signal without
per-model calibration. -94 lines.
* Hook 1 working: image captioning on text-only models
When a text-only model (no mmproj) receives an image request:
1. server-common.cpp strips image to '[image attached]' text
instead of rejecting with 500 error (when mesh hooks enabled)
2. Original image URL preserved as mesh_image_url in messages
3. server-context.cpp detects mesh_image_url, fires Hook 1
4. Rust handler extracts image URL, consults vision peer
5. Caption injected into prompt before generation
Tested end-to-end with mock hook server + Qwen3-0.6B:
- Image request accepted (was: 500 error)
- Hook 1 fires with images_no_multimodal trigger
- Caption injected (13 tokens)
- Model output matches caption content exactly
Also adds server-common.cpp to llama-patches/ (was only
server-common.h before) for the image stripping logic.
* update docs: VIRTUAL_LLM.md rewrite + TODO refresh
VIRTUAL_LLM.md:
- Removed Hook 3 (dropped earlier), background work pattern,
DashMap, mesh_request_id correlation (simplified away)
- Added Hook 2b with 3 triggers (repetition, entropy, surprise)
- Added signal detection research section with results
- Added injection framing test results
- Updated C++ file list (added server-common.cpp)
- Updated consultation section (fan-out, recursion guard, 20s timeout)
- Corrected entropy threshold (3.0 not 5.0)
TODO.md:
- Removed non-thinking model testing item (done — Gemma-4B tested)
- Added: test Hook 1 with real vision peer
- Added: push C++ to fork when stable
- Added: image caption caching
- Updated description to reflect current state
* docs: highlight inter-model collaboration across README, docs site, roadmap
README: add paragraph in 'How it works' section describing the feature,
link to VIRTUAL_LLM.md in 'More docs'.
docs/index.html: add feature card with brain emoji, add tag pill,
update research section to note collaboration is live.
ROADMAP.md: add 'Inter-model collaboration (Virtual LLM)' section
with working features, key insight, paper reference, and next steps.
* move inter-model collaboration from roadmap to README feature list
This is a shipped feature, not a roadmap item. Remove the ROADMAP
section and add a bullet to the top-level feature list instead.
* move llama.cpp to Mesh-LLM org fork, pin by SHA
Fork: github.com/Mesh-LLM/llama.cpp (master branch)
Pinned SHA in LLAMA_CPP_SHA (single source of truth)
Changes:
- All build scripts, CI workflows, Dockerfiles now use Mesh-LLM/llama.cpp
- CI reads SHA from LLAMA_CPP_SHA file, falls back to ls-remote
- Delete mesh-llm/llama-patches/ (-15k lines) — fork is the source of truth
- Add mesh-llm/docs/LLAMA_CPP_FORK.md with maintenance instructions:
how to update from upstream, how to tell an agent to sync, files we touch
- Justfile diff recipe updated for new fork layout
The fork carries 8 commits on master:
3x RPC (zero-transfer, alloc cache, B2B transfers)
4x MoE (expert mask, split tool, ranking, shared-expert fix)
1x mesh hooks (virtual LLM engine)
* AGENTS.md: document llama.cpp fork, warn agents not to update it unprompted
* build-mac: pin llama.cpp to SHA from LLAMA_CPP_SHA
Same behaviour as build-linux.sh: reads the pinned SHA, fetches it,
checks out detached. If LLAMA_CPP_SHA is missing, falls back to
pulling master HEAD.
* test: find CI model at ~/.models/ so hook test runs in CI
Was silently skipping because it only checked HuggingFace cache.
CI already downloads SmolLM2-135M to ~/.models/.
* Revert "test: find CI model at ~/.models/ so hook test runs in CI"
This reverts commit a33da501e4373a7eb5ce6776c4ba7bb66d754533.
* fix copilot review: SHA pinning consistency, UTF-8 safety, CI fail-fast
- build-release.sh: read LLAMA_CPP_SHA, hard fail if missing
- build-linux.sh, build-linux-rocm.sh: fall back to LLAMA_CPP_SHA file
when MESH_LLM_LLAMA_PIN_SHA env var not set
- ci.yml: fail if pinned SHA can't be checked out (was silent fallback)
- Fix UTF-8 byte-slice panics in consult.rs, virtual_llm.rs, test file
(use char_indices / chars().take() instead of byte slicing)
* harden mesh hooks: loopback-only guard, proper HTTP response parsing
- /mesh/hook rejects non-loopback callers with 403 — prevents remote
abuse when management API is on 0.0.0.0 via --listen-all
- consult.rs: parse HTTP status line, fail on non-200, require header
terminator, include raw response in error messages
* run virtual LLM hook integration test in CI
Test finds CI model at ~/.models/ (SmolLM2-135M, already cached).
Runs after llama-server build + model download on macOS job.
Exercises real C++ hooks: llama-server --mesh-port → mock → inject → generate.
* feat: "mesh" virtual model — auto-routes with inter-model hooks
Send "model": "mesh" to get smart routing (same as "auto") plus
mesh hook callbacks enabled. All other models pass through unhooks.
- inject_mesh_hooks_flag: byte-level injection into forwarded HTTP body
- "mesh" listed in /v1/models when any real model is served
- Smoke test: sends model=mesh, verifies response, checks llama-server
has --mesh-port in process args, verifies "mesh" in /v1/models
- 3 unit tests for byte injection (enabled/disabled/no-body)
* fix: trailing newline in lib.rs (rustfmt CI compat)
* trigger CI
* ci: retrigger after smoke test cleanup
* fix: revert trailing newline in lib.rs — cargo fmt removes it
* fix CI formatting: run cargo fmt --all from repo root
Running cargo fmt from mesh-llm/ subdir in a workspace resolves
formatting differently than from repo root. Switch both linux and
macOS CI jobs to 'cargo fmt --all -- --check' from repo root.
* fix: add trailing newline to lib.rs for Linux rustfmt
Linux and macOS rustfmt 1.94.1 disagree on trailing newlines in
short files. CI runs on Linux — match its expectation.
* fix: UTF-8 truncation safety, 403 reason phrase, doc hook count
- consult.rs/virtual_llm.rs: use char_indices().take_while() instead
of byte-slicing which can panic on multi-byte codepoints
- api/http.rs: add 403, 409, 429, 500, 503 reason phrases so
non-200 responses don't say 'HTTP/1.1 403 OK'
- VIRTUAL_LLM.md: fix 'Two hook points' → 'Three hook points'
* fix: tighten fork validation and build-mac pull error handling
- docker-precheck: require Mesh-LLM/llama.cpp URL explicitly instead
of also matching bare 'master' anywhere in the Dockerfile
- build-mac.sh: warn on git pull failure instead of silent || true
* fix: wire mesh model routing and hook injection through ingress.rs
The main API proxy path goes through ingress.rs, not transport.rs.
Without this, model=mesh fell through as an unknown model name, hit
"Model not found, trying first available", and hooks never fired.
- Add "mesh" to ingress.rs smart routing condition alongside "auto"
- Call inject_mesh_hooks_flag from ingress.rs before forwarding
- Make inject_mesh_hooks_flag pub for cross-module access
- Update proxy integration tests to tolerate injected mesh_hooks field
* fix: only inject mesh_hooks into request body for model=mesh
Previously mesh_hooks:false was injected into EVERY request going
through the proxy, mutating request bodies even for users who never
use the mesh model. Now non-mesh requests pass through completely
untouched — no body mutation, no extra JSON fields.
Restores exact-body proxy integration tests.
* pin LLAMA_CPP_SHA to ed279faf4 — mesh hooks default OFF
Hooks only fire when proxy sends mesh_hooks:true (model=mesh).
Non-mesh requests are completely unaffected.
* virtual LLM hooks are auto, not a separate model
Inter-model collaboration hooks now fire for all auto-routed requests
(model=auto or no model specified). The separate "mesh" virtual model
is removed — no extra entry in /v1/models, no new concept for users
to learn. Auto just gets smarter when peers are available.
- ingress.rs: hooks enabled for auto-routed requests
- transport.rs: same for idle/passive proxy path
- Removed "mesh" entry from /v1/models
- Updated CI smoke test to test auto with hooks
* router: drop hardcoded model table, use gossiped capabilities
Replace the static MODEL_PROFILES table (30 hand-curated entries with
vibes-based tiers and strengths) with capability-based filtering using
ModelCapabilities already gossiped from every peer.
Routing logic for auto:
- needs_tools → filter to models with tool_use capability
- Reasoning category → filter to models with reasoning capability
- Image category → filter to models with vision capability
- anything else → no filter, any model works
- fallback to all models if filter matches nothing
- spread load randomly among candidates
This means unknown models that gossip tool_use: Supported are now
routable for agentic requests without anyone updating a table.
-443 lines net, mostly the deleted profile table and tier scoring.
* capabilities: known tool-capable families fallback, test mesh_hooks injection
Add known_tool_capable_family() as a last-resort name heuristic for
models whose config files lack standard tool tokens but are known to
support tool calling: Qwen3*, MiniMax*, Hermes*, Gemma*.
Matches against path segments so repo paths like "Qwen/Qwen3-32B" work.
Only fires when no other tool_use signal was found (config scan, name
keywords). Sets tool_use: Likely, not Supported.
Also:
- Add proxy integration assertion: model=auto forwards mesh_hooks:true
to the upstream, closing the test gap between injection and hooks
- Remove unused pick_model_with_tools test helper (0 warnings now)
2026-04-16 14:54:58 +10:00
2026-05-05 11:48:40 +10:00
mesh-llm embeds the stage runtime and links patched llama.cpp static ABI
libraries. The only durable llama.cpp patch queue is
`third_party/llama.cpp/patches` , pinned by `third_party/llama.cpp/upstream.txt` .
Virtual LLM engine — callback hooks from llama-server into mesh (#225)
* docs: virtual LLM engine design — in-engine mesh hooks
Design for adding hooks inside llama-server's C++ token generation loop
so it can consult other models in the mesh during inference.
Hook points: pre-inference, per-token, pre-response, slot pause/resume.
Communication via HTTP callback to mesh-llm which routes to any model
in the mesh. Token injection into live KV cache without restarting.
Relates to #183, #165
* docs: uncertainty signals and callback design for virtual LLM hooks
Four signal types: per-token entropy, top-token margin, sequence variance
(confidence trajectory), and self-consistency via multi-completion.
Hooks are simple: measure signals in C++, call back to mesh-llm with the
data, mesh-llm decides what to do and responds with an action (inject,
continue, stop, none). All decision logic lives in the Rust side.
Single callback endpoint: POST /mesh/hook with hook type, signal data,
and generation context. Three callback modes: async fire, sync call,
poll check.
* docs: clean callback protocol and signal design for virtual LLM
Rewrite to separate concerns clearly:
- C++ side: compute signals (entropy, margin, window stats), call mesh-llm
- Rust side: all decision logic, model routing, consultation
JSON over localhost HTTP chosen over FFI/shared-memory because:
- cpp-httplib already linked, axum already running
- callback latency (~0.1ms) irrelevant vs consultation time (seconds)
- debuggable with curl, independently versionable
Five hooks, one callback shape, three actions (none/inject/stop).
Async consultation via fire-at-hook-1, poll-at-hook-3 pattern.
Per-request threshold config alongside existing task_params.
* docs: simplify to four hooks, drop per-token callback
Per-token callback was wrong — 30-100 tok/s makes it wasteful.
Signal computation stays per-token in C++ (cheap arithmetic),
but callbacks only fire at: pre-inference, post-prefill,
pre-response, and complete (telemetry).
Each hook now shows exactly what data it sends to mesh-llm:
- pre_inference: original messages array, images, model capabilities
- post_prefill: first-token entropy/margin, top-5 candidates
- pre_response: full generated text, signal summary stats
- complete: telemetry (fire-and-forget)
Mid-generation intervention deferred as optional threshold breakout.
* docs: rewrite VIRTUAL_LLM.md — hooks, callbacks, examples only
Cut all proxy discussion, architecture justification, and research notes.
Doc now covers: callback protocol, four hooks with exact JSON payloads
and worked examples, signal computation, token injection mechanics,
and the specific C++ and Rust changes needed.
* docs: add async hooks — pending action + poll for non-blocking consultation
Hook 1 can return 'pending' with an async_id instead of blocking.
llama-server stores the id and polls GET /mesh/hook/poll/{id} every
16 tokens during generation. When the result is ready (200), tokens
are injected into the live KV cache. When not ready (202), generation
continues uninterrupted.
Each hook now annotated with blocking behavior:
- pre_inference: sync or async (mesh-llm decides)
- post_prefill: always sync (just reads numbers, <1ms)
- pre_response: always sync (need verdict before sending)
- complete: fire-and-forget
* docs: drop complete hook, three hooks is enough
Telemetry can be added later if needed. The core mechanism is
pre_inference, post_prefill, pre_response + async polling.
* docs: clarify hook triggers — always fire, mesh-llm decides
* docs: Hook 1 configures hooks 2 and 3 via entropy_threshold and verify
Hook 1 always fires — it's the setup hook. Its response tells C++
what to watch for: entropy_threshold enables Hook 2, verify enables
Hook 3. Most requests: only Hook 1 fires, mesh-llm returns none.
* docs: trigger-based hooks — only fire when structural criteria met
All hooks are now conditional:
- Hook 1: images+text-only, context pressure (>75% of ctx),
long session (>10 turns), large user paste
- Hook 2: first-token entropy > threshold (set by Hook 1)
- Hook 3: max_tokens cutoff, very short response, high uncertainty,
tail entropy spike, mid-sentence cutoff
- Polling: only when async work pending
Added examples: context pressure → conversation summarization,
max_tokens cutoff → summarize + continue, very short response →
retry with encouragement, tail entropy spike → verify ending.
* docs: complete rewrite — triggers, examples, implementation details
Three sections:
1. Triggers and hooks — what fires when, one table per hook
2. Examples — concrete JSON for each scenario (captioning, summarization,
context pressure, refusal, truncation, hallucination)
3. Implementation — exact C++ code at each hook point in server-context.cpp,
signal window struct, inject helper, mesh-llm endpoints and module
* docs: full callback payloads under each hook
Each hook now shows the exact JSON mesh-llm receives:
- Hook 1: full messages array (images, audio, all turns), token counts,
context size, trigger name, model capabilities
- Hook 2: first-token signals only (entropy, margin, top-5), request_id
links back to Hook 1 data
- Hook 3: full generated text, stop reason, signal summary, request_id
links back to original request
* docs: all hooks send full messages — can't assume Hook 1 fired
Hook 2 and 3 now include the messages array. Hook 1 only fires
on specific triggers, so mesh-llm may never have seen the request.
Each hook is self-contained — sends everything mesh-llm needs.
* api: add /mesh/hook endpoint for llama-server callbacks
New route handler for mesh hook callbacks. Three hooks handled:
- pre_inference: logs trigger, returns none + entropy_threshold
- post_prefill: logs entropy signal, returns none
- pre_response: logs n_decoded + stop_reason, returns none
Poll endpoint GET /mesh/hook/poll/{id} returns 202 (stub).
All hooks return 'none' for now — plumbing first, decision logic next.
* launch: pass --mesh-port to llama-server for hook callbacks
Sets MESH_API_PORT env var at runtime startup, launch.rs reads it
and passes --mesh-port {port} to llama-server. This tells llama-server
where to POST hook callbacks on localhost.
* llama-patches: snapshot C++ mesh hook changes for co-iteration
Contains the git format-patch from llama.cpp mesh-hooks branch plus
the standalone header file. This lets us iterate on C++ and Rust
together on micn/virtual-llm. When stable, apply the patch to the
llama.cpp fork's mesh-hooks branch and remove this directory.
* llama-patches: update with trigger fixes and e2e test results
* llama-patches: update with working token injection
* TEMPORARY: inline C++ mesh hook files for single-repo iteration
llama-patches/ contains the 7 modified/new C++ files that implement
mesh hooks in llama-server. sync.sh copies them into llama.cpp/ and
build-mac.sh runs it automatically after pulling upstream-latest.
This is a temporary setup for the micn/virtual-llm branch so C++ and
Rust changes live in one repo / one PR. When stable, the C++ moves to
the mesh-hooks branch on the llama.cpp fork and this directory is deleted.
* inference: add virtual_llm decision module with documented stubs
New module inference/virtual_llm.rs — the brain behind mesh hooks.
Three handlers, one per hook type, each with:
- Documented trigger table
- Match on trigger name
- Logging with context (model, token counts, entropy, etc.)
- TODO comments describing the planned consultation
Also: HookAction enum, AsyncConsultations struct stub.
Route handler in mesh_hook.rs now delegates to virtual_llm instead
of inlining the decision logic.
* virtual_llm: add HookContext for model-aware decisions
HookContext enriches the model filename from the C++ hook payload with:
- Tier, strengths, tools from ModelProfile
- Vision/multimodal from ModelCapabilities
- Available peers by capability (vision, stronger, all)
This lets the decision engine pick complementary models: vision when
text-only, stronger when uncertain, different specialty when needed.
Struct is defined but not yet wired — handlers still take raw payload.
* simplify: remove polling, all hooks are synchronous
Delete the async poll mechanism (pending_async_ids, poll_async,
should_poll, GET /mesh/hook/poll/{id}). All hooks are now plain
synchronous POST calls.
Background work uses a simpler pattern: Hook 1 spawns a tokio task
and stores it in a DashMap keyed by request_id. Hook 3 checks the
map — if the result is ready, use it; if not, let the response go.
No polling needed.
C++ side: removed poll loop from generation, cleaned up mesh_hook_ctx.
Rust side: removed poll endpoint, removed Pending variant, updated docs.
* add mesh_request_id for request correlation, rewrite design doc
mesh-llm generates a mesh_request_id and includes it in the request
body. llama-server passes it back in every hook payload. mesh-llm
stores the original request (messages, images, etc.) keyed by this
ID so hooks can access the full conversation without C++ sending it.
Design doc rewritten to match current implementation:
- Removed polling/pending (all hooks sync now)
- Documented background work pattern (Hook 1 spawns, Hook 3 collects)
- Documented request correlation via mesh_request_id
- Updated all example payloads and scenario descriptions
- Updated implementation tables to match actual files
* implement Hook 2 KV cache injection
After prefill, if the model is uncertain (high entropy / low margin),
mesh-llm can return inject text. This text is now tokenized and
decoded into the KV cache via a temporary batch — the model 'sees'
the injected context before generating its first token.
The injection uses the same chunked decode pattern as normal prefill:
tokens are added to a temp batch, decoded in n_batch-sized chunks,
and positions are tracked via slot.prompt.tokens. Only the last
inject token requests logits, and slot.i_batch is set so sampling
reads from the correct position.
This is the key mechanism that makes the virtual LLM interesting —
a small model that's uncertain can receive context from a stronger
model mid-inference, changing the trajectory of generation.
* update docs for Hook 2 KV injection
- Design doc: Hook 2 description updated from 'informational' to
documenting inject action, how KV cache injection works, and new
scenario showing uncertain model receiving live help
- virtual_llm.rs: lifecycle diagram updated (Hook 2 now returns
inject, shows KV decode step), function doc describes injection
mechanism, removed stale 'pending' action from Hook 1
* implement real consultation logic: image captioning, summarization, second opinion, verification
New module: inference/consult.rs — peer consultation over QUIC mesh.
Provides four patterns:
- caption_image: send image to vision peer, get text description
- summarize_conversation: condense early turns to free context
- second_opinion: ask a different model the same question
- verify_response: check if a response's ending is accurate
Peer discovery finds peers by capability (vision) or by difference
(different model architecture). Prefers low-RTT peers. The value of
Hook 2 and Hook 3 verification is diversity — a different model's
perspective, not necessarily a stronger one.
Hook handlers are now async, accepting the mesh Node to find and
consult peers. All four scenarios wired:
- Hook 1 images_no_multimodal → caption via vision peer → inject
- Hook 1 context_pressure → summarize via any peer → inject
- Hook 2 high_entropy → second opinion from different model → inject into KV
- Hook 3 tail_entropy_spike/verify → verify via different model → append correction
Added MeshApi::node() helper for route handlers.
* prefer similar-tier peers, send only last message, keep injection concise
find_different_model_peer now scores by tier distance (±1 preferred)
then RTT. A 4B model asks another small model, not a 70B.
second_opinion sends only the last user message with 'answer briefly
in 2-3 sentences' — max 192 tokens. The peer returns a concise
answer, not a full essay. Keeps the blocking time and KV injection
short.
Injection text capped at 512 chars. Every injected token is decoded
into KV cache and adds latency — brevity matters.
* only annotate truncation when response looks cut off mid-sentence
max_tokens fires on every request that hits the token limit, including
intentional truncation (user set max_tokens: 100). Now we check if
the response actually ends mid-thought: no sentence-ending punctuation
means genuinely cut off, annotate it. Clean ending (period, question
mark, etc.) means the model finished naturally, skip the note.
* remove max_tokens from Hook 3 triggers
max_tokens is normal operation — the user or server set a limit and
the model hit it. finish_reason: 'length' already signals this.
Firing a hook round-trip to append a redundant note adds latency
for no value.
Hook 3 now only fires on signals that indicate the model struggled:
very_short, high_uncertainty, tail_entropy_spike, verify.
Removed from C++ trigger evaluation, Rust handler, and design doc.
* fix: add trailing newline to lib.rs for CI rustfmt
* fix: use console port (3131) for mesh hooks, not proxy port (9337)
MESH_API_PORT was set to cli.port (proxy, 9337) instead of
cli.console (management API, 3131). Hooks calling the proxy
would loop back to llama-server.
* Hook 2 always armed — default entropy_threshold 5.0
Hook 2 (post-prefill uncertainty check) no longer depends on Hook 1
firing first. It stands on its own: if the model is uncertain after
reading the prompt, ask a peer. Hook 1 can still override the
threshold for specific request types.
* Hook 1: only media triggers — drop context_pressure, long_session, large_user_message
Hook 1 is for when the model can't handle the input modality:
images without vision, audio without audio support. These are
clear problems with clear solutions (caption, transcribe).
context_pressure/long_session/large_user_message were speculative
triggers that didn't have good actions — appending a summary to
an already-long prompt makes it longer, not shorter.
The interesting hooks during inference are Hook 2 (uncertainty)
and Hook 3 (bad output signals). Those stand on their own now.
Also removes unused find_any_peer and summarize_conversation.
* mid-generation Hook 2b, Hook 3 replace, --mesh-hook-debug
Hook 2b: fires during generation when sustained entropy spike
detected (75% of rolling window has entropy > 4.0). Same KV cache
injection as Hook 2 — model continues generating but now informed
by peer context. 32-token cooldown between fires.
Hook 3: 'replace' action replaces generated_text entirely instead
of appending. When verification says the response is bad, the
peer's corrected answer replaces the model's output.
--mesh-hook-debug / MESH_HOOK_DEBUG: lowers all thresholds so hooks
fire on almost any request. entropy_threshold 0.5, mid-gen spike
ratio 0.25, cooldown 8 tokens. For testing.
* pass messages in all hook payloads, handle /mesh/hook in proxy
Messages stored in mesh_messages on task params, included in all
hook payloads (Hook 1, 2, 2b, 3). Without this, Hook 2 had no
messages to send to a peer for second opinion.
/mesh/hook handled directly in the proxy dispatcher since the
management API and proxy share port 3131 after main consolidated
ports. Proxy intercepts the route before forwarding to llama-server.
Tested: hooks fire, peer consultation initiates over QUIC mesh.
Hook 2 finds Qwen3-8B on the public mesh when local model is gemma.
Consultation latency is high (~60s) — needs investigation.
* fan-out consultation: race 2 peers, 10s timeout
Hook 2 now finds the top 2 different-model peers and races them
via JoinSet. First successful response wins, loser gets aborted.
If only 1 peer exists, falls back to single call.
10s timeout on all consultation calls — hooks block the local
model's slot, can't wait forever for a remote peer.
Verify (Hook 3) trims generated_text to last 500 chars and caps
at 256 tokens — the tail is where the spike happened.
Tested on public mesh: raced Hermes-7B vs Qwen3-8B, Qwen won
in 7s, total request time 12s (was 65s before timeout).
* fix port split: proxy on 9337, management API on 3131
Reverted api_port back to cli.port (9337) so the proxy binds there.
MESH_API_PORT stays as cli.console (3131) so llama-server hooks
hit the management API which has the /mesh/hook route.
Removed the /mesh/hook intercept from the proxy — it was a
workaround for both services fighting over port 3131.
* clean up virtual_llm: separate handlers per hook, shared helper
- handle_post_prefill (Hook 2) and handle_mid_generation (Hook 2b)
are now separate functions, both using race_second_opinion helper
- Removed dead DashMap TODO — everything is synchronous
- Removed stale doc comments about background work pattern
- 152 insertions, 218 deletions — net -66 lines
* typed handlers: handle_image, handle_uncertain, handle_drift, handle_verify
Route handler parses payload once and passes typed args to each
function — no more stringly-typed Value digging in the handlers.
Dropped very_short trigger from C++ and Rust — a short response
to a long prompt isn't necessarily wrong.
handle_uncertain: model stuck at start (high entropy first token)
handle_drift: model losing coherence mid-generation (sustained spike)
handle_verify: check output before sending (tail spike / high uncertainty)
get_peer_hint: shared helper, races 2 peers for a second opinion
* doc: crisp handler docs with args, return values, and behavior
* docs: move racing detail to get_peer_hint, simplify caller docs
* typed handle_image args, find_audio_peer, video placeholder
handle_image now takes (image_url, user_text) instead of raw payload.
extract_image moved to pub for route handler to call.
Added find_audio_peer (same pattern as find_vision_peer).
transcribe_audio finds a capable peer but audio extraction not yet wired.
video_no_support placeholder.
* lower entropy threshold from 5.0 to 3.0
5.0 was unreachable — even uncertain models rarely hit it because
it requires ~32 equally-likely tokens. 3.0 (~8 equally-likely tokens)
fires on genuine uncertainty.
Tested with Qwen3-0.6B: hooks fire on hard questions (Swahili
translation, obscure facts). Hook 3 caught high_uncertainty
(mean_entropy=3.46), Hook 2b caught mid-gen drift (tail_entropy=6.28).
Gemma-4B still doesn't trigger — it's well-calibrated.
* peer selection: exclude smaller models, prefer larger; fix context leak; eval script
Peer selection now filters out models with a lower tier than the
current model. No point asking a weaker model for help. Larger
models are slightly preferred (tier 4 > tier 3 > same tier).
Changed injection format from '[Context: ...]' to a natural
instruction ('Here is relevant information...') so models don't
echo the wrapper text in their output.
Added evals/virtual_llm_eval.py — runs a set of questions with
and without hooks, captures timing and responses to JSONL.
* deduplicate peers by model name, eval results
Two nodes running the same model don't provide diversity — they
give the same answer. Keep only the best-scored node per model name.
First eval results with Qwen3-0.6B:
- Easy questions: no hooks fire, no overhead (0.2s same)
- Swahili translation: hooks fire but model too weak to use hint
- Nauru population: improved from 200k to 24k (real ~12k)
- 2 questions got 10s slower (MiniMax timeout on verification)
* drop Hook 3 (pre_response verify)
Hook 3 mostly timed out — by the time generation is done, a 10s
sync verification adds latency for marginal value. The useful work
happens in Hook 2 (uncertain start) and Hook 2b (mid-gen drift),
both of which inject context before/during generation.
Removed from C++: entire pre_response block in send_final_response,
verify flag from mesh_hook_ctx.
Removed from Rust: handle_verify, verify_response, pre_response route.
Removed find_different_model_peer (single-peer wrapper, unused).
-197 lines across 5 files.
* injection framing test + switch to 'reference' framing
Integration test (virtual_llm_injection.rs) that spins up a mock
hook server + llama-server, sends factual/translation/reasoning
questions, and verifies the model incorporates injected hints.
Tested 4 framings against Qwen3-0.6B baseline (1/5 correct):
- 'reference': 5/5, cleanest output, no artifacts
- 'assistant_draft': 5/5, clean
- 'current': 5/5, leaks </think> into content
- 'rag': 5/5, sometimes echoes question
Switched production framing to 'Reference answer: ...' — produces
the cleanest model output with no artifacts.
Also: debug mode now always fires Hook 2 (post_prefill), regardless
of entropy. The first token is always confident for thinking models
(<think>), so entropy gating in debug mode was preventing testing.
* TODO: virtual LLM remaining work items
Track slow peer consultation investigation (MiniMax timeout issue),
peer responsiveness tracking, audio extraction, and non-thinking
model testing.
* fix TODO: MiniMax was slow, not failing
* TODO: use TTFT perf tracker for consultation peer selection (PR #271)
* prevent recursive consultation loops
Outgoing consultation requests now include mesh_hooks: false,
which tells the peer's llama-server to skip hook callbacks for
that request. Without this, peer A could consult peer B, which
could consult peer C, etc.
Flagged by GPT-5.4 code review as a critical production risk.
* per-hook timeouts and compact mid-gen injection
Hook 2 (pre-generation): 15s timeout — user is waiting for first
token anyway, longer timeout lets slower but better peers respond.
Hook 2b (mid-generation): 5s timeout — user sees a stall if we
block too long. Also uses compact 'Key fact: ...' framing (256
char cap) instead of full 'Reference answer' (512 chars) to avoid
derailing the model's continuation style.
Hook 1 (media captioning): 15s timeout — captioning is a one-time
pre-generation cost.
* drop compact mid-gen framing — tested, worse than reference
Tested 'Key fact: X' (compact) vs 'Reference answer: X. Use the
reference above...' (reference). Compact got 4/5 — Swahili
translation failed because the model echoed 'Key fact:' back
instead of using the hint. The instruction to act on it matters.
Use 'reference' framing for both Hook 2 and 2b. Simpler, tested.
* two new mid-gen triggers: repetition loop + surprise break
GPT-5.4 research: entropy alone doesn't catch 'confidently wrong' —
Gemma-4B first-token entropy is 0.03-1.33 even on hallucinated
answers. Need different signals.
New trigger 1: REPETITION LOOP
Track last 32 token IDs, compute 3-gram repeat ratio. Fire when
ratio >= 0.18. Catches degenerate loops (Gemma stuck in 'I need
to access my knowledge base' patterns). Tested: fired correctly
on Nauru query at token 43, and on p-values explanation at tokens
101 and 153.
New trigger 2: SURPRISE BREAK
EWMA of -log(p_chosen) with z-score spike detection. Fires when
2+ tokens spike >2.5 sigma after a calm run of 4+ low-z tokens.
Catches 'was flowing confidently, then suddenly broke'.
Both signals added to mesh_signal_window struct and wired into
should_fire_midgen(). Any of the three triggers (entropy spike,
repetition, surprise) can independently fire Hook 2b, all sharing
the same cooldown.
Tested on Gemma-4B with real mesh peers:
- Repetition: 3 fires across 12 queries (all genuine loops)
- Entropy spike: 0 fires (Gemma too confident, as expected)
- Hook 2 (first-token): 2 fires on creative prompts (haiku)
where Gemma genuinely didn't know how to start
- Zero false positives on factual/math queries
* 20s consultation timeout across all hooks
Triggers are rare — when they fire, it's worth waiting. The 5s
mid-gen timeout was causing all repetition-loop consultations to
fail (peers need 6-10s). Unified to a single 20s constant.
* add rank instability tracking, disable as trigger
Implemented top-8 Jaccard similarity tracking between consecutive
generation steps. Computes overlap of top-8 candidate tokens —
low Jaccard means the model is considering completely different
continuations each step.
DISABLED as a trigger: Gemma-4B has inherently unstable top-8
distributions — fires on ~100% of queries even with strict
thresholds (5/6 window, 24-token warmup). The signal needs
per-model baseline calibration or combination with another signal.
Tracking code is kept (cheap, data in signals JSON) for research.
Repetition loop remains the most reliable mid-gen trigger.
* remove rank instability signal
100% false positive rate on Gemma-4B — top-8 candidates are
inherently unstable on modern models. Not a useful signal without
per-model calibration. -94 lines.
* Hook 1 working: image captioning on text-only models
When a text-only model (no mmproj) receives an image request:
1. server-common.cpp strips image to '[image attached]' text
instead of rejecting with 500 error (when mesh hooks enabled)
2. Original image URL preserved as mesh_image_url in messages
3. server-context.cpp detects mesh_image_url, fires Hook 1
4. Rust handler extracts image URL, consults vision peer
5. Caption injected into prompt before generation
Tested end-to-end with mock hook server + Qwen3-0.6B:
- Image request accepted (was: 500 error)
- Hook 1 fires with images_no_multimodal trigger
- Caption injected (13 tokens)
- Model output matches caption content exactly
Also adds server-common.cpp to llama-patches/ (was only
server-common.h before) for the image stripping logic.
* update docs: VIRTUAL_LLM.md rewrite + TODO refresh
VIRTUAL_LLM.md:
- Removed Hook 3 (dropped earlier), background work pattern,
DashMap, mesh_request_id correlation (simplified away)
- Added Hook 2b with 3 triggers (repetition, entropy, surprise)
- Added signal detection research section with results
- Added injection framing test results
- Updated C++ file list (added server-common.cpp)
- Updated consultation section (fan-out, recursion guard, 20s timeout)
- Corrected entropy threshold (3.0 not 5.0)
TODO.md:
- Removed non-thinking model testing item (done — Gemma-4B tested)
- Added: test Hook 1 with real vision peer
- Added: push C++ to fork when stable
- Added: image caption caching
- Updated description to reflect current state
* docs: highlight inter-model collaboration across README, docs site, roadmap
README: add paragraph in 'How it works' section describing the feature,
link to VIRTUAL_LLM.md in 'More docs'.
docs/index.html: add feature card with brain emoji, add tag pill,
update research section to note collaboration is live.
ROADMAP.md: add 'Inter-model collaboration (Virtual LLM)' section
with working features, key insight, paper reference, and next steps.
* move inter-model collaboration from roadmap to README feature list
This is a shipped feature, not a roadmap item. Remove the ROADMAP
section and add a bullet to the top-level feature list instead.
* move llama.cpp to Mesh-LLM org fork, pin by SHA
Fork: github.com/Mesh-LLM/llama.cpp (master branch)
Pinned SHA in LLAMA_CPP_SHA (single source of truth)
Changes:
- All build scripts, CI workflows, Dockerfiles now use Mesh-LLM/llama.cpp
- CI reads SHA from LLAMA_CPP_SHA file, falls back to ls-remote
- Delete mesh-llm/llama-patches/ (-15k lines) — fork is the source of truth
- Add mesh-llm/docs/LLAMA_CPP_FORK.md with maintenance instructions:
how to update from upstream, how to tell an agent to sync, files we touch
- Justfile diff recipe updated for new fork layout
The fork carries 8 commits on master:
3x RPC (zero-transfer, alloc cache, B2B transfers)
4x MoE (expert mask, split tool, ranking, shared-expert fix)
1x mesh hooks (virtual LLM engine)
* AGENTS.md: document llama.cpp fork, warn agents not to update it unprompted
* build-mac: pin llama.cpp to SHA from LLAMA_CPP_SHA
Same behaviour as build-linux.sh: reads the pinned SHA, fetches it,
checks out detached. If LLAMA_CPP_SHA is missing, falls back to
pulling master HEAD.
* test: find CI model at ~/.models/ so hook test runs in CI
Was silently skipping because it only checked HuggingFace cache.
CI already downloads SmolLM2-135M to ~/.models/.
* Revert "test: find CI model at ~/.models/ so hook test runs in CI"
This reverts commit a33da501e4373a7eb5ce6776c4ba7bb66d754533.
* fix copilot review: SHA pinning consistency, UTF-8 safety, CI fail-fast
- build-release.sh: read LLAMA_CPP_SHA, hard fail if missing
- build-linux.sh, build-linux-rocm.sh: fall back to LLAMA_CPP_SHA file
when MESH_LLM_LLAMA_PIN_SHA env var not set
- ci.yml: fail if pinned SHA can't be checked out (was silent fallback)
- Fix UTF-8 byte-slice panics in consult.rs, virtual_llm.rs, test file
(use char_indices / chars().take() instead of byte slicing)
* harden mesh hooks: loopback-only guard, proper HTTP response parsing
- /mesh/hook rejects non-loopback callers with 403 — prevents remote
abuse when management API is on 0.0.0.0 via --listen-all
- consult.rs: parse HTTP status line, fail on non-200, require header
terminator, include raw response in error messages
* run virtual LLM hook integration test in CI
Test finds CI model at ~/.models/ (SmolLM2-135M, already cached).
Runs after llama-server build + model download on macOS job.
Exercises real C++ hooks: llama-server --mesh-port → mock → inject → generate.
* feat: "mesh" virtual model — auto-routes with inter-model hooks
Send "model": "mesh" to get smart routing (same as "auto") plus
mesh hook callbacks enabled. All other models pass through unhooks.
- inject_mesh_hooks_flag: byte-level injection into forwarded HTTP body
- "mesh" listed in /v1/models when any real model is served
- Smoke test: sends model=mesh, verifies response, checks llama-server
has --mesh-port in process args, verifies "mesh" in /v1/models
- 3 unit tests for byte injection (enabled/disabled/no-body)
* fix: trailing newline in lib.rs (rustfmt CI compat)
* trigger CI
* ci: retrigger after smoke test cleanup
* fix: revert trailing newline in lib.rs — cargo fmt removes it
* fix CI formatting: run cargo fmt --all from repo root
Running cargo fmt from mesh-llm/ subdir in a workspace resolves
formatting differently than from repo root. Switch both linux and
macOS CI jobs to 'cargo fmt --all -- --check' from repo root.
* fix: add trailing newline to lib.rs for Linux rustfmt
Linux and macOS rustfmt 1.94.1 disagree on trailing newlines in
short files. CI runs on Linux — match its expectation.
* fix: UTF-8 truncation safety, 403 reason phrase, doc hook count
- consult.rs/virtual_llm.rs: use char_indices().take_while() instead
of byte-slicing which can panic on multi-byte codepoints
- api/http.rs: add 403, 409, 429, 500, 503 reason phrases so
non-200 responses don't say 'HTTP/1.1 403 OK'
- VIRTUAL_LLM.md: fix 'Two hook points' → 'Three hook points'
* fix: tighten fork validation and build-mac pull error handling
- docker-precheck: require Mesh-LLM/llama.cpp URL explicitly instead
of also matching bare 'master' anywhere in the Dockerfile
- build-mac.sh: warn on git pull failure instead of silent || true
* fix: wire mesh model routing and hook injection through ingress.rs
The main API proxy path goes through ingress.rs, not transport.rs.
Without this, model=mesh fell through as an unknown model name, hit
"Model not found, trying first available", and hooks never fired.
- Add "mesh" to ingress.rs smart routing condition alongside "auto"
- Call inject_mesh_hooks_flag from ingress.rs before forwarding
- Make inject_mesh_hooks_flag pub for cross-module access
- Update proxy integration tests to tolerate injected mesh_hooks field
* fix: only inject mesh_hooks into request body for model=mesh
Previously mesh_hooks:false was injected into EVERY request going
through the proxy, mutating request bodies even for users who never
use the mesh model. Now non-mesh requests pass through completely
untouched — no body mutation, no extra JSON fields.
Restores exact-body proxy integration tests.
* pin LLAMA_CPP_SHA to ed279faf4 — mesh hooks default OFF
Hooks only fire when proxy sends mesh_hooks:true (model=mesh).
Non-mesh requests are completely unaffected.
* virtual LLM hooks are auto, not a separate model
Inter-model collaboration hooks now fire for all auto-routed requests
(model=auto or no model specified). The separate "mesh" virtual model
is removed — no extra entry in /v1/models, no new concept for users
to learn. Auto just gets smarter when peers are available.
- ingress.rs: hooks enabled for auto-routed requests
- transport.rs: same for idle/passive proxy path
- Removed "mesh" entry from /v1/models
- Updated CI smoke test to test auto with hooks
* router: drop hardcoded model table, use gossiped capabilities
Replace the static MODEL_PROFILES table (30 hand-curated entries with
vibes-based tiers and strengths) with capability-based filtering using
ModelCapabilities already gossiped from every peer.
Routing logic for auto:
- needs_tools → filter to models with tool_use capability
- Reasoning category → filter to models with reasoning capability
- Image category → filter to models with vision capability
- anything else → no filter, any model works
- fallback to all models if filter matches nothing
- spread load randomly among candidates
This means unknown models that gossip tool_use: Supported are now
routable for agentic requests without anyone updating a table.
-443 lines net, mostly the deleted profile table and tier scoring.
* capabilities: known tool-capable families fallback, test mesh_hooks injection
Add known_tool_capable_family() as a last-resort name heuristic for
models whose config files lack standard tool tokens but are known to
support tool calling: Qwen3*, MiniMax*, Hermes*, Gemma*.
Matches against path segments so repo paths like "Qwen/Qwen3-32B" work.
Only fires when no other tool_use signal was found (config scan, name
keywords). Sets tool_use: Likely, not Supported.
Also:
- Add proxy integration assertion: model=auto forwards mesh_hooks:true
to the upstream, closing the test gap between injection and hooks
- Remove unused pick_model_with_tools test helper (0 warnings now)
2026-04-16 14:54:58 +10:00
2026-07-29 10:16:23 -04:00
- `just build` builds the UI and a dynamic host, then packages the selected
local runtime next to it. The host never links a backend library.
- Static llama.cpp compilation is the explicitly named native-runtime primitive
(`just build-runtime` / `scripts/package-native-runtime.sh --build` ), used
when changing the Skippy ABI or patch queue. It is not a host build path.
2026-05-05 11:48:40 +10:00
- Do not reintroduce an external `llama-server` / `rpc-server` runtime lane.
- If you need to update upstream llama.cpp, use `scripts/prepare-llama.sh` ,
`scripts/build-llama.sh` , `scripts/update-llama-pin.sh` , and
`scripts/summarize-llama-upstream.sh` .
Virtual LLM engine — callback hooks from llama-server into mesh (#225)
* docs: virtual LLM engine design — in-engine mesh hooks
Design for adding hooks inside llama-server's C++ token generation loop
so it can consult other models in the mesh during inference.
Hook points: pre-inference, per-token, pre-response, slot pause/resume.
Communication via HTTP callback to mesh-llm which routes to any model
in the mesh. Token injection into live KV cache without restarting.
Relates to #183, #165
* docs: uncertainty signals and callback design for virtual LLM hooks
Four signal types: per-token entropy, top-token margin, sequence variance
(confidence trajectory), and self-consistency via multi-completion.
Hooks are simple: measure signals in C++, call back to mesh-llm with the
data, mesh-llm decides what to do and responds with an action (inject,
continue, stop, none). All decision logic lives in the Rust side.
Single callback endpoint: POST /mesh/hook with hook type, signal data,
and generation context. Three callback modes: async fire, sync call,
poll check.
* docs: clean callback protocol and signal design for virtual LLM
Rewrite to separate concerns clearly:
- C++ side: compute signals (entropy, margin, window stats), call mesh-llm
- Rust side: all decision logic, model routing, consultation
JSON over localhost HTTP chosen over FFI/shared-memory because:
- cpp-httplib already linked, axum already running
- callback latency (~0.1ms) irrelevant vs consultation time (seconds)
- debuggable with curl, independently versionable
Five hooks, one callback shape, three actions (none/inject/stop).
Async consultation via fire-at-hook-1, poll-at-hook-3 pattern.
Per-request threshold config alongside existing task_params.
* docs: simplify to four hooks, drop per-token callback
Per-token callback was wrong — 30-100 tok/s makes it wasteful.
Signal computation stays per-token in C++ (cheap arithmetic),
but callbacks only fire at: pre-inference, post-prefill,
pre-response, and complete (telemetry).
Each hook now shows exactly what data it sends to mesh-llm:
- pre_inference: original messages array, images, model capabilities
- post_prefill: first-token entropy/margin, top-5 candidates
- pre_response: full generated text, signal summary stats
- complete: telemetry (fire-and-forget)
Mid-generation intervention deferred as optional threshold breakout.
* docs: rewrite VIRTUAL_LLM.md — hooks, callbacks, examples only
Cut all proxy discussion, architecture justification, and research notes.
Doc now covers: callback protocol, four hooks with exact JSON payloads
and worked examples, signal computation, token injection mechanics,
and the specific C++ and Rust changes needed.
* docs: add async hooks — pending action + poll for non-blocking consultation
Hook 1 can return 'pending' with an async_id instead of blocking.
llama-server stores the id and polls GET /mesh/hook/poll/{id} every
16 tokens during generation. When the result is ready (200), tokens
are injected into the live KV cache. When not ready (202), generation
continues uninterrupted.
Each hook now annotated with blocking behavior:
- pre_inference: sync or async (mesh-llm decides)
- post_prefill: always sync (just reads numbers, <1ms)
- pre_response: always sync (need verdict before sending)
- complete: fire-and-forget
* docs: drop complete hook, three hooks is enough
Telemetry can be added later if needed. The core mechanism is
pre_inference, post_prefill, pre_response + async polling.
* docs: clarify hook triggers — always fire, mesh-llm decides
* docs: Hook 1 configures hooks 2 and 3 via entropy_threshold and verify
Hook 1 always fires — it's the setup hook. Its response tells C++
what to watch for: entropy_threshold enables Hook 2, verify enables
Hook 3. Most requests: only Hook 1 fires, mesh-llm returns none.
* docs: trigger-based hooks — only fire when structural criteria met
All hooks are now conditional:
- Hook 1: images+text-only, context pressure (>75% of ctx),
long session (>10 turns), large user paste
- Hook 2: first-token entropy > threshold (set by Hook 1)
- Hook 3: max_tokens cutoff, very short response, high uncertainty,
tail entropy spike, mid-sentence cutoff
- Polling: only when async work pending
Added examples: context pressure → conversation summarization,
max_tokens cutoff → summarize + continue, very short response →
retry with encouragement, tail entropy spike → verify ending.
* docs: complete rewrite — triggers, examples, implementation details
Three sections:
1. Triggers and hooks — what fires when, one table per hook
2. Examples — concrete JSON for each scenario (captioning, summarization,
context pressure, refusal, truncation, hallucination)
3. Implementation — exact C++ code at each hook point in server-context.cpp,
signal window struct, inject helper, mesh-llm endpoints and module
* docs: full callback payloads under each hook
Each hook now shows the exact JSON mesh-llm receives:
- Hook 1: full messages array (images, audio, all turns), token counts,
context size, trigger name, model capabilities
- Hook 2: first-token signals only (entropy, margin, top-5), request_id
links back to Hook 1 data
- Hook 3: full generated text, stop reason, signal summary, request_id
links back to original request
* docs: all hooks send full messages — can't assume Hook 1 fired
Hook 2 and 3 now include the messages array. Hook 1 only fires
on specific triggers, so mesh-llm may never have seen the request.
Each hook is self-contained — sends everything mesh-llm needs.
* api: add /mesh/hook endpoint for llama-server callbacks
New route handler for mesh hook callbacks. Three hooks handled:
- pre_inference: logs trigger, returns none + entropy_threshold
- post_prefill: logs entropy signal, returns none
- pre_response: logs n_decoded + stop_reason, returns none
Poll endpoint GET /mesh/hook/poll/{id} returns 202 (stub).
All hooks return 'none' for now — plumbing first, decision logic next.
* launch: pass --mesh-port to llama-server for hook callbacks
Sets MESH_API_PORT env var at runtime startup, launch.rs reads it
and passes --mesh-port {port} to llama-server. This tells llama-server
where to POST hook callbacks on localhost.
* llama-patches: snapshot C++ mesh hook changes for co-iteration
Contains the git format-patch from llama.cpp mesh-hooks branch plus
the standalone header file. This lets us iterate on C++ and Rust
together on micn/virtual-llm. When stable, apply the patch to the
llama.cpp fork's mesh-hooks branch and remove this directory.
* llama-patches: update with trigger fixes and e2e test results
* llama-patches: update with working token injection
* TEMPORARY: inline C++ mesh hook files for single-repo iteration
llama-patches/ contains the 7 modified/new C++ files that implement
mesh hooks in llama-server. sync.sh copies them into llama.cpp/ and
build-mac.sh runs it automatically after pulling upstream-latest.
This is a temporary setup for the micn/virtual-llm branch so C++ and
Rust changes live in one repo / one PR. When stable, the C++ moves to
the mesh-hooks branch on the llama.cpp fork and this directory is deleted.
* inference: add virtual_llm decision module with documented stubs
New module inference/virtual_llm.rs — the brain behind mesh hooks.
Three handlers, one per hook type, each with:
- Documented trigger table
- Match on trigger name
- Logging with context (model, token counts, entropy, etc.)
- TODO comments describing the planned consultation
Also: HookAction enum, AsyncConsultations struct stub.
Route handler in mesh_hook.rs now delegates to virtual_llm instead
of inlining the decision logic.
* virtual_llm: add HookContext for model-aware decisions
HookContext enriches the model filename from the C++ hook payload with:
- Tier, strengths, tools from ModelProfile
- Vision/multimodal from ModelCapabilities
- Available peers by capability (vision, stronger, all)
This lets the decision engine pick complementary models: vision when
text-only, stronger when uncertain, different specialty when needed.
Struct is defined but not yet wired — handlers still take raw payload.
* simplify: remove polling, all hooks are synchronous
Delete the async poll mechanism (pending_async_ids, poll_async,
should_poll, GET /mesh/hook/poll/{id}). All hooks are now plain
synchronous POST calls.
Background work uses a simpler pattern: Hook 1 spawns a tokio task
and stores it in a DashMap keyed by request_id. Hook 3 checks the
map — if the result is ready, use it; if not, let the response go.
No polling needed.
C++ side: removed poll loop from generation, cleaned up mesh_hook_ctx.
Rust side: removed poll endpoint, removed Pending variant, updated docs.
* add mesh_request_id for request correlation, rewrite design doc
mesh-llm generates a mesh_request_id and includes it in the request
body. llama-server passes it back in every hook payload. mesh-llm
stores the original request (messages, images, etc.) keyed by this
ID so hooks can access the full conversation without C++ sending it.
Design doc rewritten to match current implementation:
- Removed polling/pending (all hooks sync now)
- Documented background work pattern (Hook 1 spawns, Hook 3 collects)
- Documented request correlation via mesh_request_id
- Updated all example payloads and scenario descriptions
- Updated implementation tables to match actual files
* implement Hook 2 KV cache injection
After prefill, if the model is uncertain (high entropy / low margin),
mesh-llm can return inject text. This text is now tokenized and
decoded into the KV cache via a temporary batch — the model 'sees'
the injected context before generating its first token.
The injection uses the same chunked decode pattern as normal prefill:
tokens are added to a temp batch, decoded in n_batch-sized chunks,
and positions are tracked via slot.prompt.tokens. Only the last
inject token requests logits, and slot.i_batch is set so sampling
reads from the correct position.
This is the key mechanism that makes the virtual LLM interesting —
a small model that's uncertain can receive context from a stronger
model mid-inference, changing the trajectory of generation.
* update docs for Hook 2 KV injection
- Design doc: Hook 2 description updated from 'informational' to
documenting inject action, how KV cache injection works, and new
scenario showing uncertain model receiving live help
- virtual_llm.rs: lifecycle diagram updated (Hook 2 now returns
inject, shows KV decode step), function doc describes injection
mechanism, removed stale 'pending' action from Hook 1
* implement real consultation logic: image captioning, summarization, second opinion, verification
New module: inference/consult.rs — peer consultation over QUIC mesh.
Provides four patterns:
- caption_image: send image to vision peer, get text description
- summarize_conversation: condense early turns to free context
- second_opinion: ask a different model the same question
- verify_response: check if a response's ending is accurate
Peer discovery finds peers by capability (vision) or by difference
(different model architecture). Prefers low-RTT peers. The value of
Hook 2 and Hook 3 verification is diversity — a different model's
perspective, not necessarily a stronger one.
Hook handlers are now async, accepting the mesh Node to find and
consult peers. All four scenarios wired:
- Hook 1 images_no_multimodal → caption via vision peer → inject
- Hook 1 context_pressure → summarize via any peer → inject
- Hook 2 high_entropy → second opinion from different model → inject into KV
- Hook 3 tail_entropy_spike/verify → verify via different model → append correction
Added MeshApi::node() helper for route handlers.
* prefer similar-tier peers, send only last message, keep injection concise
find_different_model_peer now scores by tier distance (±1 preferred)
then RTT. A 4B model asks another small model, not a 70B.
second_opinion sends only the last user message with 'answer briefly
in 2-3 sentences' — max 192 tokens. The peer returns a concise
answer, not a full essay. Keeps the blocking time and KV injection
short.
Injection text capped at 512 chars. Every injected token is decoded
into KV cache and adds latency — brevity matters.
* only annotate truncation when response looks cut off mid-sentence
max_tokens fires on every request that hits the token limit, including
intentional truncation (user set max_tokens: 100). Now we check if
the response actually ends mid-thought: no sentence-ending punctuation
means genuinely cut off, annotate it. Clean ending (period, question
mark, etc.) means the model finished naturally, skip the note.
* remove max_tokens from Hook 3 triggers
max_tokens is normal operation — the user or server set a limit and
the model hit it. finish_reason: 'length' already signals this.
Firing a hook round-trip to append a redundant note adds latency
for no value.
Hook 3 now only fires on signals that indicate the model struggled:
very_short, high_uncertainty, tail_entropy_spike, verify.
Removed from C++ trigger evaluation, Rust handler, and design doc.
* fix: add trailing newline to lib.rs for CI rustfmt
* fix: use console port (3131) for mesh hooks, not proxy port (9337)
MESH_API_PORT was set to cli.port (proxy, 9337) instead of
cli.console (management API, 3131). Hooks calling the proxy
would loop back to llama-server.
* Hook 2 always armed — default entropy_threshold 5.0
Hook 2 (post-prefill uncertainty check) no longer depends on Hook 1
firing first. It stands on its own: if the model is uncertain after
reading the prompt, ask a peer. Hook 1 can still override the
threshold for specific request types.
* Hook 1: only media triggers — drop context_pressure, long_session, large_user_message
Hook 1 is for when the model can't handle the input modality:
images without vision, audio without audio support. These are
clear problems with clear solutions (caption, transcribe).
context_pressure/long_session/large_user_message were speculative
triggers that didn't have good actions — appending a summary to
an already-long prompt makes it longer, not shorter.
The interesting hooks during inference are Hook 2 (uncertainty)
and Hook 3 (bad output signals). Those stand on their own now.
Also removes unused find_any_peer and summarize_conversation.
* mid-generation Hook 2b, Hook 3 replace, --mesh-hook-debug
Hook 2b: fires during generation when sustained entropy spike
detected (75% of rolling window has entropy > 4.0). Same KV cache
injection as Hook 2 — model continues generating but now informed
by peer context. 32-token cooldown between fires.
Hook 3: 'replace' action replaces generated_text entirely instead
of appending. When verification says the response is bad, the
peer's corrected answer replaces the model's output.
--mesh-hook-debug / MESH_HOOK_DEBUG: lowers all thresholds so hooks
fire on almost any request. entropy_threshold 0.5, mid-gen spike
ratio 0.25, cooldown 8 tokens. For testing.
* pass messages in all hook payloads, handle /mesh/hook in proxy
Messages stored in mesh_messages on task params, included in all
hook payloads (Hook 1, 2, 2b, 3). Without this, Hook 2 had no
messages to send to a peer for second opinion.
/mesh/hook handled directly in the proxy dispatcher since the
management API and proxy share port 3131 after main consolidated
ports. Proxy intercepts the route before forwarding to llama-server.
Tested: hooks fire, peer consultation initiates over QUIC mesh.
Hook 2 finds Qwen3-8B on the public mesh when local model is gemma.
Consultation latency is high (~60s) — needs investigation.
* fan-out consultation: race 2 peers, 10s timeout
Hook 2 now finds the top 2 different-model peers and races them
via JoinSet. First successful response wins, loser gets aborted.
If only 1 peer exists, falls back to single call.
10s timeout on all consultation calls — hooks block the local
model's slot, can't wait forever for a remote peer.
Verify (Hook 3) trims generated_text to last 500 chars and caps
at 256 tokens — the tail is where the spike happened.
Tested on public mesh: raced Hermes-7B vs Qwen3-8B, Qwen won
in 7s, total request time 12s (was 65s before timeout).
* fix port split: proxy on 9337, management API on 3131
Reverted api_port back to cli.port (9337) so the proxy binds there.
MESH_API_PORT stays as cli.console (3131) so llama-server hooks
hit the management API which has the /mesh/hook route.
Removed the /mesh/hook intercept from the proxy — it was a
workaround for both services fighting over port 3131.
* clean up virtual_llm: separate handlers per hook, shared helper
- handle_post_prefill (Hook 2) and handle_mid_generation (Hook 2b)
are now separate functions, both using race_second_opinion helper
- Removed dead DashMap TODO — everything is synchronous
- Removed stale doc comments about background work pattern
- 152 insertions, 218 deletions — net -66 lines
* typed handlers: handle_image, handle_uncertain, handle_drift, handle_verify
Route handler parses payload once and passes typed args to each
function — no more stringly-typed Value digging in the handlers.
Dropped very_short trigger from C++ and Rust — a short response
to a long prompt isn't necessarily wrong.
handle_uncertain: model stuck at start (high entropy first token)
handle_drift: model losing coherence mid-generation (sustained spike)
handle_verify: check output before sending (tail spike / high uncertainty)
get_peer_hint: shared helper, races 2 peers for a second opinion
* doc: crisp handler docs with args, return values, and behavior
* docs: move racing detail to get_peer_hint, simplify caller docs
* typed handle_image args, find_audio_peer, video placeholder
handle_image now takes (image_url, user_text) instead of raw payload.
extract_image moved to pub for route handler to call.
Added find_audio_peer (same pattern as find_vision_peer).
transcribe_audio finds a capable peer but audio extraction not yet wired.
video_no_support placeholder.
* lower entropy threshold from 5.0 to 3.0
5.0 was unreachable — even uncertain models rarely hit it because
it requires ~32 equally-likely tokens. 3.0 (~8 equally-likely tokens)
fires on genuine uncertainty.
Tested with Qwen3-0.6B: hooks fire on hard questions (Swahili
translation, obscure facts). Hook 3 caught high_uncertainty
(mean_entropy=3.46), Hook 2b caught mid-gen drift (tail_entropy=6.28).
Gemma-4B still doesn't trigger — it's well-calibrated.
* peer selection: exclude smaller models, prefer larger; fix context leak; eval script
Peer selection now filters out models with a lower tier than the
current model. No point asking a weaker model for help. Larger
models are slightly preferred (tier 4 > tier 3 > same tier).
Changed injection format from '[Context: ...]' to a natural
instruction ('Here is relevant information...') so models don't
echo the wrapper text in their output.
Added evals/virtual_llm_eval.py — runs a set of questions with
and without hooks, captures timing and responses to JSONL.
* deduplicate peers by model name, eval results
Two nodes running the same model don't provide diversity — they
give the same answer. Keep only the best-scored node per model name.
First eval results with Qwen3-0.6B:
- Easy questions: no hooks fire, no overhead (0.2s same)
- Swahili translation: hooks fire but model too weak to use hint
- Nauru population: improved from 200k to 24k (real ~12k)
- 2 questions got 10s slower (MiniMax timeout on verification)
* drop Hook 3 (pre_response verify)
Hook 3 mostly timed out — by the time generation is done, a 10s
sync verification adds latency for marginal value. The useful work
happens in Hook 2 (uncertain start) and Hook 2b (mid-gen drift),
both of which inject context before/during generation.
Removed from C++: entire pre_response block in send_final_response,
verify flag from mesh_hook_ctx.
Removed from Rust: handle_verify, verify_response, pre_response route.
Removed find_different_model_peer (single-peer wrapper, unused).
-197 lines across 5 files.
* injection framing test + switch to 'reference' framing
Integration test (virtual_llm_injection.rs) that spins up a mock
hook server + llama-server, sends factual/translation/reasoning
questions, and verifies the model incorporates injected hints.
Tested 4 framings against Qwen3-0.6B baseline (1/5 correct):
- 'reference': 5/5, cleanest output, no artifacts
- 'assistant_draft': 5/5, clean
- 'current': 5/5, leaks </think> into content
- 'rag': 5/5, sometimes echoes question
Switched production framing to 'Reference answer: ...' — produces
the cleanest model output with no artifacts.
Also: debug mode now always fires Hook 2 (post_prefill), regardless
of entropy. The first token is always confident for thinking models
(<think>), so entropy gating in debug mode was preventing testing.
* TODO: virtual LLM remaining work items
Track slow peer consultation investigation (MiniMax timeout issue),
peer responsiveness tracking, audio extraction, and non-thinking
model testing.
* fix TODO: MiniMax was slow, not failing
* TODO: use TTFT perf tracker for consultation peer selection (PR #271)
* prevent recursive consultation loops
Outgoing consultation requests now include mesh_hooks: false,
which tells the peer's llama-server to skip hook callbacks for
that request. Without this, peer A could consult peer B, which
could consult peer C, etc.
Flagged by GPT-5.4 code review as a critical production risk.
* per-hook timeouts and compact mid-gen injection
Hook 2 (pre-generation): 15s timeout — user is waiting for first
token anyway, longer timeout lets slower but better peers respond.
Hook 2b (mid-generation): 5s timeout — user sees a stall if we
block too long. Also uses compact 'Key fact: ...' framing (256
char cap) instead of full 'Reference answer' (512 chars) to avoid
derailing the model's continuation style.
Hook 1 (media captioning): 15s timeout — captioning is a one-time
pre-generation cost.
* drop compact mid-gen framing — tested, worse than reference
Tested 'Key fact: X' (compact) vs 'Reference answer: X. Use the
reference above...' (reference). Compact got 4/5 — Swahili
translation failed because the model echoed 'Key fact:' back
instead of using the hint. The instruction to act on it matters.
Use 'reference' framing for both Hook 2 and 2b. Simpler, tested.
* two new mid-gen triggers: repetition loop + surprise break
GPT-5.4 research: entropy alone doesn't catch 'confidently wrong' —
Gemma-4B first-token entropy is 0.03-1.33 even on hallucinated
answers. Need different signals.
New trigger 1: REPETITION LOOP
Track last 32 token IDs, compute 3-gram repeat ratio. Fire when
ratio >= 0.18. Catches degenerate loops (Gemma stuck in 'I need
to access my knowledge base' patterns). Tested: fired correctly
on Nauru query at token 43, and on p-values explanation at tokens
101 and 153.
New trigger 2: SURPRISE BREAK
EWMA of -log(p_chosen) with z-score spike detection. Fires when
2+ tokens spike >2.5 sigma after a calm run of 4+ low-z tokens.
Catches 'was flowing confidently, then suddenly broke'.
Both signals added to mesh_signal_window struct and wired into
should_fire_midgen(). Any of the three triggers (entropy spike,
repetition, surprise) can independently fire Hook 2b, all sharing
the same cooldown.
Tested on Gemma-4B with real mesh peers:
- Repetition: 3 fires across 12 queries (all genuine loops)
- Entropy spike: 0 fires (Gemma too confident, as expected)
- Hook 2 (first-token): 2 fires on creative prompts (haiku)
where Gemma genuinely didn't know how to start
- Zero false positives on factual/math queries
* 20s consultation timeout across all hooks
Triggers are rare — when they fire, it's worth waiting. The 5s
mid-gen timeout was causing all repetition-loop consultations to
fail (peers need 6-10s). Unified to a single 20s constant.
* add rank instability tracking, disable as trigger
Implemented top-8 Jaccard similarity tracking between consecutive
generation steps. Computes overlap of top-8 candidate tokens —
low Jaccard means the model is considering completely different
continuations each step.
DISABLED as a trigger: Gemma-4B has inherently unstable top-8
distributions — fires on ~100% of queries even with strict
thresholds (5/6 window, 24-token warmup). The signal needs
per-model baseline calibration or combination with another signal.
Tracking code is kept (cheap, data in signals JSON) for research.
Repetition loop remains the most reliable mid-gen trigger.
* remove rank instability signal
100% false positive rate on Gemma-4B — top-8 candidates are
inherently unstable on modern models. Not a useful signal without
per-model calibration. -94 lines.
* Hook 1 working: image captioning on text-only models
When a text-only model (no mmproj) receives an image request:
1. server-common.cpp strips image to '[image attached]' text
instead of rejecting with 500 error (when mesh hooks enabled)
2. Original image URL preserved as mesh_image_url in messages
3. server-context.cpp detects mesh_image_url, fires Hook 1
4. Rust handler extracts image URL, consults vision peer
5. Caption injected into prompt before generation
Tested end-to-end with mock hook server + Qwen3-0.6B:
- Image request accepted (was: 500 error)
- Hook 1 fires with images_no_multimodal trigger
- Caption injected (13 tokens)
- Model output matches caption content exactly
Also adds server-common.cpp to llama-patches/ (was only
server-common.h before) for the image stripping logic.
* update docs: VIRTUAL_LLM.md rewrite + TODO refresh
VIRTUAL_LLM.md:
- Removed Hook 3 (dropped earlier), background work pattern,
DashMap, mesh_request_id correlation (simplified away)
- Added Hook 2b with 3 triggers (repetition, entropy, surprise)
- Added signal detection research section with results
- Added injection framing test results
- Updated C++ file list (added server-common.cpp)
- Updated consultation section (fan-out, recursion guard, 20s timeout)
- Corrected entropy threshold (3.0 not 5.0)
TODO.md:
- Removed non-thinking model testing item (done — Gemma-4B tested)
- Added: test Hook 1 with real vision peer
- Added: push C++ to fork when stable
- Added: image caption caching
- Updated description to reflect current state
* docs: highlight inter-model collaboration across README, docs site, roadmap
README: add paragraph in 'How it works' section describing the feature,
link to VIRTUAL_LLM.md in 'More docs'.
docs/index.html: add feature card with brain emoji, add tag pill,
update research section to note collaboration is live.
ROADMAP.md: add 'Inter-model collaboration (Virtual LLM)' section
with working features, key insight, paper reference, and next steps.
* move inter-model collaboration from roadmap to README feature list
This is a shipped feature, not a roadmap item. Remove the ROADMAP
section and add a bullet to the top-level feature list instead.
* move llama.cpp to Mesh-LLM org fork, pin by SHA
Fork: github.com/Mesh-LLM/llama.cpp (master branch)
Pinned SHA in LLAMA_CPP_SHA (single source of truth)
Changes:
- All build scripts, CI workflows, Dockerfiles now use Mesh-LLM/llama.cpp
- CI reads SHA from LLAMA_CPP_SHA file, falls back to ls-remote
- Delete mesh-llm/llama-patches/ (-15k lines) — fork is the source of truth
- Add mesh-llm/docs/LLAMA_CPP_FORK.md with maintenance instructions:
how to update from upstream, how to tell an agent to sync, files we touch
- Justfile diff recipe updated for new fork layout
The fork carries 8 commits on master:
3x RPC (zero-transfer, alloc cache, B2B transfers)
4x MoE (expert mask, split tool, ranking, shared-expert fix)
1x mesh hooks (virtual LLM engine)
* AGENTS.md: document llama.cpp fork, warn agents not to update it unprompted
* build-mac: pin llama.cpp to SHA from LLAMA_CPP_SHA
Same behaviour as build-linux.sh: reads the pinned SHA, fetches it,
checks out detached. If LLAMA_CPP_SHA is missing, falls back to
pulling master HEAD.
* test: find CI model at ~/.models/ so hook test runs in CI
Was silently skipping because it only checked HuggingFace cache.
CI already downloads SmolLM2-135M to ~/.models/.
* Revert "test: find CI model at ~/.models/ so hook test runs in CI"
This reverts commit a33da501e4373a7eb5ce6776c4ba7bb66d754533.
* fix copilot review: SHA pinning consistency, UTF-8 safety, CI fail-fast
- build-release.sh: read LLAMA_CPP_SHA, hard fail if missing
- build-linux.sh, build-linux-rocm.sh: fall back to LLAMA_CPP_SHA file
when MESH_LLM_LLAMA_PIN_SHA env var not set
- ci.yml: fail if pinned SHA can't be checked out (was silent fallback)
- Fix UTF-8 byte-slice panics in consult.rs, virtual_llm.rs, test file
(use char_indices / chars().take() instead of byte slicing)
* harden mesh hooks: loopback-only guard, proper HTTP response parsing
- /mesh/hook rejects non-loopback callers with 403 — prevents remote
abuse when management API is on 0.0.0.0 via --listen-all
- consult.rs: parse HTTP status line, fail on non-200, require header
terminator, include raw response in error messages
* run virtual LLM hook integration test in CI
Test finds CI model at ~/.models/ (SmolLM2-135M, already cached).
Runs after llama-server build + model download on macOS job.
Exercises real C++ hooks: llama-server --mesh-port → mock → inject → generate.
* feat: "mesh" virtual model — auto-routes with inter-model hooks
Send "model": "mesh" to get smart routing (same as "auto") plus
mesh hook callbacks enabled. All other models pass through unhooks.
- inject_mesh_hooks_flag: byte-level injection into forwarded HTTP body
- "mesh" listed in /v1/models when any real model is served
- Smoke test: sends model=mesh, verifies response, checks llama-server
has --mesh-port in process args, verifies "mesh" in /v1/models
- 3 unit tests for byte injection (enabled/disabled/no-body)
* fix: trailing newline in lib.rs (rustfmt CI compat)
* trigger CI
* ci: retrigger after smoke test cleanup
* fix: revert trailing newline in lib.rs — cargo fmt removes it
* fix CI formatting: run cargo fmt --all from repo root
Running cargo fmt from mesh-llm/ subdir in a workspace resolves
formatting differently than from repo root. Switch both linux and
macOS CI jobs to 'cargo fmt --all -- --check' from repo root.
* fix: add trailing newline to lib.rs for Linux rustfmt
Linux and macOS rustfmt 1.94.1 disagree on trailing newlines in
short files. CI runs on Linux — match its expectation.
* fix: UTF-8 truncation safety, 403 reason phrase, doc hook count
- consult.rs/virtual_llm.rs: use char_indices().take_while() instead
of byte-slicing which can panic on multi-byte codepoints
- api/http.rs: add 403, 409, 429, 500, 503 reason phrases so
non-200 responses don't say 'HTTP/1.1 403 OK'
- VIRTUAL_LLM.md: fix 'Two hook points' → 'Three hook points'
* fix: tighten fork validation and build-mac pull error handling
- docker-precheck: require Mesh-LLM/llama.cpp URL explicitly instead
of also matching bare 'master' anywhere in the Dockerfile
- build-mac.sh: warn on git pull failure instead of silent || true
* fix: wire mesh model routing and hook injection through ingress.rs
The main API proxy path goes through ingress.rs, not transport.rs.
Without this, model=mesh fell through as an unknown model name, hit
"Model not found, trying first available", and hooks never fired.
- Add "mesh" to ingress.rs smart routing condition alongside "auto"
- Call inject_mesh_hooks_flag from ingress.rs before forwarding
- Make inject_mesh_hooks_flag pub for cross-module access
- Update proxy integration tests to tolerate injected mesh_hooks field
* fix: only inject mesh_hooks into request body for model=mesh
Previously mesh_hooks:false was injected into EVERY request going
through the proxy, mutating request bodies even for users who never
use the mesh model. Now non-mesh requests pass through completely
untouched — no body mutation, no extra JSON fields.
Restores exact-body proxy integration tests.
* pin LLAMA_CPP_SHA to ed279faf4 — mesh hooks default OFF
Hooks only fire when proxy sends mesh_hooks:true (model=mesh).
Non-mesh requests are completely unaffected.
* virtual LLM hooks are auto, not a separate model
Inter-model collaboration hooks now fire for all auto-routed requests
(model=auto or no model specified). The separate "mesh" virtual model
is removed — no extra entry in /v1/models, no new concept for users
to learn. Auto just gets smarter when peers are available.
- ingress.rs: hooks enabled for auto-routed requests
- transport.rs: same for idle/passive proxy path
- Removed "mesh" entry from /v1/models
- Updated CI smoke test to test auto with hooks
* router: drop hardcoded model table, use gossiped capabilities
Replace the static MODEL_PROFILES table (30 hand-curated entries with
vibes-based tiers and strengths) with capability-based filtering using
ModelCapabilities already gossiped from every peer.
Routing logic for auto:
- needs_tools → filter to models with tool_use capability
- Reasoning category → filter to models with reasoning capability
- Image category → filter to models with vision capability
- anything else → no filter, any model works
- fallback to all models if filter matches nothing
- spread load randomly among candidates
This means unknown models that gossip tool_use: Supported are now
routable for agentic requests without anyone updating a table.
-443 lines net, mostly the deleted profile table and tier scoring.
* capabilities: known tool-capable families fallback, test mesh_hooks injection
Add known_tool_capable_family() as a last-resort name heuristic for
models whose config files lack standard tool tokens but are known to
support tool calling: Qwen3*, MiniMax*, Hermes*, Gemma*.
Matches against path segments so repo paths like "Qwen/Qwen3-32B" work.
Only fires when no other tool_use signal was found (config scan, name
keywords). Sets tool_use: Likely, not Supported.
Also:
- Add proxy integration assertion: model=auto forwards mesh_hooks:true
to the upstream, closing the test gap between injection and hooks
- Remove unused pick_model_with_tools test helper (0 warnings now)
2026-04-16 14:54:58 +10:00
2026-05-19 17:26:09 +10:00
## Workspace Crates
The workspace lives under `crates/` . The most important crates:
2026-06-12 18:44:36 +10:00
Shipped binary and CLI surface:
- `mesh-llm/` — shipped binary; `main.rs` builds the Tokio runtime, `lib.rs` owns `run_main` (CLI parse → one-shot command dispatch via its `commands/` module → runtime handoff), and re-exports `mesh-llm-host-runtime` as a transitional shim. No domain logic here.
- `mesh-llm-cli/` — Clap types, argument parsing, serve/client surface normalization. No handlers.
- `mesh-llm-commands/` — user-facing command handlers (auth, gpus, update, skills, agent launchers like goose/pi/opencode/claude, plugin, benchmark, model packaging).
- `mesh-llm-tui/` — terminal UI and progress output surface.
- `mesh-llm-events/` — shared runtime event and output contracts (`OutputEvent` , log formats).
Host and client runtimes:
- `mesh-llm-host-runtime/` — the host-side monolith. Owns runtime orchestration, mesh, inference, networking, management API, plugins, models, system integration. This is where most changes land.
2026-05-19 17:26:09 +10:00
- `mesh-client/` (`mesh-llm-client` ) — lighter parallel client surface with its own `inference/` , `network/` , `models/` , `mesh/` modules. Used as a dev/test surface and for client-only deployments.
2026-06-12 18:44:36 +10:00
- `mesh-llm-node/` , `mesh-llm-embedded-runtime/` — embeddable node primitives and in-process full-node embedding API.
- `mesh-llm-config/` — configuration parsing and validation (`~/.mesh-llm/config.toml` ).
2026-05-19 17:26:09 +10:00
- `mesh-llm-ui/` — React web console and embedded asset crate (shadcn/ui patterns, see https://ui.shadcn.com/llms.txt).
2026-06-12 18:44:36 +10:00
- `mesh-llm-console-server/` — static file server for embedded console assets.
Shared foundations:
2026-05-19 17:26:09 +10:00
- `mesh-llm-types/` — shared model/capability types used across crates.
- `mesh-llm-protocol/` — wire protocol types and protobuf bindings.
- `mesh-llm-routing/` — routing primitives shared across host and client.
- `mesh-llm-system/` — machine-local hardware, benchmark, autoupdate, process helpers.
2026-06-12 18:44:36 +10:00
- `mesh-llm-identity/` — owner identity and envelope crypto primitives.
- `mesh-llm-guardrails/` — guardrail and compaction primitives for OpenAI-compatible paths.
- `mesh-llm-hardware-profile/` , `mesh-llm-native-runtime/` , `mesh-llm-runtime-install/` — hardware profile detection, native runtime manifest/selection, runtime download/install/cache.
2026-05-19 17:26:09 +10:00
- `mesh-llm-plugin/` — plugin runtime/DSL primitives.
2026-06-12 18:44:36 +10:00
- `mesh-llm-plugin-manager/` — plugin package management (catalog, install, store).
- `mesh-llm-skills/` — agent skill data model and installer primitives.
SDK and API surface:
- `mesh-llm-sdk/` — Rust SDK facade for clients and embedded serving.
- `mesh-llm-api-server/` , `mesh-llm-api-client/` — public Rust SDK APIs for embedding nodes / client-only use.
- `mesh-llm-ffi/` , `mesh-llm-nodejs/` — FFI bindings and Node.js native addon.
2026-05-19 17:26:09 +10:00
- `openai-frontend/` — OpenAI-compatible HTTP frontend (chat, completions, responses, models).
2026-06-12 18:44:36 +10:00
- `mesh-mixture-of-agents/` — Mixture-of-Agents fan-out/arbitration engine.
Models:
2026-05-19 17:26:09 +10:00
- `model-artifact/` , `model-hf/` , `model-package/` , `model-ref/` , `model-resolver/` — model catalog, HuggingFace download, packaging, reference resolution.
2026-06-12 18:44:36 +10:00
Embedded staged runtime (skippy):
2026-05-19 17:26:09 +10:00
- `skippy-ffi/` — Rust ABI bindings to the patched llama.cpp staged runtime.
- `skippy-runtime/` — Rust-side staged runtime, package materialization, model info.
- `skippy-server/` — embedded staged-runtime serving (frontend, binary transport, runtime state, embedded HTTP).
- `skippy-protocol/` , `skippy-topology/` , `skippy-coordinator/` , `skippy-cache/` , `skippy-prompt/` , `skippy-metrics/` , `skippy-bench/` , `skippy-correctness/` , `skippy-model-package/` — supporting skippy infrastructure.
2026-06-12 18:44:36 +10:00
Tools and benchmarks:
2026-05-19 17:26:09 +10:00
- `metrics-server/` — standalone metrics collector binary.
- `mesh-llm-gpu-bench/` , `llama-spec-bench/` , `mesh-llm-test-harness/` — benchmarking and test harness binaries.
2026-06-12 18:44:36 +10:00
This list covers the crates you are most likely to touch; check `crates/` and each crate's `Cargo.toml` description for anything not listed.
2026-05-19 17:26:09 +10:00
Other top-level directories:
2026-06-12 18:44:36 +10:00
- `docs/` — Project docs, grouped by topic (see `docs/README.md` for the map).
2026-06-10 05:18:25 -04:00
- `website/` — Eleventy source for the public website; builds into `docs/` .
2026-05-19 17:26:09 +10:00
- `docs/design/` — Architecture, protocol, and testing docs.
2026-06-12 18:44:36 +10:00
- `docs/skippy/` — Skippy family certification, configuration, benchmarks, parity.
2026-05-19 17:26:09 +10:00
- `docs/plugins/` — Plugin architecture docs and plans.
2026-06-12 18:44:36 +10:00
- `docs/specs/` — Focused behavior specs for individual features.
2026-08-08 03:39:13 -04:00
- `.agents/agents/release-validation.md` — Canonical Markdown definition for the selectable release-validation specialist; it uses the canonical release-validation skill in `.agents/skills/` .
- `.agents/skills/` — Canonical repo-local agent skills, including per-platform deploy, mesh operations, release validation, Skippy internals, patch queues, and benchmarks.
2026-06-12 18:44:36 +10:00
- `sdk/` — SDK packaging for Node, Swift, Kotlin.
2026-05-19 17:26:09 +10:00
- `fly/` — Fly.io deployment (console + API client apps).
- `tools/relay-fly-legacy/` — Archived self-hosted iroh relay reference; production uses services.iroh.computer.
- `evals/` — Benchmarking and evaluation scripts.
- `third_party/llama.cpp/patches/` — durable llama.cpp patch queue, pinned by `upstream.txt` .
2026-03-05 08:03:51 -05:00
2026-04-03 07:19:35 +11:00
## Module Structure Rules
2026-05-19 17:26:09 +10:00
These rules apply primarily inside `crates/mesh-llm-host-runtime/src/` (the main host monolith), and by analogy inside `crates/mesh-client/src/` . New peer crates should still follow the semantic-ownership principles below.
2026-04-03 07:19:35 +11:00
2026-05-19 17:26:09 +10:00
The host-runtime crate root should stay minimal.
2026-04-03 07:19:35 +11:00
2026-05-19 17:26:09 +10:00
- Keep `crates/mesh-llm-host-runtime/src/lib.rs` slim — it is a small entry point, not a junk drawer.
- New code should go into an existing domain directory when possible.
2026-04-03 07:19:35 +11:00
2026-05-19 17:26:09 +10:00
Use semantic ownership for module placement. Inside `crates/mesh-llm-host-runtime/src/` :
- `runtime/` — top-level process orchestration, startup/runtime coordination, runtime instance, capacity, split planning, proxy lifecycle.
- `network/` — request routing, proxying, tunneling, relay/discovery networking, request-affinity logic, endpoint rewrite, target health, OpenAI transport glue.
- `inference/` — model-serving logic, election, launch, pipeline, MoE behavior, embedded skippy integration.
- `system/` — machine-local environment and platform concerns (hardware detection, benchmarking, self-update, local system integration).
- `models/` — model catalog, resolution, downloads, local model storage, model metadata.
- `mesh/` — peer membership, gossip, heartbeats, identity, peer state, mesh node behavior.
- `plugin/` — plugin host, plugin runtime, transport, config, MCP bridge support.
2026-06-12 18:44:36 +10:00
- `plugins/` — concrete in-tree plugins (currently `blobstore/` ; most plugins like blackboard, openai-endpoint, and flash-moe/ln are external packages installed via `mesh-llm plugins install` ).
2026-05-19 17:26:09 +10:00
- `api/` — management API surface and route handling.
- `protocol/` — wire protocol types, encoding/decoding, conversions.
- `runtime_data/` — runtime data collection, API views, status snapshots.
- `crypto/` — host-side crypto helpers.
2026-04-03 07:19:35 +11:00
CLI ownership rule.
2026-06-12 18:44:36 +10:00
- Clap types, argument parsing, and surface normalization belong in `crates/mesh-llm-cli/` .
- User-facing command handlers belong in `crates/mesh-llm-commands/` (or the shipped binary's `crates/mesh-llm/src/commands/` dispatch layer for wiring).
- Domain modules in `mesh-llm-host-runtime` should not own Clap parsing or top-level command dispatch.
- Domain modules may expose reusable functions that command handlers call.
2026-04-03 07:19:35 +11:00
Do not introduce generic buckets.
- Avoid directories or modules named `app` , `utils` , `misc` , `common` , or similar catch-alls.
- Name modules after the responsibility they own.
Keep shared code honest.
- If code is only used by one subsystem, keep it inside that subsystem.
2026-05-19 17:26:09 +10:00
- Only move code to a shared module (or a shared workspace crate like `mesh-llm-types` / `mesh-llm-routing` ) when it is truly cross-domain.
2026-04-03 07:19:35 +11:00
- Do not create shared helpers prematurely.
Prefer semantic grouping over symmetry.
- Do not create one directory per file just for visual symmetry.
- A single `foo.rs` file is already a Rust module; use a directory only when `foo` has meaningful substructure.
Minimize crate-root re-exports.
- Root re-exports are acceptable as temporary compatibility shims during refactors.
- New code should prefer importing from the owning module directly.
- Remove transitional re-exports once call sites have been updated.
When to split a file.
- Split a file when it contains multiple separable responsibilities, when navigation becomes difficult, or when tests naturally cluster by concern.
- Do not split purely to reduce line count if the code still represents one coherent object or subsystem.
Improve context planning, admission, and coordinator fencing (#513)
* feat: split-aware context planning with KV quant negotiation
Context planning now produces useful context windows for split models
instead of falling back to 4096.
Split-aware budget: the planner now accepts a local_layer_fraction so
it can compute the KV cache cost for just this node's layers, not the
whole model. For layer packages, the fraction is estimated from the
VRAM ratio (local / total mesh VRAM).
KV quant negotiation: when the requested KV quantisation (e.g. f16)
cannot reach the model's native context length, the planner walks a
quant ladder (f16 → q8_0 → q4_0) and picks the least aggressive
quant that fits. The negotiated quant is applied to the stage load
request automatically.
Layer package metadata: for split models, the planner now reads GGUF
architecture metadata from the layer package's shared/metadata.gguf
instead of returning None (which caused a fallback to 4096 default).
Also fixes 13 pre-existing compile errors in mesh-llm-host-runtime
test code (missing latency fields from #491, wrong function names and
stale struct fields from #485).
* fix: repair broken host-runtime tests and add to CI
Fix 13 compile errors and 3 test failures in mesh-llm-host-runtime
that were silently broken on main (CI only ran mesh-llm --lib which
has zero tests).
Compile fixes:
- Add missing latency fields (latency_ms, latency_source,
latency_age_ms, latency_observer_id) to PeerAnnouncement test
constructions in protocol/mod.rs (#491 missed these sites)
- Fix test_endpoint_id → make_test_endpoint_id in mesh/tests.rs
(#485 used wrong function name)
- Update ServedModelDescriptor to current struct shape (capabilities
+ topology instead of format + quantization + size_bytes)
- Add missing available_model_sizes field
Test fixes:
- gossip_frame_roundtrip_preserves_scanned_model_metadata: add
ModelRuntimeDescriptor with context_length to served_model_runtime
(was empty vec, then asserted on first element)
- initial_pretty_session_mode: update expectation to match current
implementation (Client surface now allows dashboard)
- Remove broken timing-dependent streaming proxy test (covered by
two other streaming tests that pass)
- Mark HF download test as #[ignore] (downloads 800MB, needs auth)
CI:
- Add cargo test -p mesh-llm-host-runtime --lib to both Linux and
macOS CI jobs
- Add cargo test -p model-artifact --lib to both jobs
* fix: respect user KV quant override, avoid blocking async, tighten test assertions
Address Copilot review feedback:
- Skip KV quant negotiation when the user explicitly set --cache-type-k
or --cache-type-v. Previously the planner would negotiate to q4_0 for
a larger context, but the downstream load honoured the user's f16
override — producing a context/memory mismatch. New kv_quant_user_locked
flag prevents this.
- Wrap scan_layer_package_metadata in spawn_blocking to avoid filesystem
I/O on the async executor (GGUF header reads, stat calls).
- Tighten negotiate_kv_quant_upgrades_to_reach_native_context assertion
to check exact expected value (16K) instead of just > 8K.
- Add user_locked_kv_quant_skips_negotiation test proving the lock
prevents negotiation and produces a smaller context than unlocked.
* Add split coordinator fencing
* simplify: remove KV quant negotiation, universal Q8_0 default
Drop the tiered KvCachePolicy (f16/q8_0/q4_0 by model size) and the
negotiation ladder in context_planning. KV cache is now Q8_0 everywhere
unless the user explicitly sets --cache-type-k/v.
The planner just does: VRAM budget ÷ per-token KV cost → context length.
No tiers, no negotiation, no negotiated_kv_quant, no kv_quant_user_locked.
Split path now runs the same planner (was hardcoded to 4096).
-216 lines net.
* fix: rename test to match actual behavior (copilot review)
* fix: solo load path no longer scales by peer VRAM
The local load path (start_runtime_local_model) incorrectly computed a
fractional layer share based on mesh VRAM ratio when loading layer-package
models. Since this path loads the entire model on one node, the fraction
should always be 1.0 — fractional scaling only applies in the split path.
This could overestimate free VRAM and plan a context window larger than
actually fits.
Also: fallback on invalid --cache-type-k/v now defaults to Q8_0 (was f16),
remove dead total_peer_vram_bytes(), fix stale doc comment.
* Bound skippy OpenAI generation admission
* Co-plan context and skippy concurrency
* Revert "Co-plan context and skippy concurrency"
This reverts commit ba211c173194aacabce9711548118859b825c238.
* fix: add generation queue fields to openai test fixtures
The bounded admission fields added in 7a00d20e were not threaded into
the multimodal smoke fixtures, breaking `cargo check --tests` in CI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: heap-allocate load_split_runtime_generation future in test
Same class of stack overflow that #504 addressed for tokio::spawn call
sites — load_split_runtime_generation_inner has grown enough (coordinator
fencing, KV quant changes) that constructing its future directly on the
test thread overflows on Linux/macOS CI runners.
Production call sites are already safe: they sit inside startup_local_model_loop
and SplitTopologyCoordinator::run, both Box::pin-ned by #504.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: add missing skippy-coordinator COPY to all Dockerfiles
04986dfe added crates/skippy-coordinator to the workspace but never
updated the Docker build contexts. cargo metadata fails inside the
container because the crate directory doesn't exist.
Also backfills ~18 other missing crates in ci/linux-test.dockerfile
which had fallen behind the workspace.
* Plan split topology from node VRAM
* fix: topology planner prefers fewest nodes before most parallel lanes
Swap the search loop order from context → lanes → nodes to
context → nodes → lanes. This means the planner picks the minimum
node count that achieves the target context, then maximizes parallel
lanes within that count, rather than greedily spreading across more
nodes to get more lanes.
Updated 5 tests to match the new priority.
* Document split topology planning
* Separate split runtime planning concerns
* Add 1k LoC refactoring agent rule
* Use 64k floor for topology context planning
* Use Metal working set for macOS VRAM
* fix: topology planner KV is shared across lanes, drop redundant headroom
KV cache is a single unified allocation shared across all lanes via
sequence IDs (kv_unified=true). The planner was multiplying KV cost
by parallel_lanes, over-budgeting and rejecting valid topologies.
Remove the 10% per-node headroom deduction — Metal's
recommendedMaxWorkingSetSize already accounts for OS reservations.
Bump non-macOS unified memory paths from 0.75 to 0.90 (macOS already
uses Metal directly via fef070f5).
* Add Studio Metal Qwen split simulation
* Log split topology planning failures
* Log split orchestration election decisions
* Align Studio split simulation with planner output
* Prefer complete cached layer package snapshots
* fix: restore tiered KV cache policy, fix split validation headroom
Models >= 50GB use Q4_0 KV cache to avoid swap thrashing on unified-memory
machines. The 480B MoE split across two Apple Silicon nodes was thrashing at
1.3 tok/s with Q8_0 KV (2GB headroom) — Q4_0 restores 13.6GB headroom and
20+ tok/s.
Split validation no longer double-counts the 10% solo-load headroom on top
of the topology planner's own VRAM budget, fixing the CI test failure in
resource_planner_returns_runtime_stage_shape.
* fix: align split capacity tests with headroom removal
The d031dd3e commit removed the 10% headroom from validate_split_capacity
to avoid double-counting the topology planner's own VRAM budget, but left
three test assertions checking the old headroom-inflated error messages.
- Update aggregate capacity assertions: 5.3GB → 4.8GB, short by 1.3GB → 0.8GB
- Reduce per-stage test node2 VRAM from 200 → 150 so the 200B assignment
still triggers a capacity rejection without the removed headroom
- Add missing positions field to skippy-bench StageWireMessage literals
* Upgrade iroh 0.98 → 1.0.0-rc.0
Breaking changes addressed:
- conn.paths() now returns PathList directly (no longer a Watcher) —
remove Watcher::get() indirection at all 5 call sites
- PathInfo::rtt() returns Duration instead of Option<Duration> —
use Duration::is_zero() to detect missing RTT
- ed25519-dalek pinned to =3.0.0-pre.7 (was =3.0.0-pre.6)
- endpoint.online() double-free (iroh#4149) is fixed — replace the
manual watch_addr() polling workaround with a simple online() call
* fix: use clamp() instead of min().max() pattern (clippy)
* fix: model name matching for auto/console in split mode
The skippy runtime (which replaced llama-server) introduced exact model
name matching in ensure_requested_model. This broke auto-routing and
console chat because:
1. /v1/models advertises 'org/repo:Q4' (no revision) but skippy-server
internally uses 'org/repo@main:Q4'. The console and auto-router send
the short form, skippy rejects it.
2. When the proxy auto-routes model="auto" to a specific model, the
resolved name was never written back into the HTTP body before
tunneling to the host's skippy-server.
Fixes:
- ensure_requested_model now normalizes @main before comparing
- Proxy rewrites body model field after auto-routing resolves
Also includes: split startup now retries on participant shortage instead
of fatally exiting, so nodes wait for peers to join rather than giving up.
* fix(ui): show split workers as serving in console
Split worker nodes report node_state=standby even though they actively
run a stage. The console now checks runtime.stages to detect split
participation and displays the node as serving with a proper share.
* test: add ensure_requested_model @main normalization coverage
* fix: relay join resilience — 30s timeout, 3 attempts, surface failure reason
Relay-only QUIC joins (WebSocket + TLS + QUIC handshake at high RTT) often
exceed the previous 15s timeout. Bump to 30s and add a third retry with
5s/10s backoff (~105s total budget). Surface the last join error in the
standalone warning so the operator can see what went wrong.
* fix: reject metadata-only HF snapshots during layer package resolution
When resolving an HF layer package from the local cache,
should_prefer_cached_snapshot_for_request now always verifies that the
snapshot has its declared layer artifacts on disk — not just for
metadata-only probes.
Previously, non-metadata-only requests (real stage loads with a layer
range) returned Ok(true) unconditionally, and metadata-only identity
probes that happened to pick a metadata-only snapshot would bake its
commit hash into the canonical package_ref. Later stage loads pinned to
that hash would then fail to find layers, causing the C++ runtime to
spin on 'failed to open GGUF file' errors.
This was triggered in practice when an HF repo pushed a new revision
after the initial metadata download: the old snapshot had
model-package.json + shared/metadata.gguf but no layer files, and the
early identity scan selected it because it was the only snapshot with
metadata at startup time.
* fix: reject skeleton HF snapshots and prefer cached snapshots with layers
Two fixes for stale/skeleton HF cache snapshot resolution:
1. cache_resolution: metadata-only identity probes now verify the snapshot
has at least one declared layer artifact on disk. A skeleton snapshot
(model-package.json + shared/ only) is rejected so the resolution falls
through to find a snapshot with actual layers.
2. materialization: after download_hf_package_to_local_sync returns for a
metadata-only request, re-scan the local cache. If the downloaded
snapshot is a skeleton (HF server HEAD differs from the local snapshot
with layers), iterate all cached snapshots and return one that has layer
artifacts. This prevents the canonical package_ref from freezing a
skeleton hash that causes downstream stage loads to fail.
* fix: eliminate stale snapshot errors from activation width probing
Two changes:
1. materialization: post-download snapshot re-scan now runs for ALL
requests, not just metadata-only probes. When the HF SDK downloads
to a skeleton snapshot that can't satisfy the caller's layer range,
re-scan all cached snapshots for one that can. This catches stage
load paths that carry a frozen skeleton hash in the topology config.
2. skippy-runtime: infer_activation_width_from_layers now checks the
layer file exists before calling ModelInfo::open. Previously the
C++ gguf_init_from_file would log 'failed to open GGUF file' errors
for missing files before the Rust error handling could suppress them.
The file existence check avoids the noisy C++ error output entirely.
* fix: collapse nested if to satisfy clippy collapsible_if
* fix: default parallel lanes to 4 instead of 16
Match llama-server's default of --parallel 4. Lanes share a unified KV
cache with eviction (kv_unified=true), so lane count does not multiply
KV memory cost. 4 concurrent request slots is a sensible default for
most model sizes; users can override via gpu.parallel in config.toml or
the per-model parallel setting.
* fix: strip_default_revision boundary match, local_model_fits for layer packages, dead code
- strip_default_revision now only removes @main when followed by : or
end-of-string, preventing corruption of repo names like @mainland.
- SplitTopologyCoordinator::local_model_fits uses package source_model_bytes
instead of stat-ing the hf:// pseudo-path (which returned 0, making local
fallback look possible when the model cannot actually fit).
- Remove unused QWEN_CODER_480B_Q4_KV_BYTES_PER_TOKEN constant.
* fix: layer package cache resolution for split nodes with divergent HF snapshots
Root cause: when two nodes had different HF cache states for the same
layer package (different snapshot commits, different model-package.json
content), the cache resolution code could pick a stale snapshot with all
layers present instead of the current HEAD snapshot with partial layers.
This caused manifest sha256 mismatches during split Load, killing the
split topology.
Three changes:
1. Cache resolution now checks only the REQUESTED layer range, not all
declared layers. Metadata-only probes (layer_start=layer_end=0) check
for at least one layer artifact (anti-skeleton). Real stage loads
check their assigned range only.
2. Removed the pre-download floating-revision fallback scan that walked
all cached snapshots looking for one with layers. This was the code
that picked stale snapshots with different manifests.
3. Scoped the post-download fallback scan to metadata-only probes only.
Real stage loads always download their assigned layers into the HEAD
snapshot, so no fallback is needed.
Also adds debug-level stage control tracing in handle_stage_control for
future split debugging.
Validated: 480B split across Studio (stage-0, layers 0-49) and James
(stage-1, layers 50-61) over relay — inference working end-to-end with
divergent HF cache states on both nodes.
* test: unit tests for cache_resolution layer range and skeleton checks
Adds 10 focused tests for the cache resolution functions introduced in
the previous commit:
- cached_snapshot_has_any_layer_artifact: skeleton rejection, partial acceptance
- cached_snapshot_has_requested_layers: range checks, partial ranges, missing layers
- should_prefer_cached_snapshot_for_request: dispatch to correct check based on metadata-only vs stage load
* fix: split diagnostic bytes_per_layer should not multiply KV by lanes
KV cache is a unified allocation shared across parallel lanes with
eviction. The diagnostic function was multiplying by lane count,
overstating memory needs in failure messages.
---------
Co-authored-by: James Dumay <jameswdumay@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 18:03:58 +10:00
1k LoC refactoring rule.
- When touching a source file that is already over 1,000 lines, first check whether the change adds or exposes a separable responsibility.
- If it does, split that responsibility into a semantically named module as part of the change, and keep the new file under 1,000 lines.
- If a full split is too risky for the current task, make the smallest useful extraction and call out the remaining oversized file in the final summary.
- Add or move tests so the extracted module owns tests for the behavior it now owns.
- Do not create generic buckets just to reduce line count; split by domain responsibility and keep ownership obvious.
2026-04-03 07:19:35 +11:00
Naming rule.
- File and module names should describe responsibility, not implementation detail.
- Prefer names like `affinity` , `discovery` , `transport` , `maintenance` , `warnings` .
- Avoid vague names like `helpers` , `stuff` , `logic` , or `manager` unless the abstraction is genuinely that broad.
2026-05-19 17:26:09 +10:00
When to add a new workspace crate.
- Prefer adding modules inside an existing crate first.
- Add a new `crates/<name>/` only when the responsibility is genuinely cross-cutting (used by host and client, or host and a separate binary) or when isolating compile time / dependencies for a specific binary or FFI surface.
- New crates should be named after the responsibility they own, not the consumer (e.g., `model-resolver` not `mesh-llm-model-helpers` ).
2026-04-03 07:19:35 +11:00
Current structure notes.
2026-05-19 17:26:09 +10:00
- Request-affinity code belongs with networking/routing behavior (`network/affinity.rs` ), not `system/` .
- Plugin MCP support belongs inside `mesh-llm-host-runtime/src/plugin/` , not as a separate root module.
2026-06-12 18:44:36 +10:00
- Model command handlers belong in `mesh-llm-commands/` (or `crates/mesh-llm/src/commands/` for dispatch wiring); host-runtime `models/` should stay domain-focused.
- The shipped binary crate (`crates/mesh-llm/` ) carries CLI dispatch wiring only; do not move domain logic into it.
2026-04-03 07:19:35 +11:00
2026-05-19 18:22:47 -04:00
## Code Quality Rules for New Code
- Do not add Rust methods or functions over the configured Clippy line-count
limit. Split long logic into semantically named helpers before it reaches the
configured `too_many_lines` threshold.
- Do not add Rust source files over 2,000 lines. If a file is approaching that
size, split it by responsibility into an owning module instead of adding more
code to the oversized file.
- Do not add Rust code over the configured cognitive-complexity limit. Prefer
small, named decision helpers and clear control-flow phases instead of nested
branching.
- Treat these as design constraints for new code, not cleanup suggestions after
the fact. CI runs Clippy with warnings denied, so configured Clippy warnings
must be resolved before a PR can pass.
2026-03-05 08:03:51 -05:00
## Key Source Files
2026-05-19 17:26:09 +10:00
Host runtime (main monolith — `crates/mesh-llm-host-runtime/src/` ):
2026-06-12 18:44:36 +10:00
- `lib.rs` — crate entry; exposes the runtime entrypoints (`run_runtime_initialized` , `initialize_host_runtime` ) called from `crates/mesh-llm/src/lib.rs` .
2026-05-19 17:26:09 +10:00
- `runtime/mod.rs` — top-level startup flows, runtime orchestration, command dispatch.
- `runtime/instance.rs` — per-instance runtime directory management: `InstanceRuntime` , pidfiles, flock liveness, scoped orphan reaping, local instance scanning.
- `runtime/local.rs` — local model startup loop.
- `runtime/discovery.rs` — discovery loops and auto-mode coordination.
- `runtime/proxy.rs` , `runtime/proxy/` — HTTP proxy lifecycle from the runtime side.
- `runtime/capacity.rs` , `runtime/split_planning.rs` , `runtime/context_planning.rs` — placement/sizing decisions.
- `mesh/mod.rs` — `Node` struct, mesh_id, peer management.
- `mesh/gossip.rs` — gossip wire format and peer state updates.
- `mesh/heartbeat.rs` — heartbeat publishing and freshness.
- `inference/election.rs` — host election, tensor split calculation.
- `inference/skippy/` — embedded staged runtime integration.
- `inference/pipeline.rs` — inference pipeline coordination.
- `inference/virtual_llm.rs` — virtual LLM (inter-model collaboration).
- `network/proxy.rs` — HTTP proxy: request parsing, model routing, response helpers.
- `network/router.rs` — request classification, model scoring, multimodal routing.
- `network/nostr.rs` — Nostr discovery, `score_mesh()` , `smart_auto()` .
- `network/tunnel.rs` — TCP ↔ QUIC relay (RPC + HTTP).
- `network/affinity.rs` — request-affinity tracking.
- `network/target_health.rs` — target health tracking.
- `network/openai/` — OpenAI transport glue.
- `api/mod.rs` , `api/routes/` — management API (:3131): `/api/status` , `/api/events` , `/api/discover` .
- `models/catalog.rs` — model catalog, HuggingFace downloads.
- `models/capabilities.rs` — multimodal/vision/audio/reasoning capability inference.
- `models/resolve/` — model reference resolution.
- `plugins/blobstore/mod.rs` — request-scoped media object storage for multimodal.
2026-06-12 18:44:36 +10:00
- `plugin/` — plugin host, runtime, transport, config, MCP bridge (external plugins install via `mesh-llm plugins install` ).
2026-05-19 17:26:09 +10:00
2026-06-12 18:44:36 +10:00
Shipped binary and CLI (`crates/mesh-llm/src/` , `crates/mesh-llm-cli/src/` , `crates/mesh-llm-commands/src/` ):
2026-05-19 17:26:09 +10:00
2026-06-12 18:44:36 +10:00
- `mesh-llm/src/main.rs` — builds the Tokio runtime (custom stack size via `MESH_TOKIO_STACK_SIZE` ) and calls `mesh_llm::run_main()` .
- `mesh-llm/src/lib.rs` — `run_main` : CLI parse, one-shot command dispatch, runtime handoff; plus a transitional `pub use mesh_llm_host_runtime::*;` re-export.
- `mesh-llm/src/commands/` — dispatch wiring from parsed `Command` values to handlers.
- `mesh-llm-cli/src/parser.rs` — Clap surface, serve/client arg normalization, advanced help.
- `mesh-llm-commands/src/` — user-facing handlers (auth, gpus, update, skills, agent launchers, plugin, benchmark).
2026-05-19 17:26:09 +10:00
Embedded staged runtime (`crates/skippy-*` ):
- `skippy-ffi/src/lib.rs` — Rust ABI mirror of the patched llama.cpp staged runtime; `ABI_VERSION_*` constants must stay in sync with `skippy/common.h` in the patch queue.
- `skippy-runtime/src/package.rs` — layer-package materialization, identity-bound cache.
- `skippy-runtime/src/devices.rs` — backend device enumeration.
- `skippy-server/src/frontend.rs` , `skippy-server/src/frontend/` — embedded chat/generation frontend.
- `skippy-server/src/runtime_state.rs` — KV-slot, lane, session state machine.
- `skippy-server/src/binary_transport.rs` , `binary_transport/` — binary transport to embedded server.
OpenAI-compatible HTTP frontend (`crates/openai-frontend/src/` ):
- `router.rs` , `chat.rs` , `completions.rs` , `responses.rs` , `models.rs` , `sse.rs` , `backend.rs` — OpenAI surface.
2026-04-10 03:26:21 +10:00
## Mesh Protocol Compatibility
Mesh compatibility across versions is critical. Nodes in the wild run different versions and must interoperate.
- The mesh supports mixed-version operation: QUIC ALPN `mesh-llm/1` (protobuf) and `mesh-llm/0` (legacy JSON) nodes coexist. Do not break this.
- Gossip fields, stream types, and protobuf schemas must be additive. New fields should be optional and ignored by older nodes. Do not repurpose or remove existing fields.
- When adding new gossip fields, stream types, or changing wire format, explicitly consider what happens when an older node receives the new data and when a newer node talks to an older peer.
- Capability advertisement (vision, audio, multimodal, reasoning, tool_use, moe) is gossiped to all peers and consumed by routing, the API, and the UI. Changes to capability semantics affect the whole mesh, not just the local node.
- If a change would break mixed-version meshes, explicitly flag it as a breaking protocol change and ask the developer before proceeding.
- Test compatibility by running the current branch against a released binary on a second node. Verify gossip, routing, and inference work across the version boundary.
2026-03-05 08:03:51 -05:00
2026-03-27 10:56:37 +11:00
## Plugin Protocol Compatibility
When iterating on the plugin protocol, always consider protocol compatibility.
- If a protocol change may be breaking, explicitly ask the developer whether the change is intended to be breaking.
- If the change is not intended to be breaking, the previous version of the plugin protocol must continue to be supported.
- Do not silently ship plugin protocol changes that strand older plugins or hosts without confirming that outcome is acceptable.
2026-05-19 17:26:09 +10:00
## Skippy ABI Compatibility
The patched llama.cpp staged runtime has its own ABI version, tracked in `skippy/common.h` (inside the patch queue) and mirrored by `SKIPPY_ABI_VERSION_*` constants in `crates/skippy-ffi/src/lib.rs` .
- When changing the staged-runtime ABI in the patch queue, bump `SKIPPY_ABI_VERSION_PATCH` (or MINOR/MAJOR) in `skippy/common.h` AND keep the Rust constants in `skippy-ffi/src/lib.rs` in sync in the same change.
- `skippy-runtime` consumes the ABI version for package loading and feature probing; an out-of-sync mirror will silently advertise the wrong version.
- Treat the staged-runtime ABI the same as the mesh wire protocol: additive changes preferred, breaking changes need explicit acknowledgement.
2026-03-05 08:03:51 -05:00
## UI Notes
2026-05-08 12:11:14 +10:00
For changes in `crates/mesh-llm-ui/` , use components and compose interfaces consistently with shadcn/ui patterns. Prefer extending existing primitives in `src/components/ui/` over ad-hoc markup.
2026-03-05 08:03:51 -05:00
## Testing
2026-05-02 09:44:52 +10:00
Read `docs/design/TESTING.md` before running tests. It has all test scenarios, remote deploy instructions, and cleanup commands.
2026-03-05 08:03:51 -05:00
2026-04-10 03:26:21 +10:00
Testing matters more than usual in this project because:
- Nodes run on different machines with different hardware and OS versions. Bugs that don't reproduce locally can appear in real deployments.
- The mesh protocol is a distributed system — gossip, election, and routing interact across nodes. Single-node unit tests don't catch protocol-level regressions.
2026-05-07 21:25:40 -04:00
- The public mesh at meshllm.cloud runs continuously. Breaking changes that pass local tests can take down live inference for real users.
2026-04-10 03:26:21 +10:00
- Multimodal, MoE splitting, and multi-model routing all have complex interaction paths that are hard to reason about statically.
2026-05-19 17:26:09 +10:00
When making changes that touch gossip, routing, proxy, election, or capability advertisement, test against at least two nodes before merging. The deploy checklist below is not optional.
2026-04-10 03:26:21 +10:00
2026-05-21 12:54:18 +10:00
### Confidence Testing (multi-node, when warranted)
For changes that affect routing, MoA, gossip, the OpenAI surface, agent harnesses, or anything multi-node, validate with these three shapes before declaring a branch ready:
1. **2-node private mesh** — start one node with `mesh-llm serve --model <big> --port 9337 --console 3131` , grab its invite token from the JSON log, and start the second node with `mesh-llm serve --gguf <small.gguf> --port 9447 --console 3145 --join <token>` . Confirm peers=1 on both consoles and `/v1/models` returns the union. Exercises QUIC tunnelling and cross-node routing.
2026-07-29 10:16:23 -04:00
2. **Public mesh as a client** — `mesh-llm client --auto` from a workstation. Confirm `discovery_joined` plus a structured client-ready event (`passive_mode` , `status=ready` , `role=client` ) in the log and an inference call against a mesh-advertised model returns. Exercises the read-only routing path agent users hit.
2026-05-21 12:54:18 +10:00
3. **Agent harness** — run ≥ 1 of the harnesses (“mini-agent” Python loops at `/tmp/mini-agent*.py` , Goose, OpenCode) against the local proxy with both `model=auto` and `model=mesh` to catch tool-call and reducer regressions that simple curl checks miss.
2026-04-10 10:16:56 +10:00
### Cargo Concurrency
Run `cargo` commands serially. Do not run multiple `cargo` commands in parallel (including parallel test runs), because this repo frequently hits Cargo lock conflicts (`package cache` / `artifact directory` ) under concurrent invocation.
2026-05-19 17:26:09 +10:00
### Which crate to `-p`
- Touched `mesh-llm-host-runtime` or the shipped `mesh-llm` binary — use `-p mesh-llm` for build/check (it pulls the host runtime through its single dep) and `-p mesh-llm-host-runtime` for focused tests.
- Touched a specific workspace crate (e.g., `skippy-runtime` , `openai-frontend` , `mesh-client` ) — run `cargo check -p <crate>` and `cargo test -p <crate> --lib` for fast iteration.
- For broad refactors, fall back to `cargo check --workspace` (serially!).
mesh: drop peers below v0.60.0 from gossip ingest and re-broadcast (#576)
* mesh: drop peers below v0.60.0 from gossip ingest and re-broadcast
Reject sub-floor peers in `add_peer` (direct ingest) and
`update_transitive_peer` (transitive ingest), and exclude them from
`collect_announcements` as a safety net. Peers below the floor do not
appear in /api/status, the UI, routing, or outbound gossip.
Observed on the public mesh: peer count drops from ~329 to ~17 on a
fresh `mesh-llm client --auto`, with the v0.57.x cohort going from
306 peers to zero. Real hosts and workers above v0.60 are unaffected.
v0.60.0 is chosen because that release added the on-wire `hardware`
block; below it, gossip metadata is missing fields the current mesh
relies on. Build metadata (`+skippy.…`) and pre-release tags
(`-rc5`) are stripped before comparison. Peers that advertise no
version at all are conservatively accepted, on the theory that a
missing version is more likely to be a legitimate old node than a
targeted bypass.
Tested live in both client and serve mode against the public mesh.
17/17 gossip tests pass including five new tests covering version
parsing edges, transitive-ingest rejection, and direct-add rejection.
* fix(clippy): rewrite version-floor comparison for absurd_extreme_comparisons
clippy flags 'major < 0' as always-false. Restructure as != + > so clippy
is happy while keeping the comparison shape for the day the floor bumps
to a non-zero major.
* docs(agents): clarify --headless vs --log-format json for local runs
--headless disables the embedded web UI, not the TUI. Reaching for it to
'go quiet' is a recurring mistake — use --log-format json instead.
* mesh(gossip): drop idle transitive clients from local table
Refuse transitive ingest of peers that are:
- Client role, AND
- have empty requested_models, AND
- have empty serving_models, AND
- have empty (or absent) hosted_models.
Such a peer contributes nothing through transitive propagation:
- not routable to (no model to serve)
- not findable (clients don't dial clients by design)
- no demand signal (empty requested_models)
- not relaying for us (purely transitive, no connection)
Direct ingest in add_peer is unchanged — a client we actually connect to
is admitted regardless of what they advertise. The moment a transitive
client starts asking for, serving, or hosting anything, the filter stops
firing and they are admitted normally.
Pairs with the v0.60.0 version floor (95eac80a): the floor catches the
current v0.57.x swarm cohort, this filter catches any future cohort that
adopts the same empty-client shape under a different version string.
Tests cover the predicate shapes, transitive rejection, and an explicit
test that direct add_peer still admits idle clients.
* mesh(gossip): refine idle-transitive-client predicate to add hostname + direct-measurement checks
The previous predicate caught real idle ≥v0.60 clients (Olympus.local,
Sams-MacBook-Pro.local, perseus.local, etc.) that were just idle users
on the public mesh — anyone running 'mesh-llm client --auto' without
asking for a model right now.
Tighten the predicate to require BOTH:
- hostname is None (no machine identity), AND
- latency_source != Direct (no direct measurement by any peer)
hostname is populated synchronously by system::hardware::survey() at node
construction, so every real client on every supported platform has one
from its first gossip frame. latency_source=Direct is set by any peer in
the mesh that has measured this peer's RTT via direct contact; a peer
that has been directly measured by anyone is real.
Verified against live /api/status data from the public mesh:
- Catches all 306 v0.57.x swarm members (already caught by version
floor; redundant safety here for shape-only attack)
- Catches 2 pre-v0.60 hostname-less ghosts (already caught by floor)
- Keeps all 11 real ≥v0.60 idle clients with hostname
- Keeps the 1 real ≥v0.60 client without hostname that someone in the
mesh directly measured (rtt_ms=1, latency_source=direct)
Zero false positives on the live snapshot.
Tests updated to lock in the new behaviour: hostname-only, direct-only,
estimated-doesn't-count, hostname-and-direct-both, and the model-interest
exits.
2026-05-18 14:22:28 +10:00
## Running mesh-llm locally
Default the launch to a normal foreground run (TUI visible) unless you have a
specific reason to suppress UI surfaces. Most observation/debug tasks do not
need the TUI suppressed.
- `mesh-llm client --auto` — normal foreground run with the TUI. Use this by
default.
- `--log-format json` — emits machine-parseable JSON log lines. Use this when
you want to programmatically read events.
- `--headless` — disables the **embedded web UI** , not the TUI. The TUI still
draws. Only use `--headless` when you are intentionally avoiding the
management web console — it is **not** the way to get a quiet background run.
- `--no-console` — fully disables the management console (HTTP API on the
console port).
- `nohup … &` with a foreground binary that draws a TUI will appear to run but
often exits or behaves oddly when the TUI cannot attach to a terminal. Prefer
letting the developer launch the binary in their own terminal and observing
via `/api/status` , `--log-format json` , or by reading stderr.
Do not reach for `--headless` to "go quiet" — that is a recurring mistake. If
you want quiet output, use `--log-format json` and parse what you need.
2026-04-11 14:04:48 +10:00
## Pre-Commit Checklist
2026-03-27 22:29:40 +11:00
2026-04-11 14:04:48 +10:00
Before committing, run the local checks most likely to fail in CI for the files you touched. Do not rely on CI to catch basic formatting, compile, or stale UI build issues.
### Minimum bar before every commit
2026-05-29 17:11:54 +10:00
- Rust-only change — format the changed Rust files and run `cargo check -p <touched-crate>` plus `cargo clippy -p <touched-crate> --all-targets -- -D warnings` (and both commands with `-p mesh-llm` if you touched anything reachable from the shipped binary).
2026-04-11 14:04:48 +10:00
- UI-only change — run `just build` .
- Mixed Rust and UI change — run `just build` .
### Rust changes
2026-06-17 10:01:37 +10:00
- The preferred Rust edition for this workspace is Rust 2024. Determine the edition from the owning crate's `Cargo.toml` ; if it uses `edition.workspace = true` , read `workspace.package.edition` from the root `Cargo.toml` . Most crates inherit `edition = "2024"` from the root; any crate that opts out declares its own edition in its `Cargo.toml` .
- Format Rust files in a way that preserves the owning crate's edition metadata. Prefer `cargo fmt -p <crate> -- path/to/file.rs` for a narrow edit, or `cargo fmt --all` when changes span packages. Do not use `cargo fmt --all -- path/to/file.rs` : workspace-level file arguments can be parsed without the owning crate's Rust 2024 edition metadata and fail on let-chains.
- If you must invoke `rustfmt` directly on a standalone file, pass the edition resolved from that manifest lookup, for example `--edition 2024` for the current workspace default; otherwise use `cargo fmt` through the owning package.
- Before committing Rust changes, ensure the formatting check passes with `cargo fmt --all --check` .
2026-05-29 17:11:54 +10:00
- After Rust changes, run `cargo check` and `cargo clippy --all-targets -- -D warnings` for each touched crate (`-p <crate>` ), and at least `cargo check -p mesh-llm` plus `cargo clippy -p mesh-llm --all-targets -- -D warnings` if the change is reachable from the shipped binary.
- Treat Clippy as a required local gate, not a CI-only cleanup step. `cargo check` , `just build` , and formatter success do not catch lints such as `clippy::collapsible-if` ; run the warning-denying Clippy command before opening or updating a PR.
2026-05-19 17:26:09 +10:00
- If you touched tests, public APIs, routing, inference, gossip, plugin protocol, skippy ABI, or CLI behavior, run the relevant tests before committing.
- If you touched `proto/` , any `protocol/` module, `mesh-llm-host-runtime/src/mesh/gossip.rs` , `mesh-llm-host-runtime/src/mesh/mod.rs` , routing, election, API serialization, or `skippy-ffi` ABI constants, do not stop at build-only validation: run at least `cargo test -p mesh-llm-host-runtime --lib` (plus `cargo test -p skippy-ffi --lib` / `-p skippy-runtime --lib` when ABI is touched) and wait for it to exit successfully before committing.
2026-04-14 09:54:50 +10:00
- Do not report a build or test step as complete until the command has actually exited with code `0` .
2026-04-11 14:04:48 +10:00
- Run Rust validation serially. Do not run multiple `cargo` commands at the same time.
2026-07-23 10:45:39 -04:00
### CI changes
Before inspecting, running, defining, editing, reviewing, or documenting CI,
read `.agents/skills/manage-ci/SKILL.md` completely. The `manage-ci` skill is
the canonical source for workflow, dependency, runner/worker, image, cache,
artifact, variable, secret, permission, release, deployment, operational, and
validation rules. Start every `.github/` , CI-script, or runner-integration edit
there.
Keep `.agents/skills/manage-ci/references/current-inventory.md` synchronized
with the checked-in CI contract and `ci/ci.md` synchronized with topology. When
a CI rule changes, update the skill first rather than adding duplicate guidance
to this file or `.github/AGENTS.md` .
2026-05-23 20:19:32 +10:00
2026-04-11 14:04:48 +10:00
### UI changes
- Use the repo's supported workflow and run `just build` .
2026-06-12 18:44:36 +10:00
- If `just build` fails on the UI step with `npm error Exit handler never called!` , run `just ui-clean` and then rerun `just build` .
2026-04-11 14:04:48 +10:00
### Commit standard
- Do not commit if formatting has not been applied.
- Do not commit if basic local validation for your change type has not been run.
- Do not commit known warnings in code you touched.
2026-03-27 22:29:40 +11:00
2026-04-03 14:44:25 +11:00
## Warnings
Do not leave Rust compiler warnings behind in code you touched.
- Fix or remove unused code, dead code, and other warnings introduced or surfaced by your change before committing.
- Do not silence warnings with `#[allow(...)]` unless there is a clear reason and the developer has asked for that tradeoff.
2026-04-03 15:03:27 +11:00
## Pull Requests
Pull request titles and descriptions should be user-focused by default.
2026-06-17 10:01:37 +10:00
- Prefer the GitHub CLI (`gh` ) for GitHub operations in this repo, including inspecting issues/PRs, editing PR descriptions, pushing branches, and opening PRs. Use built-in MCP/GitHub connector tools only as a fallback or for read-only lookup when `gh` cannot provide the needed data.
2026-04-03 15:03:27 +11:00
- Title PRs around the user-visible change or capability, not the implementation detail.
- Start the description with what the user can now do, see, or understand after the change.
- Keep architectural refactors, internal state reshaping, and code-organization notes out of the opening summary unless they directly change user behavior.
- If there are important architectural changes, add a separate `## Architecture` section.
- If there are protocol or compatibility implications, add a separate `## Protocol` section that clearly calls out compatibility, migration, or breaking-change impact.
- If the PR changes CLI behavior or touches user-facing CLI flows, include example commands and representative output in the PR description.
- If the PR changes the UI, include at least one screenshot in the PR description.
- Validation and screenshots should stay separate from the user-facing summary.
2026-03-05 08:03:51 -05:00
### Deploy to Remote
```bash
2026-06-12 18:44:36 +10:00
just bundle # /tmp/mesh-llm-bundle.tar.gz — single mesh-llm binary
# scp bundle to remote, tar xzf, then on macOS: codesign -s - mesh-llm && xattr -cr <dir>
2026-03-05 08:03:51 -05:00
```
2026-08-08 03:39:13 -04:00
For the full per-platform deploy flows, see the repo skills
`.agents/skills/deploy-macos/` , `.agents/skills/deploy-linux-gpu/` , and
`.agents/skills/deploy-windows/` .
2026-06-12 18:44:36 +10:00
2026-03-05 08:03:51 -05:00
### Cleanup
runtime: introduce scoped instance runtime with safe process lifecycle management
Replace global pkill-based process management with a per-instance runtime
directory system backed by flock liveness, atomic JSON pidfiles, and scoped
orphan reaping. Each mesh-llm process now owns a directory under the runtime
root, writes pidfiles for its child processes, and cleans up on exit. A
background scanner tracks all local instances and exposes them through the
management API and UI.
Key changes:
- InstanceRuntime: scoped runtime directory with flock-based liveness,
atomic JSON pidfile RAII guards, portable PID comm/start-time validation,
and pure decision logic for cross-instance orphan reaping.
- Process lifecycle: llama-server and rpc-server are now launched with
PID-tracked handles (LlamaServerHandle, RpcServerHandle) that write
pidfiles on start and reap on drop. Signal delivery validates PID comm
before sending, guarded by is_safe_kill_target to prevent kill(0/-1).
- Legacy removal: kill_llama_server, kill_orphan_rpc_servers,
terminate_process_by_name, and pkill-based helpers are removed.
A regression test asserts zero occurrences of these patterns in source.
- API: /api/status now includes version and local_instances fields,
populated from the background scanner with a self-entry safety net.
- UI: peer version is shown in the node table and sidebar; warning banners
surface mixed-version meshes and multi-instance conflicts.
2026-04-08 04:38:18 -04:00
Clean shutdown removes the instance's runtime directory automatically. Prefer the scoped runtime-aware commands first:
```bash
mesh-llm stop
just stop
```
Those paths use the runtime metadata under `~/.mesh-llm/runtime/` to stop the tracked mesh-llm instance and its child servers cleanly.
If an instance is wedged badly enough that the scoped stop path cannot reach it, fall back to an emergency kill:
2026-03-05 08:03:51 -05:00
```bash
2026-05-05 11:48:40 +10:00
pkill -f mesh-llm
2026-03-05 08:03:51 -05:00
```
MoA: mesh mode and many inference critical fixes, and quic keep alive (#566)
* feat: MoA gateway — stateful mixture-of-agents with tool arbitration
New standalone crate (moa-gateway) that fans out to N heterogeneous LLM
endpoints in parallel, normalizes dirty worker outputs, arbitrates with
deterministic logic, and manages the full tool call lifecycle across turns.
Tested live against 3 ollama models (llama3.2:3b, qwen3:4b, qwen3.6:27b):
- Knowledge/reasoning: picks highest-confidence answer across models
- Tool calling: correctly produces tool_calls when workers propose tools
- Tool lifecycle: full cycle query → tool_call → result → final answer
- Tool results bypass fan-out, go to reducer only (one transcript)
The gateway is transport-agnostic — works against any OpenAI-compatible
endpoint (ollama, mesh-llm, remote APIs). Integration with mesh model
discovery is the next step.
* feat: multi-turn context efficiency + mesh endpoint discovery
Progressive running summary: workers get compact deterministic summaries
instead of raw message history. Tested across 3-turn conversations —
workers retain context (Melbourne → restaurant recommendation) through
the summary, not through replaying 20k tokens of history.
New moa-mesh binary discovers models from any OpenAI-compatible endpoint
(mesh-llm proxy, ollama, vLLM) and runs the full MoA test suite against
it. Designed to work with 'mesh-llm client --auto' out of the box.
Multi-turn tool lifecycle proven end-to-end:
Turn 1: weather query → fan-out → tool_call (get_weather)
Turn 2: tool result → reducer only → text answer
Turn 3: follow-up 'bring jacket?' → fan-out with running summary → contextual answer
* fix: increase running summary budget to ~2k tokens
Per-fact truncation: 200 → 500 chars
Recent fact window: 5 → 15 facts
Tool result truncation: 80 → 300 chars
Turn outcome capture: first sentence → first 400 chars
200 tokens was too aggressive for real agent sessions where system
prompts alone can be 500+ tokens. 2k tokens gives enough room for
15 turns of meaningful context while still being much cheaper than
replaying raw history.
* feat: agentic workload test + improved prose tool detection
New moa-agent binary simulates a multi-step coding agent: read file,
analyze bug, edit fix, run tests, diagnose failure, iterate. Exercises
7+ turns with accumulating context, repeated tool use, and loop detection.
Improved normalizer: small models that describe tool usage in prose
("I'll use the edit_file tool") are now correctly classified as
tool proposals via known-tool-name + action-verb heuristic.
Key findings from agentic testing:
- Gateway routing, context management, and tool lifecycle are solid
through 7 turns / 15 messages / 6 reducer calls
- Loop detection catches repeated identical tool calls
- Small models (3b/4b) produce correct tool calls for read/search
but describe edits in prose instead of calling edit_file
- Multi-step plan execution (tool→analyze→tool→fix) needs a stronger
model in the reducer role — the arbiter and routing aren't the
bottleneck, model capability is
* fix: mesh-tested agentic flow — local reducer, model dedup, KV parse fix
Tested against live mesh: GLM-4.7-Flash (local) + Qwen3-8B (remote peer).
Three fixes from live mesh testing:
1. Local reducer: prefer first endpoint (local model) as reducer instead
of last. Remote models over QUIC relay are fine as parallel workers
but timeout as the sequential reducer. Result: reducer response
times dropped from 180s (remote timeout) to 2-18s (local).
2. Model dedup: mesh-llm exposes the same model under multiple aliases
(e.g. unsloth/GLM-4.7-Flash-GGUF and @main:Q4_K_M variant).
discover_endpoints() now deduplicates by normalized display name.
Extracted as shared lib function used by all three test binaries.
3. KV parse fix: models that say 'kind: answer' but also include
'tool: read_file' are now correctly classified as ToolProposal.
This was the #1 cause of missed tool calls in the agentic flow.
* moa: integrate into mesh proxy, SSE streaming, Goose support
MoA is now available as model="moa" through the mesh proxy on :9337.
When ≥2 models are available (local + mesh peers), the MoA virtual model
appears in /v1/models automatically.
Integration:
- mesh-llm-host-runtime depends on moa-gateway
- ingress.rs intercepts model="moa" requests, builds endpoints from
callable models, calls Gateway::turn(), returns the result
- SSE framing: converts the non-streaming MoA response into SSE chunks
for streaming clients (Goose, pi, etc.)
- Tool calls passed through in SSE format with finish_reason="tool_calls"
Passthrough mode (tools present):
- When the request includes tools (agentic use via Goose/pi), workers
receive the original messages+tools unmodified — no MoA envelope
- This avoids conflicting system prompts confusing small models
- First successful worker response is returned, providing redundancy
Content cleanup:
- Think tags (<think>...</think>) stripped from all responses
- Orphan </think> tags cleaned up
- KV envelope lines (kind:/confidence:/payload:) stripped when they
leak into heuristic-classified output
- Normalizer pre-cleans think tags before trying JSON/KV parse
Tested with:
- Direct curl (non-streaming + streaming)
- Goose CLI: factual questions, tool execution (shell, execute_typescript)
- 22 unit tests passing
* fix: add moa-gateway to Docker builds
The docker-client CI job failed because crates/moa-gateway/ was missing
from both Dockerfile.client and fly/Dockerfile. cargo metadata couldn't
resolve the workspace member, breaking the cargo-chef prepare step.
* moa: early-exit on worker consensus
Instead of waiting for all workers before arbitrating, check for
consensus after each worker returns. When 2+ workers agree on an
answer or tool call, return immediately and abort remaining workers.
This eliminates the 'slowest worker' bottleneck. In testing:
- Average latency dropped from 21.0s to 5.8s (single model: 7.1s)
- MoA is now 19% faster than querying a single model
- Worst case (code-debug) went from 120s timeout to 4.2s
The key insight: with parallel fan-out, we only need to wait for the
fastest N workers that agree, not all of them. Slow/dead remote
workers no longer block the response.
Also adds 5 new arbiter tests for early decision logic (27 total).
* moa: real context slices, not synthetic envelopes
Major architectural change to how MoA packs context for workers.
Before: workers got a synthetic system prompt ('You are a fast analysis
worker...') that replaced the agent's real system prompt, tool schemas,
and conversation history. Workers were asked to respond in a KV envelope
format (kind:/confidence:/payload:) that small models followed unreliably.
When tools were present, the entire MoA pipeline was bypassed via a
'passthrough mode' that raced identical requests to all workers.
After: workers get slices of the REAL context — the agent's actual system
prompt and messages — with depth varying by role:
- Fast: system prompt + last user msg + tool names only
- Specialist: system prompt + last 4 msgs + tool summaries
- Strong: system prompt + full recent history + native tool schemas
- Reducer: system prompt + worker outputs + full tool schemas
The gateway augments with a one-line preamble, not a replacement. The
passthrough mode is removed — tool-use goes through the full normalize →
arbitrate pipeline. Strong workers get native tool schemas forwarded so
they can produce real tool_calls.
Also:
- Dedup model aliases in ingress (GLM and GLM@main:Q4_K_M are the same)
- Early exit handles failed workers (sole survivor returns immediately)
- Worker timeout reduced from 120s to 30s
- 28 unit tests (up from 27)
* moa: rename to mesh-mixture-of-agents, mesh-native transport, model='mesh'
Renamed crate from moa-gateway to mesh-mixture-of-agents. Keeps the
crate isolated (own tests, own compilation unit) while connecting it
to mesh transport via a ModelBackend trait.
Transport is now mesh-native instead of HTTP loopback:
- LocalModelBackend: direct HTTP to skippy port (bypasses proxy)
- RemoteModelBackend: QUIC tunnel to peer (bypasses proxy + tunnel layer)
- Both set mesh_hooks: false to prevent recursive consultation
The ModelBackend trait keeps the crate testable in isolation — the
default HttpBackend works against any OpenAI-compatible endpoint.
The mesh backends are implemented in ingress.rs where Node and
InferenceTarget are available.
Virtual model renamed from 'moa' to 'mesh'. Appears in /v1/models
when ≥2 distinct models are available.
Test bins removed (used old Gateway API). 29 unit tests remain.
* fix: update Dockerfiles for moa-gateway → mesh-mixture-of-agents rename
* moa: remove 'mesh' from /v1/models list
The 'mesh' virtual model is a routing directive like 'auto', not a
real model. It should not appear in the models list. Clients that
want MoA fan-out use model: "mesh" explicitly.
* fix: clippy warnings in mesh-mixture-of-agents
* fix: clippy unnecessary_lazy_evaluations in ingress build_moa_config
* docs: update MoA design doc with current architecture and test plan
Reflects: mesh-mixture-of-agents crate rename, ModelBackend trait,
handle_turn() stateless API, mesh-native transport, model='mesh'
virtual routing, early-exit consensus, and eval plan.
* moa: fix tool call arguments lost in arbitration
Two fixes:
- Arbiter now prefers tool proposals with actual arguments over
proposals that only have the tool name (from fast workers that
don't get native tool schemas).
- Specialist workers now receive native tool schemas so they can
produce structured tool_calls with arguments, not just mention
tool names in text.
Before: read_file({})
After: read_file({"path":"/tmp/test.txt"})
* moa: worker diversity sampling, 429 retry, faster timeouts, sole-survivor early exit
Three improvements to MoA reliability and response quality:
- Workers get high temperature (0.8) + top_p (0.95) for diverse
exploration; reducer gets low temperature (0.3) for precise synthesis.
SamplingParams flows through the ModelBackend trait.
- 429 rate-limit errors trigger one automatic retry after the server's
retry-after delay (default 1s).
- Worker timeout 30s → 15s, reducer 45s → 30s.
- Sole survivor returns immediately when majority of other workers have
already failed, instead of waiting for remaining stragglers.
39 unit tests (up from 29).
* Fix MoA tool result handling and NaN confidence (PR review feedback)
Three issues from Copilot review on PR #534:
1. Tool result turns now include actual tool output content.
pack_for_tool_result_turn was reading from pending_tools which
is always empty on a fresh session (stateless per request).
Now forwards the raw message sequence including assistant
tool_call + tool result messages so the reducer sees the full
context. Added regression test.
2. NaN confidence no longer panics arbiter comparisons.
Replaced partial_cmp().unwrap() with total_cmp() in arbiter,
and added a sanitizer in normalize that clamps non-finite
confidence to 0.5. Added test.
3. Exclude spec-prefill-poc from workspace members (Docker fix).
Moved to Cargo.toml exclude list so Docker builds don't fail
on the missing experimental crate.
* ci: align WORKSPACE_MEMBERS with current workspace
- Add mesh-mixture-of-agents (new crate on this branch)
- Rename mesh-llm-client → mesh-client (renamed on main)
Fixes the scripts/affected-crates.sh consistency check that gates the
Linux CPU CI build.
* moa: size-aware role assignment + hallucinated tool name filtering
Two surgical fixes from real-world goose testing:
1. Role assignment by capacity tier, not list-order.
assign_roles previously used list order: first=fast, last=strong.
When a small local model (e.g. Qwen2.5-3B) was loaded last, it got
tagged Strong and used as the reducer — exactly when goose needs a
capable model for tool arbitration.
Now we sort by size tier using the same is_single_digit_b_name
heuristic as the main router's pick_model_classified, so MoA's
strong worker matches what auto would pick. MiniMax-M2.5 and
Qwen3-32B (big tier) become Strong; Qwen3-8B and Qwen2.5-3B (small
tier) become Fast.
2. Filter hallucinated tool names from worker proposals.
A worker proposing a tool not declared in the request (e.g. local
3B hallucinating 'execute_typescript' when only 'shell' was offered)
would bypass arbitration and reach the client, causing silent
failures when goose tried to dispatch the unknown tool.
gather_workers_incremental and the reducer output paths now demote
such proposals to Uncertainty with a tracing warning. The arbiter
sees a clean set of valid proposals and resolves correctly.
Threading: handle_query, handle_tool_result, gather_workers_incremental,
and resolve_decision all now take &[String] allowed_tools derived from
session.tool_names(). When allowed_tools is empty (no tools on request)
the filter is a no-op.
* moa: reducer candidate fallback on 5xx / timeout
When the chosen reducer peer is broken (e.g. stale binary returning 502
on tool grammars) or unreachable, the tool-result turn or NeedsReducer
arbitration would fail with that single error, even though other strong
peers were available.
Replace single-pick pick_reducer with reducer_candidates returning all
big-tier models (multi-digit B or no size in name) followed by small-tier
as last-resort fallback. Both call sites — handle_tool_result and
resolve_decision NeedsReducer — now iterate candidates and break on the
first success.
This rescues the common goose-on-public-mesh case where one strong peer
(e.g. Qwen3-32B host) is running a stale binary that 502s on tool calls,
while another (e.g. MiniMax) is healthy. Without this, MoA's tool-result
turn was as fragile as auto routing.
* ci: fix workspace member drift — keep mesh-llm-client package name, add mesh-mixture-of-agents to clippy script
Same fix as the prefill-draft branch:
- The mesh-client directory rename did not change the package name —
the crate is still published as 'mesh-llm-client'. Revert the
scripts/affected-crates.sh edit that broke the CI consistency check.
- Add mesh-mixture-of-agents to plan-clippy-batches.sh which carries
its own WORKSPACE_MEMBERS list with the same drift constraint.
* moa: support model:"mesh" on client/standby nodes via forward-to-host
Pure --client nodes and standby GPU nodes accept inbound HTTP via
handle_mesh_request (in transport.rs) instead of the model-aware api_proxy
in ingress.rs. The MoA fan-out intercept lives in api_proxy, so when a
client received "model": "mesh" it fell through to the "no host serves
this model" branch and 429d.
* moa: fix clippy lints surfaced by CI
Two pre-existing lint violations in mesh-mixture-of-agents that CI didn't
see before because the crate wasn't in the affected-crates / clippy
workspace lists. Now that the WORKSPACE_MEMBERS drift is fixed they show
up on every PR clippy run.
- worker.rs:67 `x == false` -> `!x` (clippy::bool_comparison)
- lib.rs:523 `&name` -> `name` (clippy::needless_borrow)
No behavior change — tool_call_response takes &str either way, and the
sort key inverts identically.
* moa: hedge reducer candidates instead of sequential fallback
Cut worst-case reducer latency from N×timeout to roughly
reducer_timeout + (N-1)·hedge_delay. Big win when a peer is slow or
broken; zero cost on the happy path.
Before:
for candidate in candidates:
call(candidate, timeout=30s) # wait up to 30s per stale peer
if ok: return
# 3 stale big-tier peers ⇒ 90s before falling through to small-tier
After:
spawn candidate[0]
loop:
select:
a candidate finished:
ok → cancel rest, return
err → spawn next candidate immediately (no hedge wait)
hedge_delay elapsed and more candidates remain:
spawn next alongside in-flight ones (race)
Cost shape:
- Happy path (cand 0 OK in <hedge_delay): exactly 1 backend call. Free.
- Slow first (cand 0 takes hedge_delay..reducer_timeout): up to 2
overlapping calls, accept whichever wins, cancel loser.
- Fast-fail (cand 0 errors quickly): next candidate immediately, 1 call.
- All fail: ≤N calls, capped at reducer_timeout + (N-1)·hedge_delay.
Wall-clock improvement for 3 stale big-tier peers (worker_timeout=15s,
reducer_timeout=15s, hedge_delay=5s):
- Before: 3 × 30s = 90s before reaching small-tier fallback.
- After: 15s + 2 × 5s = 25s. Plus reducer_timeout itself drops 30s → 15s
now that the hedged ladder makes a single per-attempt cap safe to
shorten.
Changes:
- Add hedged_reducer_call() in mesh-mixture-of-agents/src/lib.rs.
- Replace the for-loop in handle_tool_result() with it.
- Replace the for-loop in resolve_decision()'s NeedsReducer arm with it.
- Add hedge_delay field to GatewayConfig (defaults set at the single
construction site, build_moa_config in ingress.rs).
- Lower reducer_timeout 30s → 15s in build_moa_config.
- 4 new unit tests cover happy path, hedge-on-slow, fast-fail, all-fail.
- Refresh stale numbers in docs/design/MOA_GATEWAY.md and replace the
"first model wins" reducer paragraph with the hedged-ladder description.
Verified:
cargo fmt --all -- --check # clean
cargo check -p mesh-llm-host-runtime # clean
cargo clippy -p mesh-llm-host-runtime --lib # clean
cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings # clean
cargo test -p mesh-llm-host-runtime --lib # 1381 passed
cargo test -p mesh-mixture-of-agents --lib # 47 passed (4 new hedge tests)
* evals: add bench-moa.sh — quick wall-clock benchmark for model:"mesh"
POSTs N chat completion requests to a running mesh-llm endpoint with
model="mesh" and reports p50/p95/p99 wall-clock latency. Probes /v1/models
first and warns if fewer than 2 models are present (MoA returns 503).
Usage:
./evals/bench-moa.sh # 20 requests, localhost:9337
N=50 ./evals/bench-moa.sh
BASE_URL=http://host:9337 ./evals/bench-moa.sh
PROMPT="why is the sky blue?" ./evals/bench-moa.sh
Output is per-request lines plus a summary block (min/p50/p95/p99/max/
mean/stdev). Failed requests are reported but excluded from percentiles.
Raw TSV results are kept in a tmpdir so before/after comparisons are easy.
Dependencies: curl, jq, python3 — nothing exotic. Run on the same machine
as a serving host so wall-clock is dominated by inference + arbitration.
* moa: split lib.rs into backend / reducer / fanout modules
lib.rs was past the 1k LoC refactoring threshold and had three separable responsibilities. Extract them into named modules and keep lib.rs as the orchestration entrypoint (handle_turn, GatewayConfig, TurnResult, response builders).
- backend.rs: ModelBackend trait, HttpBackend, SamplingParams, ModelEntry, call_backend + retry-after parsing
- reducer.rs: reducer_candidates ordering, hedged_reducer_call ladder
- fanout.rs: gather_workers_incremental
ModelEntry, HttpBackend, ModelBackend, and SamplingParams are re-exported from lib.rs so existing callers (worker.rs, host-runtime ingress) keep working.
Tests move with their owner: backend.rs gets the sampling/retry-after tests, reducer.rs gets the 4 hedged-reducer tests plus the FakeBackend helper. No behavior change.
lib.rs: 1267 -> 545 LoC
47 existing tests still pass.
* moa: cover role-shaped context packing with tests
context.rs owned the role-shaped packing logic (fast/specialist/strong/reducer depth contract) without any tests. Pin the design claims:
- per-role token budgets (256 / 512 / 1024)
- fast: system + last user only, tool names only, no tools field
- specialist: tool summaries in system, native tools populated
- strong: deep history (>=6 msgs), native tools populated
- generalist/reducer roles alias the strong shape
- MoA preamble augments rather than replaces the agent's system prompt
- reducer context includes the conflict reason and labeled worker payloads
- long worker payloads are truncated with an ellipsis to bound context
8 tests, all green.
* docs: surface model:"mesh" (MoA) in README
Add a workflow-table row pointing at MOA_GATEWAY.md and a short 'Mixture-of-Agents' section with a curl example and the two-models-required gate, so the feature is discoverable from the project entrypoint.
* moa: extract tool_guard module, drop dead Endpoint/discover_endpoints
Two small cleanups on lib.rs:
- Move enforce_allowed_tools into its own tool_guard.rs module. It's a
content-policy concern (demote hallucinated tool names to Uncertainty
before arbitration), not orchestration, so it doesn't belong in the
handle_turn entrypoint file. Comes with 4 unit tests covering allowed
pass-through, unknown-tool demotion (incl. confidence drop),
empty-allowed-list noop, and non-proposal outputs untouched.
- Delete the Endpoint struct and discover_endpoints helper from lib.rs.
Their doc comments described them as 'convenience for test harnesses'
but they have zero call sites in-tree and no out-of-tree consumers we
know of. Dead code from the standalone phase before mesh-native
backends landed.
lib.rs: 545 -> 454 LoC.
Tests: 55 -> 59 (4 new in tool_guard).
* docs(moa): drop the speculative hook-integration line
MoA and hooks are intentionally independent — worker requests set
mesh_hooks: false so the hook pipeline can't re-enter a worker call.
The old design doc closed the relationship section with 'they could
integrate later (hook signals as arbiter weights)', which makes it
look like roadmap. It isn't — keeping them separate is the design.
Replace that line with one that states the separation as intentional
and points at the mesh_hooks: false invariant that enforces it.
* router: weight 'auto' selection by locally observed tok/s
Before, 'auto' picked uniformly at random within the multi-digit-B
tier. On the public mesh this meant a fast MiniMax on a 4090 and a
slow 35B-A3B on an M2 Air were equally likely to be chosen, even
though we'd already measured the throughput gap in routing_metrics
and were just not reading it.
Now: each big-tier candidate is weighted by its locally observed
avg_tokens_per_second (clamped to [5, 100] tok/s so nothing fully
starves and no outlier monopolizes). Models without enough samples
(< 3) get a neutral weight so they compete fairly until data
accumulates. A 15% exploration probability ignores weights and
picks uniformly, which keeps the system from locking onto stale
rankings and guarantees cold peers see traffic.
Plumbing:
- RoutingMetrics::tps_for_model(name) -> Option<(f64, u64)>: cheap
per-model lookup that locks only the relevant shard, avoiding the
per-call HashMap allocation model_snapshots() does in the hot path.
- Node::routing_metrics() public accessor (Arc-backed, cheap).
- RoutingCandidate { name, caps, tps_hint, throughput_samples }
replaces the anonymous (&str, f64, ModelCapabilities) tuple whose
middle slot was literally always 0.0 at every populated call site.
The struct makes the tps hint a real, typed concept rather than a
dangling hook.
Behaviour preserved:
- Single-digit-B partition (smalls stay last-resort) unchanged.
- All-cold candidate pool falls back to ~uniform pick (regression
test confirms no model is starved when there's no data yet).
- Capability filtering for tools / reasoning / vision unchanged.
Plumbing per call site:
- ingress.rs + transport.rs: live routing path, look up tps_hint
from the local RoutingMetrics handle for each candidate.
- discovery.rs + integrations.rs: pre-startup paths with no live
metrics; build candidates with RoutingCandidate::unscored() so
they get the cold-neutral weight.
Tests:
- weighted_pick_all_cold_is_roughly_uniform — regression safety.
- weighted_pick_fast_wins_majority_but_slow_still_gets_some —
fast wins by >=1.5x but slow still gets >30/600 picks.
- weighted_pick_cold_model_competes_with_hot_fast — newcomer gets
>100/600 picks against an established fast peer (so it can
actually accumulate samples and earn its score).
- weighted_pick_low_sample_count_treated_as_cold — 1-sample
measurements don't dominate routing.
- candidate_weight_clamps_extremes — weight stays in [5, 100],
cold = 25.
Removed:
- shuffle_in_place (replaced by SplitMix64 + pick_weighted).
- The dishonest 0.0 f64 slot in the candidate tuple, everywhere.
Validation:
cargo fmt --all -- --check # clean
cargo check -p mesh-llm-host-runtime # clean
cargo clippy -p mesh-llm-host-runtime --lib # clean
cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings # clean
cargo test -p mesh-llm-host-runtime --lib # 1398 passed (17 in router)
cargo test -p mesh-mixture-of-agents --lib # 59 passed (no regression)
* docs(moa): clarify topology — N workers + serial 2-call shape
- Replace topology diagram with one that shows N workers fanned out in
parallel and the serial fan-out → arbiter → reducer path.
- Add explicit "how many models" table (2..N) and the worker → role
mapping so readers don't have to infer it from worker.rs.
- Spell out that a worst-case MoA turn is 2 LLM round-trips serially
(fan-out wall-clock = slowest worker, then optional reducer), and
that happy paths collapse to 1 (consensus or tool-result turn).
- Refresh stale crate-structure table: post-split LoC + test counts
for backend / reducer / fanout / tool_guard / arbiter / context /
worker / session / normalize / lib.
* moa: emit x-moa-* observability headers from gateway
Extend TurnResult with turn_kind (Fanout / EarlyExit / ToolResult / Failed)
and reducer_attempts (candidates actually spawned). hedged_reducer_call
now returns a named HedgedReducerOk struct carrying winner, text, and
spawn count so the caller can attribute hedge cost.
The ingress MoA intercept reads these and emits:
x-moa-elapsed-ms
x-moa-turn fanout | early-exit | tool-result | failed
x-moa-workers total workers dispatched
x-moa-workers-ok workers that returned a usable answer
x-moa-reducer true | false
x-moa-reducer-attempts 0 on no-reducer path, 1 happy, >=2 hedged
Headers are emitted on both the JSON and SSE response paths via a new
send_json_ok_with_headers helper and an extra_headers arg to
send_moa_as_sse. Normal OpenAI clients ignore unknown headers; benches
and ops tooling can read them without parsing the body.
Side fix: handle_tool_result previously reported attempts = total
candidate pool size rather than candidates actually spawned. Now
correctly reports the spawn count from HedgedReducerOk.
* bench-moa: aggregate gateway path, reducer, and hedge stats
Read x-moa-* response headers per request and roll them up in the
summary. New aggregates:
Gateway paths histogram of fanout / early-exit / tool-result / failed
Reducer invocation rate + hedge rate + avg/max attempts
Worker fan-out average width + histogram by N
Latency p50 split by gateway path (so 'reducer turns are 4x slower'
is visible at a glance)
Per-request log line now shows the turn kind, worker count, and reducer
status alongside the latency, making it easier to eyeball individual
outliers.
Older mesh-llm binaries that don't emit x-moa-* headers degrade to the
old summary (latency-only) with a note that headers were not seen, so
this works against any mesh-llm version.
* evals: remove bench-moa.sh — never actually run
The script was written but never executed against a live mesh. MoA verification is the manual live-mesh testing called out in the PR body. Real aggregates, if we want them, should come from passive counters fed by real traffic, not a synthetic curl loop.
* moa: orchestrate from any node, build worker pool from mesh-wide gossip
Before this change MoA only ran when the request hit `api_proxy` on a
serving host. A pure `--client` node received `model: "mesh"` in
`handle_mesh_request`, fell through to the "forward to any host"
fallback, and the receiving host either ran an older binary that
ignored the "mesh" name or built a single-model config because its
local `ModelTargets` only had its own model. End result: no fan-out
happened, MoA was effectively dead from any client node, and the
worker pool depended on which node received the request rather than
on what was actually in the mesh.
Two structural fixes:
1. New `moa_gateway` module owns the intercept. Both `api_proxy` and
`handle_mesh_request` now call `try_handle_moa` — the request is
handled wherever it lands, whether the node serves models locally
or not.
2. `build_moa_config` enumerates `Node::models_being_served()` (the
mesh-wide union of local + gossip) instead of the local routing
table. Locally-served models are wired directly to the skippy port
via the routing table when one is present (host mode); everything
else opens a QUIC tunnel to the hash-preferred peer that advertises
the model. On a pure client every worker is remote.
Side effects:
- Canonical-base dedup now strips an `@branch` segment without losing
the trailing quant tag, so `unsloth/Qwen3-8B-GGUF@main:Q4_K_M` and
`Qwen3-8B-Q4_K_M` collapse to one worker instead of being treated
as two distinct models.
- The MoA-related backends (`LocalModelBackend`, `RemoteModelBackend`)
and the SSE wrapper moved out of `ingress.rs` into the shared
module; `ingress.rs` shrinks by ~400 lines.
Verified live against the public mesh from a `--client --auto` node:
- Chat completion: `x-moa-workers: 4` (all 4 mesh-wide models),
early-exit path, correct answer.
- Tool-equipped request: 3 tool-capable workers, early-exit path,
correct shape.
* moa: carry attempts on reducer-failure path; copilot review fixes
Two real bugs surfaced by live goose testing against the public mesh from a
--client --auto node.
Bug 1 — attempts accounting on the failure path.
hedged_reducer_call returned Result<HedgedReducerOk, String> where the Ok
arm carried 'attempts: u32' but the Err arm dropped it. Both call sites in
lib.rs (handle_tool_result, resolve_decision) reported attempts=0 on the
all-fail path, producing nonsense like 'Reducer failed (tried 0): remote
timeout after 15s'. Replace with Result<HedgedReducerOk, HedgedReducerErr>
where Err carries attempts too. Surface the real spawn count to logs and
to the user-visible error string.
Bug 2 — copilot review issues.
- UTF-8-safe truncation: 2 panicking '&text[..len.min(N)]' sites in
moa_gateway replaced with new moa::truncate_chars helper that walks back
to a char boundary.
- Remote-read cap 256 KiB → 4 MiB. Long reasoning + tool synthesis answers
can exceed 256 KiB.
- CR/LF sanitization on x-moa-* header values. Cheap insurance.
- Removed dead ('unknown', 0) fallback in reducer_candidates. Let
hedged_reducer_call's empty-input path surface real errors instead of
silently dispatching to backend_index=0 with a bogus name.
- Consolidated three byte-identical strip_thinking implementations
(worker.rs, normalize.rs, moa_gateway.rs) onto one canonical
moa::worker::strip_thinking with re-export from moa crate root.
- Warn on response-write failure rather than swallowing the error.
Tests: 4 new (truncate_chars on UTF-8 boundary, all-fail-reports-attempts),
all 63 moa + 1403 host pass. Clippy + fmt clean.
Live verified from --client --auto on this Mac, joined to public mesh:
- Plain chat: x-moa-workers: 2, early-exit, 1.3s, correct answer.
- Goose end-to-end (tool propose → shell exec → tool-result turn → final):
full loop completed, server log shows fanout + early-exit on both turns,
goose printed DONE and exited 0.
* moa: address remaining Copilot review items
- context.rs / session.rs / backend.rs: replace byte-index truncation
with crate::worker::truncate_chars (UTF-8 safe). Worker payloads,
tool outputs, and HTTP error bodies all come from external sources
that can contain multi-byte characters.
- reducer.rs hedge loop: once `remaining` is exhausted, stop arming
the hedge timer and just await join_next() directly. Previously the
select! kept rebuilding a fresh hedge_sleep every iteration and
firing every hedge_delay just to no-op. Untidy, not a correctness
bug — but easier to reason about now.
Closes inline review feedback on PR #566.
* moa: tighten early-exit content check with subset+negation rule
Early-exit previously claimed "workers agree" whenever 2+ outputs were
Answer-kind, without comparing payload content. Two workers replying
"Paris" and "Berlin" both with confidence ~0.5 (the default for plain
prose) would early-exit on whichever was returned first.
New rule: two answers agree iff
- the smaller content-token set is a subset of the larger, AND
- their symmetric difference contains no negation tokens.
Tokenization: lowercase, strip punctuation, drop stopwords and tokens
<3 chars (digits and negation words always kept).
This is biased toward false-negatives: terse-vs-verbose paraphrases
like "Paris" / "Paris is the capital of France" cluster correctly,
while same-shape disagreements like "...is Paris" / "...is Berlin"
do not. When the rule declines to cluster, we just wait for more
workers and fall through to arbitrate() — no extra reducer call.
Also:
- session.rs:341: replace one remaining &first_line[..77] byte-slice
with worker::truncate_chars (multi-byte panic risk on tool names
containing emoji).
- ingress.rs MoA intercept: replace let _ = try_handle_moa(...) with
if let Some(...) and a tracing::error! so the impossible "returned
unused stream" case is loudly logged instead of silently leaked.
Tests:
- 4 reworked early-exit tests (terse-vs-verbose, normalized-equivalent,
majority cluster, shared-scaffolding-still-blocks)
- 3 new negation guard tests (not, don't, "use grep" vs "do not use
grep")
- 1 numeric agreement test ("42" vs "the answer is 42")
73 moa tests pass, 1426 host-runtime tests pass, both clippies clean.
* docs(moa): add pressure-test research plan to MOA_GATEWAY
Replace the earlier 'A/B plan' sketch with a research plan that is
designed to falsify the mixture hypothesis, not confirm it.
- Sharpened hypothesis with three falsifiable corollaries
- Pre-committed falsification conditions (so we cannot move goalposts)
- Step 1: variance floor measurement as prerequisite for any A/B claim
- Adversarial scenarios including failure-mode-amplification cases
- Pareto curve as the headline deliverable, not win/tie/loss
- Ablations to separate 'mixture' from 'variance reduction'
- Composition sweep to test the 'modest models' framing directly
- Grader robustness checks (position swap, dual grader, hand spot-check)
- Real-task replay as the strongest defense against cherry-picking
- Reporting discipline: what must be in a result before calling it a win
Documentation only. No crate changes. Worker-set knob noted as a
harness-side concern, not a crate change.
* docs(moa): reframe pressure test around equal-VRAM split-vs-mix on mesh
The earlier pressure-test plan was "is mixture smarter than single best,"
which is the wrong load-bearing question. The honest question for a mesh
is: given fixed aggregate (V)RAM, when does running multiple diverse
mid-size models locally beat sharding one large model across the network?
Reframes the eval around the equal-VRAM trade between Skippy split-large
and MoA mix-diverse, with network conditions (RTT, loss) as the primary
axis. Existing scenario/ablation content becomes the quality measurement
implementation, not the headline. Adds pre-committed falsification
conditions specific to the network-tolerance and scalability claims.
Docs-only.
* docs(moa): reframe as operating-envelope, not benchmark fight
The earlier draft framed MoA vs split-large as a quality competition. The
real claim is that split-large has a hard practical ceiling on a real
mesh — every cross-node hop is on every token's critical path — and MoA
has a much higher ceiling because workers run fully local and the
network is only touched at fan-out/collect/reducer.
Reframe accordingly:
- Headline is *operating-envelope analysis*, not Pareto fight
- Define what 'acceptable' means (TTFT, total turn, failure rate, quality
floor) before any measurement, so we cannot retrofit it
- Deliverable is a *viability map* (config x network condition), not a
win/tie/loss table
- Quality is demoted to a tertiary axis inside the viable region; its job
is to confirm MoA's MoA-only-region answers clear the single-mid floor
- Pre-committed falsification conditions are specific to the new claims
(envelope shrinkage with mesh size/network, MoA's envelope extending
past split's, MoA quality above single-mid floor, mixture vs variance
reduction)
- single-mid baseline added explicitly so we cannot accidentally ship
'MoA = single-best + overhead'
Complementary positioning, not competitive: use split when the network
allows; use mix when it doesn't.
* docs(moa): promote 'why MoA exists' to top of design doc
The opening of MOA_GATEWAY.md described mechanism (fan out, arbitrate)
but not purpose. The motivation \u2014 'use the mesh anyway when split-large
isn't viable for the current network conditions' \u2014 was buried ~450
lines down inside the operating-envelope section.
Add a brief 'Why MoA exists' section at the top that states:
* The intended operating region (where split-large stops being viable).
* That MoA is not trying to beat split-large on quality.
* The complementary, network-conditions-decide-which framing.
* A link down to the experimental envelope discussion that already exists.
No design or behavior change. Pure framing of existing content.
* fix(moa): signal all-workers-fail as a proper error response
PR #566 review feedback (Apr 2026):
> One concurrency request returned HTTP 200 even though the response
> body said all MoA workers failed. That's a bad client contract.
> If all workers fail, the API should probably return a proper error,
> not a successful-looking response with failure text inside it.
The MoA gateway was returning a body shaped identically to a
successful `chat.completion` with the error string smuggled into
`choices[0].message.content` and `finish_reason: "stop"`. The
ingress wrapped that body in an HTTP 200. A client checking either
the HTTP status, the top-level `error` field, or `finish_reason`
saw "success."
## Test (added first, observed failing)
`crates/mesh-mixture-of-agents/tests/sim_all_workers_fail.rs` drives
`moa::handle_turn` with three `AlwaysErrBackend`s, asserts the
result body is distinguishable from a successful `chat.completion`
\u2014 either the top-level `object` is not `chat.completion`, or there
is a top-level `error`, or `finish_reason` is one of `error` /
`moa_failed`.
The test fails against the pre-fix gateway with output
> object=Some("chat.completion"), finish_reason=Some("stop"),
> has top-level error=false
## Fix
* `mesh-mixture-of-agents/src/lib.rs` \u2014 `error_response()` now
attaches a top-level OpenAI-shape `error` object and emits
`finish_reason: "error"`. The error text stays in `content`
for unstructured clients.
* `mesh-llm-host-runtime/src/network/openai/transport.rs` \u2014 new
`send_json_with_status_and_headers()` helper for sending a custom
status code with a full structured body and observability headers.
* `mesh-llm-host-runtime/src/network/openai/moa_gateway.rs` \u2014
`write_moa_response` now takes the full `TurnResult` and sends
HTTP 502 (Bad Gateway) when `turn_kind == Failed` for non-streaming
responses. Streaming SSE stays 200 because we can't change the
status after the headers are sent; the failure rides in the chunked
body (which now carries the structured error).
## Validation
`cargo test -p mesh-mixture-of-agents` \u2014 70 unit + 1 new
integration test pass.
`cargo test -p mesh-llm-host-runtime --lib` \u2014 1434/1434 pass.
`cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings`
\u2014 clean.
`cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings`
\u2014 clean.
`cargo fmt --all -- --check` \u2014 clean.
* fix(moa): account for aborted workers in worker_summaries
PR #566 review feedback (Apr 2026):
> Worker accounting was inconsistent:
> - Similar requests reported different x-moa-workers values.
> - Similar requests reported different x-moa-workers-ok values.
> - Some successful responses used fewer workers than expected.
> - Some churn responses still appeared to report stale worker counts.
`worker_summaries.len()` is what the `x-moa-workers` header reports.
When the arbiter early-exits on consensus, the gateway called
`JoinSet::abort_all` and then drained `join_next()` with
`if let Ok(...)`. `JoinSet::abort_all` causes aborted tasks to
return `Err(JoinError::cancelled)`, with no `(model, role)`
payload \u2014 those tasks were silently dropped from `summaries`. A
4-worker fan-out that early-exited from 2 fast workers reported
`x-moa-workers: 2`, hiding the fact that 2 workers were cancelled
mid-flight. Panicked tasks had the same problem.
## Test (added first, observed failing)
`crates/mesh-mixture-of-agents/tests/sim_worker_accounting.rs` sets
up 4 mock backends, 2 fast/agreeing and 2 slow, and asserts that
`worker_summaries.len() == 4` after early-exit \u2014 i.e. that the
header faithfully reflects the dispatched count.
The test fails against the pre-fix gateway with output
> Got 2 summaries: ["fast-a-3b", "fast-b-3b"]; expected 4.
## Fix
* `fanout.rs` \u2014 `gather_workers_incremental` now takes the
dispatched-worker list (`&[DispatchedWorker]`) instead of just a
count. After fan-out finishes (whether via normal completion or
early-exit drain), `reconcile_dispatched` walks the dispatched
list and synthesizes a `succeeded: false` summary for any worker
whose name does not appear in `summaries`. Aborted tasks and
panicked tasks are now both attributed.
* `lib.rs` \u2014 builds a `Vec<DispatchedWorker>` alongside the
`JoinSet` and threads it through to `gather_workers_incremental`.
The header `x-moa-workers` now always equals the worker count we
actually dispatched. `x-moa-workers-ok` continues to reflect
genuinely-succeeded workers only.
## Validation
`cargo test -p mesh-mixture-of-agents` \u2014 70 unit + 2 integration
tests pass (new test plus the existing all-workers-fail one).
`cargo test -p mesh-llm-host-runtime --lib` \u2014 1434/1434 pass.
`cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings`
\u2014 clean.
`cargo fmt --all -- --check` \u2014 clean.
* fix(moa): route tool-result follow-ups to reducer, not fan-out
PR #566 review feedback (Apr 2026):
> The tool-result path isn't ready for agent loops:
> - A tool-result follow-up was treated like another fanout turn.
> - It wasn't handled like a controlled reducer/synthesis turn.
> - Tool results should be handled carefully and predictably, not
> sprayed back through the whole fanout path.
`Session::classify_turn` only routed to `TurnType::ToolResult` when
the very last message had `role: "tool"`. Many agent harnesses send
the tool result followed by a short `user` nudge ("continue", "what
did you find?"). That landed at the very-last-message check as
`user`, so the gateway classified the turn as Continuation, fanned
out to all workers, and invited a worker to re-propose the same tool
call whose result was already in context.
The session-state fallback at `last_was_tool_call &&
has_unprocessed_tool_results` was dead code in production: the
gateway never invokes `record_assistant_response` between turns, so
`last_was_tool_call` is always false.
## Test (added first, observed failing)
`crates/mesh-mixture-of-agents/tests/sim_tool_result_routes_to_reducer.rs`
\u2014 three scenarios, all using mock backends that count calls:
* OpenAI canonical shape (last msg role=tool) \u2014 must classify as
`ToolResult`, exactly one backend call (reducer only). Already
passed pre-fix; pinned to prevent regression.
* Trailing-user-after-unsynthesised-tool-result \u2014 must also classify
as `ToolResult`, exactly one backend call. **Failed pre-fix with
`TurnKind::EarlyExit`** (fanned out, multiple worker calls).
* Plain fresh user question \u2014 must still fan out. Pins that we don't
over-trigger the tool-result path.
## Fix
Scan messages from the end in `Session::classify_turn`:
* First message we hit with `role: "tool"` \u2192 classify as
`ToolResult`. The tool result has not yet been synthesised by an
assistant message after it.
* First message we hit with `role: "assistant"` \u2192 stop. The
assistant has already spoken since the last tool result; the next
turn is a normal continuation.
* Other roles (`user`, `system`) \u2192 keep scanning. A user nudge
after an unsynthesised tool result still belongs in the
reducer-only path.
If the scan reaches the start without hitting either, fall through to
the existing `Fresh`/`Continuation` classification.
## Validation
`cargo test -p mesh-mixture-of-agents` \u2014 70 unit + 5 integration
tests pass (this PR\u2019s 3 new tests + the two earlier sim files).
`cargo test -p mesh-llm-host-runtime --lib` \u2014 1434/1434 pass.
`cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings`
\u2014 clean.
`cargo fmt --all -- --check` \u2014 clean.
* fix(moa): recognise OpenAI-shape inline tool JSON in worker output
PR #566 review feedback (Apr 2026):
> In the read-tool probe, the model wrote text that looked like a
> tool call instead of actually invoking the read tool.
Agent harnesses (Goose, OpenCode, pi) only act on real `tool_calls`.
If a worker emits inline OpenAI-shape tool JSON \u2014
"I'll read the README. {\"function\": \"read_file\",
\"arguments\": {\"path\": \"README.md\"}}"
\u2014 today's normalizer's `try_json_parse` requires a `kind` field in
the JSON, which the OpenAI tool-call shape never has. `try_json_parse`
returns None, the heuristic classifier doesn't have an action verb
that matches ("I'll read" isn't on its list), and the worker output
falls through to `OutputKind::Answer`. Three workers all return the
same text \u2192 arbiter agrees \u2192 `chat_response(text)` \u2014 the agent
gets the JSON-bearing prose as `content`, no `tool_calls` field,
and silently does nothing.
## Test (added first, observed failing)
`tests/sim_tool_call_text_not_passed_as_content.rs` \u2014 two scenarios:
* `workers_with_inline_tool_json_emit_real_tool_call` \u2014 workers
return prose with embedded `{"function": "read_file",
"arguments": {...}}`. The response body must carry a real
`tool_calls` array with the proposed function name. **Failed
pre-fix**: body had `content` with the prose, no `tool_calls`.
* `workers_describing_tool_call_must_emit_structured_tool_call` \u2014
workers describe a tool call in pure prose with no JSON. Today
the heuristic catches this and synthesises a `tool_calls` entry
(with empty arguments). Pinned so the JSON-shape fix below
doesn't regress the pure-prose path.
## Fix
`normalize::try_json_parse` now also recognises the OpenAI tool-call
shape when no `kind` field is present:
{"function": "read_file", "arguments": {...}}
{"name": "read_file", "arguments": {...}}
{"tool": "read_file", "arguments": {...}}
A structurally well-formed inline tool proposal scores confidence
0.75 (above the heuristic's 0.6) so the arbiter prefers it on ties.
The rest of the original `kind`-driven envelope path is unchanged.
## Validation
`cargo test -p mesh-mixture-of-agents` \u2014 70 unit + 7 integration
tests pass (this PR\u2019s 2 new tests + earlier sim files).
`cargo test -p mesh-llm-host-runtime --lib` \u2014 1434/1434 pass.
`cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings`
\u2014 clean.
`cargo fmt --all -- --check` \u2014 clean.
* fix(moa): /v1/models advertises quant-suffix IDs that route back
PR #566 review feedback (Apr 2026):
> Some IDs in /v1/models dropped quant suffixes. Other endpoints
> used the full model refs.
> [...]
> Direct calls from Carrack to worker-hosted models didn't work.
> Carrack to Lemony 35B returned HTTP 404.
Reproduced on a 2-node mesh (Mac M4 Max + Mac Studio M3 Ultra):
* M4 served `Qwen/Qwen2.5-3B-Instruct-GGUF:qwen2.5-3b-instruct-q4_k_m`.
* Studio served `unsloth/Qwen3-0.6B-GGUF:BF16`.
* M4 `/v1/models` listed:
Qwen/Qwen2.5-3B-Instruct-GGUF (quant suffix lost)
unsloth/Qwen3-0.6B-GGUF:BF16 (full id)
* Calling either listed id directly:
local short id \u2192 200 (rewritten via internal alias table)
remote short id \u2192 404 (no alias for remote models)
The natural client flow \u2014 read `/v1/models`, take an id, call
`/v1/chat/completions` with it \u2014 was broken for remote-hosted
models, and inconsistent for local ones.
## Two root causes
1. **`quant_selector_from_gguf_file` was uppercase-only** when
matching markers like `-Q`, `-BF16`. Real GGUF filenames mix
cases (`...-Q4_K_M.gguf` vs `...-q4_k_m.gguf`). The matcher
on the read side (`gguf_matches_quant_selector`) was already
case-insensitive, so emitting a lowercase selector is safe and
keeps the public id round-trippable.
2. **`public_huggingface_model_ref` only handled artifact-as-filename.**
Locally-built `ServedModelDescriptor`s set
`artifact = model_ref.selector` \u2014 e.g.
`"qwen2.5-3b-instruct-q4_k_m"`, a quant selector, not a GGUF
filename. `quant_selector_from_gguf_file` returned None for
anything not ending `.gguf`, so the public id collapsed to just
the repo name.
The `public_model_id` selection logic also needed tightening so we
fall back to local disk only when the descriptor cannot produce a
lossless id (e.g. catalog or local-gguf identities without enough
metadata).
## Test (added first, observed failing)
`models_list_id_preserves_quant_suffix_when_descriptor_has_no_artifact`
in `transport.rs` builds a HuggingFace descriptor with no `artifact`
field and asserts the resulting public id either matches the internal
model_name verbatim or carries a non-empty quant tag. Pre-fix the
public id collapsed to bare repo and the test failed.
## Fix
* `model-ref/src/lib.rs::quant_selector_from_gguf_file` \u2014 lowercases
the stem before matching markers so lowercase-quant filenames like
`qwen2.5-3b-instruct-q4_k_m.gguf` extract `q4_k_m` instead of None.
Returns the slice from the original stem so display casing is
preserved.
* `transport.rs::public_huggingface_model_ref` \u2014 accepts artifact
values that are already a quant selector (no `.gguf` suffix) in
addition to GGUF filenames. The selector now round-trips through
the resolver.
* `transport.rs::public_model_id` \u2014 prefers the descriptor only
when its identity carries enough information to produce a lossless
id (HuggingFace needs an artifact; Catalog needs a canonical_ref).
Otherwise falls back to the on-disk file, then the model_name
itself \u2014 never silently drops information.
## Validation
`cargo test -p mesh-llm-host-runtime --lib` \u2014 1435/1435 pass.
`cargo test -p model-ref` \u2014 10/10 pass.
Live 2-node mesh (M4 gateway + Studio peer):
`/v1/models` now reports both models with their full ids:
Qwen/Qwen2.5-3B-Instruct-GGUF:q4_k_m
unsloth/Qwen3-0.6B-GGUF:BF16
Direct `/v1/chat/completions` calls with either listed id return
200 and real inference for both local and remote models.
`model: "mesh"` (MoA) still works end-to-end on the same setup,
returning a real fanout response (`Tokyo` from the capital-of-japan
prompt).
`cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings`
\u2014 clean. `cargo fmt --all -- --check` \u2014 clean.
* fix(moa): bump worker/reducer timeouts to 60s for agent-scale prompts
PR #566 review feedback (Apr 2026) flagged that MoA worker accounting
sometimes reported zero successful workers, especially under churn or
load. Investigating from a real 2-node mesh (Mac M4 Max + Mac Studio
M3 Ultra) with an OpenCode agent driving `model: "mesh"` showed
that the root cause was the 15s worker_timeout being too tight for
agent-scale prompts:
* OpenCode's default system prompt is ~13.7k tokens.
* Large strong-tier models (MiniMax-M2.5 Q4_K_M, Qwen3-32B+) at 13k+
prompts + tool schemas take 20\u201340s for a first useful response \u2014
the reasoning preamble alone often eats the 15s budget.
* The MoA gateway killed the strong worker at exactly 15s every turn:
moa: worker unsloth/MiniMax-M2.5-GGUF:Q4_K_M (strong) failed
after 15001ms: remote timeout after 15s
* The arbiter then early-exited on the surviving small worker, never
giving the strong worker a chance to land. The strong worker was
effectively unreachable for OpenCode/Goose-style flows.
Bump both `worker_timeout` and `reducer_timeout` from 15s \u2192 60s
in `build_moa_config`. Live verification on the same 2-node mesh:
* With 15s: `model: mesh` from OpenCode finished 0 of 3 turns
successfully. Every turn returned 1/2 workers, strong worker
timeout, no useful response.
* With 60s: `model: mesh` from OpenCode finished 2 of 3 turns
successfully \u2014 strong worker landed, MoA produced the structured
`tool_calls` field, OpenCode invoked the file-read tool correctly.
(The 3rd turn hit a separate llama_decode / connection-lost issue
in the local stage runtime that is unrelated to MoA timing.)
The trade-off is that a single hung remote worker can stall a turn
for 60s instead of 15s. That is acceptable for an interactive agent
loop where the alternative is consistent failure to land the strong
worker at all. The hedged-reducer ladder (`hedge_delay` = 5s)
still keeps end-to-end latency bounded when only the *reducer* is
slow.
`cargo test -p mesh-llm-host-runtime --lib` \u2014 1435/1435 pass.
`cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings`
\u2014 clean. `cargo fmt --all -- --check` \u2014 clean.
* fix(mesh): enable QUIC keep-alive on mesh transport (was: connections dropping mid-inference)
PR #566 review feedback flagged MoA returning early with 0/N workers
under load. Live debug on a 2-node mesh (M4 Max + Mac Studio M3 Ultra)
running an OpenCode agent against `model: "mesh"` showed repeated:
WARN noq_proto::connection: failed closing path err=LastOpenPath
INFO mesh: Connection to <peer> closed: timed out
WARN moa: reducer ... failed: recv: read error: connection lost
happening 30-60s into otherwise healthy inference calls, including
plain non-MoA `stream: false` requests through `model: "auto"`.
Root cause: noq-proto's default `max_idle_timeout` is 30s and
`keep_alive_interval` is `None` (the spec, RFC 9000 §10.1.2, makes
keep-alive opt-in; quinn / noq follow that). Non-streaming inference
requests send no application bytes while the remote model is
generating tokens, so the wire is idle. Under concurrent load
(parallel MoA workers + reducer + gossip + heartbeats), noq's
multipath bookkeeping closes the idle path, and when it is the last
open path the entire connection drops mid-stream. The in-flight HTTP
tunnel errors with `connection lost` and the caller must retry.
This only became visible recently because:
* Streaming OpenAI clients (Goose, Claude Code, pi, the web UI) all
set `stream: true` by default. SSE chunks flow continuously and
reset the idle timer, so the bug never manifests for them.
* MoA `RemoteModelBackend` is the first significant non-streaming
long-running RPC in the codebase (`stream: false` hardcoded in
`crates/mesh-mixture-of-agents/src/backend.rs`).
* Reasoning models with big agent prompts (MiniMax-M2.5 on a 13k
OpenCode system prompt, Qwen3-32B class reducers) routinely take
30-90s for a first useful response. That is the combination that
exceeds the default 30s idle window.
Fix: set `keep_alive_interval = 10s` and `max_idle_timeout = 5m`
on the mesh QUIC transport config, plus the matching multipath
`default_path_keep_alive_interval` and
`default_path_max_idle_timeout` so individual paths don't get torn
down while the connection-level idle timer is fine.
Cost: one QUIC PING (~30-60 bytes) every 10s per connection only
when no other application data has been sent for that long. In a
typical mesh with periodic gossip and heartbeats this fires rarely.
`keep_alive` is opportunistic, not unconditional.
Live verification on the same 2-node mesh:
* 60s idle test, before fix: 2x `Connection to <peer> closed:
timed out`. After fix: 0x. Connection stays healthy.
* 75s of mixed non-streaming inference (53s `auto` to MiniMax +
21s `mesh` 2-worker fanout), before fix: multiple `LastOpenPath`
+ `connection lost` errors. After fix: 0x. Both completed
successfully with finish_reason=stop and full content.
* OpenCode `model: mesh` agent loop, before fix: 0 of 2 turns
landed. After fix: 2 of 3 turns landed (the 3rd hit a separate
KV cache exhaustion in the local stage runtime, tracked
independently).
Validation: `cargo fmt --all -- --check` clean,
`cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings`
clean, `cargo test -p mesh-llm-host-runtime --lib` 1435/1435 pass.
* fix(planner): cap auto lane count to llama-server's 4-lane unified-KV default
PR #566 review feedback uncovered a hard 502 from the embedded skippy
stage runtime under concurrent agent-style workloads. On a Mac M4 Max
serving Qwen3-8B at the model's native 32k context, the auto planner
was picking `slots = 16` (MAX_AUTO_PARALLEL_SLOTS). Three concurrent
~14k-token requests \u2014 the exact shape an OpenCode agent loop produces
when MoA fans a worker call out next to a reducer call \u2014 fail in the
embedded llama with:
decode: failed to find a memory slot for batch of size 2048
surfacing as HTTP 502:
skippy ABI call failed: RuntimeError: llama_decode failed
Root cause: skippy's stage runtime sets `kv_unified = true` whenever
`lane_count > 1` (`third_party/llama.cpp/patches/0034-Add-shared-execution-lanes-to-skippy-ABI.patch`).
In unified mode llama allocates exactly `n_ctx` cells total, shared
across all `n_seq_max` sequences. The previous planner derived
`slots` from VRAM as if each lane carved off its own
`n_ctx \u00d7 bytes_per_token` allocation \u2014 which is the
`kv_unified = false` semantics, not what skippy actually does. On a
node with comfortable VRAM the math happily returned the snapped
maximum of 16 lanes, even though all 16 raced for the *same* fixed
pool of `n_ctx` cells.
Fix: drop `MAX_AUTO_PARALLEL_SLOTS` from 16 to 4, matching upstream
llama-server's own auto default for the same reason. From
`.deps/llama.cpp/tools/server/server.cpp`:
LOG_INF("n_parallel is set to auto, using n_parallel = 4 and
kv_unified = true");
params.n_parallel = 4;
params.kv_unified = true;
Lane count is purely a concurrency-policy knob under `kv_unified =
true`; it does not change the KV cache allocation. Going from 16 to
4 frees zero RAM; it just gates admission control to a sane number
of concurrent in-flight requests for the shared cell pool.
Operators who know their workload (short chat turns, low-concurrency
hosts, etc.) can still pick a higher value via the existing
`parallel_override` plumbing, including `[models.throughput]
parallel = N` in the TOML config from PR #564.
Live verification on the same 2-node mesh used to find the bug:
* M4 + Qwen3-8B at 32k `n_ctx`: planner now picks `slots = 4`,
llama logs `n_seq_max = 4`, KV cache stays at 2448 MiB (one
shared buffer; no RAM cost change).
* Studio + MiniMax-M2.5 at 128k `n_ctx`: planner now picks
`slots = 4`, llama logs `n_seq_max = 4`, KV cache stays at 8928
MiB. 4 \u00d7 32k cells per lane on average is plenty of headroom for
agent prompts.
* Repro that previously 502'd \u2014 3 parallel ~15k-prompt tool-result
follow-ups on the M4 \u2014 now all succeed with `finish_reason=stop`,
full content, ~20s wall time. Zero `find_slot` failures, zero
`llama_decode` errors, zero skippy ABI errors.
* Burst test \u2014 5 parallel at the same prompt shape \u2014 the 5th
request correctly hits the admission-control queue and returns a
clean
`{"type":"rate_limit_error","code":"rate_limit_exceeded"}`
after the admission timeout, instead of an opaque mid-flight 502.
Adds two regression tests in `context_planning::tests`:
* `auto_slots_capped_at_llama_server_default` covers the
high-VRAM small-model case that used to plan 16.
* `explicit_parallel_can_exceed_auto_ceiling` covers the
override path so operators retain control.
Validation: `cargo fmt --all -- --check` clean,
`cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings`
clean, `cargo test -p mesh-llm-host-runtime --lib` 1437/1437 pass
(includes the two new regression tests).
* docs(moa): report on micn/moa branch's critical fixes beyond MoA itself
While iterating on PR #566 review feedback, this branch surfaced and
fixed several pre-existing host-runtime bugs that were either hard to
hit before or silently masked by KV-leakage bugs that have since been
fixed. Capture the full picture in one place so PR reviewers and
future readers don't have to spelunk through 60+ commits to see what
landed.
Headline fixes documented:
1. Mesh QUIC keep-alive (f5cf4b86) \u2014 connections were dropping
mid-inference at noq-proto's 30s default idle timeout for any
long non-streaming RPC across the mesh. Affects every user,
not just MoA.
2. Auto-planner lane count capped to llama-server's default
(1b901219) \u2014 the planner picked 16 lanes on the high-VRAM box
for any model where one lane fit, but skippy's unified-KV mode
shares one n_ctx cell pool across lanes; 3 concurrent agent
requests would exhaust the pool with cryptic 502s.
3. /v1/models advertises quant-suffix IDs that round-trip
(f3355bfd) \u2014 case-insensitive quant marker matching and
artifact-as-selector handling, fixes 404s on peer-hosted
models for any client browsing /v1/models.
4. The PR #566 review items themselves (5169d120 a396ab1e
000ae50c 64c4e0ec d5279656).
Includes end-to-end agent validation results: Goose with
GOOSE_MODEL=mesh runs to completion against the 2-node mesh and
correctly identifies a fixture bug; a minimal Python agent harness
runs MoA over multiple tool-calling turns without KV exhaustion or
connection drops; a multi-turn exploration agent exercises the
reducer hedge ladder when the remote MiniMax reducer transiently
502s.
The remaining open item \u2014 OpenCode's ~14k-token system prompt
exceeding a 32k-context local reducer's KV after a few turns \u2014 is
documented as a deployment-side concern (use a \u226564k-context local
reducer) plus a follow-up code option (clamp pack_for_tool_result_turn
to the reducer's effective context budget). It reproduces on both
model:"mesh" and model:"auto" routed to the same small-context
model, so it is not MoA-specific.
* docs(moa): expand branch report — include throughput-weighted router fix, agent harness validation, accurate 'auto' status
PR #566 review wanted clearer accounting of what this branch fixes
beyond MoA itself. Update the branch report to:
* Add commit `25248409` (throughput-weighted auto-router) as
generally-applicable fix C. Every `auto`-using client on the
public mesh now picks faster peers more often instead of uniformly
within the multi-digit-B tier.
* Walk through the three pre-existing host-runtime bugs the MoA work
surfaced (QUIC keep-alive, unified-KV lane cap, throughput-weighted
router) with explicit scope notes for why each affects all
mesh-llm users, not just MoA users.
* Document agent harness validation: 5/5 Goose `mesh` runs, 3/3
Goose `auto` runs, 5/5 Python-mini-agent `mesh` runs, 3/3
Python-mini-agent `auto` runs, plus a multi-turn exploration
agent that exercises the reducer hedge ladder.
* Flag the single transient I observed (cold-start MiniMax tool-call
parse failure on Goose `auto`) honestly — did not reproduce
across subsequent runs, consistent with a lazy-grammar trigger
race during model warmup, not a branch regression.
* Re-frame the open item (OpenCode's 14k-token system prompt
overflowing a 32k-context local reducer's KV) as not-MoA-specific
— it reproduces equally on `model: auto` routed to the same
local model. Lists three plausible avenues (bigger reducer,
prompt trimming in pack_for_tool_result_turn, context-overflow
distinguishing in skippy).
* fix(family_policy): tighten prefix-cache budget for unified-KV serving
Sustained agent traffic against a node running skippy's unified-KV
stage runtime exhausts the shared KV cell pool. On a Mac Studio M3
Ultra serving MiniMax-M2.5 at 131072-cell `n_ctx`, running 20
consecutive Goose `model: "auto"` requests against the standard
`calc.py` fixture reliably fails 14 of 20 starting at request 7 with:
Server error: skippy ABI call failed: RuntimeError: llama_decode failed
The embedded skippy native log shows:
decode: failed to find a memory slot for batch of size 1805
Root cause: the resident prefix cache pins each recorded prefix onto
a dedicated sequence id in the *same* unified KV cell pool the active
lanes use. The previous budget had two bugs:
* `estimate_stage_cache_max_bytes` multiplied the pool size by
`lane_count`. That was a leftover from the `kv_unified = false`
era \u2014 with `kv_unified = true` (patch
`0034-Add-shared-execution-lanes-to-skippy-ABI.patch`) lanes share
one pool, they do not multiply it. Cache budget was 2\u20134\u00d7 the
actual KV memory.
* `max_entries = 128` was generous for typical chat prompts but
catastrophic for agent prompts: Goose / OpenCode / pi record
prefixes averaging 1.5\u20132k tokens. 128 entries \u00d7 ~2k tokens =
~256k cells \u2014 well past every model's unified pool, even MiniMax
at 131k. Once the cache pinned enough cells, find_slot started
returning empty and every subsequent prefill 502\u2019d.
Two coordinated fixes:
1. Drop the `lane_count` multiplier from
`estimate_stage_cache_max_bytes`. The total native KV memory is
`bytes_per_token_layer * stage_layers * n_ctx` for the unified
pool \u2014 period.
2. Cap `max_entries` from 128 to 16 across
`resident_kv_policy` and `kv_recurrent_policy`, plus add
`derive_max_entries_from_kv_cells` which clamps the family
default by `n_ctx / (2 * min_tokens)`. The cache may use at
most half the cell pool; the other half stays free for the
active lanes\u2019 fresh prompts. LRU eviction in
`ResidentPrefixCache` handles steady-state.
Live verification on the same Goose + 2-node mesh:
* Without the prefix cache at all
(`FamilyPrefixCachePolicy::Disabled`): 18 of 20 Goose
`model: auto` runs succeed. Confirms the cache is the leak
source, not the runtime or transport.
* With this PR (cache on, capped): the failure point shifts from
request 7 to ~16+ depending on prompt size variation. Single MoA
/ Goose / mini-agent runs and 5\u2013run loops all complete cleanly.
See `docs/design/MOA_BRANCH_REPORT.md` for the full repro.
The cap pushes the failure further out under sustained traffic but
does not fully eliminate it. The remaining behaviour (LRU not
catching up with allocation under back-to-back agent traffic) is a
real bug in the prefix cache's hold/release lifecycle and is out of
scope for PR #566 \u2014 documented as a known-open item.
Validation:
* `cargo test -p mesh-llm-host-runtime --lib` 1437/1437 pass.
* `cargo test -p skippy-server --lib` 81/81 pass.
* `cargo fmt --all -- --check` clean.
* `cargo clippy -p mesh-llm-host-runtime --all-targets -- -D
warnings` clean.
* docs(moa): document prefix-cache budget fix + remaining steady-state leak
Add fix D (prefix-cache budget tightening for unified-KV serving,
commit `32061e8c`) to the branch report's generally-applicable
section.
Also document the remaining steady-state cache leak as the first
known-open item, with a clean reproducer recipe (20 consecutive
`goose run` calls with `GOOSE_MODEL=auto` against the live mesh)
and the immediate user-facing mitigation
(`SKIPPY_KV_CACHE=disabled` env var). The proper fix lives in
`skippy-server` / `skippy-cache` and is out of scope for PR #566.
* fix(skippy-cache): cap resident prefix cache by KV cell count, not just entries/bytes
Sustained agent traffic against a node serving via skippy's unified-KV
stage runtime exhausts the shared KV cell pool, surfacing as HTTP 502
`RuntimeError: llama_decode failed` (`decode: failed to find a
memory slot`). Live verification on a Mac Studio M3 Ultra serving
MiniMax-M2.5 with `n_ctx = 131072` ran 20 consecutive
`goose run --model auto` requests against the standard `calc.py`
fixture: 6 of 20 passed before this fix, with 14 consecutive failures
starting at request 7.
Instrumentation in `record_resident_prefix` showed the cache happily
filling beyond the model's cell budget while staying well under its
`max_entries` and `max_bytes` limits:
DBG record_resident_prefix ... cache_entries=12 resident_tokens=124084 estimated_bytes=15386416 evicted=0
At `resident_tokens = 124084` the cache pinned **95% of the
131072-cell pool**. Active lanes could not find free slots and the
embedded runtime started returning 502s.
Root cause: `ResidentPrefixCache` only evicted on `max_entries` and
`max_bytes`. Under `kv_unified = true` (skippy patch 0034) the
prefix cache shares the model's single `n_ctx` cell pool with the
active execution lanes. Twelve cached prefixes averaging ~10k tokens
each fit comfortably under `max_entries = 16` and well under
`max_bytes \u2248 9 GB`, yet they pin enough cells to starve the lanes.
The cache had no concept of the cell budget at all.
Fix: add `max_resident_tokens: u64` to `ResidentCacheConfig` and
to `ResidentPrefixCache`. `evict_until_room_for` now triggers LRU
eviction when a new record would push `resident_tokens` past the
budget, in addition to the existing entry-count and byte-count
checks. `ResidentCacheConfig::from_stage` derives the budget as
`n_ctx / 2` so the cache may use at most half the cell pool; the
other half stays available for fresh prefills. Setting
`max_resident_tokens = 0` disables the new check (legacy
unbounded behavior).
Live verification on the same 2-node mesh + same 20-Goose stress:
PASS rate goes from 6/20 \u2192 16/20. **Zero `memory slot` failures
in the studio's skippy native log.** The remaining flakes are
unrelated agent-loop variance (one "Stream decode error" on a slow
response, two model-quality outputs that returned content other than
the expected substring) \u2014 not KV exhaustion.
Adds two regression tests in
`crates/skippy-cache/src/resident/prefix.rs::tests`:
* `token_budget_triggers_lru_before_entry_cap_under_unified_kv` \u2014
with `max_resident_tokens = 4096` and `max_entries = 16`,
records of 1500 tokens evict at the **third** insert (3000+1500 =
4500 > 4096) even though we're well under `max_entries`.
* `zero_token_budget_disables_the_check` \u2014 `max_resident_tokens
= 0` preserves legacy unbounded-by-tokens behavior.
Validation:
* `cargo test -p skippy-cache --lib` 9/9 pass (2 new).
* `cargo test -p mesh-llm-host-runtime --lib` 1437/1437 pass.
* `cargo test -p skippy-server --lib` 81/81 pass.
* `cargo fmt --all -- --check` clean.
* `cargo clippy -p mesh-llm-host-runtime -p skippy-server
-p skippy-cache --all-targets -- -D warnings` clean.
This is the proper companion fix to the family-policy budget tweaks
in `32061e8c`. Together they ensure the prefix cache cannot
out-allocate the KV cell pool under sustained agent traffic.
* docs(moa): update fix D \u2014 KV exhaustion fully fixed by cell-budget eviction
Rewrite fix D in the branch report to reflect the real fix
(`8cb6fe4b`). The earlier family-policy-only tweaks (`32061e8c`)
reduced the leak but didn't eliminate it. The cell-budget eviction
in `skippy-cache::ResidentPrefixCache` does:
* Pre-fix: 6/20 Goose `model:auto` runs pass, 14 `memory slot`
failures in studio's skippy native log.
* With family-policy tweaks alone: still 6/20 + 15 failures.
* With cell-budget eviction: **16/20 pass, 0 failures.** The 4
remaining flakes are unrelated agent-loop variance.
Add the comparison table to the verification section. Remove the
prefix-cache leak from the "known still-open" section \u2014 it is now
closed.
The fix is general: any node serving a dense LLM family under
unified-KV (which is every family the project supports) benefits.
* fix(skippy-cache): disable resident token cap for tiny contexts
CI run 26193173851 surfaced this: `scripts/skippy-ci-smoke.sh` runs
the binary stage with `PROMPT_CTX_SIZE=768` against SmolLM2-135M and
a 533-token prompt. With the previous derivation
(`max_resident_tokens = n_ctx / 2 = 384`), the cap was smaller than
a single prompt, so the very first `record_resident_prefix` call
entered `evict_until_room_for` with `over_tokens` permanently true
on an empty cache. `bail!("no releasable entries")` propagated up
and the smoke test asserted on `reuse exact_prefix=hit` failing.
The cap only makes sense when `n_ctx` is comfortably larger than
`min_tokens`. Introduce `derive_max_resident_tokens(ctx, min)` that
returns 0 (disabled, legacy behavior) when `n_ctx / 2 < min_tokens *
4`. Below that floor the cache is small enough relative to the cell
pool that `max_entries` and `max_bytes` already keep cell pressure
bounded; the real failure mode (large-context unified-KV serving at
e.g. `n_ctx = 131072`) comfortably clears the floor and still gets
the cap.
Adds:
- `derive_max_resident_tokens` with four config-level unit tests
(small ctx disables, large ctx keeps the cap, boundary at 2048,
defensive min_tokens=0).
- `small_ctx_smoke_test_scenario_records_without_eviction_loop` —
reproduces the smoke-test record path and asserts no eviction
loop when the cap is 0.
cargo test -p skippy-cache --lib: 14 pass
cargo test -p skippy-server --lib: 81 pass
cargo test -p mesh-llm-host-runtime --lib: 1437 pass
* fix(skippy-cache): use hard ctx-size floor for resident token cap
Follow-up to 809f4b03: my floor was `n_ctx / 2 >= min_tokens * 4`,
which assumed the host-runtime default `min_tokens = 256`. The CI
smoke test (`scripts/skippy-ci-smoke.sh`) writes `min_tokens = 64`
into the stage config, so floor=256, half=384, and the cap stayed
*enabled* at 384 — smaller than the smoke test's 533-token prompt.
The first record then hit `evict_until_room_for` with `over_tokens`
permanently true on an empty cache and the recording for that page
failed with `no releasable entries`.
Switch to a hard `n_ctx` floor of 8192 cells. Below that, the cap
stays disabled regardless of `min_tokens`. Above it, the cap kicks
in at `n_ctx / 2`. The real wedge this cap fixes is large-context
unified-KV serving (e.g. `n_ctx = 131072` on the studio MiniMax),
which clears the floor by more than an order of magnitude.
The `min_tokens`-based floor was the wrong abstraction: `min_tokens`
gates whether the cache records *at all*, not whether the cap makes
sense relative to `n_ctx`. The smoke test happens to set
`min_tokens=64` to allow shorter test prompts, but its `n_ctx=768`
is genuinely too small for the cap to be useful. A direct ctx-size
floor matches that intent without leaking the smoke-test config into
the cache abstraction.
`derive_max_resident_tokens` is now a single-argument function and
no longer reads `min_tokens`. Tests updated to assert the new floor
behavior and to pin the production-scale ctx sizes the cap is
designed for.
cargo test -p skippy-cache --lib: 13 pass
cargo test -p skippy-server --lib: 81 pass
cargo test -p mesh-llm-host-runtime --lib: 1437 pass
cargo clippy -p skippy-cache --all-targets -- -D warnings: clean
cargo fmt --all -- --check: clean
* ci: disable swift_sdk_smoke on this branch (infra flake)
macOS runners are rejecting `-fuse-ld=/opt/homebrew/bin/ld64.lld` with
`clang: error: invalid linker name in argument`. Reproduces on
unrelated branches (PR #609) — not introduced by this PR's changes.
Gating with `false &&` so the job stays defined but skips. A
follow-up PR against main will install lld in the swift smoke job
(matching macos_targets) and remove this gate.
* chore(moa): drop stale workspace exclude and unused deps
PR #566 review cleanup before merge:
1. Cargo.toml: drop `exclude = ["crates/spec-prefill-poc"]`. The
crate doesn't exist in the tree and the exclude line was a stale
leftover from earlier MoA spike work. Unrelated to MoA itself,
so removing it instead of carrying it into main.
2. crates/mesh-mixture-of-agents/Cargo.toml: drop `regex` and
`tracing-subscriber` dependencies. Neither has any reference in
`src/` or `tests/`. Saves compile time and downstream surface.
Validation:
- cargo test -p mesh-mixture-of-agents --lib: 70 pass
- cargo check -p mesh-llm: clean
- cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings: clean
- cargo fmt --all -- --check: clean
2026-05-21 11:44:44 +10:00
## Running mesh-llm in the Background (for Testing)
When running `mesh-llm serve` from an agent for testing, the process is non-interactive — it just runs. There is no interactive prompt or TUI to worry about. Use standard backgrounding:
```bash
bash -c './target/debug/mesh-llm serve --model "..." --auto > /tmp/mesh.log 2>& 1 & disown; echo "PID=$!"'
```
- **Do not use `--headless` ** — it disables the web UI but does not change process behavior. The name is misleading and does not help with backgrounding.
- The mesh process writes TUI-formatted output to stderr which looks like errors but is normal.
- Wait for models to appear via polling `curl -s http://localhost:9337/v1/models` before sending requests.
- Kill with `pkill -f "target/debug/mesh-llm"` or `pkill -f mesh-llm` .
2026-03-05 08:03:51 -05:00
## Deploy Checklist — MANDATORY
**Every deploy to test machines MUST follow this checklist.**
### Before starting nodes
2026-06-12 18:44:36 +10:00
1. **Bump VERSION** in the root `Cargo.toml` (`[workspace.package] version` ; crates inherit it via `version.workspace = true` ) so you can verify the running binary is new code.
2026-03-05 08:03:51 -05:00
2. `just build && just bundle`
2026-05-05 11:48:40 +10:00
3. Kill ALL processes on ALL nodes — `pkill -9 -f mesh-llm`
4. Verify clean — `ps -eo pid,args | grep -E 'mesh-llm' | grep -v grep` must be empty.
2026-03-05 08:03:51 -05:00
5. Deploy bundle — scp + tar + codesign on remote nodes.
6. Verify version — `mesh-llm --version` on every node.
### After starting nodes
7. Verify exactly 1 mesh-llm process per node.
2026-05-05 11:48:40 +10:00
8. Verify no external llama serving child processes are required.
2026-03-05 08:03:51 -05:00
9. `curl -s http://localhost:3131/api/status` returns valid JSON on every node.
10. Check `/api/status` peers for new version string.
11. Verify expected peer count.
12. Test inference through every model in `/v1/models` .
13. Test `/v1/` passthrough on port 3131.
2026-05-05 11:48:40 +10:00
### Debugging Embedded Runtime Startup
2026-04-04 18:47:32 +11:00
2026-05-05 11:48:40 +10:00
If the embedded runtime fails to load, check mesh-llm stderr/log output and
2026-05-12 16:18:14 -04:00
`~/.mesh-llm/runtime/` for the active instance metadata. Embedded
2026-05-05 19:26:40 -04:00
skippy/llama.cpp native logs are redirected away from the TUI into the active
instance runtime directory:
```text
< runtime-root > /< pid > /logs/skippy-native.log
```
runtime: introduce scoped instance runtime with safe process lifecycle management
Replace global pkill-based process management with a per-instance runtime
directory system backed by flock liveness, atomic JSON pidfiles, and scoped
orphan reaping. Each mesh-llm process now owns a directory under the runtime
root, writes pidfiles for its child processes, and cleans up on exit. A
background scanner tracks all local instances and exposes them through the
management API and UI.
Key changes:
- InstanceRuntime: scoped runtime directory with flock-based liveness,
atomic JSON pidfile RAII guards, portable PID comm/start-time validation,
and pure decision logic for cross-instance orphan reaping.
- Process lifecycle: llama-server and rpc-server are now launched with
PID-tracked handles (LlamaServerHandle, RpcServerHandle) that write
pidfiles on start and reap on drop. Signal delivery validates PID comm
before sending, guarded by is_safe_kill_target to prevent kill(0/-1).
- Legacy removal: kill_llama_server, kill_orphan_rpc_servers,
terminate_process_by_name, and pkill-based helpers are removed.
A regression test asserts zero occurrences of these patterns in source.
- API: /api/status now includes version and local_instances fields,
populated from the background scanner with a self-entry safety net.
- UI: peer version is shown in the node table and sidebar; warning banners
surface mixed-version meshes and multi-instance conflicts.
2026-04-08 04:38:18 -04:00
To override the runtime root (e.g., for tests or systemd):
- `MESH_LLM_RUNTIME_ROOT=/path/to/custom/root` — highest priority
- `XDG_RUNTIME_DIR` — if set (typical on systemd: `/run/user/{uid}/mesh-llm/runtime` )
- `$HOME/.mesh-llm/runtime` — default fallback
For stale instances (crashed mesh-llm leaving behind a runtime dir):
- Other running mesh-llm instances GC dead-owner dirs older than 1 hour on startup
- Manual cleanup: `rm -rf ~/.mesh-llm/runtime/<stale_pid>/`
2026-04-04 18:47:32 +11:00
2026-03-05 08:03:51 -05:00
### Common failures
- **nohup over SSH doesn't stick** — use `bash -c "nohup ... & disown"` , verify process survives disconnect.
- **Duplicate processes** — always kill-verify-start.
- **codesign changes the hash** — don't compare local vs codesigned remote.
## Releasing
2026-03-27 18:39:15 +11:00
See `RELEASE.md` for the full process.
2026-06-12 18:44:36 +10:00
Current release flow: kick off the **Release** workflow (`.github/workflows/release.yml` ) from the GitHub Actions UI via `workflow_dispatch` with the version input (e.g. `v0.X.Y` ).
The dispatched workflow handles everything: it bumps versions via `scripts/release-version.sh` , generates and patches the SwiftPM manifest, packages SDK console assets, creates and pushes the release tag at a release-prep commit, builds the full artifact matrix (macOS, Linux CPU/ARM64/CUDA/CUDA-Blackwell/ROCm/Vulkan, Windows CPU/CUDA/ROCm/Vulkan), and publishes the GitHub release. Dispatch inputs include `skip_gpu_bundles` and `canary` (dry-run: build + smoke without publishing).
2026-03-27 18:39:15 +11:00
2026-06-12 18:44:36 +10:00
Pushing a `v*` tag manually also triggers the workflow, but that path requires preparing `Package.swift` and SDK console assets in the tag commit yourself — see `RELEASE.md` . Prefer the dispatch path.
2026-03-05 08:02:26 -05:00
2026-06-10 05:18:25 -04:00
### Installer checksum sidecars
Release/package scripts should keep generating `.sha256` sidecars for new
release archives. Do not rely on backfilling old release assets, because pinned
versions and alternate repos may not have sidecars.
`install.sh` and `install.ps1` must treat release-archive checksums as
backward-compatible rollout metadata:
- If `<archive>.sha256` exists, verify it and fail the install on malformed
checksum data or checksum mismatch.
- If the sidecar is missing for a legacy/current release, warn and continue by
default.
- If `MESH_LLM_REQUIRE_CHECKSUM=1` is set, a missing sidecar is fatal.
Do not change installer behavior to hard-require sidecars by default unless the
release policy also guarantees every supported/pinned release and alternate
install repo has matching checksum assets.
2026-03-05 08:02:26 -05:00
## Credentials
Test machine IPs, SSH details, and passwords are in `~/Documents/private-note.txt` (outside the repo). **Never commit credentials to any tracked file.**
## What NOT to add
2026-05-19 17:26:09 +10:00
- **No `api_key_token` feature** — explicitly rejected, removed in v0.26.0.
- **No credentials in tracked files** — IPs, passwords, SSH commands belong in `~/Documents/private-note.txt` only.
2026-06-12 18:44:36 +10:00
- **No domain logic in `crates/mesh-llm/src/` ** — that crate is CLI dispatch wiring over `mesh-llm-cli` / `mesh-llm-commands` / `mesh-llm-host-runtime` ; put new domain code in the host-runtime crate (or a more specific peer crate).
2026-05-19 17:26:09 +10:00
- **No external `llama-server` / `rpc-server` runtime lane** — the embedded staged runtime via patched llama.cpp is the only supported path.
