mesh-llm/docs/design/MULTI_MODAL.md

Multi-Modal Roadmap

Status

Implemented for the current multimodal scope.

Phases 1 through 5 in this plan are complete:

• multimodal capability advertisement across status, models, routing, and UI
• request-scoped ingress-local blob/object transport
• console support for image, audio, and file attachments
• multimodal /v1/chat/completions
• multimodal /v1/responses for both non-streaming and streaming requests

What remains is polish and hardening of the existing path, not new endpoint scope.

Goals

• Support multimodal inference through mesh-llm using llama.cpp-compatible models.
• Route image and audio requests to compatible hosts automatically.
• Keep media transport bounded and private by default.
• Preserve protocol compatibility unless we explicitly choose a breaking change.
• Keep the crate/plugin split clean: core owns inference routing, plugin owns media object storage.

Non-Goals

• Permanent distributed file storage
• IPFS/libp2p-first design
• POST /v1/audio/transcriptions
• POST /v1/audio/speech
• v1/realtime
• Audio generation from llama alone
• Native end-to-end video inference on the current llama.cpp path

API Targets

Phase 1

• POST /v1/chat/completions
• GET /v1/models

This is the shortest path because llama.cpp already supports multimodal chat here for supported models.

Phase 2

• POST /v1/responses

Implement as a mesh-llm compatibility shim after chat completions are solid.

Capability Model

Do not collapse everything into vision.

Recommended model capabilities:

• multimodal: bool
• vision: CapabilityLevel
• audio: CapabilityLevel
• reasoning: CapabilityLevel
• tool_use: CapabilityLevel

Why:

• multimodal is a useful umbrella signal for UI and coarse filtering.
• vision and audio are still required for correct routing.
• A model can be multimodal without supporting both image and audio equally.

Supported Family Matrix

This is the current mesh-llm capability matrix, based on the inference logic in crates/mesh-llm-host-runtime/src/models/capabilities.rs.

Important:

• supported means mesh-llm will treat the family/model as runtime-capable for routing and UI.
• likely means mesh-llm will surface a weaker multimodal hint, but not rely on it as a hard runtime capability.
• Catalog/download metadata can upgrade a family from likely to supported, for example:
  • vision models with an mmproj
  • models whose metadata exposes vision_config, audio_config, or modality token IDs

Vision

Family / signal	mesh-llm status	Notes / examples
Qwen3-VL, Qwen3VL	supported	Example: Qwen3VL-2B-Instruct-Q4_K_M
Qwen2-VL, Qwen2.5-VL	supported	Any matching qwen2-vl, qwen2.5-vl family name
LLaVA	supported	Covers llava family names
mllama	supported	Llama vision variants exposed as mllama
PaliGemma	supported	Family-name detection
Idefics	supported	Family-name detection
Molmo	supported	Family-name detection
InternVL	supported	Family-name detection
GLM-4V / GLM4V	supported	Family-name detection
Ovis	supported	Family-name detection
Florence	supported	Family-name detection
Any catalog model with mmproj	supported	This is the strongest repo-local signal for GGUF vision models
Any model with vision_config or vision token IDs	supported	Derived from local/remote metadata JSON
Generic -vl, _vl, video, multimodal, image names	likely	Hint only until promoted by stronger metadata

Audio

Family / signal	mesh-llm status	Notes / examples
Qwen2-Audio	supported	Covers qwen2-audio / qwen2_audio
SeaLLM-Audio	supported	Covers seallm-audio / seallm_audio
Ultravox	supported	Family-name detection
Omni	supported	Example: Qwen2.5-Omni-3B-Q4_K_M
Whisper	supported	Family-name detection
Generic audio / speech family names	supported	Strong signal in current inference code
Any model with audio_config or audio token IDs	supported	Derived from local/remote metadata JSON
Generic voice naming only	likely	Hint only unless stronger metadata is present

Multimodal Umbrella

Case	mesh-llm status	Notes
vision != none	multimodal = true	Vision implies multimodal
audio != none	multimodal = true	Audio implies multimodal
Both vision and audio supported	multimodal = true	Best fit for mixed image+audio routing
Name-only generic multimodal signal	likely for modality-specific support	Not enough on its own for hard vision/audio routing

Current Practical Examples

Model	Vision	Audio	Notes
Qwen3VL-2B-Instruct-Q4_K_M	supported	none	Good current image test model
Qwen2.5-Omni-3B-Q4_K_M	none or metadata-dependent	supported	Good current audio test model
Catalog GGUF with mmproj	supported	depends	Vision promoted by sidecar
Generic multimodal name with no sidecar/metadata	likely	likely/none	UI hint only; not a strong routing guarantee

Protocol Plan

Preferred path: additive change.

• add multimodal
• add audio
• keep vision meaning image/vision support

Repurposing vision to mean generic multimodal would be a breaking semantic change:

• old nodes would misinterpret the flag
• routing and UI would become incorrect in mixed-version meshes
• protobuf only protects unknown fields, not changed meaning of existing fields

If we ever want to make that breaking change anyway, it should be an explicit protocol-version decision.

Request Formats

Chat Completions

Accept and preserve OpenAI-style content parts:

• text
• image URL / data URL
• audio URL / data URL
• future file-style references

Video is not a Phase 1 request shape target. Treat video separately from image/audio until the serving stack can handle it natively.

mesh-llm should detect multimodal intent from structured content blocks, not just text keywords.

Responses API

Add a translation layer from responses input items into chat-completions-style message content after Phase 1 is working.

Routing Notes

• audio and multimodal are part of capability inference and surfaced in /api/status and /v1/models.
• Multimodal payloads are detected by structured content inspection, not just prompt text.
• model=auto prefers:
  • vision-capable hosts for image inputs
  • audio-capable hosts for audio inputs
  • models supporting both when both are present
• Media requests bypass the fragile pre-plan path where needed.

Video Support

Some model families support video at the model level, but that should not be treated as available in mesh-llm yet.

Current working assumption:

• open multimodal models with video support exist
• the current llama.cpp integration path in mesh-llm should be treated as image/audio only
• native video input should stay out of scope until upstream serving support is real and reliable

Recommended first implementation path for video:

• accept uploaded video into the same request-scoped blob plugin
• decode and sample frames server-side
• send sampled frames as ordered image inputs to a vision-capable model
• optionally include timestamp metadata in the prompt or content structure

Why this path first:

• it reuses the existing blob/object lifecycle
• it works with current image-capable serving paths
• it avoids blocking on native video support in llama.cpp

Follow-up requirements when we do video:

• add video capability metadata only when there is a real serving path behind it
• define upload limits, codec/container acceptance, and frame-sampling defaults
• decide whether video should be exposed only through responses, through chat/completions, or both
• make it explicit in the UI when video is being converted into sampled images rather than handled natively

Media Transport Plan

Large media should not travel inside request JSON by default.

First Pass

Use a request-scoped mesh content store implemented as a plugin.

Properties:

• ingress node stores the uploaded object locally
• no replication
• client receives an opaque secret token, not a content hash
• completion request references the token
• serving node fetches the object from the ingress node if needed
• ingress node deletes the object when the request reaches terminal state
• short TTL cleanup handles crashes, disconnects, and abandoned requests

External vs Internal Identity

• external object ID: secret high-entropy token
• internal storage key: content hash for integrity and dedupe only

The token should be user-visible and fetch-authorizing. The content hash should remain internal.

Blob Plugin Shape

The blob/content store should be a plugin, not core storage logic.

Core responsibilities:

• parse inference requests
• choose inference targets
• decide request lifecycle
• tell the plugin when a request starts and ends

Plugin responsibilities:

• store request-scoped media on ingress
• mint and validate opaque tokens
• serve media fetches to remote hosts
• enforce TTL and cleanup

Initial plugin operations:

• put_request_object
• get_request_object
• complete_request
• abort_request
• reap_expired_objects

Object Lifecycle

Request-Scoped Media

1. Client uploads image/audio/file to the ingress node.
2. Plugin stores bytes locally and returns a short-lived secret token.
3. Client sends completion request referencing that token.
4. Chosen host fetches from ingress if the object is not local.
5. Request completes, fails definitively, or is canceled.
6. Ingress plugin deletes token and blob after a short grace window.
7. TTL fallback cleans up leaks.

Recommended first-pass behavior:

• no replication
• token bound to one request
• allow a small retry budget for reroute / transport retries
• keep only a short grace period after terminal state

Console State

Existing

• image, audio, and file attachment UI
• upload-before-send for request-scoped objects
• token-reference request construction for chat/completions and responses
• attachment previews / badges before send:
  • image thumbnail
  • audio filename, duration if available, and remove action
  • generic file name, size, mime type, and remove action
• upload state handling:
  • pending
  • uploaded
  • failed
  • retrying
• model=auto switching when attachments are present
• capability hints in the model picker:
  • vision
  • audio
  • multimodal
• attachment preservation through retries and reroutes for the same request
• clear fallback UX when no compatible warm model is available
• explicit transcript attachment rendering instead of flattening everything into plain text

Suggested Console Sequence

1. User attaches image/audio/file.
2. Console creates a pending attachment entry in local UI state.
3. Console uploads the object to the ingress node and receives a request-scoped token.
4. Pending entry becomes uploaded attachment metadata.
5. Completion request is sent with structured message content referring to the token.
6. If the request is retried, reuse the same token while the request is still alive.
7. When the request finishes, normal cleanup happens on the ingress node.

Nice-to-Have Later

• drag-and-drop attachments
• paste image support
• microphone capture for audio input
• waveform / duration preview for audio
• video upload once there is either frame-sampling support or native serving support
• richer attachment rendering in the transcript
• attachment reuse inside the same conversation only if we later decide to support longer-lived object leases

Body Size and Transport Limits

Today the proxy body limits are tuned for JSON chat, not large media.

We should:

• keep small inline image support where it is practical
• prefer upload-and-reference for audio and larger files
• avoid raising global HTTP body limits as the main solution

Implementation Phases

Phase 1: Capability Plumbing

Complete.

Phase 2: Routing and Request Parsing

Complete.

Phase 3: Blob Plugin

Complete.

Phase 4: Console

Complete.

Phase 5: Compatibility Shims

Complete for /v1/responses.

Phase 6: Optional Extended Backends

• video ingestion and frame sampling pipeline
• richer persistent asset handling if we ever want reusable attachments

Follow-Up Work

• polish and harden the existing multimodal responses path where needed
• improve capability inference for edge-case model families
• refine attachment UX and transcript rendering
• keep video, if ever pursued, as frame sampling over the existing image path

Open Questions

• exact token format and request binding rules
• whether tiny inline images should remain supported in addition to blob upload
• whether responses should become the primary public API once parity is good enough
• whether multimodal requests should always disable pre-plan, or only for audio
• whether plugin-host fetches should flow through the existing mesh transport or a dedicated plugin RPC path
• when to introduce a real video capability instead of treating video as a higher-level image sequence workflow