dschlueter/llamacppctl
1
0
Fork 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity

feat(docker): support a multimodal projector via --mmproj

Browse source 
Vision-capable GGUFs need a separate projector (mmproj) that maps image
embeddings into the text model's space. Add `mmproj` and `mmproj_offload`
as config keys and CLI overrides, and pass them through to llama-server.

The projector path resolves under hf_home exactly like model_path, so it
is covered by the existing read-only mount. validate_model_path() now also
checks the projector, which means --change rejects a missing one *before*
it removes the running container.

--no-mmproj-offload is suppressed when no projector is configured, since
llama.cpp rejects the flag on its own.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Dieter Schlüter 2026-07-10 16:24:47 +02:00
commit 218c3fc791
9 changed files with 199 additions and 9 deletions
Download patch file Download diff file Expand all files Collapse all files

18
llama.cpp.config.example
View file

@ -88,6 +88,24 @@ host_port = 8003
gpu_device = 1
ctx_size = 131072
# Vision profile: `mmproj` points at the multimodal projector (vision encoder)
# GGUF. Like `model_path` it is resolved relative to `hf_home` unless absolute.
# The projector MUST match the base model's vision tower -- a projector built for
# a different base will not load. A missing projector aborts with exit code 3
# (for --change: before the running container is removed).
#
# llama.cpp offloads the projector to the GPU by default; set mmproj_offload to
# false to keep it on the CPU when VRAM is tight. Images are then sent to
# /v1/chat/completions as an image_url content part -- `llamacppctl --chat`
# itself only sends text.
[model.vision]
model_path = qwen3/Qwen3-35B-A3B-Q4_K_M.gguf
mmproj = qwen3/mmproj.gguf
mmproj_offload = true
container_name = llama_cpp_vision
host_port = 8004
gpu_device = 1
[prompt.concise]
system_prompt = Du antwortest kurz, präzise und technisch.