dschlueter/llamacppctl
1
0
Fork 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity

feat(chat): make streaming and chat timeouts per-profile configurable

Browse source 
The chat request previously used a hard-coded 30 s timeout and ignored
--read-timeout entirely (that flag only bounded URL prompt fetching), so
slow reasoning models were cut off mid-generation unless one remembered
to pass --stream (which used a separate hard-coded 600 s).

Resolve `stream`, `read_timeout` and `connect_timeout` from
[default]/[model.<profile>] into PromptConfig and wire the (connect, read)
timeout into both the streaming and non-streaming chat calls. CLI
--stream/--read-timeout/--connect-timeout still override; the two timeout
flags default to None so a config value can win, with the URL-fetch
fallbacks (3 s/10 s) preserved. Default chat read_timeout is now 600 s.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Dieter Schlüter 2026-07-09 18:26:33 +02:00
commit 6076e3bcb8
9 changed files with 91 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

8
llama.cpp.config.example
View file

@ -62,6 +62,14 @@ api_key =
max_tokens = 2048
# chat_temperature leer lassen -> die Server-Temperatur (temp) gilt.
chat_temperature =
# Chat-Antwort standardmäßig token-weise streamen (wie --stream). Für langsame
# Reasoning-Modelle angenehm, weil sofort Ausgabe erscheint.
stream = false
# (connect, read)-Timeout des Chat-Requests in Sekunden. read_timeout muss die
# volle Generierungszeit abdecken; Reasoning-Modelle brauchen viel -> hoch
# setzen. --read-timeout/--connect-timeout auf der CLI überschreiben das.
read_timeout = 600
connect_timeout = 10
# Example second model profile, pinned to the second GPU (e.g. RTX 3090 #2)
# with its own port and container name so it can run alongside [default].