dschlueter/llamacppctl
1
0
Fork 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity
llamacppctl/CHANGELOG.md
dschlueter 6076e3bcb8 feat(chat): make streaming and chat timeouts per-profile configurable 
The chat request previously used a hard-coded 30 s timeout and ignored
--read-timeout entirely (that flag only bounded URL prompt fetching), so
slow reasoning models were cut off mid-generation unless one remembered
to pass --stream (which used a separate hard-coded 600 s).

Resolve `stream`, `read_timeout` and `connect_timeout` from
[default]/[model.<profile>] into PromptConfig and wire the (connect, read)
timeout into both the streaming and non-streaming chat calls. CLI
--stream/--read-timeout/--connect-timeout still override; the two timeout
flags default to None so a config value can win, with the URL-fetch
fallbacks (3 s/10 s) preserved. Default chat read_timeout is now 600 s.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-09 18:26:33 +02:00

2.7 KiB
Raw Blame History

Changelog

All notable changes to this project are documented here.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

Added

  • Per-profile chat streaming and chat-request timeouts: stream, read_timeout and connect_timeout are now resolvable in [default]/[model.<profile>] (CLI --stream/--read-timeout/--connect-timeout still override).

Changed

  • The chat request now honors the configured (connect, read) timeout instead of a hard-coded 30 s; the default read_timeout is 600 s so slow reasoning models are no longer cut off mid-generation.

0.1.0 - 2026-07-07

Initial release. Replaces the previous collection of shell scripts (start-llm-server.sh, status-llm-server.sh, switch-llm.sh, …) with a single Python package.

Added

  • CLI to start, check, stop, and switch (--change) llama.cpp OpenAI-compatible Docker server instances, plus --chat and --dry-run.
  • Layered configuration (builtin defaults → [default] → [model.<profile>] → CLI overrides) with [prompt.<profile>] fallback system prompts and ${ENV}/~ expansion.
  • Security-hardened prompt input layer for --system-*/--prompt-*: file reads limited to regular UTF-8 files with a size cap and symlink rejection; URL reads are HTTPS-only with SSRF defenses (IP-literal and private/loopback/link-local/ metadata blocking, DNS pinning against rebinding, no-credentials URLs, redirect re-validation, content-type allowlist, streaming size limit).
  • Loopback-only port publishing by default; --expose + --api-key for LAN use.
  • Readiness check via a real chat completion (not just an open port) and an exclusive non-blocking file lock for --change.
  • MIT LICENSE, man page, and an opt-in end-to-end smoke test (scripts/smoke.sh).
  • Continuous integration on Forgejo: pytest (Python 3.10 and 3.12), ruff, and mypy.

Changed

  • Pin the default llama.cpp Docker image by digest for reproducibility instead of the moving :server-cuda tag.
  • Model-size profile names carry a b suffix (qwen35b, qwen27b) so the digits read as parameter count, not Qwen version; documentation examples run against the [default] section so they work by copy & paste.

Fixed

  • Override the image's baked-in Docker HEALTHCHECK (hardcoded to port 8080) so it targets the configured container_port, ending the false "unhealthy" status when the server runs on a different port.