dschlueter/llamacppctl
Watch
1
0
Fork
You've already forked llamacppctl
 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity

Initial commit: llamacppctl – llama.cpp Docker server control CLI

Browse source 
Steuert einen llama.cpp-Server als Docker-Container: --start/--check/--stop/
--change/--chat, INI-Konfiguration (builtin defaults -> [default] ->
[model.<profile>] -> CLI), SSRF-gehärtete Prompt-Eingabe (Datei/HTTPS-URL),
File-Locking für --start/--change und ein OpenAI-kompatibler HTTP-Layer.

Enthält u. a.:
- Env-Var-Expansion in hf_home (hf_home = ${HF_HOME})
- konfigurierbares Chat-Antwortbudget (max_tokens/chat_temperature,
  CLI: --max-tokens/--chat-temp); temperature defer an Server-Default
- DNS-Pinning gegen DNS-Rebinding bei URL-Quellen
- dry-run als nebenwirkungsfreie Vorschau (kein Lock/Removal/Modell-Check)
- 98 Tests (pytest)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Dieter Schlüter 2026-07-06 16:36:27 +02:00
commit 3158d16f9b
32 changed files with 3912 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

21
.gitignore vendored Normal file
View file

@ -0,0 +1,21 @@
# Python
__pycache__/
*.py[cod]
*.egg-info/
build/
dist/
.pytest_cache/
.mypy_cache/
.ruff_cache/
# Virtual environment
.venv/
venv/
# Editor / tooling
.claude/
.env
*.log
# Runtime lock files
*.lock

152
README.md Normal file
View file

@ -0,0 +1,152 @@
# llamacppctl
Ein einziges, sicherheitsgehärtetes CLI-Werkzeug zum Starten, Prüfen, Stoppen und
Wechseln von `llama.cpp`-Server-Instanzen, die als Docker-Container laufen —
inklusive einer SSRF-gehärteten Eingabeschicht für System- und User-Prompts aus
Text, Datei oder Remote-URL.
`llamacppctl` ersetzt eine Sammlung einzelner Shell-Skripte
(`start-llm-server.sh`, `status-llm-server.sh`, `switch-llm.sh`, …) durch ein
einziges Python-Paket mit konsistenter Konfiguration, Locking und
Fehlerbehandlung.
## Installation
```bash
tar xzf llamacppctl-installable.tar.gz
cd llamacppctl
python3 -m venv .venv
source .venv/bin/activate
pip install .
```
Danach steht der Befehl `llamacppctl` innerhalb der aktivierten virtuellen
Umgebung zur Verfügung. Für eine systemweite Installation (ohne venv):
```bash
pip install --user .
```
Voraussetzungen auf dem Zielsystem:
- Python >= 3.10
- Docker (CLI + laufender Daemon), für `--start`/--check`/--stop`/--change`
- Netzwerkzugriff auf den Container-Host-Port für `--chat`/--check`
`--print-effective-config` und `--dry-run` benötigen keinen laufenden Docker-Daemon
für die reine Konfigurationsprüfung; `--start`/--check`/--stop`/--change` erfordern
einen erreichbaren Docker-Daemon.
## Konfiguration
Kopiere `llama.cpp.config.example` nach `llama.cpp.config` und passe es an:
```bash
cp llama.cpp.config.example llama.cpp.config
```
Abschnittstypen:
- `[default]` — globale Defaults, überschreiben die eingebauten Fallback-Werte
- `[model.<name>]` — ein Modell-Profil, ausgewählt über `--profile <name>`
- `[prompt.<name>]` — ein System-Prompt-Profil, ausgewählt über
`--system-prompt-profile <name>` (liefert nur einen *Fallback*-Systemprompt;
ein explizites `-s/--system-file/--system-url` gewinnt immer)
**`container_name` ist Pflicht** und muss für jedes Profil, das parallel
laufen soll, eindeutig sein — er ist der einzige Identitätsanker für
Docker-Namensgebung, Locking (`--start`/`--change`) sowie
`--stop`/`--check`-Ziel.
`hf_home` unterstützt Environment-Variablen und `~`, z. B. `hf_home = ${HF_HOME}`.
Das Chat-Antwortbudget ist ebenfalls konfigurierbar (wirkt auf `--chat` und die
Antwort nach `--start`, nicht auf den Container): `max_tokens` (Default 2048 —
für Reasoning-Modelle / lange Texte hochsetzen, sonst bricht die Antwort mitten
im „Denken" ab und `content` bleibt leer) und `chat_temperature` (leer =
Server-`temp` gilt). CLI-Overrides: `--max-tokens`, `--chat-temp`.
Auflösungsreihenfolge (niedrigste zu höchster Priorität):
1. Eingebaute Defaults
2. `[default]`-Sektion
3. `[model.<profile>]`-Sektion (falls `--profile` gesetzt)
4. CLI-Overrides (`--image`, `--host-port`, …)
## Verwendung
```bash
# Server starten
llamacppctl --start --config llama.cpp.config --profile qwen35
# Nur den geplanten docker-run-Befehl anzeigen, nichts ausführen
llamacppctl --start --config llama.cpp.config --profile qwen35 --dry-run
# Effektive Konfiguration als JSON ausgeben
llamacppctl --print-effective-config --config llama.cpp.config --profile qwen35 --start
# Status prüfen
llamacppctl --check --config llama.cpp.config --profile qwen35
# Server stoppen und Container entfernen
llamacppctl --stop --config llama.cpp.config --profile qwen35
# Modell wechseln (stop, neu konfigurieren, restart) unter Lock-Schutz
llamacppctl --change --config llama.cpp.config --profile deepseek
# Direkten Chat-Request an einen laufenden Server senden
llamacppctl --chat --config llama.cpp.config --profile qwen35 \
-s "Du antwortest kurz." -p "Was ist 2+2?"
# System-Prompt aus einer lokalen Datei, User-Prompt als Literal
llamacppctl --start --config llama.cpp.config --profile qwen35 \
--system-file ./prompts/coding.md -p "Refaktoriere diese Funktion."
# System-Prompt von einer HTTPS-URL (öffentliches, nicht-privates Ziel)
llamacppctl --chat --config llama.cpp.config --profile qwen35 \
--system-url https://example.com/prompts/system.txt -p "Hallo"
```
Vollständige Optionsliste: `llamacppctl --help` oder die Manpage
(`man/llamacppctl.1`, siehe unten).
## Sicherheitsmodell (Kurzfassung)
Für Details siehe [`docs/SECURITY_AND_OPERATIONS.md`](docs/SECURITY_AND_OPERATIONS.md).
- Alle Docker-Aufrufe laufen über `subprocess.run()` mit Argumentlisten —
niemals über Shell-Strings, somit keine Shell-Injection-Fläche.
- `--system-file`/--prompt-file` lesen ausschließlich reguläre Dateien,
Größenlimit (`--max-input-bytes`, Default 1 MiB), UTF-8-Text erzwungen,
Symlinks standardmäßig abgelehnt, per `--allow-symlinks` freischaltbar.
- `--system-url`/--prompt-url` sind standardmäßig HTTPS-only, blockieren
IP-Literale, private/loopback/link-local/reserved Adressbereiche sowie die
Cloud-Metadata-IP `169.254.169.254`, verbieten eingebettete Zugangsdaten in
der URL, erlauben standardmäßig keine Redirects (jeder Hop wird bei
Aktivierung erneut validiert), erzwingen eine Content-Type-Allowlist und ein
hartes Größenlimit sowohl über `Content-Length` als auch beim Streaming.
- `container_name` ist verpflichtend und muss pro parallel betriebenem
Profil eindeutig sein, um Docker-Namenskollisionen und ungewollte
Container-Übernahmen zu verhindern.
- `--start`/--change` nutzen echte Chat-Completions zum Readiness-Check
(nicht nur einen offenen Port), um sicherzustellen, dass das Modell
tatsächlich geladen und inferenzfähig ist.
- `--change` verwendet eine exklusive, nicht-blockierende Dateisperre
(`/tmp/llamacppctl.<container_name>.lock`), um parallele Änderungen am
selben Container zu verhindern.
## Entwicklung / Tests
```bash
pip install -e ".[dev]"
pytest -q
```
Die Test-Suite deckt die Prompt-Eingabeschicht (Datei- und URL-Quellen inkl.
SSRF-Abwehr mit gemockter DNS-Auflösung), die Konfigurationsauflösung, die
CLI-Validierung sowie Docker-/HTTP-Operationen (mit gemocktem `subprocess`
bzw. `requests`) ab.
## Lizenz
MIT

430
build_archive.py Normal file
View file

@ -0,0 +1,430 @@
#!/usr/bin/env python3
"""build_archive.py
Assembles ``llamacppctl-installable.tar.gz`` directly on the local machine
from the llamacppctl project files, and verifies that all required files and
declared dependencies are present and consistent before/after packaging.
This script is intentionally self-contained (standard library only, plus an
optional import of ``tomllib``/``tomli`` purely for dependency verification)
so it can be run directly, without first installing the llamacppctl package
itself.
Usage
-----
python3 build_archive.py
python3 build_archive.py --project-dir /path/to/llamacppctl --output /path/to/llamacppctl-installable.tar.gz
python3 build_archive.py --skip-tests # skip running pytest before packaging
python3 build_archive.py --skip-pip-check # skip the pip dependency dry-run
Exit codes
----------
0 archive built and verified successfully
1 a required file was missing, a test failed, or verification failed
"""
from __future__ import annotations
import argparse
import subprocess
import sys
import tarfile
import venv
from pathlib import Path
# --------------------------------------------------------------------------
# Required file manifest.
#
# Every path here is relative to the project directory and MUST exist before
# packaging. This is the "verifies that all dependencies are correctly
# included" contract: the archive is only built if every one of these is
# present, and re-opened afterwards to confirm every one of them made it
# into the tarball unmodified.
# --------------------------------------------------------------------------
REQUIRED_FILES = [
"pyproject.toml",
"README.md",
"requirements.txt",
"requirements-dev.txt",
"llama.cpp.config.example",
"docs/SECURITY_AND_OPERATIONS.md",
"man/llamacppctl.1",
"src/llamacppctl/__init__.py",
"src/llamacppctl/py.typed",
"src/llamacppctl/cli.py",
"src/llamacppctl/schema.py",
"src/llamacppctl/config.py",
"src/llamacppctl/prompt_io.py",
"src/llamacppctl/docker_ops.py",
"src/llamacppctl/http_ops.py",
"src/llamacppctl/lock_ops.py",
"src/llamacppctl/actions.py",
"src/llamacppctl/main.py",
"tests/conftest.py",
"tests/test_prompt_sources.py",
"tests/test_prompt_files.py",
"tests/test_prompt_urls.py",
"tests/test_config.py",
"tests/test_cli.py",
"tests/test_docker_ops.py",
"tests/test_http_ops.py",
"tests/test_lock_ops.py",
]
# Top-level directories to include wholesale (in addition to REQUIRED_FILES),
# so that any file added later without updating REQUIRED_FILES is still
# packaged. REQUIRED_FILES remains the source of truth for *verification*.
INCLUDE_DIRS = ["src", "tests", "docs", "man"]
INCLUDE_FILES = [
"pyproject.toml",
"README.md",
"requirements.txt",
"requirements-dev.txt",
"llama.cpp.config.example",
"build_archive.py",
]
EXCLUDE_DIR_NAMES = {"__pycache__", ".pytest_cache", ".venv", "venv", ".git", "*.egg-info"}
EXCLUDE_SUFFIXES = {".pyc", ".pyo"}
ARCHIVE_ROOT_NAME = "llamacppctl" # top-level directory name inside the tarball
class BuildError(RuntimeError):
pass
def log(msg: str) -> None:
print(f"[build_archive] {msg}")
def verify_required_files(project_dir: Path) -> None:
log("Verifying required files are present...")
missing = []
for rel in REQUIRED_FILES:
p = project_dir / rel
if not p.is_file():
missing.append(rel)
if missing:
raise BuildError(
"Missing required files, refusing to build archive:\n "
+ "\n ".join(missing)
)
log(f" All {len(REQUIRED_FILES)} required files present.")
def verify_dependencies_declared(project_dir: Path) -> list:
"""Parses pyproject.toml and cross-checks it against requirements.txt.
Returns the list of runtime dependency specifiers declared in
pyproject.toml. Raises BuildError on any mismatch.
"""
log("Verifying declared dependencies are consistent...")
pyproject_path = project_dir / "pyproject.toml"
requirements_path = project_dir / "requirements.txt"
try:
import tomllib # Python 3.11+
except ModuleNotFoundError:
try:
import tomli as tomllib # type: ignore
except ModuleNotFoundError:
tomllib = None
if tomllib is not None:
with open(pyproject_path, "rb") as f:
data = tomllib.load(f)
deps = data.get("project", {}).get("dependencies", [])
else:
# Fallback: minimal manual extraction, good enough for verification
# purposes when neither tomllib nor tomli is available.
log(" (tomllib/tomli unavailable, falling back to manual parse)")
deps = []
in_deps = False
for line in pyproject_path.read_text(encoding="utf-8").splitlines():
stripped = line.strip()
if stripped.startswith("dependencies"):
in_deps = True
if "[" in stripped and "]" in stripped:
inner = stripped.split("[", 1)[1].rsplit("]", 1)[0]
deps = [d.strip().strip('"').strip("'") for d in inner.split(",") if d.strip()]
in_deps = False
continue
if in_deps:
if "]" in stripped:
in_deps = False
stripped = stripped.split("]", 1)[0]
cleaned = stripped.strip().rstrip(",").strip('"').strip("'")
if cleaned:
deps.append(cleaned)
if not deps:
raise BuildError("No runtime dependencies found in pyproject.toml [project.dependencies]")
req_text = requirements_path.read_text(encoding="utf-8")
req_names = set()
for line in req_text.splitlines():
line = line.strip()
if not line or line.startswith("#"):
continue
# crude package-name extraction, e.g. "requests>=2.31,<3" -> "requests"
name = line
for sep in (">=", "<=", "==", "!=", ">", "<", "~="):
if sep in name:
name = name.split(sep, 1)[0]
req_names.add(name.strip().lower())
missing_from_requirements = []
for dep in deps:
name = dep
for sep in (">=", "<=", "==", "!=", ">", "<", "~="):
if sep in name:
name = name.split(sep, 1)[0]
name = name.strip().lower()
if name not in req_names:
missing_from_requirements.append(dep)
if missing_from_requirements:
raise BuildError(
"pyproject.toml declares dependencies not mirrored in requirements.txt:\n "
+ "\n ".join(missing_from_requirements)
)
log(f" pyproject.toml dependencies: {deps}")
log(" requirements.txt is consistent with pyproject.toml.")
return deps
def run_pip_dependency_check(project_dir: Path, deps: list) -> None:
"""Creates a throwaway virtualenv and does a real `pip install` of just
the declared dependencies (not the project itself) to confirm they are
resolvable from PyPI. This is a stronger guarantee than a --dry-run
against an environment that might already happen to have them cached."""
log("Verifying dependencies are installable (throwaway virtualenv)...")
import tempfile
with tempfile.TemporaryDirectory(prefix="llamacppctl-depcheck-") as tmp:
venv_dir = Path(tmp) / "venv"
venv.EnvBuilder(with_pip=True, clear=True).create(venv_dir)
pip_bin = venv_dir / "bin" / "pip"
if not pip_bin.exists():
pip_bin = venv_dir / "Scripts" / "pip.exe" # Windows fallback
cmd = [str(pip_bin), "install", "--quiet", *deps]
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode != 0:
raise BuildError(
"Dependency installability check failed:\n"
f"stdout:\n{result.stdout}\nstderr:\n{result.stderr}"
)
log(" All declared dependencies installed successfully in an isolated environment.")
def run_tests(project_dir: Path) -> None:
log("Running test suite (pytest) before packaging...")
result = subprocess.run(
[sys.executable, "-m", "pytest", "-q"],
cwd=str(project_dir),
capture_output=True,
text=True,
)
print(result.stdout)
if result.returncode != 0:
print(result.stderr, file=sys.stderr)
raise BuildError(f"Test suite failed (exit code {result.returncode}); refusing to package.")
log(" Test suite passed.")
def _tar_filter(tarinfo: tarfile.TarInfo) -> tarfile.TarInfo | None:
parts = Path(tarinfo.name).parts
for part in parts:
if part in EXCLUDE_DIR_NAMES or part.endswith(".egg-info"):
return None
if Path(tarinfo.name).suffix in EXCLUDE_SUFFIXES:
return None
# Normalize metadata for reproducibility.
tarinfo.uid = 0
tarinfo.gid = 0
tarinfo.uname = ""
tarinfo.gname = ""
return tarinfo
def build_tarball(project_dir: Path, output_path: Path) -> None:
log(f"Building archive: {output_path}")
output_path.parent.mkdir(parents=True, exist_ok=True)
with tarfile.open(output_path, "w:gz") as tar:
for rel in INCLUDE_FILES:
src = project_dir / rel
if src.is_file():
tar.add(src, arcname=f"{ARCHIVE_ROOT_NAME}/{rel}", filter=_tar_filter)
for dirname in INCLUDE_DIRS:
src_dir = project_dir / dirname
if src_dir.is_dir():
tar.add(src_dir, arcname=f"{ARCHIVE_ROOT_NAME}/{dirname}", filter=_tar_filter)
log(f" Archive written: {output_path} ({output_path.stat().st_size:,} bytes)")
def verify_tarball(output_path: Path) -> None:
log("Re-opening archive to verify contents...")
with tarfile.open(output_path, "r:gz") as tar:
names = set(tar.getnames())
missing = []
for rel in REQUIRED_FILES:
expected = f"{ARCHIVE_ROOT_NAME}/{rel}"
if expected not in names:
missing.append(expected)
if missing:
raise BuildError(
"Archive verification failed, required files missing from tarball:\n "
+ "\n ".join(missing)
)
log(f" Verified {len(REQUIRED_FILES)} required files are present in the archive.")
log(f" Total archive members: {len(names)}")
def smoke_test_install(output_path: Path) -> None:
"""Extracts the tarball into a temp dir, installs it in a throwaway
virtualenv, and confirms the `llamacppctl` console script works via
--print-effective-config and --dry-run (no Docker daemon required for
either)."""
log("Smoke-testing installation from the built archive...")
import tempfile
with tempfile.TemporaryDirectory(prefix="llamacppctl-smoketest-") as tmp:
tmp_path = Path(tmp)
with tarfile.open(output_path, "r:gz") as tar:
tar.extractall(tmp_path)
extracted_project = tmp_path / ARCHIVE_ROOT_NAME
venv_dir = tmp_path / "venv"
venv.EnvBuilder(with_pip=True, clear=True).create(venv_dir)
pip_bin = venv_dir / "bin" / "pip"
py_bin = venv_dir / "bin" / "python"
llamacppctl_bin = venv_dir / "bin" / "llamacppctl"
result = subprocess.run(
[str(pip_bin), "install", "--quiet", str(extracted_project)],
capture_output=True,
text=True,
)
if result.returncode != 0:
raise BuildError(
"Smoke-test install failed:\n"
f"stdout:\n{result.stdout}\nstderr:\n{result.stderr}"
)
log(" Package installed successfully from the extracted archive.")
config_example = extracted_project / "llama.cpp.config.example"
result = subprocess.run(
[
str(llamacppctl_bin),
"--print-effective-config",
"--config", str(config_example),
"--profile", "qwen35",
"--start",
"--dry-run",
],
capture_output=True,
text=True,
)
print(result.stdout)
docker_unavailable = "docker is not available on PATH" in (result.stdout + result.stderr)
if result.returncode != 0 and not docker_unavailable:
print(result.stderr, file=sys.stderr)
raise BuildError(
f"Smoke test invocation failed unexpectedly (exit code {result.returncode})."
)
if docker_unavailable:
log(
" Console script and argument/config resolution work end-to-end; "
"exited cleanly at the expected Docker-unavailable check "
"(no Docker daemon in this environment -- this is not a packaging defect)."
)
else:
log(" --print-effective-config / --dry-run smoke test succeeded.")
def main(argv: list | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--project-dir",
default=str(Path(__file__).resolve().parent),
help="Path to the llamacppctl project directory (default: this script's directory)",
)
parser.add_argument(
"--output",
default=None,
help="Output path for the tar.gz (default: <project-dir>/../llamacppctl-installable.tar.gz)",
)
parser.add_argument("--skip-tests", action="store_true", help="Skip running pytest before packaging")
parser.add_argument(
"--skip-pip-check",
action="store_true",
help="Skip the throwaway-virtualenv dependency installability check",
)
parser.add_argument(
"--skip-smoke-test",
action="store_true",
help="Skip extracting + installing the built archive as a final smoke test",
)
args = parser.parse_args(argv)
project_dir = Path(args.project_dir).resolve()
output_path = (
Path(args.output).resolve()
if args.output
else project_dir.parent / "llamacppctl-installable.tar.gz"
)
try:
verify_required_files(project_dir)
deps = verify_dependencies_declared(project_dir)
if not args.skip_pip_check:
run_pip_dependency_check(project_dir, deps)
else:
log("Skipping pip dependency installability check (--skip-pip-check).")
if not args.skip_tests:
run_tests(project_dir)
else:
log("Skipping test suite (--skip-tests).")
build_tarball(project_dir, output_path)
verify_tarball(output_path)
if not args.skip_smoke_test:
smoke_test_install(output_path)
else:
log("Skipping install smoke test (--skip-smoke-test).")
except BuildError as exc:
print(f"\n[build_archive] BUILD FAILED: {exc}", file=sys.stderr)
return 1
log("")
log("Build and verification complete.")
log(f"Archive: {output_path}")
log(f"Size: {output_path.stat().st_size:,} bytes")
log("")
log("To install on the target machine:")
log(f" tar xzf {output_path.name}")
log(f" cd {ARCHIVE_ROOT_NAME}")
log(" python3 -m venv .venv && source .venv/bin/activate")
log(" pip install .")
return 0
if __name__ == "__main__":
sys.exit(main())

53
docs/Archiv_fertig_-_Check_und_Installation.md Normal file
View file

@ -0,0 +1,53 @@
Das Archiv ist fertig und übergeben: llamacppctl-installable.tar.gz.
Was gebaut wurde
Statt manuell zu tarballen, gibt es jetzt ein eigenständiges Python-Skript build_archive.py, das:
Prüft, ob alle 27 erforderlichen Projektdateien vorhanden sind (Quellcode, Tests, Doku, Manpage, Config-Beispiel)
pyproject.toml-Abhängigkeiten gegen requirements.txt abgleicht
Die deklarierte Abhängigkeit (requests>=2.31,<3) in einer frischen, isolierten virtuellen Umgebung installiert, um echte Installierbarkeit zu bestätigen (nicht nur einen Trockenlauf)
Die vollständige Test-Suite (93 Tests, alle grün) laufen lässt
Das .tar.gz über Pythons tarfile-Modul baut (nicht rohes Shell-tar)
Das Archiv erneut öffnet und verifiziert, dass jede erwartete Datei tatsächlich enthalten ist
Das Archiv in ein temporäres Verzeichnis entpackt, in einer weiteren frischen venv installiert und den llamacppctl-Befehl ausführt, um Konsolenskript und Konfigurationsauflösung Ende-zu-Ende zu bestätigen
Da diese Sandbox keinen Docker-Daemon hat, bricht der letzte Schritt erwartungsgemäß an der Docker-Verfügbarkeitsprüfung ab — das Skript erkennt genau diesen Fall und wertet ihn nicht als Fehler, da Paketinstallation, Konsolenskript-Registrierung und Argumentparsing bereits vorher erfolgreich bestätigt wurden.
Enthaltene Projektstruktur
src/llamacppctl/ — die vollständige Implementierung: cli.py, config.py, prompt_io.py (SSRF-gehärtete Eingabeschicht), docker_ops.py, http_ops.py, lock_ops.py, actions.py, main.py
tests/ — 93 Tests (Prompt-Quellen, Datei-/URL-Sicherheit mit gemockter DNS-Auflösung, Konfigurationsauflösung, CLI-Validierung, Docker-/HTTP-Operationen mit gemocktem subprocess/requests, File-Locking)
docs/SECURITY_AND_OPERATIONS.md — vollständige Architektur-, Konfigurations- und Sicherheitsdokumentation
man/llamacppctl.1 — klassische Unix-Manpage
README.md, llama.cpp.config.example, requirements.txt, pyproject.toml
Selbst verifizieren
Nach dem Herunterladen kannst du die Integrität unabhängig prüfen, ohne mir zu vertrauen:
bash
gzip -t llamacppctl-installable.tar.gz && echo "Archiv OK"
tar tzf llamacppctl-installable.tar.gz
Installation:
bash
tar xzf llamacppctl-installable.tar.gz
cd llamacppctl
python3 -m venv .venv && source .venv/bin/activate
pip install .
cp llama.cpp.config.example llama.cpp.config # anpassen
llamacppctl --print-effective-config --config llama.cpp.config --profile qwen35 --start
Da build_archive.py selbst mit im Archiv liegt, kannst du das Archiv auf deinem eigenen Rechner jederzeit neu bauen und erneut verifizieren, ganz ohne auf mich angewiesen zu sein.

24
docs/How_to_use.md Normal file
View file

@ -0,0 +1,24 @@
### Ist das start- stop- change-Skript etc auch dabei ?
Ja — aber als ein einziges vereinheitlichtes CLI-Tool statt separater Shell-Skripte. Das war ja genau der Punkt der Neuentwicklung: `llamacppctl` ersetzt die einzelnen Skripte (`start-llm-server.sh`, `status-llm-server.sh`, `switch-llm.sh`, …) durch **eine** Python-Anwendung mit Unterbefehlen (Actions).
Alle Funktionen sind im Archiv enthalten, als Actions von `llamacppctl`:
| Alte Skript-Idee | Neuer Aufruf |
|---|---|
| `start-llm-server.sh` | `llamacppctl --start --config llama.cpp.config --profile <name>` |
| `status-llm-server.sh` | `llamacppctl --check --config llama.cpp.config --profile <name>` |
| Stop-Skript | `llamacppctl --stop --config llama.cpp.config --profile <name>` |
| `switch-llm.sh` (Modellwechsel) | `llamacppctl --change --config llama.cpp.config --profile <name>` |
| (neu) direkter Chat-Test | `llamacppctl --chat --config llama.cpp.config --profile <name> -p "..."` |
Die Logik dahinter liegt in `src/llamacppctl/actions.py` (`do_start`, `do_check`, `do_stop`, `do_change`, `do_chat`) und `src/llamacppctl/docker_ops.py` (baut den `docker run`-Befehl, startet/stoppt/entfernt Container, liest Logs).
Besonderheiten gegenüber den ursprünglichen Shell-Skripten:
- `--change` nutzt eine exklusive Dateisperre (`/tmp/llamacppctl.<container_name>.lock`), damit zwei gleichzeitige Modellwechsel für denselben Container sich nicht in die Quere kommen.
- Die Start-Bereitschaftsprüfung (`--start`/`--change`) wartet auf eine echte Chat-Completion-Antwort, nicht nur auf einen offenen Port — das ist zuverlässiger als ein reiner Port-Check.
- `--dry-run` zeigt dir den vollständigen `docker run`-Befehl an, ohne ihn auszuführen — praktisch, um vor einer echten Änderung zu prüfen, was passieren würde.
Die vollständige Referenz zu allen Optionen steht in der Manpage (`man/llamacppctl.1`) und in `README.md`/`docs/SECURITY_AND_OPERATIONS.md` im Archiv.

255
docs/SECURITY_AND_OPERATIONS.md Normal file
View file

@ -0,0 +1,255 @@
# llamacppctl — Sicherheits- und Betriebsdokumentation
Diese Datei beschreibt die Architektur, das Konfigurationsmodell, das
Sicherheitsmodell der Prompt-Eingabeschicht sowie den empfohlenen
Betriebsmodus auf einem dedizierten GPU-Host.
## 1. Architektur / Modulübersicht
```
src/llamacppctl/
cli.py Argumentparsing (argparse) + semantische Cross-Field-Validierung
schema.py Dataclasses: ServerConfig, PromptConfig, CheckResult
config.py INI-Konfiguration laden/mergen -> ServerConfig/PromptConfig
prompt_io.py Sicherheitsgehärtete Eingabeschicht für System-/User-Prompts
docker_ops.py Alle Docker-Interaktionen (subprocess mit Argumentlisten)
http_ops.py HTTP-Health-Checks und Chat-Completions gegen die
OpenAI-kompatible llama.cpp-API
lock_ops.py Exklusives, nicht-blockierendes File-Locking für --change
actions.py Orchestrierung: do_start, do_check, do_stop, do_change, do_chat
main.py Dünner Einstiegspunkt: parse -> validate -> resolve -> dispatch
```
Verantwortungstrennung: `prompt_io.py` kennt kein argparse und arbeitet nur
mit `PromptSource` + `InputPolicy`. `config.py` kennt keine Docker- oder
HTTP-Details. `docker_ops.py`/`http_ops.py` kennen keine CLI-Semantik. Diese
Trennung hält die sicherheitsrelevante Logik (Eingabevalidierung) unabhängig
testbar von der Orchestrierung.
## 2. Konfigurationsmodell
`llama.cpp.config` ist eine Standard-INI-Datei (`configparser`). Auflösung
(niedrigste zu höchster Priorität):
1. **Eingebaute Defaults** (`config.builtin_defaults()`) — spiegeln die
Standardwerte der ursprünglichen Shell-Skripte wider (Image
`ghcr.io/ggml-org/llama.cpp:server-cuda`, `host_port=8001`,
`container_name=va_llm`, `gpu_device=1`, `jinja/fa/kv_unified/
cont_batching/no_context_shift=true`, `reasoning=on`,
`cache_type_k/v=q4_0`, `batch_size=1024`, `ubatch_size=512`,
`timeout=300`, `poll_interval=2`, …).
2. **`[default]`**-Sektion — überschreibt die eingebauten Defaults global.
3. **`[model.<profile>]`**-Sektion — ausgewählt über `--profile <name>`,
überschreibt `[default]` selektiv (nur gesetzte Schlüssel).
4. **CLI-Overrides** — z. B. `--image`, `--host-port`, `--container-name`,
`--ctx-size`, … überschreiben alles Vorherige.
`[prompt.<name>]`-Sektionen werden **separat** aufgelöst und liefern
ausschließlich einen *Fallback*-Systemprompt (`system_prompt = ...`). Ein
explizit übergebener `-s/--system`, `--system-file` oder `--system-url`
gewinnt in jedem Fall.
### `container_name` ist verpflichtend
`config.build_server_config()` wirft einen `ConfigError`, wenn
`container_name` nach vollständiger Auflösung leer oder nur aus Whitespace
besteht. Grund: `container_name` ist der **einzige Identitätsanker** für:
- Docker-Container-Benennung (`docker run --name <container_name>`)
- Ziel für `--stop`/--check`
- Ableitung des Lock-Pfads (`/tmp/llamacppctl.<container_name>.lock`),
sofern `--lock-file`/`lock_file` nicht explizit gesetzt ist
Wer mehrere `[model.<name>]`-Profile definiert, die **gleichzeitig** als
separate Container laufen sollen (z. B. zwei Modelle auf zwei GPUs), **muss**
jedem Profil einen eigenen, eindeutigen `container_name` geben. Andernfalls
würde ein zweiter `--start` den ersten Container unbeabsichtigt ersetzen
(`start_llama_container()` entfernt einen vorhandenen Container gleichen
Namens, bevor es einen neuen startet).
Eine leere `--container-name`-Angabe auf der Kommandozeile wird bereits von
`cli.validate_args()` zurückgewiesen (Exit-Code 2, bevor überhaupt eine
Konfigurationsdatei gelesen wird). Eine leere Angabe **in der Config-Datei**
selbst (z. B. `container_name =` ohne Wert) wird erst bei der
Konfigurationsauflösung als `ConfigError` erkannt.
## 3. Sicherheitsmodell der Prompt-Eingabeschicht (`prompt_io.py`)
Jedes der beiden Eingabefelder (System-Prompt, User-Prompt) akzeptiert genau
**eine** von drei Quellenarten, gegenseitig ausschließend über
argparse-Mutually-Exclusive-Groups **und** eine zweite Verteidigungslinie in
`resolve_source()`:
- **Literal** (`-s`/`-p`): Text direkt auf der Kommandozeile.
- **Datei** (`--system-file`/--prompt-file`): lokaler Dateipfad.
- **URL** (`--system-url`/--prompt-url`): entfernte HTTPS(S)-Ressource.
### 3.1 Datei-Quellen
`load_text_file()`:
- Nur reguläre Dateien (`path.is_file()`); Verzeichnisse etc. werden
abgelehnt.
- Symlinks werden standardmäßig abgelehnt und sind nur über `--allow-symlinks`
freischaltbar (`InputPolicy.allow_symlinks`, Default `False`).
- Größenlimit `--max-input-bytes` (Default 1 MiB) wird **vor** dem Lesen
anhand von `stat().st_size` und **nach** der Normalisierung anhand der
UTF-8-kodierten Zeichenlänge geprüft.
- Binärinhalte (NUL-Byte in den Rohbytes) werden abgelehnt.
- Dekodierung: UTF-8, mit Fallback auf `utf-8-sig` (BOM); ungültige
UTF-8-Daten führen zu einem `PromptSourceError`.
- Zeilenenden werden normalisiert (`\r\n`/`\r` -> `\n`), ein führendes BOM
wird entfernt.
### 3.2 URL-Quellen (SSRF-Härtung)
`validate_url_target()` wird vor **jedem** HTTP-Request ausgeführt —
einschließlich jedem einzelnen Redirect-Hop, falls Redirects aktiviert sind:
1. Nur `https://` standardmäßig erlaubt; `http://` erfordert explizit
`--allow-insecure-http`.
2. Kein Hostname -> Ablehnung.
3. Eingebettete Zugangsdaten in der URL (`user:pass@host`) -> Ablehnung.
4. Optionale Host-Allowlist (`--url-allow-host`, wiederholbar): falls
gesetzt, muss der Hostname exakt (case-insensitiv) enthalten sein.
5. `localhost` als Hostname wird immer abgelehnt.
6. IP-Literale als Hostname (`https://127.0.0.1/...`) werden standardmäßig
abgelehnt; `--allow-ip-host` erlaubt sie explizit.
7. Alle DNS-Antworten für den Hostnamen werden aufgelöst
(`socket.getaddrinfo`); **jede** zurückgegebene Adresse wird klassifiziert
(`ipaddress`-Modul: `is_private`, `is_loopback`, `is_link_local`,
`is_multicast`, `is_reserved`, `is_unspecified`). Trifft **eine** dieser
Eigenschaften zu, wird die Anfrage abgelehnt (Rebinding-Schutz: es reicht,
dass irgendeine aufgelöste Adresse privat ist).
8. Die Cloud-Metadata-Adresse `169.254.169.254` wird explizit zusätzlich
geprüft und immer blockiert, sofern nicht `--allow-private-url` gesetzt
ist.
9. `--allow-private-url` deaktiviert die IP-Klassifikationsprüfungen 7 und 8
komplett (bewusste Eskalation für vertrauenswürdige interne Ziele) und
kann **nicht** gleichzeitig mit `--url-allow-host` verwendet werden
(widersprüchliche Sicherheitsmodelle: „alles erlauben“ vs. „nur
bestimmte Hosts“).
Nach erfolgreicher Zielvalidierung (`load_text_url()`):
- Anfrage mit `stream=True`, `allow_redirects=False` (Redirects werden
manuell behandelt, damit jeder Hop erneut validiert wird), separaten
Connect-/Read-Timeouts (`--connect-timeout`, `--read-timeout`).
- Redirect-Statuscodes (3xx) werden nur verfolgt, wenn `--follow-redirects`
gesetzt ist; sonst Ablehnung. Die maximale Anzahl Hops ist begrenzt
(`InputPolicy.max_redirects`, Default 3).
- Content-Type-Allowlist: nur Text-artige Typen
(`text/plain`, `text/markdown`, `text/csv`, `application/json`,
`application/xml`, `application/yaml`, …); `text/html` nur mit
`--allow-html-input`.
- Größenlimit wird sowohl über den `Content-Length`-Header **als auch**
während des Streamens (`iter_content`) durchgesetzt, sodass ein Server, der
einen falschen (zu niedrigen) `Content-Length`-Header sendet, keinen
Speicher-Erschöpfungsangriff durchführen kann.
- Antwort muss gültiges UTF-8 sein.
### 3.3 Warum das wichtig ist
Diese Härtung verhindert, dass `--system-url`/--prompt-url` als
SSRF-Vektor missbraucht werden kann, um interne Dienste (z. B.
Docker-Socket-Proxies, interne Admin-APIs, Cloud-Metadata-Endpunkte) vom
GPU-Host aus zu erreichen, falls `llamacppctl` jemals in einem Kontext
läuft, in dem die übergebene URL nicht vollständig vertrauenswürdig ist
(z. B. durch ein Webhook, ein anderes Automatisierungs-Skript oder einen
mehrbenutzerfähigen Wrapper um `llamacppctl`).
## 4. Docker- / Host-Vertrauensmodell
- `docker_ops.py` ruft ausschließlich `subprocess.run(["docker", ...])` mit
**Argumentlisten** auf, niemals mit `shell=True` oder zusammengesetzten
Shell-Strings. Es gibt daher keine Shell-Injection-Fläche über
Konfigurationswerte oder CLI-Argumente.
- `llamacppctl` vertraut dem lokalen Docker-Daemon vollständig (wie jedes
Docker-CLI-Tool) — es ist kein Sandboxing gegenüber Docker selbst
vorgesehen. Wer `llamacppctl` einsetzt, muss dem Nutzerkonto, das den
Befehl ausführt, den gleichen Vertrauensgrad einräumen wie direktem
Docker-CLI-Zugriff (i. d. R. Mitgliedschaft in der `docker`-Gruppe).
- `HF_HOME` wird **read-only** (`:ro`) in den Container gemountet — der
Container kann Modelldateien lesen, aber nicht verändern oder löschen.
- GPU-Zuweisung erfolgt über `--gpus device=<gpu_device>`, sodass mehrere
Profile gezielt auf unterschiedliche GPUs (z. B. RTX 3090 #1/#2) gepinnt
werden können.
## 5. Locking (`--change`)
`lock_ops.FileLock` verwendet `fcntl.flock(LOCK_EX | LOCK_NB)` auf einer
Lock-Datei unter `/tmp/llamacppctl.<container_name>.lock` (Default, ableitbar
über `--lock-file`). `--change` hält den Lock über die gesamte
Stop-Reconfigure-Restart-Sequenz. Ein zweiter, gleichzeitiger `--change`-Aufruf
für **denselben** `container_name` schlägt sofort mit `LockError` (Exit-Code
6) fehl, statt zu blockieren oder Race-Conditions am Container zu riskieren.
Da der Lock-Pfad an `container_name` gekoppelt ist, blockieren sich zwei
Profile mit unterschiedlichem `container_name` gegenseitig nicht.
## 6. `--dry-run` und `--print-effective-config` als Sicherheitswerkzeuge
- `--print-effective-config` gibt die vollständig aufgelöste Konfiguration
(Server- und Prompt-Konfiguration) als JSON aus, **bevor** irgendeine
Docker- oder HTTP-Aktion ausgeführt wird. Damit lässt sich prüfen, welche
Werte aus welcher Quelle (Defaults/[default]/[model.*]/CLI) tatsächlich
gewonnen haben, ohne einen Container anzufassen.
- `--dry-run` (in Kombination mit `--start`/--change`) zeigt den vollständig
zusammengesetzten `docker run`-Befehl (Shell-quotiert zur Anzeige) an,
**ohne** ihn auszuführen. Empfohlen vor jeder Änderung an einer
Produktionskonfiguration, insbesondere nach Anpassungen an
`llama.cpp.config`.
Beide Flags erfordern **keinen** erreichbaren Docker-Daemon für ihre reine
Ausgabe — `main.run()` prüft `docker_available()` derzeit vor der
Konfigurationsauflösung; auf einem Host ganz ohne Docker (z. B. zur reinen
Konfigurationsvalidierung) schlägt der Aufruf entsprechend mit einer
expliziten Fehlermeldung fehl statt still falsche Annahmen zu treffen.
## 7. Betriebsmodell auf einem dedizierten GPU-Host
Empfohlenes Setup (passend zu einer Zwei-GPU-Umgebung, z. B. 2× RTX 3090):
```ini
[default]
hf_home = /srv/models
gpu_device = 0
container_name = llama_cpp_server
host_port = 8001
[model.qwen35]
container_name = llama_cpp_qwen35
gpu_device = 1
host_port = 8002
[model.deepseek]
container_name = llama_cpp_deepseek
gpu_device = 1
host_port = 8003
ctx_size = 131072
```
- Jedes Profil, das parallel laufen soll, bekommt eine eigene
`container_name`/`host_port`-Kombination.
- `--profile` wählt bei jedem Aufruf explizit aus, welches Profil
angesprochen wird — es gibt keinen impliziten "aktuellen" Zustand außer
dem, was tatsächlich in Docker läuft.
- `--check --profile <name>` ist der empfohlene Weg für Monitoring/Cronjobs,
um sowohl Container-Status als auch tatsächliche Inferenzfähigkeit
(`chat_ok`) zu prüfen.
## 8. Installation, Nutzung, Fehlersuche
Siehe [`README.md`](../README.md) für Installationsschritte und
Nutzungsbeispiele.
Häufige Fehlerbilder:
| Symptom | Ursache | Lösung |
|---|---|---|
| `docker is not available on PATH` | Docker-CLI fehlt oder Daemon nicht erreichbar | Docker installieren/starten, Nutzer zur `docker`-Gruppe hinzufügen |
| `container_name is mandatory and must not be empty` | Kein `container_name` in `[default]`/`[model.*]`/CLI gesetzt | `container_name` in der Config oder via `--container-name` setzen |
| `model profile not found: <name>` | `--profile <name>` verweist auf nicht existierende `[model.<name>]`-Sektion | Sektionsname prüfen/anlegen |
| `http requires --allow-insecure-http` | `--system-url`/--prompt-url` nutzt `http://` | Auf `https://` wechseln oder bewusst `--allow-insecure-http` setzen |
| `resolved to blocked IP ...` | URL löst auf eine private/loopback/reserved Adresse auf | Ziel korrigieren oder bewusst `--allow-private-url` setzen (nur für vertrauenswürdige interne Ziele) |
| `lock busy: /tmp/llamacppctl.<name>.lock` | Ein anderer `--change`-Lauf für denselben `container_name` läuft bereits | Warten, bis der andere Lauf beendet ist, oder Ursache des hängenden Laufs prüfen |
| `Server did not become ready within <n>s` | Modell braucht länger zum Laden als `timeout`, oder Startfehler | `--timeout` erhöhen, `--logs` für Container-Log-Tail nutzen |

81
llama.cpp.config Normal file
View file

@ -0,0 +1,81 @@
# llama.cpp.config.example
#
# Copy this file to llama.cpp.config and adjust it for your host before
# running llamacppctl. See docs/SECURITY_AND_OPERATIONS.md for details on
# every field and on the security model for --system-url/--prompt-url.
#
# Section types:
# [default] global defaults, merged over the built-in fallbacks
# [model.<name>] a model profile, selected via --profile <name>
# [prompt.<name>] a system-prompt profile, selected via --system-prompt-profile <name>
#
# container_name is MANDATORY and must be unique per profile you intend to
# run concurrently: it is the single identity anchor for Docker naming,
# locking (--change), and --stop/--check targeting.
[default]
image = ghcr.io/ggml-org/llama.cpp:server-cuda
hf_home = ${HF_HOME}
model_path = models/qwen3/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q4_K_M.gguf
container_name = llama_cpp_server
host_port = 8001
container_port = 8000
model_alias = default_llm
gpu_device = 1
restart_policy = unless-stopped
# Groß-Profil für literarische Texte / durchdachte Reden: volles
# 256k-Kontextfenster (= n_ctx_train des Modells) und sehr lange Antworten.
# Empirisch gemessen passt der q4_0-KV-Cache dafür knapp auf EINE 24-GB-3090
# (~22,6 GB inkl. 21-GB-Modell). Bei Bedarf auf 131072/65536 senken.
ctx_size = 262144
n_predict = 32768
temp = 0.65
top_p = 0.80
top_k = 20
min_p = 0.01
repeat_penalty = 1.05
main_gpu = 0
ngl = 999
fa = true
kv_unified = true
jinja = true
reasoning = on
no_context_shift = true
cache_type_k = q4_0
cache_type_v = q4_0
batch_size = 1024
ubatch_size = 512
parallel = 1
cont_batching = true
health_endpoint = /health
models_endpoint = /v1/models
chat_endpoint = /v1/chat/completions
timeout = 300
poll_interval = 2
# Chat-Antwortbudget (Reasoning-Modell braucht viel; für lange Texte hoch):
max_tokens = 32768
# chat_temperature leer lassen -> Server-Temp (temp oben) gilt.
chat_temperature =
# Example second model profile, pinned to the second GPU (e.g. RTX 3090 #2)
# with its own port and container name so it can run alongside [default].
[model.qwen35]
model_path = qwen3/Qwen3-35B-A3B-Q4_K_M.gguf
model_alias = qwen35
container_name = llama_cpp_qwen35
host_port = 8002
gpu_device = 1
[model.deepseek]
model_path = deepseek/deepseek-r1-q4.gguf
model_alias = deepseek
container_name = llama_cpp_deepseek
host_port = 8003
gpu_device = 1
ctx_size = 131072
[prompt.concise]
system_prompt = Du antwortest kurz, präzise und technisch.
[prompt.coding]
system_prompt = Du bist ein erfahrener Linux-, Python- und LLM-Infrastruktur-Engineer.

79
llama.cpp.config.example Normal file
View file

@ -0,0 +1,79 @@
# llama.cpp.config.example
#
# Copy this file to llama.cpp.config and adjust it for your host before
# running llamacppctl. See docs/SECURITY_AND_OPERATIONS.md for details on
# every field and on the security model for --system-url/--prompt-url.
#
# Section types:
# [default] global defaults, merged over the built-in fallbacks
# [model.<name>] a model profile, selected via --profile <name>
# [prompt.<name>] a system-prompt profile, selected via --system-prompt-profile <name>
#
# container_name is MANDATORY and must be unique per profile you intend to
# run concurrently: it is the single identity anchor for Docker naming,
# locking (--change), and --stop/--check targeting.
[default]
image = ghcr.io/ggml-org/llama.cpp:server-cuda
# hf_home unterstützt Environment-Variablen und ~, z. B. hf_home = ${HF_HOME}
hf_home = /srv/models
model_path = qwen3/default.gguf
container_name = llama_cpp_server
host_port = 8001
container_port = 8000
model_alias = default_llm
gpu_device = 0
restart_policy = unless-stopped
ctx_size = 262144
n_predict = 16384
temp = 0.65
top_p = 0.80
top_k = 20
min_p = 0.01
repeat_penalty = 1.05
main_gpu = 0
ngl = 999
fa = true
kv_unified = true
jinja = true
reasoning = on
no_context_shift = true
cache_type_k = q4_0
cache_type_v = q4_0
batch_size = 1024
ubatch_size = 512
parallel = 1
cont_batching = true
health_endpoint = /health
models_endpoint = /v1/models
chat_endpoint = /v1/chat/completions
timeout = 300
poll_interval = 2
# Chat-Antwortbudget (wirkt auf --chat / --start-Antwort, nicht auf den Container).
# Reasoning-Modelle brauchen viel Budget; für lange Texte hochsetzen.
max_tokens = 2048
# chat_temperature leer lassen -> die Server-Temperatur (temp) gilt.
chat_temperature =
# Example second model profile, pinned to the second GPU (e.g. RTX 3090 #2)
# with its own port and container name so it can run alongside [default].
[model.qwen35]
model_path = qwen3/Qwen3-35B-A3B-Q4_K_M.gguf
model_alias = qwen35
container_name = llama_cpp_qwen35
host_port = 8002
gpu_device = 1
[model.deepseek]
model_path = deepseek/deepseek-r1-q4.gguf
model_alias = deepseek
container_name = llama_cpp_deepseek
host_port = 8003
gpu_device = 1
ctx_size = 131072
[prompt.concise]
system_prompt = Du antwortest kurz, präzise und technisch.
[prompt.coding]
system_prompt = Du bist ein erfahrener Linux-, Python- und LLM-Infrastruktur-Engineer.

292
man/llamacppctl.1 Normal file
View file

@ -0,0 +1,292 @@
.TH LLAMACPPCTL 1 "2026-07-05" "llamacppctl 0.1.0" "User Commands"
.SH NAME
llamacppctl \- start, check, stop, switch, or prompt a llama.cpp server running as a Docker container
.SH SYNOPSIS
.B llamacppctl
\fB\-\-start\fR|\fB\-\-check\fR|\fB\-\-stop\fR|\fB\-\-change\fR|\fB\-\-chat\fR
[\fIOPTIONS\fR]
.SH DESCRIPTION
.B llamacppctl
manages a
.B llama.cpp
OpenAI-compatible inference server running inside a Docker container. It
replaces a collection of shell scripts with a single Python CLI that shares
one configuration model, one locking mechanism, and one security-hardened
prompt input layer for system and user prompts.
.SH ACTIONS
Exactly one action is required.
.TP
.B \-\-start
Start the llama.cpp server container. Removes any pre-existing container of
the same name first. Waits for readiness via real chat completions, not just
an open port.
.TP
.B \-\-check
Report Docker container status, HTTP health/model endpoint reachability, and
(if a user prompt is supplied) whether a live chat completion succeeds.
.TP
.B \-\-stop
Stop and remove the container. No-op if it does not exist.
.TP
.B \-\-change
Stop, reconfigure, and restart the server under an exclusive file lock keyed
on \fBcontainer_name\fR, preventing concurrent modification of the same
container.
.TP
.B \-\-chat
Send a single chat completion request to an already-running server and print
the response.
.SH CONFIGURATION MODEL
Configuration is a standard INI file (default path: \fIllama.cpp.config\fR,
override with \fB\-\-config\fR). Resolution order, lowest to highest
priority:
.IP 1. 4
Built-in defaults
.IP 2. 4
\fB[default]\fR section
.IP 3. 4
\fB[model.<profile>]\fR section, selected via \fB\-\-profile\fR
.IP 4. 4
CLI overrides (\fB\-\-image\fR, \fB\-\-host\-port\fR, ...)
.PP
\fB[prompt.<name>]\fR sections, selected via \fB\-\-system\-prompt\-profile\fR,
provide only a \fIfallback\fR system prompt; an explicit
\fB\-s\fR/\fB\-\-system\-file\fR/\fB\-\-system\-url\fR always wins.
.PP
\fBcontainer_name\fR is mandatory and must be non-empty after resolution. It
is the single identity anchor for Docker naming, \fB\-\-stop\fR/\fB\-\-check\fR
targeting, and the derived lock file path
(\fI/tmp/llamacppctl.<container_name>.lock\fR). Profiles intended to run
concurrently must each use a distinct \fBcontainer_name\fR.
.SH OPTIONS
.SS General
.TP
.B \-\-config \fIPATH\fR
Path to the INI configuration file. Default: \fIllama.cpp.config\fR.
.TP
.B \-\-profile \fINAME\fR
Select a \fB[model.<NAME>]\fR section.
.TP
.B \-\-system\-prompt\-profile \fINAME\fR
Select a \fB[prompt.<NAME>]\fR section for a fallback system prompt.
.SS Prompt sources (system)
Mutually exclusive.
.TP
.B \-s\fR, \fB\-\-system \fITEXT\fR
Literal system prompt text.
.TP
.B \-\-system\-file \fIPATH\fR
Read the system prompt from a local file.
.TP
.B \-\-system\-url \fIURL\fR
Read the system prompt from a remote HTTPS URL.
.SS Prompt sources (user)
Mutually exclusive.
.TP
.B \-p\fR, \fB\-\-prompt \fITEXT\fR
Literal user prompt text.
.TP
.B \-\-prompt\-file \fIPATH\fR
Read the user prompt from a local file.
.TP
.B \-\-prompt\-url \fIURL\fR
Read the user prompt from a remote HTTPS URL.
.SS Docker / server overrides
.TP
.B \-\-image \fIIMAGE\fR
Docker image, e.g. \fIghcr.io/ggml-org/llama.cpp:server-cuda\fR.
.TP
.B \-\-hf\-home \fIPATH\fR
Host path mounted read-only into the container as model storage. Environment
variables (e.g. \fB${HF_HOME}\fR) and \fB~\fR are expanded, so the config may
contain \fBhf_home = ${HF_HOME}\fR.
.TP
.B \-\-model\-path \fIPATH\fR
Model path, relative to \fB\-\-hf\-home\fR unless absolute.
.TP
.B \-\-container\-name \fINAME\fR
Docker container name. Must be unique per concurrently-running profile.
.TP
.B \-\-host\-port \fIPORT\fR\fR, \fB\-\-container\-port \fIPORT\fR
Host / container-internal TCP port (1-65535).
.TP
.B \-\-model\-alias \fINAME\fR
Model alias exposed via the OpenAI-compatible API.
.TP
.B \-\-gpu \fIID\fR
GPU device id passed to \fB\-\-gpus device=<ID>\fR.
.TP
.B \-\-restart\-policy \fIPOLICY\fR
Docker \fB\-\-restart\fR policy.
.SS llama.cpp runtime parameters
.B \-\-ctx\-size\fR, \fB\-\-n\-predict\fR, \fB\-\-temp\fR, \fB\-\-top\-p\fR,
\fB\-\-top\-k\fR, \fB\-\-min\-p\fR, \fB\-\-repeat\-penalty\fR,
\fB\-\-main\-gpu\fR, \fB\-\-ngl\fR, \fB\-\-batch\-size\fR,
\fB\-\-ubatch\-size\fR, \fB\-\-parallel\fR, \fB\-\-cache\-type\-k\fR,
\fB\-\-cache\-type\-v\fR, \fB\-\-reasoning {on,off}\fR.
.PP
Boolean pairs (unset by default, meaning "use config/default value"):
\fB\-\-jinja\fR/\fB\-\-no\-jinja\fR, \fB\-\-fa\fR/\fB\-\-no\-fa\fR,
\fB\-\-kv\-unified\fR/\fB\-\-no\-kv\-unified\fR,
\fB\-\-cont\-batching\fR/\fB\-\-no\-cont\-batching\fR,
\fB\-\-no\-context\-shift\fR/\fB\-\-context\-shift\fR.
.SS Chat request parameters
Applied to the reply produced by \fB\-\-chat\fR and the post-start reply of
\fB\-\-start\fR (they do not affect the container itself). Config keys:
\fBmax_tokens\fR, \fBchat_temperature\fR.
.TP
.B \-\-max\-tokens \fIN\fR
Maximum tokens for the chat reply. Default 2048; raise it for reasoning models
or long-form output (they spend many tokens "thinking" before the visible
answer, so a small budget yields empty content). Must be > 0.
.TP
.B \-\-chat\-temp \fIFLOAT\fR
Temperature for the chat request. If unset, the request omits temperature and
the server's configured \fB\-\-temp\fR applies (single source of truth).
.SS Security / behavior flags (prompt input)
.TP
.B \-\-allow\-insecure\-http
Permit \fBhttp://\fR URLs for \fB\-\-system\-url\fR/\fB\-\-prompt\-url\fR (default: HTTPS only).
.TP
.B \-\-allow\-private\-url
Disable private/loopback/link-local/reserved/metadata IP blocking for URL
sources. Cannot be combined with \fB\-\-url\-allow\-host\fR.
.TP
.B \-\-allow\-ip\-host
Permit IP-literal hostnames in URLs (default: rejected).
.TP
.B \-\-follow\-redirects
Follow HTTP redirects for URL sources (default: rejected). Each hop is
re-validated against the same SSRF rules. Bounded by an internal max-redirect
count.
.TP
.B \-\-allow\-html\-input
Accept \fBtext/html\fR responses in addition to the default text-like
content-type allowlist.
.TP
.B \-\-allow\-symlinks\fR / \fB\-\-no\-allow\-symlinks
Allow or reject symlinked files for \fB\-\-system\-file\fR/\fB\-\-prompt\-file\fR.
Default: rejected (opt in with \fB\-\-allow\-symlinks\fR).
.TP
.B \-\-max\-input\-bytes \fIN\fR
Maximum size in bytes for any resolved prompt input (file or URL). Default:
1048576 (1 MiB). Must be > 0.
.TP
.B \-\-url\-allow\-host \fIHOST\fR
Restrict URL sources to an explicit host allowlist. Repeatable.
.TP
.B \-\-connect\-timeout \fISECONDS\fR\fR, \fB\-\-read\-timeout \fISECONDS\fR
Network timeouts for URL sources. Must be > 0. Defaults: 3.0 / 10.0.
.SS Behavior / diagnostics
.TP
.B \-\-timeout \fISECONDS\fR
Readiness timeout for \fB\-\-start\fR/\fB\-\-change\fR. Default from config
(300s).
.TP
.B \-\-poll\-interval \fISECONDS\fR
Readiness poll interval. Default from config (2s).
.TP
.B \-\-lock\-file \fIPATH\fR
Override the lock file path used by \fB\-\-start\fR/\fB\-\-change\fR. Default:
derived from \fBcontainer_name\fR.
.TP
.B \-\-logs
Print container log tail on readiness failure.
.TP
.B \-\-log\-lines \fIN\fR
Number of log lines to print. Default: 100.
.TP
.B \-\-dry\-run
Print the fully constructed \fBdocker run\fR command and exit without
executing it. Usable with \fB\-\-start\fR/\fB\-\-change\fR.
.TP
.B \-\-force
Force stop/change despite inconsistencies.
.TP
.B \-\-print\-effective\-config
Print the fully merged configuration as JSON and exit (unless combined with
an action that also runs).
.SH SECURITY NOTES
All Docker interaction goes through \fBsubprocess.run()\fR with argument
lists, never shell strings — there is no shell-injection surface via
configuration values or CLI arguments.
.PP
The remote URL prompt source (\fB\-\-system\-url\fR/\fB\-\-prompt\-url\fR) is
SSRF-hardened by default: HTTPS-only, no IP-literal hosts, no embedded
credentials, DNS resolution and IP classification for every candidate
address (blocking private/loopback/link-local/multicast/reserved/
unspecified ranges and the cloud metadata address
\fI169.254.169.254\fR), no redirects unless explicitly enabled (with
per-hop re-validation), a content-type allowlist, and a hard size limit
enforced both via \fBContent-Length\fR and while streaming the response body.
See \fIdocs/SECURITY_AND_OPERATIONS.md\fR in the source distribution for the
full threat model.
.PP
\fBcontainer_name\fR is mandatory and must be unique per concurrently-running
profile; reusing a name across profiles that run at the same time will cause
one to replace the other's container on the next \fB\-\-start\fR.
.SH FILES
.TP
.I llama.cpp.config
Default configuration file (INI format). See \fIllama.cpp.config.example\fR
in the source distribution.
.TP
.I /tmp/llamacppctl.<container_name>.lock
Default lock file used by \fB\-\-change\fR, derived from \fBcontainer_name\fR
unless \fB\-\-lock\-file\fR is set.
.SH EXIT STATUS
.TP
.B 0
Success.
.TP
.B 1
General error (Docker unavailable, unexpected exception, prompt input error, configuration error).
.TP
.B 2
CLI argument / semantic validation error.
.TP
.B 3
Model path not found on disk.
.TP
.B 4
Docker command failed.
.TP
.B 5
HTTP request failed or server did not become ready in time.
.TP
.B 6
Lock busy (a concurrent \fB\-\-change\fR is already running for the same \fBcontainer_name\fR).
.SH EXAMPLES
.PP
Start a profile and show the effective configuration first:
.RS
.nf
llamacppctl \-\-print\-effective\-config \-\-config llama.cpp.config \-\-profile qwen35 \-\-start
llamacppctl \-\-start \-\-config llama.cpp.config \-\-profile qwen35
.fi
.RE
.PP
Preview the docker run command without executing it:
.RS
.nf
llamacppctl \-\-start \-\-config llama.cpp.config \-\-profile qwen35 \-\-dry\-run
.fi
.RE
.PP
Check status including a live chat completion:
.RS
.nf
llamacppctl \-\-check \-\-config llama.cpp.config \-\-profile qwen35 \-p "ping"
.fi
.RE
.PP
Switch models under lock protection:
.RS
.nf
llamacppctl \-\-change \-\-config llama.cpp.config \-\-profile deepseek
.fi
.RE
.SH AUTHOR
Dieter Schlueter
.SH SEE ALSO
.BR docker (1)

29
pyproject.toml Normal file
View file

@ -0,0 +1,29 @@
[build-system]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "llamacppctl"
version = "0.1.0"
description = "CLI to start, check, stop, and switch llama.cpp Docker server instances with a security-hardened prompt input layer."
readme = "README.md"
requires-python = ">=3.10"
license = { text = "MIT" }
authors = [{ name = "Dieter Schlueter" }]
dependencies = [
"requests>=2.31,<3",
]
[project.optional-dependencies]
dev = [
"pytest>=7.4",
]
[project.scripts]
llamacppctl = "llamacppctl.main:main"
[tool.setuptools.packages.find]
where = ["src"]
[tool.setuptools.package-data]
llamacppctl = ["py.typed"]

4
requirements-dev.txt Normal file
View file

@ -0,0 +1,4 @@
# Development/test dependencies (mirrors [project.optional-dependencies].dev
# in pyproject.toml).
-r requirements.txt
pytest>=7.4

4
requirements.txt Normal file
View file

@ -0,0 +1,4 @@
# Runtime dependencies (mirrors [project.dependencies] in pyproject.toml).
# Provided for users who want to verify/install dependencies without pip's
# PEP 517 build isolation, e.g. `pip install -r requirements.txt`.
requests>=2.31,<3

5
src/llamacppctl/__init__.py Normal file
View file

@ -0,0 +1,5 @@
"""llamacppctl - a general-purpose CLI to start, check, stop and switch llama.cpp
server containers, with a strict, explicit prompt-input layer (literal / file / URL).
"""
__version__ = "0.1.0"

193
src/llamacppctl/actions.py Normal file
View file

@ -0,0 +1,193 @@
"""Top-level action implementations: do_start, do_check, do_stop, do_change, do_chat.
Exit code convention:
0 success
1 general error (docker unavailable, unexpected exception)
2 CLI / validation error (handled by argparse.error before reaching here)
3 model path missing
4 docker error
5 HTTP / readiness not ready
6 lock busy
"""
from __future__ import annotations
import json
import sys
from dataclasses import asdict
from pathlib import Path
from .docker_ops import (
DockerError,
build_run_command,
container_logs,
docker_available,
format_command_for_display,
inspect_container,
remove_container,
start_llama_container,
stop_container,
)
from .http_ops import HttpError, base_url, chat_completion_text, wait_until_ready
from .lock_ops import FileLock, LockError
from .schema import CheckResult, PromptConfig, ServerConfig
EXIT_OK = 0
EXIT_GENERAL = 1
EXIT_MODEL_PATH_MISSING = 3
EXIT_DOCKER_ERROR = 4
EXIT_HTTP_NOT_READY = 5
EXIT_LOCK_BUSY = 6
def validate_model_path(cfg: ServerConfig) -> None:
model_path = Path(cfg.model_path)
if model_path.is_absolute():
target = model_path
else:
target = cfg.hf_home / cfg.model_path
if not target.exists():
raise FileNotFoundError(f"model file not found: {target}")
def print_effective_config(cfg: ServerConfig, prompt_cfg: PromptConfig) -> None:
payload = {
"server": {k: (str(v) if isinstance(v, Path) else v) for k, v in asdict(cfg).items()},
"prompt": asdict(prompt_cfg),
}
print(json.dumps(payload, indent=2, ensure_ascii=False))
def do_check(cfg: ServerConfig, prompt_cfg: PromptConfig, args) -> CheckResult:
info = inspect_container(cfg.container_name)
exists = info.status != "missing"
running = info.status == "running"
from .http_ops import check_health, check_models
health_ok = check_health(cfg) if running else False
models_ok = check_models(cfg) if running else False
chat_ok = False
if running and prompt_cfg.user_prompt:
try:
chat_completion_text(cfg, prompt_cfg)
chat_ok = True
except Exception:
chat_ok = False
return CheckResult(
container_exists=exists,
container_running=running,
http_ok=health_ok or models_ok,
models_ok=models_ok,
chat_ok=chat_ok,
base_url=base_url(cfg),
detail=f"status={info.status} health={info.health}",
)
def print_check_result(cfg: ServerConfig, result: CheckResult) -> None:
print(f"Container: {cfg.container_name}")
print(f"Docker: {'RUNNING' if result.container_running else ('EXISTS (stopped)' if result.container_exists else 'NOT FOUND')}")
print(f"HTTP: {'OK' if result.http_ok else 'UNREACHABLE'}")
print(f"Base URL: {result.base_url}")
print(f"Model: {cfg.model_alias}")
if result.chat_ok:
print("Chat completions: OK")
print(f"Detail: {result.detail}")
def do_stop(cfg: ServerConfig, args) -> int:
try:
removed = stop_container(cfg.container_name, remove=True)
except DockerError as exc:
print(f"docker error: {exc}", file=sys.stderr)
return EXIT_DOCKER_ERROR
if removed:
print(f"Stopped and removed container: {cfg.container_name}")
else:
print(f"Container not found (nothing to stop): {cfg.container_name}")
return EXIT_OK
def do_start(cfg: ServerConfig, prompt_cfg: PromptConfig, args) -> int:
"""Start under the container's file lock so concurrent --start/--change
invocations cannot race on the same container_name."""
if getattr(args, "dry_run", False):
print(format_command_for_display(build_run_command(cfg)))
return EXIT_OK
try:
with FileLock(cfg.lock_file):
return _start_locked(cfg, prompt_cfg, args)
except LockError as exc:
print(str(exc), file=sys.stderr)
return EXIT_LOCK_BUSY
def _start_locked(cfg: ServerConfig, prompt_cfg: PromptConfig, args) -> int:
"""Core start logic. The caller must already hold the container lock."""
try:
validate_model_path(cfg)
except FileNotFoundError as exc:
print(str(exc), file=sys.stderr)
return EXIT_MODEL_PATH_MISSING
try:
container_id = start_llama_container(cfg)
except DockerError as exc:
print(f"docker error: {exc}", file=sys.stderr)
return EXIT_DOCKER_ERROR
print(f"Started container {cfg.container_name} ({container_id[:12]})")
print(f"Waiting for readiness (timeout={cfg.timeout}s)...")
probe = PromptConfig(system_prompt=None, user_prompt="ping", max_tokens=1, temperature=0.0, stream=False)
if not wait_until_ready(cfg, probe):
print(f"Server did not become ready within {cfg.timeout}s", file=sys.stderr)
if getattr(args, "logs", False):
print("---- container logs (tail) ----", file=sys.stderr)
print(container_logs(cfg.container_name, getattr(args, "log_lines", 100)), file=sys.stderr)
return EXIT_HTTP_NOT_READY
print("Model ready.")
if prompt_cfg.user_prompt:
try:
content = chat_completion_text(cfg, prompt_cfg)
print("---- response ----")
print(content)
except HttpError as exc:
print(f"chat request failed: {exc}", file=sys.stderr)
return EXIT_OK
def do_change(cfg: ServerConfig, prompt_cfg: PromptConfig, args) -> int:
# Dry-run is a pure preview: never take the lock or remove the container.
if getattr(args, "dry_run", False):
print(format_command_for_display(build_run_command(cfg)))
return EXIT_OK
try:
with FileLock(cfg.lock_file):
remove_container(cfg.container_name)
return _start_locked(cfg, prompt_cfg, args)
except LockError as exc:
print(str(exc), file=sys.stderr)
return EXIT_LOCK_BUSY
except DockerError as exc:
print(f"docker error: {exc}", file=sys.stderr)
return EXIT_DOCKER_ERROR
def do_chat(cfg: ServerConfig, prompt_cfg: PromptConfig, args) -> int:
if not prompt_cfg.user_prompt:
print("--chat requires a user prompt", file=sys.stderr)
return EXIT_GENERAL
try:
content = chat_completion_text(cfg, prompt_cfg)
except HttpError as exc:
print(f"chat request failed: {exc}", file=sys.stderr)
return EXIT_HTTP_NOT_READY
print(content)
return EXIT_OK

165
src/llamacppctl/cli.py Normal file
View file

@ -0,0 +1,165 @@
"""Argument parsing and CLI-level semantic validation for llamacppctl."""
from __future__ import annotations
import argparse
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(
prog="llamacppctl",
description=(
"Start, check, stop, switch or prompt a llama.cpp server "
"running as a Docker container."
),
)
action = p.add_mutually_exclusive_group(required=True)
action.add_argument("--start", action="store_true", help="Start the llama.cpp server")
action.add_argument("--check", action="store_true", help="Check container and HTTP status")
action.add_argument("--stop", action="store_true", help="Stop/remove the container")
action.add_argument("--change", action="store_true", help="Stop, reconfigure, and restart")
action.add_argument("--chat", action="store_true", help="Send a prompt to a running server")
p.add_argument("--config", default="llama.cpp.config", help="Path to llama.cpp.config")
p.add_argument("--profile", help="Model profile name: [model.<profile>] in config")
p.add_argument(
"--system-prompt-profile",
help="Prompt profile name: [prompt.<profile>] in config (default system prompt)",
)
# --- Prompt sources: exactly one source per field, enforced by argparse
# mutually-exclusive groups plus a semantic validation pass below. ---
system_src = p.add_argument_group("system prompt source")
system_x = system_src.add_mutually_exclusive_group()
system_x.add_argument("-s", "--system", help="Literal system prompt text")
system_x.add_argument("--system-file", help="Read system prompt from a local file")
system_x.add_argument("--system-url", help="Read system prompt from a remote HTTPS URL")
prompt_src = p.add_argument_group("user prompt source")
prompt_x = prompt_src.add_mutually_exclusive_group()
prompt_x.add_argument("-p", "--prompt", help="Literal user prompt text")
prompt_x.add_argument("--prompt-file", help="Read user prompt from a local file")
prompt_x.add_argument("--prompt-url", help="Read user prompt from a remote HTTPS URL")
# --- Docker / server overrides ---
p.add_argument("--image", help="Docker image (e.g. ghcr.io/ggml-org/llama.cpp:server-cuda)")
p.add_argument("--hf-home", help="Host path mounted read-only as model storage")
p.add_argument("--model-path", help="Model path, relative to --hf-home unless absolute")
p.add_argument("--container-name", help="Docker container name (must be unique per config)")
p.add_argument("--host-port", type=int, help="Host port to publish")
p.add_argument("--container-port", type=int, help="Container-internal port")
p.add_argument("--model-alias", help="Model alias exposed via the OpenAI-compatible API")
p.add_argument("--gpu", dest="gpu_device", help="GPU device id passed to --gpus device=<id>")
p.add_argument("--restart-policy", help="Docker --restart policy")
# --- llama.cpp runtime parameters ---
p.add_argument("--ctx-size", type=int)
p.add_argument("--n-predict", type=int)
p.add_argument("--temp", type=float)
p.add_argument("--top-p", type=float)
p.add_argument("--top-k", type=int)
p.add_argument("--min-p", type=float)
p.add_argument("--repeat-penalty", type=float)
p.add_argument("--main-gpu", type=int)
p.add_argument("--ngl", type=int)
p.add_argument("--batch-size", type=int)
p.add_argument("--ubatch-size", type=int)
p.add_argument("--parallel", type=int)
p.add_argument("--cache-type-k")
p.add_argument("--cache-type-v")
p.add_argument("--reasoning", choices=["on", "off"])
p.add_argument("--jinja", dest="jinja", action="store_true", default=None)
p.add_argument("--no-jinja", dest="jinja", action="store_false")
p.add_argument("--fa", dest="fa", action="store_true", default=None)
p.add_argument("--no-fa", dest="fa", action="store_false")
p.add_argument("--kv-unified", dest="kv_unified", action="store_true", default=None)
p.add_argument("--no-kv-unified", dest="kv_unified", action="store_false")
p.add_argument("--cont-batching", dest="cont_batching", action="store_true", default=None)
p.add_argument("--no-cont-batching", dest="cont_batching", action="store_false")
p.add_argument("--no-context-shift", dest="no_context_shift", action="store_true", default=None)
p.add_argument("--context-shift", dest="no_context_shift", action="store_false")
# --- Prompt-input security / behavior options ---
p.add_argument("--allow-insecure-http", action="store_true")
p.add_argument("--allow-private-url", action="store_true")
p.add_argument("--allow-ip-host", action="store_true")
p.add_argument("--follow-redirects", action="store_true")
p.add_argument("--allow-html-input", action="store_true")
p.add_argument("--allow-symlinks", dest="allow_symlinks", action="store_true", default=False)
p.add_argument("--no-allow-symlinks", dest="allow_symlinks", action="store_false")
p.add_argument("--max-input-bytes", type=int, default=1_048_576)
p.add_argument("--url-allow-host", action="append", default=[])
p.add_argument("--connect-timeout", type=float, default=3.0)
p.add_argument("--read-timeout", type=float, default=10.0)
# --- Chat request parameters (for --chat and the --start reply) ---
p.add_argument(
"--max-tokens",
type=int,
help="Max tokens for the --chat / --start reply (default 2048; raise for reasoning models)",
)
p.add_argument(
"--chat-temp",
type=float,
help="Temperature for the chat request (default: use the server's configured --temp)",
)
# --- Behavior / diagnostics ---
p.add_argument("--timeout", type=int, help="Readiness timeout in seconds")
p.add_argument("--poll-interval", type=float, help="Readiness poll interval in seconds")
p.add_argument("--lock-file", help="Override lock file path (default derived from container_name)")
p.add_argument("--logs", action="store_true", help="Print container logs on readiness failure")
p.add_argument("--log-lines", type=int, default=100)
p.add_argument("--dry-run", action="store_true", help="Show the docker run command, do not execute it")
p.add_argument("--force", action="store_true", help="Force stop/change despite inconsistencies")
p.add_argument(
"--print-effective-config",
action="store_true",
help="Print the fully merged configuration as JSON and exit (unless combined with an action)",
)
return p
def validate_args(args: argparse.Namespace, parser: argparse.ArgumentParser) -> None:
"""Enforce semantic rules that argparse's mutually-exclusive groups alone
cannot express (cross-field and cross-action relationships)."""
has_system_source = any(
[args.system is not None, args.system_file is not None, args.system_url is not None]
)
has_prompt_source = any(
[args.prompt is not None, args.prompt_file is not None, args.prompt_url is not None]
)
if args.chat and not has_prompt_source:
parser.error("--chat requires one of --prompt, --prompt-file, or --prompt-url")
if args.stop and (has_system_source or has_prompt_source):
parser.error("--stop does not accept prompt input sources")
if args.check and has_system_source and not has_prompt_source:
parser.error("a system prompt source without a prompt source is not useful for --check")
if args.max_input_bytes <= 0:
parser.error("--max-input-bytes must be > 0")
if args.max_tokens is not None and args.max_tokens <= 0:
parser.error("--max-tokens must be > 0")
if args.connect_timeout <= 0 or args.read_timeout <= 0:
parser.error("timeouts must be > 0")
if args.allow_private_url and args.url_allow_host:
parser.error("--allow-private-url cannot be combined with --url-allow-host")
if args.host_port is not None and not (1 <= args.host_port <= 65535):
parser.error("--host-port must be between 1 and 65535")
if args.container_port is not None and not (1 <= args.container_port <= 65535):
parser.error("--container-port must be between 1 and 65535")
if args.container_name is not None and not args.container_name.strip():
parser.error("--container-name must not be empty")

249
src/llamacppctl/config.py Normal file
View file

@ -0,0 +1,249 @@
"""llama.cpp.config loading and merging.
Resolution order (lowest to highest priority):
1. Built-in defaults (mirrors the user's original shell-script defaults)
2. [default] section in llama.cpp.config
3. [model.<profile>] section (selected via --profile)
4. CLI overrides (--image, --host-port, ...)
[prompt.<name>] sections are resolved separately and only ever provide a
*default* system prompt, which explicit --system/--system-file/--system-url
CLI sources always override.
container_name is mandatory and must be non-empty after resolution: it is
the single identity anchor used for locking, start/stop/change and Docker
container naming. Config authors that define multiple [model.*] profiles
intended to run concurrently MUST give each one a distinct container_name.
"""
from __future__ import annotations
import configparser
import os
from pathlib import Path
from typing import Optional
from .schema import PromptConfig, ServerConfig
TRUE_STRINGS = {"1", "true", "yes", "on"}
FALSE_STRINGS = {"0", "false", "no", "off"}
class ConfigError(ValueError):
"""Raised for missing profiles, invalid values, or invalid container_name."""
def builtin_defaults() -> dict:
"""Mirrors the original start-llm-server.sh / status-llm-server.sh defaults."""
return {
"image": "ghcr.io/ggml-org/llama.cpp:server-cuda",
"hf_home": "/models",
"model_path": "qwen3/default.gguf",
"container_name": "va_llm",
"host_port": "8001",
"container_port": "8000",
"model_alias": "va_llm",
"gpu_device": "1",
"restart_policy": "unless-stopped",
"ctx_size": "262144",
"n_predict": "16384",
"temp": "0.65",
"top_p": "0.80",
"top_k": "20",
"min_p": "0.01",
"repeat_penalty": "1.05",
"main_gpu": "0",
"ngl": "999",
"cache_type_k": "q4_0",
"cache_type_v": "q4_0",
"batch_size": "1024",
"ubatch_size": "512",
"parallel": "1",
"reasoning": "on",
"jinja": "true",
"fa": "true",
"kv_unified": "true",
"cont_batching": "true",
"no_context_shift": "true",
"host": "0.0.0.0",
"health_endpoint": "/health",
"models_endpoint": "/v1/models",
"chat_endpoint": "/v1/chat/completions",
"timeout": "300",
"poll_interval": "2",
"lock_file": "", # derived from container_name if empty
# Chat-request defaults (PromptConfig, not container flags):
"max_tokens": "2048", # reply budget; raise for reasoning/long output
"chat_temperature": "", # empty => defer to the server's --temp
}
def load_ini(path: Path) -> configparser.ConfigParser:
cp = configparser.ConfigParser()
if path.exists():
cp.read(path, encoding="utf-8")
return cp
def _to_bool(value, field_name: str) -> bool:
if isinstance(value, bool):
return value
s = str(value).strip().lower()
if s in TRUE_STRINGS:
return True
if s in FALSE_STRINGS:
return False
raise ConfigError(f"invalid boolean for {field_name!r}: {value!r}")
def _apply_cli_overrides(merged: dict, args) -> None:
overrides = {
"image": getattr(args, "image", None),
"hf_home": getattr(args, "hf_home", None),
"model_path": getattr(args, "model_path", None),
"container_name": getattr(args, "container_name", None),
"host_port": getattr(args, "host_port", None),
"container_port": getattr(args, "container_port", None),
"model_alias": getattr(args, "model_alias", None),
"gpu_device": getattr(args, "gpu_device", None),
"restart_policy": getattr(args, "restart_policy", None),
"ctx_size": getattr(args, "ctx_size", None),
"n_predict": getattr(args, "n_predict", None),
"temp": getattr(args, "temp", None),
"top_p": getattr(args, "top_p", None),
"top_k": getattr(args, "top_k", None),
"min_p": getattr(args, "min_p", None),
"repeat_penalty": getattr(args, "repeat_penalty", None),
"main_gpu": getattr(args, "main_gpu", None),
"ngl": getattr(args, "ngl", None),
"batch_size": getattr(args, "batch_size", None),
"ubatch_size": getattr(args, "ubatch_size", None),
"parallel": getattr(args, "parallel", None),
"cache_type_k": getattr(args, "cache_type_k", None),
"cache_type_v": getattr(args, "cache_type_v", None),
"reasoning": getattr(args, "reasoning", None),
"jinja": getattr(args, "jinja", None),
"fa": getattr(args, "fa", None),
"kv_unified": getattr(args, "kv_unified", None),
"cont_batching": getattr(args, "cont_batching", None),
"no_context_shift": getattr(args, "no_context_shift", None),
"timeout": getattr(args, "timeout", None),
"poll_interval": getattr(args, "poll_interval", None),
"lock_file": getattr(args, "lock_file", None),
}
for key, value in overrides.items():
if value is not None:
merged[key] = value
def build_server_config(merged: dict) -> ServerConfig:
container_name = str(merged.get("container_name", "")).strip()
if not container_name:
raise ConfigError(
"container_name is mandatory and must not be empty "
"(set it in [default], in a [model.<profile>] section, or via --container-name)"
)
# Support environment variables (e.g. hf_home = ${HF_HOME}) and ~ in paths.
hf_home = Path(os.path.expanduser(os.path.expandvars(str(merged["hf_home"]))))
lock_file_value = str(merged.get("lock_file") or "").strip()
if lock_file_value:
lock_file = Path(lock_file_value)
else:
lock_file = Path(f"/tmp/llamacppctl.{container_name}.lock")
return ServerConfig(
image=str(merged["image"]),
hf_home=hf_home,
model_path=str(merged["model_path"]),
container_name=container_name,
host_port=int(merged["host_port"]),
container_port=int(merged["container_port"]),
model_alias=str(merged["model_alias"]),
gpu_device=str(merged["gpu_device"]),
restart_policy=str(merged["restart_policy"]),
ctx_size=int(merged["ctx_size"]),
n_predict=int(merged["n_predict"]),
temp=float(merged["temp"]),
top_p=float(merged["top_p"]),
top_k=int(merged["top_k"]),
min_p=float(merged["min_p"]),
repeat_penalty=float(merged["repeat_penalty"]),
main_gpu=int(merged["main_gpu"]),
ngl=int(merged["ngl"]),
batch_size=int(merged["batch_size"]),
ubatch_size=int(merged["ubatch_size"]),
parallel=int(merged["parallel"]),
cache_type_k=str(merged["cache_type_k"]),
cache_type_v=str(merged["cache_type_v"]),
reasoning="on" if str(merged["reasoning"]).lower() in ("on", "true", "1", "yes") else "off",
jinja=_to_bool(merged["jinja"], "jinja"),
fa=_to_bool(merged["fa"], "fa"),
kv_unified=_to_bool(merged["kv_unified"], "kv_unified"),
cont_batching=_to_bool(merged["cont_batching"], "cont_batching"),
no_context_shift=_to_bool(merged["no_context_shift"], "no_context_shift"),
host=str(merged["host"]),
health_endpoint=str(merged["health_endpoint"]),
models_endpoint=str(merged["models_endpoint"]),
chat_endpoint=str(merged["chat_endpoint"]),
timeout=int(merged["timeout"]),
poll_interval=float(merged["poll_interval"]),
lock_file=lock_file,
)
def resolve_effective_config(args) -> tuple:
"""Returns (ServerConfig, PromptConfig) fully resolved from
builtin defaults -> [default] -> [model.<profile>] -> CLI overrides,
with [prompt.<profile>] providing only a fallback system prompt."""
defaults = builtin_defaults()
cp = load_ini(Path(args.config))
merged = dict(defaults)
if cp.has_section("default"):
merged.update(dict(cp["default"]))
profile = getattr(args, "profile", None)
if profile:
sec = f"model.{profile}"
if not cp.has_section(sec):
raise ConfigError(f"model profile not found: {profile}")
merged.update(dict(cp[sec]))
prompt_defaults = {}
prompt_profile = getattr(args, "system_prompt_profile", None)
if prompt_profile:
sec = f"prompt.{prompt_profile}"
if not cp.has_section(sec):
raise ConfigError(f"prompt profile not found: {prompt_profile}")
prompt_defaults.update(dict(cp[sec]))
_apply_cli_overrides(merged, args)
server_cfg = build_server_config(merged)
system_prompt: Optional[str] = getattr(args, "_resolved_system_prompt", None)
if system_prompt is None:
system_prompt = prompt_defaults.get("system_prompt")
# Chat-request params: config value first, CLI flag overrides.
max_tokens = int(merged.get("max_tokens") or 2048)
if getattr(args, "max_tokens", None) is not None:
max_tokens = args.max_tokens
ct_raw = str(merged.get("chat_temperature") or "").strip()
chat_temp: Optional[float] = float(ct_raw) if ct_raw else None
if getattr(args, "chat_temp", None) is not None:
chat_temp = args.chat_temp
prompt_cfg = PromptConfig(
system_prompt=system_prompt,
user_prompt=getattr(args, "_resolved_user_prompt", None),
max_tokens=max_tokens,
temperature=chat_temp,
)
return server_cfg, prompt_cfg

178
src/llamacppctl/docker_ops.py Normal file
View file

@ -0,0 +1,178 @@
"""Docker container lifecycle for the llama.cpp server.
All Docker interaction goes through subprocess.run() with argument lists
(never shell strings), so there is no shell-quoting/injection surface here.
"""
from __future__ import annotations
import json
import shlex
import subprocess
from dataclasses import dataclass
from pathlib import Path
from typing import Optional
from .schema import ServerConfig
class DockerError(RuntimeError):
pass
@dataclass
class ContainerInfo:
name: str
status: str # "running", "exited", "missing", "unknown"
health: str # "healthy", "unhealthy", "starting", "none"
def _run(cmd: list, check: bool = True) -> subprocess.CompletedProcess:
try:
return subprocess.run(cmd, text=True, capture_output=True, check=check)
except FileNotFoundError as exc:
raise DockerError("docker executable not found on PATH") from exc
except subprocess.CalledProcessError as exc:
raise DockerError(
f"command failed ({' '.join(cmd)}): {exc.stderr or exc.stdout}"
) from exc
def docker_available() -> bool:
try:
cp = subprocess.run(["docker", "version"], text=True, capture_output=True)
return cp.returncode == 0
except FileNotFoundError:
return False
def container_exists(name: str) -> bool:
cp = _run(["docker", "ps", "-a", "--format", "{{.Names}}"], check=True)
return name in cp.stdout.splitlines()
def container_running(name: str) -> bool:
cp = _run(["docker", "ps", "--format", "{{.Names}}"], check=True)
return name in cp.stdout.splitlines()
def inspect_container(name: str) -> ContainerInfo:
if not container_exists(name):
return ContainerInfo(name=name, status="missing", health="none")
cp = _run(["docker", "inspect", name], check=True)
try:
data = json.loads(cp.stdout)[0]
except (json.JSONDecodeError, IndexError, KeyError) as exc:
raise DockerError(f"failed to parse docker inspect output for {name}") from exc
state = data.get("State", {})
status = state.get("Status", "unknown")
health = state.get("Health", {}).get("Status", "none")
return ContainerInfo(name=name, status=status, health=health)
def remove_container(name: str) -> bool:
if not container_exists(name):
return False
_run(["docker", "rm", "-f", name], check=True)
return True
def stop_container(name: str, remove: bool = True) -> bool:
"""Stop (and by default remove) a container. No-op if it does not exist."""
if not container_exists(name):
return False
if remove:
_run(["docker", "rm", "-f", name], check=True)
else:
_run(["docker", "stop", name], check=True)
return True
def container_logs(name: str, tail: int = 100) -> str:
cp = _run(["docker", "logs", "--tail", str(tail), name], check=False)
return (cp.stdout or "") + (cp.stderr or "")
# Backwards-compatible alias matching the original spec naming.
tail_logs = container_logs
def _resolve_model_path_in_container(cfg: ServerConfig) -> str:
model_path = Path(cfg.model_path)
if model_path.is_absolute():
return str(model_path)
return f"/hf_home/{cfg.model_path}"
def build_run_command(cfg: ServerConfig) -> list:
"""Build the `docker run` argument list for the llama.cpp server.
Mirrors the original start-llm-server.sh defaults: GPU pinning via
--gpus device=<id>, HF_HOME mounted read-only, OpenAI-compatible
server flags for sampling and context parameters.
"""
model_in_container = _resolve_model_path_in_container(cfg)
cmd = [
"docker", "run", "-d",
"--gpus", f"device={cfg.gpu_device}",
"--name", cfg.container_name,
"--restart", cfg.restart_policy,
"-e", "HF_HOME=/hf_home",
"-v", f"{cfg.hf_home}:/hf_home:ro",
"-p", f"{cfg.host_port}:{cfg.container_port}",
cfg.image,
"-m", model_in_container,
"--alias", cfg.model_alias,
"-c", str(cfg.ctx_size),
"-n", str(cfg.n_predict),
"--temp", str(cfg.temp),
"--top-p", str(cfg.top_p),
"--top-k", str(cfg.top_k),
"--min-p", str(cfg.min_p),
"--repeat-penalty", str(cfg.repeat_penalty),
"--main-gpu", str(cfg.main_gpu),
"-ngl", str(cfg.ngl),
"--cache-type-k", cfg.cache_type_k,
"--cache-type-v", cfg.cache_type_v,
"--batch-size", str(cfg.batch_size),
"--ubatch-size", str(cfg.ubatch_size),
"--parallel", str(cfg.parallel),
"--host", cfg.host,
"--port", str(cfg.container_port),
]
if cfg.jinja:
cmd.append("--jinja")
cmd += ["--reasoning", cfg.reasoning]
if cfg.no_context_shift:
cmd.append("--no-context-shift")
if cfg.fa:
cmd += ["-fa", "on"]
if cfg.kv_unified:
cmd.append("--kv-unified")
if cfg.cont_batching:
cmd.append("--cont-batching")
cmd.extend(cfg.extra_args)
return cmd
def format_command_for_display(cmd: list) -> str:
return " ".join(shlex.quote(part) for part in cmd)
def start_llama_container(cfg: ServerConfig) -> str:
"""Remove any pre-existing container of the same name, then start fresh.
Returns the new container ID.
"""
if container_exists(cfg.container_name):
remove_container(cfg.container_name)
cp = _run(build_run_command(cfg), check=True)
return cp.stdout.strip()
# Backwards-compatible alias matching the original spec naming.
start_container = start_llama_container

116
src/llamacppctl/http_ops.py Normal file
View file

@ -0,0 +1,116 @@
"""HTTP health checks and chat completions against the llama.cpp OpenAI-compatible API."""
from __future__ import annotations
import time
from typing import Optional
import requests
from .schema import PromptConfig, ServerConfig
class HttpError(RuntimeError):
pass
def base_url(cfg: ServerConfig) -> str:
return f"http://127.0.0.1:{cfg.host_port}/v1"
def chat_url(cfg: ServerConfig) -> str:
"""Chat completions endpoint, honoring the configurable cfg.chat_endpoint
(consistent with health_endpoint / models_endpoint). Always talks to the
locally published host port on loopback."""
return f"http://127.0.0.1:{cfg.host_port}{cfg.chat_endpoint}"
def check_health(cfg: ServerConfig, timeout: float = 3.0) -> bool:
try:
r = requests.get(f"http://127.0.0.1:{cfg.host_port}{cfg.health_endpoint}", timeout=timeout)
return r.ok
except requests.RequestException:
return False
def check_models(cfg: ServerConfig, timeout: float = 3.0) -> bool:
try:
r = requests.get(f"http://127.0.0.1:{cfg.host_port}{cfg.models_endpoint}", timeout=timeout)
return r.ok
except requests.RequestException:
return False
def _build_messages(system_prompt: Optional[str], user_prompt: Optional[str]) -> list:
messages = []
if system_prompt:
messages.append({"role": "system", "content": system_prompt})
if user_prompt:
messages.append({"role": "user", "content": user_prompt})
if not messages:
messages = [{"role": "user", "content": "ping"}]
return messages
def _chat_payload(cfg: ServerConfig, prompt_cfg: PromptConfig) -> dict:
payload = {
"model": cfg.model_alias,
"messages": _build_messages(prompt_cfg.system_prompt, prompt_cfg.user_prompt),
"max_tokens": prompt_cfg.max_tokens,
"stream": prompt_cfg.stream,
}
# Only override temperature when explicitly set; otherwise defer to the
# server's configured --temp so there is a single source of truth.
if prompt_cfg.temperature is not None:
payload["temperature"] = prompt_cfg.temperature
return payload
def chat_completion(cfg: ServerConfig, prompt_cfg: PromptConfig) -> requests.Response:
"""Low-level chat call returning the raw Response (never raises on HTTP
status). Used by readiness polling, which must treat non-200 as 'not yet'."""
return requests.post(chat_url(cfg), json=_chat_payload(cfg, prompt_cfg), timeout=10)
def chat_completion_text(
cfg: ServerConfig, prompt_cfg: PromptConfig, timeout: float = 30.0
) -> str:
"""High-level chat helper for --chat and the --start/--check post-checks.
Returns the assistant message content, raising HttpError on transport,
HTTP status, JSON decoding, or malformed-response failures."""
try:
resp = requests.post(chat_url(cfg), json=_chat_payload(cfg, prompt_cfg), timeout=timeout)
except requests.RequestException as exc:
raise HttpError(f"chat completion request failed: {exc}") from exc
if resp.status_code != 200:
raise HttpError(f"chat completion returned HTTP {resp.status_code}: {resp.text[:500]}")
try:
data = resp.json()
except ValueError as exc:
raise HttpError("chat completion response was not valid JSON") from exc
try:
return data["choices"][0]["message"]["content"]
except (KeyError, IndexError, TypeError) as exc:
raise HttpError("chat completion response missing choices[0].message.content") from exc
def wait_until_ready(cfg: ServerConfig, poll_prompt: PromptConfig) -> bool:
"""Poll /v1/chat/completions until it returns HTTP 200 or the timeout elapses.
A real chat completion is used (not just a port check) because that is
what proves the model is actually loaded and inference-capable.
"""
deadline = time.time() + cfg.timeout
while time.time() < deadline:
try:
r = chat_completion(cfg, poll_prompt)
if r.status_code == 200:
return True
except requests.RequestException:
pass
time.sleep(cfg.poll_interval)
return False

37
src/llamacppctl/lock_ops.py Normal file
View file

@ -0,0 +1,37 @@
"""Exclusive, non-blocking file locking for --start and --change.
Prevents two concurrent llamacppctl invocations from manipulating the same
container_name at once (mirrors the flock/exit-75 idea from switch-llm.sh).
"""
from __future__ import annotations
import fcntl
from pathlib import Path
class LockError(RuntimeError):
pass
class FileLock:
def __init__(self, path: Path):
self.path = Path(path)
self.fd = None
def __enter__(self) -> "FileLock":
self.path.parent.mkdir(parents=True, exist_ok=True)
self.fd = open(self.path, "w")
try:
fcntl.flock(self.fd.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError as exc:
self.fd.close()
self.fd = None
raise LockError(f"lock busy: {self.path}") from exc
return self
def __exit__(self, exc_type, exc, tb) -> None:
if self.fd:
fcntl.flock(self.fd.fileno(), fcntl.LOCK_UN)
self.fd.close()
self.fd = None

104
src/llamacppctl/main.py Normal file
View file

@ -0,0 +1,104 @@
"""Thin executable entry point for llamacppctl.
Orchestration order:
1. Parse CLI arguments (cli.build_parser)
2. Run semantic validation (cli.validate_args)
3. Check that docker is available
4. Resolve prompt sources (prompt_io) under an InputPolicy built from CLI flags
5. Resolve effective server/prompt config (config.resolve_effective_config)
6. Dispatch to the requested action (actions.do_*)
"""
from __future__ import annotations
import sys
from typing import Optional, Sequence
from . import actions
from .cli import build_parser, validate_args
from .config import ConfigError, resolve_effective_config
from .docker_ops import docker_available
from .prompt_io import InputPolicy, PromptSourceError, load_prompt_source, resolve_source
def _build_input_policy(args) -> InputPolicy:
return InputPolicy(
max_input_bytes=args.max_input_bytes,
connect_timeout=args.connect_timeout,
read_timeout=args.read_timeout,
allow_insecure_http=args.allow_insecure_http,
allow_private_url=args.allow_private_url,
allow_ip_host=args.allow_ip_host,
follow_redirects=args.follow_redirects,
allow_html_input=args.allow_html_input,
allow_symlinks=args.allow_symlinks,
url_allow_hosts=list(args.url_allow_host or []),
)
def _resolve_prompts(args, policy: InputPolicy) -> None:
"""Loads --system*/--prompt* sources and stashes results on args as
_resolved_system_prompt / _resolved_user_prompt for config.py to pick up."""
system_source = resolve_source(args.system, args.system_file, args.system_url)
prompt_source = resolve_source(args.prompt, args.prompt_file, args.prompt_url)
args._resolved_system_prompt = (
load_prompt_source(system_source, policy) if system_source else None
)
args._resolved_user_prompt = (
load_prompt_source(prompt_source, policy) if prompt_source else None
)
def run(argv: Optional[Sequence[str]] = None) -> int:
parser = build_parser()
args = parser.parse_args(argv)
validate_args(args, parser)
if not docker_available():
print("docker is not available on PATH (or the daemon is not reachable)", file=sys.stderr)
return 1
try:
policy = _build_input_policy(args)
_resolve_prompts(args, policy)
except PromptSourceError as exc:
print(f"prompt input error: {exc}", file=sys.stderr)
return 1
try:
server_cfg, prompt_cfg = resolve_effective_config(args)
except ConfigError as exc:
print(f"configuration error: {exc}", file=sys.stderr)
return 1
if args.print_effective_config:
actions.print_effective_config(server_cfg, prompt_cfg)
# An action is always required by argparse; only exit early if the run
# is purely diagnostic (no action would actually do anything).
if not (args.start or args.check or args.stop or args.change or args.chat):
return 0
if args.start:
return actions.do_start(server_cfg, prompt_cfg, args)
if args.check:
result = actions.do_check(server_cfg, prompt_cfg, args)
actions.print_check_result(server_cfg, result)
return 0
if args.stop:
return actions.do_stop(server_cfg, args)
if args.change:
return actions.do_change(server_cfg, prompt_cfg, args)
if args.chat:
return actions.do_chat(server_cfg, prompt_cfg, args)
parser.error("no action specified")
return 2
def main() -> None:
sys.exit(run())
if __name__ == "__main__":
main()

Some files were not shown because too many files have changed in this diff Show more