Compare commits
No commits in common. "bee6a71de02719fd53c0fe2f715e18616baf1c82" and "489af21fd2615777461ec2556fa9360fc0e1a070" have entirely different histories.
bee6a71de0
...
489af21fd2
24 changed files with 41 additions and 1265 deletions
|
|
@ -19,7 +19,7 @@ jobs:
|
||||||
run: pip install -e ".[dev]"
|
run: pip install -e ".[dev]"
|
||||||
|
|
||||||
- name: Ruff (lint)
|
- name: Ruff (lint)
|
||||||
run: ruff check src/ tests/ build_archive.py scripts/eval_prompt_tests.py
|
run: ruff check src/ tests/ build_archive.py
|
||||||
|
|
||||||
- name: Mypy (type check)
|
- name: Mypy (type check)
|
||||||
run: mypy
|
run: mypy
|
||||||
|
|
|
||||||
25
CHANGELOG.md
25
CHANGELOG.md
|
|
@ -11,36 +11,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||||
- Per-profile chat streaming and chat-request timeouts: `stream`, `read_timeout`
|
- Per-profile chat streaming and chat-request timeouts: `stream`, `read_timeout`
|
||||||
and `connect_timeout` are now resolvable in `[default]`/`[model.<profile>]`
|
and `connect_timeout` are now resolvable in `[default]`/`[model.<profile>]`
|
||||||
(CLI `--stream`/`--read-timeout`/`--connect-timeout` still override).
|
(CLI `--stream`/`--read-timeout`/`--connect-timeout` still override).
|
||||||
- Multimodal support: `mmproj` and `mmproj_offload` config keys (CLI `--mmproj`,
|
|
||||||
`--no-mmproj-offload`) pass a vision projector to the server. The path resolves
|
|
||||||
under `hf_home` like `model_path`, and a configured-but-missing projector now
|
|
||||||
fails with exit code 3 — for `--change` before the running container is removed.
|
|
||||||
- `docs/KI_TOOLS_PROFILES.md`: which local model suits which task, plus mmproj
|
|
||||||
compatibility per base model.
|
|
||||||
- `docs/EVAL_RUBRIC.md` and `scripts/eval_prompt_tests.py`: scoring rubric and an
|
|
||||||
evaluator for the manual prompt-test archive that measures length compliance and
|
|
||||||
**executes** the model-generated code against its own pytest suite.
|
|
||||||
- `scripts/run_prompt_suite.sh`: runs a prompt domain against a running profile and
|
|
||||||
stores the outputs under the archive's naming convention.
|
|
||||||
- `[model.qwen35base]` profile: the non-abliterated Qwen3.6-35B-A3B (bartowski
|
|
||||||
imatrix Q4_K_M) as a control for the abliteration comparison. Inherits sampling,
|
|
||||||
`ctx_size` and cache types from `[default]`; only model, container, port and GPU
|
|
||||||
differ, so it runs alongside the production server.
|
|
||||||
|
|
||||||
### Changed
|
### Changed
|
||||||
- The chat request now honors the configured `(connect, read)` timeout instead
|
- The chat request now honors the configured `(connect, read)` timeout instead
|
||||||
of a hard-coded 30 s; the default `read_timeout` is 600 s so slow reasoning
|
of a hard-coded 30 s; the default `read_timeout` is 600 s so slow reasoning
|
||||||
models are no longer cut off mid-generation.
|
models are no longer cut off mid-generation.
|
||||||
- **Breaking:** `--print-effective-config` is now an action in the mutually
|
|
||||||
exclusive action group instead of a flag, and it skips the `docker_available()`
|
|
||||||
check. `--print-effective-config --start` is therefore an argparse error
|
|
||||||
(exit 2) rather than a config dump followed by a real container start.
|
|
||||||
|
|
||||||
### Fixed
|
|
||||||
- `--print-effective-config` combined with an action silently performed that
|
|
||||||
action. Documented as a diagnostic command (README, installation guide, manual),
|
|
||||||
`--print-effective-config --config … --start` printed the resolved config and
|
|
||||||
then ran `do_start()`, replacing a running container with the `[default]` model.
|
|
||||||
|
|
||||||
## [0.1.0] - 2026-07-07
|
## [0.1.0] - 2026-07-07
|
||||||
|
|
||||||
|
|
|
||||||
104
CLAUDE.md
104
CLAUDE.md
|
|
@ -1,104 +0,0 @@
|
||||||
# CLAUDE.md
|
|
||||||
|
|
||||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
||||||
|
|
||||||
## Commands
|
|
||||||
|
|
||||||
```bash
|
|
||||||
pip install -e ".[dev]" # dev install
|
|
||||||
scripts/check.sh # full local CI gate (ruff + mypy + pytest) — mirrors .forgejo/workflows/ci.yml
|
|
||||||
python -m pytest -q # tests
|
|
||||||
python -m pytest tests/test_prompt_urls.py -q # one file
|
|
||||||
python -m pytest tests/test_actions.py::test_do_start -q # one test
|
|
||||||
ruff check src/ tests/ build_archive.py
|
|
||||||
mypy # config in pyproject.toml (files = src/llamacppctl)
|
|
||||||
```
|
|
||||||
|
|
||||||
Always use `python -m pytest`, never the `pytest` console script: the test modules do
|
|
||||||
`from tests.test_docker_ops import ...`, which needs the repo root on `sys.path`.
|
|
||||||
|
|
||||||
`git config core.hooksPath .githooks` (once per clone) enables the pre-push hook that runs
|
|
||||||
`scripts/check.sh`. CI runs on Python 3.10 and 3.12; `requires-python = ">=3.10"`.
|
|
||||||
|
|
||||||
Other entry points:
|
|
||||||
- `python3 build_archive.py` — builds and self-verifies `llamacppctl-installable.tar.gz`
|
|
||||||
(has its own required-file manifest; adding a top-level file that must ship means updating it).
|
|
||||||
- `SMOKE_GPU=1 scripts/smoke.sh` — opt-in end-to-end test against a real Docker + GPU + GGUF model.
|
|
||||||
Not part of the pytest suite.
|
|
||||||
|
|
||||||
## Architecture
|
|
||||||
|
|
||||||
A single CLI (`llamacppctl`) that starts/checks/stops/switches llama.cpp servers running as Docker
|
|
||||||
containers, plus a security-hardened input layer for system/user prompts from text, file, or URL.
|
|
||||||
|
|
||||||
`main.py` is a thin pipeline: parse (`cli.build_parser`) → validate (`cli.validate_args`) →
|
|
||||||
`docker_available()` → resolve prompt sources (`prompt_io`) → resolve config
|
|
||||||
(`config.resolve_effective_config`) → dispatch to `actions.do_*`.
|
|
||||||
|
|
||||||
Layering is deliberate and must be preserved — it keeps the security-critical input validation
|
|
||||||
independently testable from orchestration:
|
|
||||||
- `prompt_io.py` knows nothing about argparse; it works only with `PromptSource` + `InputPolicy`.
|
|
||||||
- `config.py` knows nothing about Docker or HTTP.
|
|
||||||
- `docker_ops.py` / `http_ops.py` know nothing about CLI semantics.
|
|
||||||
- `actions.py` orchestrates; `schema.py` holds the shared dataclasses (`ServerConfig`,
|
|
||||||
`PromptConfig`, `ChatReply`, `CheckResult`).
|
|
||||||
|
|
||||||
Two cross-module couplings to know about:
|
|
||||||
- `main._resolve_prompts()` stashes loaded prompts on the argparse namespace as
|
|
||||||
`args._resolved_system_prompt` / `args._resolved_user_prompt`; `config.py` picks them up there.
|
|
||||||
- `main.run()` calls `docker_available()` *before* config resolution, so `--dry-run` still
|
|
||||||
needs a reachable Docker daemon. Only `--print-effective-config` skips that check — it is a
|
|
||||||
standalone action in the mutually-exclusive group, never a modifier, so it cannot start
|
|
||||||
anything.
|
|
||||||
|
|
||||||
### Config resolution
|
|
||||||
|
|
||||||
INI file (`llama.cpp.config`, template in `llama.cpp.config.example`), lowest to highest priority:
|
|
||||||
builtin defaults (`config.builtin_defaults()`) → `[default]` → `[model.<profile>]` (via `--profile`)
|
|
||||||
→ CLI overrides. `[prompt.<name>]` sections resolve separately and only supply a *fallback* system
|
|
||||||
prompt — an explicit `-s`/`--system-file`/`--system-url` always wins.
|
|
||||||
|
|
||||||
`container_name` is mandatory (empty → `ConfigError`). It is the only identity anchor: Docker
|
|
||||||
`--name`, the `--stop`/`--check` target, and the derived lock path
|
|
||||||
`/tmp/llamacppctl.<container_name>.lock`. Profiles meant to run in parallel need distinct
|
|
||||||
`container_name` *and* `host_port`, otherwise a second `--start` silently replaces the first
|
|
||||||
container.
|
|
||||||
|
|
||||||
`max_tokens` / `chat_temperature` / `connect_timeout` / `read_timeout` affect only the chat request
|
|
||||||
(`--chat` and the reply printed after `--start`), never the container.
|
|
||||||
|
|
||||||
### Exit codes (`actions.py`)
|
|
||||||
|
|
||||||
`0` ok · `1` general/config/prompt error · `2` argparse validation · `3` model path missing ·
|
|
||||||
`4` docker error · `5` HTTP not ready (also `--check` failing) · `6` lock busy.
|
|
||||||
`--check` is meant to be scriptable; keep those codes stable.
|
|
||||||
|
|
||||||
## Invariants worth not breaking
|
|
||||||
|
|
||||||
- All Docker calls go through `subprocess.run()` with **argument lists** — never `shell=True`,
|
|
||||||
never composed shell strings.
|
|
||||||
- The port publishes to `127.0.0.1` by default (`docker_ops._port_publish()`); the llama.cpp
|
|
||||||
OpenAI endpoint is unauthenticated unless `api_key` is set. `--expose` binds all interfaces.
|
|
||||||
- Every HTTP request in `prompt_io` — including each redirect hop — passes `validate_url_target()`:
|
|
||||||
HTTPS-only, no IP literals, no embedded credentials, no `localhost`, all resolved addresses
|
|
||||||
classified (private/loopback/link-local/multicast/reserved/unspecified → reject), metadata IP
|
|
||||||
`169.254.169.254` blocked. `_pin_dns()` then restricts `getaddrinfo` to the already-validated IPs
|
|
||||||
for the duration of the request, closing the TOCTOU/DNS-rebinding window. Size limits are enforced
|
|
||||||
both from `Content-Length` and while streaming.
|
|
||||||
- Escape hatches (`--allow-private-url`, `--allow-insecure-http`, `--allow-ip-host`,
|
|
||||||
`--allow-symlinks`, `--follow-redirects`, `--allow-html-input`) are opt-in and off by default.
|
|
||||||
`--allow-private-url` and `--url-allow-host` are mutually exclusive.
|
|
||||||
- `--start` and `--change` hold an exclusive non-blocking `fcntl.flock`; `--change` validates the
|
|
||||||
model path *before* removing the running container. `--dry-run` takes no lock and touches nothing.
|
|
||||||
|
|
||||||
Tests mock `subprocess` and `requests` (and DNS resolution for the SSRF cases); nothing in the
|
|
||||||
pytest suite requires Docker or the network.
|
|
||||||
|
|
||||||
## Conventions
|
|
||||||
|
|
||||||
- Docs, comments in `docs/`, and the example prompts are in German; source code, docstrings, and
|
|
||||||
commit subjects are in English. Commits follow Conventional Commits (`feat(chat):`, `docs(prompts):`).
|
|
||||||
- Line length 100, `from __future__ import annotations` in every module, `py.typed` is shipped —
|
|
||||||
keep annotations complete enough for `mypy` to pass.
|
|
||||||
- Behavior changes should be reflected in `man/llamacppctl.1`, `CHANGELOG.md`, and — for security-
|
|
||||||
relevant ones — `docs/SECURITY_AND_OPERATIONS.md`.
|
|
||||||
11
README.md
11
README.md
|
|
@ -44,10 +44,9 @@ Voraussetzungen auf dem Zielsystem:
|
||||||
- Docker (CLI + laufender Daemon), für `--start`/--check`/--stop`/--change`
|
- Docker (CLI + laufender Daemon), für `--start`/--check`/--stop`/--change`
|
||||||
- Netzwerkzugriff auf den Container-Host-Port für `--chat`/--check`
|
- Netzwerkzugriff auf den Container-Host-Port für `--chat`/--check`
|
||||||
|
|
||||||
`--print-effective-config` ist eine eigenständige, nebenwirkungsfreie Aktion und
|
`--print-effective-config` und `--dry-run` benötigen keinen laufenden Docker-Daemon
|
||||||
benötigt keinen laufenden Docker-Daemon. `--start`/--check`/--stop`/--change`
|
für die reine Konfigurationsprüfung; `--start`/--check`/--stop`/--change` erfordern
|
||||||
erfordern einen erreichbaren Docker-Daemon — auch zusammen mit `--dry-run`, weil
|
einen erreichbaren Docker-Daemon.
|
||||||
die Verfügbarkeit geprüft wird, bevor der Trockenlauf greift.
|
|
||||||
|
|
||||||
## Konfiguration
|
## Konfiguration
|
||||||
|
|
||||||
|
|
@ -94,8 +93,8 @@ llamacppctl --start --config llama.cpp.config
|
||||||
# Nur den geplanten docker-run-Befehl anzeigen, nichts ausführen
|
# Nur den geplanten docker-run-Befehl anzeigen, nichts ausführen
|
||||||
llamacppctl --start --config llama.cpp.config --dry-run
|
llamacppctl --start --config llama.cpp.config --dry-run
|
||||||
|
|
||||||
# Effektive Konfiguration als JSON ausgeben (startet nichts, braucht kein Docker)
|
# Effektive Konfiguration als JSON ausgeben
|
||||||
llamacppctl --print-effective-config --config llama.cpp.config
|
llamacppctl --print-effective-config --config llama.cpp.config --start
|
||||||
|
|
||||||
# Status prüfen
|
# Status prüfen
|
||||||
llamacppctl --check --config llama.cpp.config
|
llamacppctl --check --config llama.cpp.config
|
||||||
|
|
|
||||||
|
|
@ -49,7 +49,7 @@ Siehe [`INSTALL_FROM_ARCHIVE.md`](INSTALL_FROM_ARCHIVE.md).
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
llamacppctl --help
|
llamacppctl --help
|
||||||
llamacppctl --print-effective-config --config llama.cpp.config
|
llamacppctl --print-effective-config --config llama.cpp.config --start --dry-run
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
|
||||||
|
|
@ -1,108 +0,0 @@
|
||||||
# Bewertungsrubrik für Modell-Prompt-Tests
|
|
||||||
|
|
||||||
Zweck: Modellwechsel (anderes Modell, andere Quantisierung, `Balanced` statt
|
|
||||||
`Aggressive`, MTP, KV-Cache-Einstellungen) sollen an **Zahlen** entschieden
|
|
||||||
werden, nicht an Bauchgefühl. Ohne Vorher-Nachher-Messung ist nicht feststellbar,
|
|
||||||
ob eine Änderung verbessert oder verschlechtert hat.
|
|
||||||
|
|
||||||
Grundlage sind die Läufe unter `~/llamacppctl_prompt_tests/` gegen die Prompts in
|
|
||||||
`example_user_prompts/`. Modelleignung siehe [`KI_TOOLS_PROFILES.md`](KI_TOOLS_PROFILES.md).
|
|
||||||
|
|
||||||
## Zwei Ebenen
|
|
||||||
|
|
||||||
**Objektiv** — von `scripts/eval_prompt_tests.py` automatisch gemessen, keine
|
|
||||||
Meinung nötig:
|
|
||||||
|
|
||||||
| Metrik | Domäne | Bedeutung |
|
|
||||||
|---|---|---|
|
|
||||||
| Wortzahl, Abweichung vom Ziel | prosa, reden | Der Prompt nennt ein Ziel („etwa 1200 Wörtern"). |
|
|
||||||
| Trunkierungsverdacht | alle | Text endet ohne Satzzeichen → Budget zu klein. |
|
|
||||||
| Tests laufen durch | coding | Der generierte Code wird **wirklich ausgeführt**. |
|
|
||||||
| Anteil bestandener Tests | coding | `passed / (passed + failed)`. |
|
|
||||||
|
|
||||||
**Subjektiv** — von Hand oder per LLM-Judge, Skala 1–5. Die Anker unten sind
|
|
||||||
bewusst so formuliert, dass 3 „brauchbar, aber mit Mängeln" bedeutet; ein
|
|
||||||
durchschnittlich guter Text landet nicht automatisch bei 4.
|
|
||||||
|
|
||||||
## Skala
|
|
||||||
|
|
||||||
| Punkte | Bedeutung |
|
|
||||||
|---|---|
|
|
||||||
| 5 | Kriterium durchgängig erfüllt; kein Eingriff nötig. |
|
|
||||||
| 4 | Erfüllt, mit einzelnen Schwächen, die den Gesamteindruck nicht tragen. |
|
|
||||||
| 3 | Brauchbar, aber erkennbare Mängel; Überarbeitung nötig. |
|
|
||||||
| 2 | Kriterium überwiegend verfehlt; Grundgerüst vorhanden. |
|
|
||||||
| 1 | Verfehlt. |
|
|
||||||
|
|
||||||
## Prosa
|
|
||||||
|
|
||||||
1. **Show, don't tell** — Innenleben ausschließlich über Handlung, Gegenstand,
|
|
||||||
Wahrnehmung. *1 = benannte Gefühle („sie war traurig"); 5 = kein einziges
|
|
||||||
benanntes Gefühl, Zustand trotzdem eindeutig.*
|
|
||||||
2. **Schlussbild** — konkretes, bedeutungstragendes Bild statt Moral oder
|
|
||||||
Zusammenfassung. *1 = explizite Lehre; 5 = Bild, das die Geschichte trägt.*
|
|
||||||
3. **Ton- und Registertreue** — hält die geforderte Tonlage (z. B. „nüchtern,
|
|
||||||
ohne Pathos") über den gesamten Text.
|
|
||||||
4. **Sprachliche Präzision** — konkrete Substantive, keine Füllattribute, keine
|
|
||||||
Klischees.
|
|
||||||
5. **Komposition** — Aufbau trägt; kein Leerlauf, kein abrupter Abbruch.
|
|
||||||
6. **Sprachrichtigkeit (Deutsch)** — Grammatik, Kasus, Idiomatik. *Eigener
|
|
||||||
Punkt, weil abliterierte, primär englisch trainierte Modelle hier auffällig
|
|
||||||
sind.*
|
|
||||||
|
|
||||||
## Reden
|
|
||||||
|
|
||||||
1. **Argumentative Substanz** — trägt der Gedankengang, oder reiht er Behauptungen?
|
|
||||||
2. **Vorweggenommene Einwände** — wird der stärkste Gegeneinwand benannt und
|
|
||||||
beantwortet, nicht der schwächste?
|
|
||||||
3. **Benannte Zielkonflikte** — werden echte Interessengegensätze offen
|
|
||||||
ausgesprochen statt harmonisiert?
|
|
||||||
4. **Rhetorische Mittel** — bewusst und sparsam eingesetzt (Trikolon, Anapher,
|
|
||||||
Antithese), nicht dekorativ.
|
|
||||||
5. **Adressatenbezug** — Sprache, Beispiele und Anrede passen zum Publikum.
|
|
||||||
6. **Sprachrichtigkeit (Deutsch)** — siehe oben.
|
|
||||||
|
|
||||||
## Coding
|
|
||||||
|
|
||||||
1. **Korrektheit** — *primär aus dem automatischen Testlauf.* Ein Modul, dessen
|
|
||||||
eigene Tests durchfallen, kann in diesem Kriterium nicht über 2 kommen.
|
|
||||||
2. **Anforderungsabdeckung** — sind alle Punkte des Prompts umgesetzt?
|
|
||||||
3. **Fehlerbehandlung** — die im Prompt geforderten Fehlerfälle, sauber getrennt.
|
|
||||||
4. **Testqualität** — decken die Tests die genannten Fälle ab, und sind sie
|
|
||||||
konsistent zum eigenen Code (richtiger Exception-Typ, Fixtures erfüllen das
|
|
||||||
Schema)?
|
|
||||||
5. **Keine Halluzinationen** — keine erfundenen APIs, Signaturen,
|
|
||||||
Framework-Aussagen oder Benchmark-Messwerte. *Automatisch nicht prüfbar;
|
|
||||||
erfundene Messwerte sind in den bisherigen Läufen wiederholt aufgetreten.*
|
|
||||||
6. **Lesbarkeit** — Benennung, Struktur, Kommentardichte.
|
|
||||||
|
|
||||||
## Durchführung
|
|
||||||
|
|
||||||
- **Blind bewerten.** Die Modellzuordnung steckt im Dateinamen; beim Scoren
|
|
||||||
ausblenden (`scripts/eval_prompt_tests.py --anonymize` schreibt anonymisierte
|
|
||||||
Kopien mit Zufalls-IDs und eine Auflösungstabelle).
|
|
||||||
- **Ein Kriterium über alle Texte**, nicht ein Text über alle Kriterien. Das
|
|
||||||
hält den Maßstab konstant.
|
|
||||||
- **LLM-Judge:** möglich gegen den lokalen Server, aber der Judge darf **nicht**
|
|
||||||
das bewertete Modell sein. Ein Modell bewertet seine eigenen Texte zu gut.
|
|
||||||
Der Judge ersetzt die menschliche Stichprobe nicht — mindestens 20 % der Texte
|
|
||||||
gegenlesen und die Übereinstimmung prüfen.
|
|
||||||
- **Mindestens zwei Läufe pro Zelle**, weil bei `temp > 0` ein Einzellauf wenig
|
|
||||||
aussagt. Für Coding-Vergleiche `--seed` fixieren.
|
|
||||||
|
|
||||||
## Was die Zahlen nicht sagen
|
|
||||||
|
|
||||||
Ein bestandener Testlauf beweist, dass der Code *seine eigenen* Tests besteht —
|
|
||||||
nicht, dass er korrekt ist. Wenn Modell und Test aus derselben Halluzination
|
|
||||||
stammen, sind beide konsistent falsch. Kriterium 4 (Testqualität) ist deshalb
|
|
||||||
nicht redundant zu Kriterium 1, sondern dessen Korrektiv.
|
|
||||||
|
|
||||||
## Sicherheitshinweis
|
|
||||||
|
|
||||||
`scripts/eval_prompt_tests.py` führt **modellgenerierten Code aus**. Das ist der
|
|
||||||
Sinn der Übung, aber es ist Codeausführung aus einer nicht vertrauenswürdigen
|
|
||||||
Quelle. Der Runner arbeitet in einem temporären Verzeichnis, erzwingt ein
|
|
||||||
Zeitlimit und verweigert den Start als `root`. Er bietet **keine** Netz- oder
|
|
||||||
Dateisystem-Isolation. Wer die Läufe auf einem Rechner mit Produktionsdaten
|
|
||||||
auswertet, sollte `--no-exec` verwenden oder das Skript in einem Container
|
|
||||||
starten.
|
|
||||||
|
|
@ -64,7 +64,7 @@ cp llama.cpp.config.example llama.cpp.config
|
||||||
Auflösung prüfen, ohne etwas zu starten:
|
Auflösung prüfen, ohne etwas zu starten:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
llamacppctl --print-effective-config --config llama.cpp.config
|
llamacppctl --print-effective-config --config llama.cpp.config --start --dry-run
|
||||||
```
|
```
|
||||||
|
|
||||||
Das zeigt die vollständig aufgelöste Konfiguration als JSON **und** das
|
Das zeigt die vollständig aufgelöste Konfiguration als JSON **und** das
|
||||||
|
|
|
||||||
|
|
@ -1,169 +0,0 @@
|
||||||
# Modellprofile — Eignung der lokalen GGUF-Modelle
|
|
||||||
|
|
||||||
Dieses Dokument beschreibt, **wofür** sich die Modelle unter
|
|
||||||
`$HF_HOME/models/qwen3/` jeweils eignen, unabhängig von den Beispiel-Prompts in
|
|
||||||
`example_system_prompts/`. Grundlage sind die Modellkarten auf Hugging Face
|
|
||||||
(Stand Juli 2026) sowie die eigenen Testläufe unter `~/llamacppctl_prompt_tests/`.
|
|
||||||
|
|
||||||
> Die Angaben zu Fähigkeiten und Benchmarks stammen, wo nicht anders vermerkt,
|
|
||||||
> **von den Modellautoren selbst** und sind nicht unabhängig verifiziert.
|
|
||||||
|
|
||||||
## Übersicht
|
|
||||||
|
|
||||||
| Datei | Profil | Basis | Primäre Eignung |
|
|
||||||
|---|---|---|---|
|
|
||||||
| `ornith-1.0-35b-Q4_K_M.gguf` | `ornith35b` | Qwen3.5-MoE | Agentisches Coding |
|
|
||||||
| `Carnice-Qwen3.6-MoE-35B-A3B-Q4_K_M.gguf` | `carnice` | Qwen3.6-35B-A3B | Agenten-/Tool-Calling-Workflows |
|
|
||||||
| `Qwopus3.6-35B-A3B-v1-Q4_K_M.gguf` | `qwopus` | Qwen3.6-35B-A3B | Reasoning, Vision, Tool-Calling |
|
|
||||||
| `Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q4_K_M.gguf` | `[default]` | Qwen3.6-35B-A3B | Kreatives Schreiben ohne Verweigerungen |
|
|
||||||
| `Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-IQ4_XS.gguf` | `qwen27b` | Qwen3.6-27B (dense) | Wie oben, kleinerer VRAM-Bedarf |
|
|
||||||
| `mmproj.gguf` | — | Qwen3.6-35B-A3B | Vision-Projektor (kein eigenständiges Modell) |
|
|
||||||
|
|
||||||
Alle 35B-A3B-Modelle sind MoE mit **3 B aktiven Parametern** pro Token: hoher
|
|
||||||
Durchsatz bei 21 GB VRAM-Bedarf. Das 27B ist dense — langsamer pro Token, aber
|
|
||||||
nur ~15 GB.
|
|
||||||
|
|
||||||
## Ornith-1.0-35B — agentisches Coding
|
|
||||||
|
|
||||||
Quelle: [`deepreinforce-ai/Ornith-1.0-35B-GGUF`](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B-GGUF) (MIT)
|
|
||||||
|
|
||||||
Ein dediziertes Coding-Modell aus einer selbstverbessernden Modellfamilie. Das
|
|
||||||
Trainings-Framework erzeugt per RL nicht nur Lösungs-Rollouts, sondern optimiert
|
|
||||||
auch das treibende Scaffold gemeinsam mit ihnen. Beworben mit SOTA unter
|
|
||||||
vergleichbar großen Open-Source-Modellen auf Terminal-Bench 2.1, SWE-Bench,
|
|
||||||
NL2Repo und OpenClaw. Reasoning-Modell: die Assistant-Antwort beginnt mit einem
|
|
||||||
`<think>`-Block.
|
|
||||||
|
|
||||||
**Einsetzen für:** Code-Generierung, Refactoring, Repo-weite Aufgaben,
|
|
||||||
Agenten-Loops mit Tool-Calls.
|
|
||||||
**Nicht einsetzen für:** Prosa und Reden — dafür ist das Sampling im Profil
|
|
||||||
(`temp = 0.20`) bewusst deterministisch gewählt.
|
|
||||||
|
|
||||||
Beachte: Basis ist **Qwen3.5**-MoE (`qwen3_5_moe`), nicht 3.6. Die lokale
|
|
||||||
`mmproj.gguf` passt deshalb *nicht* zu diesem Modell.
|
|
||||||
|
|
||||||
## Carnice-Qwen3.6-MoE-35B-A3B — Agenten-Workflows
|
|
||||||
|
|
||||||
Quelle: [`samuelcardillo/Carnice-Qwen3.6-MoE-35B-A3B`](https://huggingface.co/samuelcardillo/Carnice-Qwen3.6-MoE-35B-A3B)
|
|
||||||
|
|
||||||
QLoRA-Finetune von Qwen3.6-35B-A3B, trainiert auf echten Execution-Traces der
|
|
||||||
Hermes-Agent-Runtime. Das erklärte Ziel ist, dem Modell die konkreten
|
|
||||||
Konversationsmuster beizubringen, die ein Agenten-Runtime erwartet — nicht
|
|
||||||
generisches Reasoning.
|
|
||||||
|
|
||||||
**Einsetzen für:** Tool-Calling-Pipelines, Agenten-Orchestrierung (z. B. n8n),
|
|
||||||
strukturierte Mehrschritt-Abläufe.
|
|
||||||
|
|
||||||
## Qwopus3.6-35B-A3B-v1 — Reasoning + Vision
|
|
||||||
|
|
||||||
Quelle: [`Jackrong/Qwopus3.6-35B-A3B-v1`](https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-v1)
|
|
||||||
|
|
||||||
Reasoning-verstärkter Finetune, dreistufiges SFT mit progressiv steigender
|
|
||||||
Reasoning-Komplexität. Unterstützt Vision und Tool-Calling.
|
|
||||||
|
|
||||||
**Einsetzen für:** Aufgaben mit langer Gedankenkette, multimodale Eingaben.
|
|
||||||
**Vorbehalt:** Der Autor kennzeichnet das Modell ausdrücklich als experimentell,
|
|
||||||
ohne vollständige Performance- oder Sicherheitsevaluation.
|
|
||||||
|
|
||||||
Für reines Coding gibt es die separate Variante
|
|
||||||
[`Qwopus3.6-35B-A3B-Coder-MTP`](https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-Coder-MTP-GGUF)
|
|
||||||
(thinking-off, 62,4 % auf einem 300-Fall-SWE-Bench-Lauf) — lokal nicht vorhanden.
|
|
||||||
|
|
||||||
## HauhauCS Uncensored (35B-A3B und 27B) — kreatives Schreiben
|
|
||||||
|
|
||||||
Quellen: [35B-A3B](https://huggingface.co/HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive),
|
|
||||||
[27B](https://huggingface.co/HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive)
|
|
||||||
|
|
||||||
**Abliterierte** Modelle: keine neuen Fähigkeiten, entfernte Verweigerungen
|
|
||||||
(Modellkarte: „0/465 Refusals"). Alle Quants mit imatrix erzeugt. Beide sind
|
|
||||||
multimodal.
|
|
||||||
|
|
||||||
**Einsetzen für:** Prosa, Reden, Dialog, Rollenspiel — Themen, bei denen ein
|
|
||||||
aligntes Modell abbricht.
|
|
||||||
**Nicht einsetzen für:** Coding. Eigene Testläufe zeigen wiederkehrend Tests, die
|
|
||||||
inkonsistent zum selbst erzeugten Schema sind, erfundene Framework-Aussagen und
|
|
||||||
Benchmarks mit erfundenen Messwerten.
|
|
||||||
|
|
||||||
Zwei Punkte, die in der aktuellen Konfiguration relevant sind:
|
|
||||||
|
|
||||||
1. Die Karte des 27B empfiehlt selbst, dass **99,9 % der Nutzer die
|
|
||||||
`Balanced`-Variante** nehmen sollten statt `Aggressive`. Beide erreichen
|
|
||||||
dieselbe Refusal-Rate; `Balanced` bietet zusätzlich stabileres Sampling für
|
|
||||||
agentic coding, Tool-Use und Reasoning. `Aggressive` spart lediglich
|
|
||||||
Vorreden. Aktuell sind beide lokalen HauhauCS-Modelle `Aggressive`.
|
|
||||||
2. Die Behauptungen „lossless" und „zero capability loss" stammen vom Autor der
|
|
||||||
Modellkarte. Abliteration kostet in aller Regel *etwas* Fähigkeit.
|
|
||||||
|
|
||||||
## Empirische Befunde (eigene Läufe)
|
|
||||||
|
|
||||||
Getestet gegen das Default-Modell (HauhauCS 35B-A3B), Reasoning aktiv, ctx 262144:
|
|
||||||
|
|
||||||
- **Längenvorgaben werden nicht präzise getroffen** (−27 % bis +26 % über sechs
|
|
||||||
Läufe). Länge als Spanne vorgeben oder im zweiten Turn nachjustieren.
|
|
||||||
- **Reasoning frisst das Antwortbudget.** Bei `--max-tokens 10000` endeten
|
|
||||||
schwere Aufgaben in `finish_reason=length`. Für Prosa/Reden/Dialog
|
|
||||||
`--max-tokens` ≥ 16000 wählen.
|
|
||||||
- **`--chat` ohne `--stream`** läuft bei langem Output in den Read-Timeout.
|
|
||||||
Immer `--stream` verwenden (jeder Token setzt den Timeout zurück).
|
|
||||||
- **Coding: Struktur gut, Korrektheit unzuverlässig.** Generierten Code und
|
|
||||||
generierte Tests immer wirklich ausführen.
|
|
||||||
|
|
||||||
Rohdaten: `~/llamacppctl_prompt_tests/`.
|
|
||||||
|
|
||||||
## Vision (mmproj)
|
|
||||||
|
|
||||||
Die lokale `mmproj.gguf` (902 MB) ist der Vision-Encoder des **Basismodells
|
|
||||||
Qwen3.6-35B-A3B** (aus dem unsloth-Repo). GGUF-Metadaten:
|
|
||||||
|
|
||||||
```
|
|
||||||
general.architecture = clip
|
|
||||||
general.name = Qwen3.6 35B A3B
|
|
||||||
clip.projector_type = qwen3vl_merger
|
|
||||||
clip.has_vision_encoder
|
|
||||||
```
|
|
||||||
|
|
||||||
Ein mmproj ist kein eigenständiges Modell, sondern projiziert Bild-Embeddings in
|
|
||||||
den Textmodell-Raum. Er muss zum Vision-Tower der jeweiligen Basis passen.
|
|
||||||
|
|
||||||
| Modell | mmproj.gguf passt? |
|
|
||||||
|---|---|
|
|
||||||
| HauhauCS 35B-A3B | ja (Basis Qwen3.6-35B-A3B) |
|
|
||||||
| Qwopus3.6-35B-A3B | ja (dieselbe Basis) |
|
|
||||||
| Carnice-Qwen3.6-MoE-35B-A3B | ja (dieselbe Basis) |
|
|
||||||
| `ornith-1.0-35b` | **nein** — Basis Qwen3.5-MoE, eigener Projektor nötig |
|
|
||||||
| HauhauCS 27B (dense) | **nein** — anderer Vision-Encoder |
|
|
||||||
|
|
||||||
Verwendung über `llamacppctl` (siehe `[model.vision]` in `llama.cpp.config`):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
llamacppctl --start --config llama.cpp.config --profile vision
|
|
||||||
```
|
|
||||||
|
|
||||||
Das Profil setzt `mmproj = models/qwen3/mmproj.gguf`; der Pfad wird — wie
|
|
||||||
`model_path` — relativ zu `hf_home` aufgelöst und als `--mmproj` an den
|
|
||||||
llama.cpp-Server durchgereicht. Der Projektor wird standardmäßig auf die GPU
|
|
||||||
ausgelagert; `mmproj_offload = false` bzw. `--no-mmproj-offload` verhindert das,
|
|
||||||
wenn der VRAM knapp ist.
|
|
||||||
|
|
||||||
Bilder gehen anschließend über den OpenAI-kompatiblen Endpunkt
|
|
||||||
`/v1/chat/completions` als `image_url`-Content-Part. `llamacppctl --chat` sendet
|
|
||||||
derzeit nur Text; für Bild-Requests den Endpunkt direkt ansprechen.
|
|
||||||
|
|
||||||
## MTP / Speculative Decoding (nicht lokal vorhanden)
|
|
||||||
|
|
||||||
Für Ornith, Carnice und Qwopus existieren **MTP-Quants** mit eingebettetem
|
|
||||||
Multi-Token-Prediction-Head. Mit hinreichend aktuellem llama.cpp erlauben sie
|
|
||||||
self-speculative Decoding ohne separates Draft-Modell (`--draft-mtp`);
|
|
||||||
Community-Berichte nennen etwa doppelten Decode-Durchsatz bei ~2,5 % größerer
|
|
||||||
Datei. Die im pyproject gepinnte Image-Version muss das unterstützen.
|
|
||||||
|
|
||||||
## Quellen
|
|
||||||
|
|
||||||
- <https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B-GGUF>
|
|
||||||
- <https://huggingface.co/SC117/Ornith-1.0-35B-MTP-APEX-GGUF>
|
|
||||||
- <https://huggingface.co/samuelcardillo/Carnice-Qwen3.6-MoE-35B-A3B>
|
|
||||||
- <https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-v1>
|
|
||||||
- <https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-Coder-MTP-GGUF>
|
|
||||||
- <https://huggingface.co/HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive>
|
|
||||||
- <https://huggingface.co/HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive>
|
|
||||||
- <https://github.com/ggml-org/llama.cpp/blob/master/docs/multimodal.md>
|
|
||||||
|
|
@ -258,29 +258,22 @@ keinen vermeidbaren Ausfall verursacht.
|
||||||
|
|
||||||
## 6. `--dry-run` und `--print-effective-config` als Sicherheitswerkzeuge
|
## 6. `--dry-run` und `--print-effective-config` als Sicherheitswerkzeuge
|
||||||
|
|
||||||
- `--print-effective-config` ist eine **eigene Aktion** in der
|
- `--print-effective-config` gibt die vollständig aufgelöste Konfiguration
|
||||||
Mutually-Exclusive-Gruppe: sie gibt die vollständig aufgelöste Konfiguration
|
(Server- und Prompt-Konfiguration) als JSON aus, **bevor** irgendeine
|
||||||
(Server- und Prompt-Konfiguration) als JSON aus und beendet sich. Damit lässt
|
Docker- oder HTTP-Aktion ausgeführt wird. Damit lässt sich prüfen, welche
|
||||||
sich prüfen, welche Werte aus welcher Quelle (Defaults/[default]/[model.*]/CLI)
|
Werte aus welcher Quelle (Defaults/[default]/[model.*]/CLI) tatsächlich
|
||||||
tatsächlich gewonnen haben, ohne einen Container anzufassen.
|
gewonnen haben, ohne einen Container anzufassen.
|
||||||
|
|
||||||
Bis einschließlich 0.1.0 war dies ein *Flag*. Da argparse immer eine Aktion
|
|
||||||
verlangt, führte `--print-effective-config --start` die Konfigurationsausgabe
|
|
||||||
**und anschließend einen echten `do_start()`** aus — der einen laufenden
|
|
||||||
Container stillschweigend ersetzte. Die Kombination ist jetzt ein
|
|
||||||
argparse-Fehler (Exit-Code 2).
|
|
||||||
- `--dry-run` (in Kombination mit `--start`/--change`) zeigt den vollständig
|
- `--dry-run` (in Kombination mit `--start`/--change`) zeigt den vollständig
|
||||||
zusammengesetzten `docker run`-Befehl (Shell-quotiert zur Anzeige) an,
|
zusammengesetzten `docker run`-Befehl (Shell-quotiert zur Anzeige) an,
|
||||||
**ohne** ihn auszuführen. Empfohlen vor jeder Änderung an einer
|
**ohne** ihn auszuführen. Empfohlen vor jeder Änderung an einer
|
||||||
Produktionskonfiguration, insbesondere nach Anpassungen an
|
Produktionskonfiguration, insbesondere nach Anpassungen an
|
||||||
`llama.cpp.config`.
|
`llama.cpp.config`.
|
||||||
|
|
||||||
`--print-effective-config` erfordert **keinen** erreichbaren Docker-Daemon;
|
Beide Flags erfordern **keinen** erreichbaren Docker-Daemon für ihre reine
|
||||||
`main.run()` überspringt die `docker_available()`-Prüfung für diese Aktion. Für
|
Ausgabe — `main.run()` prüft `docker_available()` derzeit vor der
|
||||||
`--dry-run` gilt das **nicht**: es ist ein Modifikator von `--start`/`--change`,
|
Konfigurationsauflösung; auf einem Host ganz ohne Docker (z. B. zur reinen
|
||||||
und die Docker-Prüfung läuft vor der Konfigurationsauflösung. Auf einem Host ganz
|
Konfigurationsvalidierung) schlägt der Aufruf entsprechend mit einer
|
||||||
ohne Docker schlägt `--start --dry-run` daher mit einer expliziten Fehlermeldung
|
expliziten Fehlermeldung fehl statt still falsche Annahmen zu treffen.
|
||||||
fehl, statt still falsche Annahmen zu treffen.
|
|
||||||
|
|
||||||
## 7. Betriebsmodell auf einem dedizierten GPU-Host
|
## 7. Betriebsmodell auf einem dedizierten GPU-Host
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -88,24 +88,6 @@ host_port = 8003
|
||||||
gpu_device = 1
|
gpu_device = 1
|
||||||
ctx_size = 131072
|
ctx_size = 131072
|
||||||
|
|
||||||
# Vision profile: `mmproj` points at the multimodal projector (vision encoder)
|
|
||||||
# GGUF. Like `model_path` it is resolved relative to `hf_home` unless absolute.
|
|
||||||
# The projector MUST match the base model's vision tower -- a projector built for
|
|
||||||
# a different base will not load. A missing projector aborts with exit code 3
|
|
||||||
# (for --change: before the running container is removed).
|
|
||||||
#
|
|
||||||
# llama.cpp offloads the projector to the GPU by default; set mmproj_offload to
|
|
||||||
# false to keep it on the CPU when VRAM is tight. Images are then sent to
|
|
||||||
# /v1/chat/completions as an image_url content part -- `llamacppctl --chat`
|
|
||||||
# itself only sends text.
|
|
||||||
[model.vision]
|
|
||||||
model_path = qwen3/Qwen3-35B-A3B-Q4_K_M.gguf
|
|
||||||
mmproj = qwen3/mmproj.gguf
|
|
||||||
mmproj_offload = true
|
|
||||||
container_name = llama_cpp_vision
|
|
||||||
host_port = 8004
|
|
||||||
gpu_device = 1
|
|
||||||
|
|
||||||
[prompt.concise]
|
[prompt.concise]
|
||||||
system_prompt = Du antwortest kurz, präzise und technisch.
|
system_prompt = Du antwortest kurz, präzise und technisch.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -27,7 +27,7 @@ run() {
|
||||||
fi
|
fi
|
||||||
}
|
}
|
||||||
|
|
||||||
run "ruff (lint)" "${BIN}ruff" check src/ tests/ build_archive.py scripts/eval_prompt_tests.py
|
run "ruff (lint)" "${BIN}ruff" check src/ tests/ build_archive.py
|
||||||
run "mypy (types)" "${BIN}mypy"
|
run "mypy (types)" "${BIN}mypy"
|
||||||
# Use `python -m pytest` (not the pytest console script) so the repo root is on
|
# Use `python -m pytest` (not the pytest console script) so the repo root is on
|
||||||
# sys.path — the test modules import `from tests.test_docker_ops import ...`.
|
# sys.path — the test modules import `from tests.test_docker_ops import ...`.
|
||||||
|
|
|
||||||
|
|
@ -1,457 +0,0 @@
|
||||||
#!/usr/bin/env python3
|
|
||||||
"""eval_prompt_tests.py
|
|
||||||
|
|
||||||
Auswertung der manuellen Modell-Prompt-Testlaeufe unter
|
|
||||||
``~/llamacppctl_prompt_tests/`` gegen die Prompts in ``example_user_prompts/``.
|
|
||||||
|
|
||||||
Misst die objektive Ebene der Rubrik (docs/EVAL_RUBRIC.md):
|
|
||||||
|
|
||||||
* prosa/reden: Wortzahl und Abweichung vom im Prompt genannten Zielwert,
|
|
||||||
Trunkierungsverdacht.
|
|
||||||
* coding: extrahiert die Python-Codebloecke, schreibt Modul und Tests in
|
|
||||||
ein temporaeres Verzeichnis und **fuehrt pytest tatsaechlich aus**.
|
|
||||||
|
|
||||||
WARNUNG: ``--exec`` fuehrt modellgenerierten Code aus. Der Runner nutzt ein
|
|
||||||
temporaeres Verzeichnis, ein Zeitlimit und verweigert den Start als root, bietet
|
|
||||||
aber keine Netz- oder Dateisystem-Isolation. Siehe docs/EVAL_RUBRIC.md.
|
|
||||||
|
|
||||||
Beispiele
|
|
||||||
---------
|
|
||||||
scripts/eval_prompt_tests.py # messen + Tests ausfuehren
|
|
||||||
scripts/eval_prompt_tests.py --no-exec # nur statisch messen
|
|
||||||
scripts/eval_prompt_tests.py --json report.json # maschinenlesbar
|
|
||||||
scripts/eval_prompt_tests.py --anonymize blind/ # blinde Bewertung vorbereiten
|
|
||||||
|
|
||||||
Exit-Codes
|
|
||||||
----------
|
|
||||||
0 Auswertung durchgelaufen (auch wenn einzelne Testlaeufe rot sind)
|
|
||||||
1 Eingabeverzeichnis fehlt oder unsichere Ausfuehrungsumgebung
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import random
|
|
||||||
import re
|
|
||||||
import shutil
|
|
||||||
import subprocess
|
|
||||||
import sys
|
|
||||||
import tempfile
|
|
||||||
from dataclasses import asdict, dataclass, field
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Optional
|
|
||||||
|
|
||||||
DEFAULT_RESULTS_DIR = Path.home() / "llamacppctl_prompt_tests"
|
|
||||||
DEFAULT_PROMPTS_DIR = Path(__file__).resolve().parents[1] / "example_user_prompts"
|
|
||||||
|
|
||||||
DOMAINS = ("prosa", "reden", "coding")
|
|
||||||
# Dateien ohne Modellpraefix stammen vom Default-Modell aus llama.cpp.config.
|
|
||||||
KNOWN_MODELS = ("carnice", "qwen27b", "qwopus", "ornith", "qwen35base")
|
|
||||||
DEFAULT_MODEL_LABEL = "default"
|
|
||||||
VARIANT_MARKERS = frozenset({"r2", "highbudget", "explizit"})
|
|
||||||
|
|
||||||
# "etwa 1200 Wörtern", "ca. 700 Wörter", "rund 900 Worten"
|
|
||||||
LENGTH_TARGET_RE = re.compile(
|
|
||||||
r"(?:etwa|ca\.|circa|rund)\s*(\d{2,5})\s*(?:Wörter|Wörtern|Worte|Worten)",
|
|
||||||
re.IGNORECASE,
|
|
||||||
)
|
|
||||||
FENCE_RE = re.compile(r"^```([A-Za-z+#]*)\s*$(.*?)^```\s*$", re.MULTILINE | re.DOTALL)
|
|
||||||
PY_FILENAME_RE = re.compile(r"`([A-Za-z_][A-Za-z0-9_]*)\.py`")
|
|
||||||
FROM_IMPORT_RE = re.compile(r"^\s*from\s+([A-Za-z_][A-Za-z0-9_]*)\s+import", re.MULTILINE)
|
|
||||||
PLAIN_IMPORT_RE = re.compile(r"^\s*import\s+([A-Za-z_][A-Za-z0-9_]*)", re.MULTILINE)
|
|
||||||
PYTEST_COUNT_RE = re.compile(r"(\d+)\s+(passed|failed|error|errors)")
|
|
||||||
|
|
||||||
# Endet der Text sauber? Fehlendes Satzzeichen deutet auf finish_reason=length.
|
|
||||||
SENTENCE_END = tuple('.!?"»“’—-)')
|
|
||||||
|
|
||||||
# Ein Modul darf niemals nach einem Stdlib-Modul benannt werden: eine Datei
|
|
||||||
# sqlite3.py im Arbeitsverzeichnis ueberschattet die echte Stdlib und laesst den
|
|
||||||
# Testlauf mit einem Importfehler scheitern, der faelschlich dem Modell
|
|
||||||
# angelastet wuerde. sys.stdlib_module_names ist exakt (Python >= 3.10).
|
|
||||||
RESERVED_MODULE_NAMES = frozenset(sys.stdlib_module_names) | {"pytest", "conftest"}
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class Run:
|
|
||||||
"""Ein einzelner Testlauf, abgeleitet aus dem Dateinamen."""
|
|
||||||
|
|
||||||
path: Path
|
|
||||||
model: str
|
|
||||||
domain: str
|
|
||||||
case: str
|
|
||||||
variant: str = ""
|
|
||||||
|
|
||||||
@property
|
|
||||||
def label(self) -> str:
|
|
||||||
base = f"{self.model}/{self.domain}_{self.case}"
|
|
||||||
return f"{base}[{self.variant}]" if self.variant else base
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class Metrics:
|
|
||||||
label: str
|
|
||||||
model: str
|
|
||||||
domain: str
|
|
||||||
case: str
|
|
||||||
variant: str
|
|
||||||
words: int = 0
|
|
||||||
target_words: Optional[int] = None
|
|
||||||
deviation_pct: Optional[float] = None
|
|
||||||
looks_truncated: bool = False
|
|
||||||
# coding
|
|
||||||
executed: bool = False
|
|
||||||
exec_status: str = "" # ok | failed | timeout | no-tests | skipped-<grund>
|
|
||||||
tests_passed: int = 0
|
|
||||||
tests_failed: int = 0
|
|
||||||
detail: str = ""
|
|
||||||
code_langs: list = field(default_factory=list)
|
|
||||||
|
|
||||||
|
|
||||||
def parse_run(path: Path) -> Optional[Run]:
|
|
||||||
"""Zerlegt ``[<modell>_]<domaene>_<nr>[_<variante>].out.txt``.
|
|
||||||
|
|
||||||
Die Namenskonvention ist historisch uneinheitlich (mal steht der Fallname
|
|
||||||
hinter der Nummer, mal eine Variante wie ``r2``), deshalb wird tolerant
|
|
||||||
geparst statt streng validiert.
|
|
||||||
"""
|
|
||||||
stem = path.name
|
|
||||||
for suffix in (".out.txt", ".txt", ".md"):
|
|
||||||
if stem.endswith(suffix):
|
|
||||||
stem = stem[: -len(suffix)]
|
|
||||||
break
|
|
||||||
|
|
||||||
tokens = stem.split("_")
|
|
||||||
if not tokens:
|
|
||||||
return None
|
|
||||||
|
|
||||||
model = DEFAULT_MODEL_LABEL
|
|
||||||
if tokens[0] in KNOWN_MODELS:
|
|
||||||
model = tokens[0]
|
|
||||||
tokens = tokens[1:]
|
|
||||||
|
|
||||||
domain = next((t for t in tokens if t in DOMAINS), None)
|
|
||||||
if domain is None:
|
|
||||||
return None
|
|
||||||
rest = tokens[tokens.index(domain) + 1 :]
|
|
||||||
|
|
||||||
case = next((t for t in rest if t.isdigit()), None)
|
|
||||||
if case is None:
|
|
||||||
return None
|
|
||||||
|
|
||||||
# Der Fallname (z. B. "jsonl_validator") ist keine Variante. Nur bekannte
|
|
||||||
# Marker zaehlen -- sie muessen erhalten bleiben, sonst kollidieren zwei
|
|
||||||
# verschiedene Laeufe (prosa_04_dialog vs. prosa_04_dialog_highbudget) unter
|
|
||||||
# demselben Label.
|
|
||||||
variant = "_".join(t for t in rest[rest.index(case) + 1 :] if t in VARIANT_MARKERS)
|
|
||||||
|
|
||||||
return Run(path=path, model=model, domain=domain, case=case, variant=variant)
|
|
||||||
|
|
||||||
|
|
||||||
def load_length_targets(prompts_dir: Path) -> dict:
|
|
||||||
"""Mappt (domaene, nr) auf die im Prompt genannte Ziel-Wortzahl."""
|
|
||||||
targets: dict = {}
|
|
||||||
if not prompts_dir.is_dir():
|
|
||||||
return targets
|
|
||||||
for prompt in sorted(prompts_dir.glob("*.md")):
|
|
||||||
tokens = prompt.stem.split("_")
|
|
||||||
if len(tokens) < 2 or tokens[0] not in DOMAINS or not tokens[1].isdigit():
|
|
||||||
continue
|
|
||||||
match = LENGTH_TARGET_RE.search(prompt.read_text(encoding="utf-8"))
|
|
||||||
if match:
|
|
||||||
targets[(tokens[0], tokens[1])] = int(match.group(1))
|
|
||||||
return targets
|
|
||||||
|
|
||||||
|
|
||||||
def count_words(text: str) -> int:
|
|
||||||
"""Woerter ohne Markdown-Auszeichnung und ohne Codebloecke."""
|
|
||||||
without_code = FENCE_RE.sub("", text)
|
|
||||||
cleaned = re.sub(r"[#*_>`]", " ", without_code)
|
|
||||||
return len(cleaned.split())
|
|
||||||
|
|
||||||
|
|
||||||
def looks_truncated(text: str) -> bool:
|
|
||||||
stripped = text.strip()
|
|
||||||
if not stripped:
|
|
||||||
return True
|
|
||||||
return not stripped.endswith(SENTENCE_END)
|
|
||||||
|
|
||||||
|
|
||||||
def extract_code_blocks(text: str) -> list:
|
|
||||||
"""Liefert (sprache, code, start_offset) je Fence-Block.
|
|
||||||
|
|
||||||
Der Offset wird gebraucht, um den im Fliesstext *vor* einem Block genannten
|
|
||||||
Dateinamen zu finden; ein Textvergleich waere mehrdeutig, wenn derselbe Code
|
|
||||||
mehrfach vorkommt.
|
|
||||||
"""
|
|
||||||
return [
|
|
||||||
(match.group(1).lower(), match.group(2), match.start())
|
|
||||||
for match in FENCE_RE.finditer(text)
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
def is_test_block(code: str) -> bool:
|
|
||||||
return "def test_" in code or "import pytest" in code
|
|
||||||
|
|
||||||
|
|
||||||
def _usable(name: str) -> bool:
|
|
||||||
return bool(name) and name not in RESERVED_MODULE_NAMES and not name.startswith("test_")
|
|
||||||
|
|
||||||
|
|
||||||
def infer_module_name(markdown: str, block_start: int, tests: list) -> str:
|
|
||||||
"""Ermittelt den Dateinamen des Moduls.
|
|
||||||
|
|
||||||
Reihenfolge: (1) letzte im Fliesstext *vor* dem Block genannte ``*.py``-Datei,
|
|
||||||
(2) ``from X import`` in den Tests -- die Tests importieren das Modul unter
|
|
||||||
genau diesem Namen, (3) ``import X``. Stdlib-Namen und ``test_*`` werden in
|
|
||||||
jeder Stufe verworfen.
|
|
||||||
"""
|
|
||||||
for name in reversed(PY_FILENAME_RE.findall(markdown[:block_start])):
|
|
||||||
if _usable(name):
|
|
||||||
return name
|
|
||||||
|
|
||||||
for pattern in (FROM_IMPORT_RE, PLAIN_IMPORT_RE):
|
|
||||||
for test_code in tests:
|
|
||||||
for candidate in pattern.findall(test_code):
|
|
||||||
if _usable(candidate):
|
|
||||||
return candidate
|
|
||||||
|
|
||||||
return "generated_module"
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_pytest_counts(output: str) -> tuple:
|
|
||||||
passed = failed = 0
|
|
||||||
for count, kind in PYTEST_COUNT_RE.findall(output):
|
|
||||||
if kind == "passed":
|
|
||||||
passed = int(count)
|
|
||||||
else:
|
|
||||||
failed += int(count)
|
|
||||||
return passed, failed
|
|
||||||
|
|
||||||
|
|
||||||
def run_python_case(markdown: str, timeout: int, keep_dir: Optional[Path]) -> dict:
|
|
||||||
"""Schreibt Modul + Tests in ein Temp-Verzeichnis und ruft pytest auf."""
|
|
||||||
blocks = extract_code_blocks(markdown)
|
|
||||||
py_blocks = [(body, start) for lang, body, start in blocks if lang in ("python", "py")]
|
|
||||||
if not py_blocks:
|
|
||||||
return {"exec_status": "no-code", "detail": "kein Python-Block gefunden"}
|
|
||||||
|
|
||||||
tests = [body for body, _ in py_blocks if is_test_block(body)]
|
|
||||||
modules = [(body, start) for body, start in py_blocks if not is_test_block(body)]
|
|
||||||
if not tests:
|
|
||||||
return {"exec_status": "no-tests", "detail": f"{len(modules)} Modul-Block(loecke), keine Tests"}
|
|
||||||
|
|
||||||
workdir = Path(tempfile.mkdtemp(prefix="llamaeval_"))
|
|
||||||
warnings = []
|
|
||||||
try:
|
|
||||||
used: set = set()
|
|
||||||
for module_body, block_start in modules:
|
|
||||||
name = infer_module_name(markdown, block_start, tests)
|
|
||||||
if name == "generated_module":
|
|
||||||
# Weder der Fliesstext nennt einen Dateinamen noch importieren die
|
|
||||||
# Tests das Modul: ein echter Mangel der Testqualitaet, kein
|
|
||||||
# Ratefehler des Runners.
|
|
||||||
warnings.append("Tests importieren das Modul nicht")
|
|
||||||
while name in used: # zwei Modul-Bloecke, gleicher geratener Name
|
|
||||||
name += "_x"
|
|
||||||
used.add(name)
|
|
||||||
(workdir / f"{name}.py").write_text(module_body, encoding="utf-8")
|
|
||||||
for index, test_body in enumerate(tests):
|
|
||||||
suffix = "" if index == 0 else f"_{index}"
|
|
||||||
(workdir / f"test_generated{suffix}.py").write_text(test_body, encoding="utf-8")
|
|
||||||
|
|
||||||
try:
|
|
||||||
proc = subprocess.run(
|
|
||||||
[sys.executable, "-m", "pytest", "-q", "--tb=no", "-p", "no:cacheprovider"],
|
|
||||||
cwd=workdir,
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
timeout=timeout,
|
|
||||||
)
|
|
||||||
except subprocess.TimeoutExpired:
|
|
||||||
return {"exec_status": "timeout", "detail": f"pytest > {timeout}s"}
|
|
||||||
except FileNotFoundError:
|
|
||||||
return {"exec_status": "skipped-no-pytest", "detail": "pytest nicht gefunden"}
|
|
||||||
|
|
||||||
output = proc.stdout + proc.stderr
|
|
||||||
passed, failed = _parse_pytest_counts(output)
|
|
||||||
status = "ok" if proc.returncode == 0 else "failed"
|
|
||||||
last = [ln for ln in output.strip().splitlines() if ln.strip()]
|
|
||||||
detail = "; ".join(warnings + ([last[-1]] if last else []))
|
|
||||||
return {
|
|
||||||
"exec_status": status,
|
|
||||||
"tests_passed": passed,
|
|
||||||
"tests_failed": failed,
|
|
||||||
"detail": detail[:120],
|
|
||||||
}
|
|
||||||
finally:
|
|
||||||
if keep_dir is not None:
|
|
||||||
shutil.move(str(workdir), str(keep_dir / workdir.name))
|
|
||||||
else:
|
|
||||||
shutil.rmtree(workdir, ignore_errors=True)
|
|
||||||
|
|
||||||
|
|
||||||
def evaluate(run: Run, targets: dict, do_exec: bool, timeout: int, keep_dir) -> Metrics:
|
|
||||||
text = run.path.read_text(encoding="utf-8", errors="replace")
|
|
||||||
metrics = Metrics(
|
|
||||||
label=run.label,
|
|
||||||
model=run.model,
|
|
||||||
domain=run.domain,
|
|
||||||
case=run.case,
|
|
||||||
variant=run.variant,
|
|
||||||
words=count_words(text),
|
|
||||||
looks_truncated=looks_truncated(text),
|
|
||||||
code_langs=sorted({lang for lang, _, _ in extract_code_blocks(text) if lang}),
|
|
||||||
)
|
|
||||||
|
|
||||||
target = targets.get((run.domain, run.case))
|
|
||||||
if target:
|
|
||||||
metrics.target_words = target
|
|
||||||
metrics.deviation_pct = round((metrics.words - target) / target * 100, 1)
|
|
||||||
|
|
||||||
if run.domain != "coding":
|
|
||||||
return metrics
|
|
||||||
|
|
||||||
if not do_exec:
|
|
||||||
metrics.exec_status = "skipped-no-exec"
|
|
||||||
return metrics
|
|
||||||
if "typescript" in metrics.code_langs and "python" not in metrics.code_langs:
|
|
||||||
metrics.exec_status = "skipped-typescript"
|
|
||||||
metrics.detail = "kein tsc im PATH; TypeScript-Lauf nicht implementiert"
|
|
||||||
return metrics
|
|
||||||
|
|
||||||
result = run_python_case(text, timeout, keep_dir)
|
|
||||||
metrics.executed = result["exec_status"] in ("ok", "failed")
|
|
||||||
metrics.exec_status = result["exec_status"]
|
|
||||||
metrics.tests_passed = result.get("tests_passed", 0)
|
|
||||||
metrics.tests_failed = result.get("tests_failed", 0)
|
|
||||||
metrics.detail = result.get("detail", "")
|
|
||||||
return metrics
|
|
||||||
|
|
||||||
|
|
||||||
def anonymize(runs: list, out_dir: Path) -> Path:
|
|
||||||
"""Kopiert die Laeufe unter Zufalls-IDs und schreibt die Aufloesungstabelle.
|
|
||||||
|
|
||||||
Fuer blinde Bewertung: der Bewertende darf das Modell nicht kennen.
|
|
||||||
"""
|
|
||||||
out_dir.mkdir(parents=True, exist_ok=True)
|
|
||||||
ids = [f"text_{i:03d}" for i in range(len(runs))]
|
|
||||||
random.shuffle(ids)
|
|
||||||
mapping = {}
|
|
||||||
for run, blind_id in zip(runs, ids):
|
|
||||||
shutil.copyfile(run.path, out_dir / f"{blind_id}.txt")
|
|
||||||
mapping[blind_id] = {"model": run.model, "domain": run.domain, "case": run.case,
|
|
||||||
"variant": run.variant, "source": run.path.name}
|
|
||||||
key_path = out_dir / "AUFLOESUNG.json"
|
|
||||||
key_path.write_text(json.dumps(mapping, indent=2, ensure_ascii=False), encoding="utf-8")
|
|
||||||
return key_path
|
|
||||||
|
|
||||||
|
|
||||||
def _fmt_dev(metrics: Metrics) -> str:
|
|
||||||
if metrics.deviation_pct is None:
|
|
||||||
return "—"
|
|
||||||
return f"{metrics.deviation_pct:+.1f}%"
|
|
||||||
|
|
||||||
|
|
||||||
def print_report(all_metrics: list) -> None:
|
|
||||||
for domain in DOMAINS:
|
|
||||||
rows = [m for m in all_metrics if m.domain == domain]
|
|
||||||
if not rows:
|
|
||||||
continue
|
|
||||||
print(f"\n=== {domain} ===")
|
|
||||||
if domain == "coding":
|
|
||||||
print(f"{'Lauf':<28} {'Status':<18} {'passed':>6} {'failed':>6} Detail")
|
|
||||||
for m in sorted(rows, key=lambda r: (r.model, r.case)):
|
|
||||||
print(f"{m.label:<28} {m.exec_status:<18} {m.tests_passed:>6} "
|
|
||||||
f"{m.tests_failed:>6} {m.detail[:60]}")
|
|
||||||
else:
|
|
||||||
print(f"{'Lauf':<28} {'Wörter':>7} {'Ziel':>6} {'Abw.':>8} {'trunkiert':<9}")
|
|
||||||
for m in sorted(rows, key=lambda r: (r.model, r.case)):
|
|
||||||
target = str(m.target_words) if m.target_words else "—"
|
|
||||||
trunc = "ja" if m.looks_truncated else ""
|
|
||||||
print(f"{m.label:<28} {m.words:>7} {target:>6} {_fmt_dev(m):>8} {trunc:<9}")
|
|
||||||
|
|
||||||
coding = [m for m in all_metrics if m.domain == "coding" and m.executed]
|
|
||||||
if coding:
|
|
||||||
green = sum(1 for m in coding if m.exec_status == "ok")
|
|
||||||
total_p = sum(m.tests_passed for m in coding)
|
|
||||||
total_f = sum(m.tests_failed for m in coding)
|
|
||||||
print(f"\nCoding gesamt: {green}/{len(coding)} Läufe grün, "
|
|
||||||
f"{total_p} Tests bestanden, {total_f} durchgefallen.")
|
|
||||||
|
|
||||||
lengths = [m for m in all_metrics if m.deviation_pct is not None]
|
|
||||||
if lengths:
|
|
||||||
within5 = sum(1 for m in lengths if abs(m.deviation_pct) <= 5)
|
|
||||||
worst = max(lengths, key=lambda m: abs(m.deviation_pct))
|
|
||||||
print(f"Längentreue: {within5}/{len(lengths)} Läufe im ±5%-Fenster; "
|
|
||||||
f"größte Abweichung {_fmt_dev(worst)} ({worst.label}).")
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> int:
|
|
||||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[1] if __doc__ else "")
|
|
||||||
parser.add_argument("--results-dir", type=Path, default=DEFAULT_RESULTS_DIR)
|
|
||||||
parser.add_argument("--prompts-dir", type=Path, default=DEFAULT_PROMPTS_DIR)
|
|
||||||
parser.add_argument("--no-exec", dest="exec_code", action="store_false", default=True,
|
|
||||||
help="Generierten Code NICHT ausführen, nur statisch messen")
|
|
||||||
parser.add_argument("--timeout", type=int, default=60, help="Sekunden pro pytest-Lauf")
|
|
||||||
parser.add_argument("--json", type=Path, help="Report zusätzlich als JSON schreiben")
|
|
||||||
parser.add_argument("--keep-workdirs", type=Path,
|
|
||||||
help="Temp-Verzeichnisse der Testläufe hierhin retten (Debugging)")
|
|
||||||
parser.add_argument("--anonymize", type=Path,
|
|
||||||
help="Anonymisierte Kopien + Auflösungstabelle für blinde Bewertung")
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
if not args.results_dir.is_dir():
|
|
||||||
print(f"Ergebnisverzeichnis nicht gefunden: {args.results_dir}", file=sys.stderr)
|
|
||||||
return 1
|
|
||||||
|
|
||||||
if args.exec_code and hasattr(os, "geteuid") and os.geteuid() == 0:
|
|
||||||
print("Verweigert: modellgenerierten Code nicht als root ausführen "
|
|
||||||
"(--no-exec erzwingt die statische Auswertung).", file=sys.stderr)
|
|
||||||
return 1
|
|
||||||
|
|
||||||
runs = []
|
|
||||||
skipped = []
|
|
||||||
for path in sorted(args.results_dir.iterdir()):
|
|
||||||
if not path.is_file():
|
|
||||||
continue
|
|
||||||
run = parse_run(path)
|
|
||||||
(runs.append(run) if run else skipped.append(path.name))
|
|
||||||
|
|
||||||
if not runs:
|
|
||||||
print(f"Keine auswertbaren Läufe in {args.results_dir}", file=sys.stderr)
|
|
||||||
return 1
|
|
||||||
|
|
||||||
if args.anonymize:
|
|
||||||
key = anonymize(runs, args.anonymize)
|
|
||||||
print(f"{len(runs)} Texte anonymisiert nach {args.anonymize}; Auflösung: {key}")
|
|
||||||
|
|
||||||
if args.keep_workdirs:
|
|
||||||
args.keep_workdirs.mkdir(parents=True, exist_ok=True)
|
|
||||||
|
|
||||||
targets = load_length_targets(args.prompts_dir)
|
|
||||||
if not targets:
|
|
||||||
print(f"Warnung: keine Ziel-Wortzahlen aus {args.prompts_dir} gelesen", file=sys.stderr)
|
|
||||||
|
|
||||||
all_metrics = [
|
|
||||||
evaluate(run, targets, args.exec_code, args.timeout, args.keep_workdirs)
|
|
||||||
for run in runs
|
|
||||||
]
|
|
||||||
|
|
||||||
print_report(all_metrics)
|
|
||||||
if skipped:
|
|
||||||
print(f"\nÜbersprungen ({len(skipped)}): {', '.join(skipped)}")
|
|
||||||
|
|
||||||
if args.json:
|
|
||||||
args.json.write_text(
|
|
||||||
json.dumps([asdict(m) for m in all_metrics], indent=2, ensure_ascii=False),
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
print(f"JSON-Report: {args.json}")
|
|
||||||
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
sys.exit(main())
|
|
||||||
|
|
@ -1,97 +0,0 @@
|
||||||
#!/usr/bin/env bash
|
|
||||||
#
|
|
||||||
# Faehrt eine Domaene der Beispiel-Prompts gegen ein laufendes Modell-Profil und
|
|
||||||
# legt die Ausgaben unter der etablierten Namenskonvention im Test-Archiv ab:
|
|
||||||
#
|
|
||||||
# <label>_<domaene>_<nr>.out.txt
|
|
||||||
#
|
|
||||||
# Der Server fuer <profil> muss bereits laufen (llamacppctl --start --profile ...).
|
|
||||||
# Ausgewertet wird anschliessend mit scripts/eval_prompt_tests.py.
|
|
||||||
#
|
|
||||||
# Verwendung:
|
|
||||||
# scripts/run_prompt_suite.sh <profil> <label> <domaene> [<domaene>...]
|
|
||||||
#
|
|
||||||
# Beispiel (Kontrolllauf des nicht-abliterierten Basismodells auf Prosa):
|
|
||||||
# scripts/run_prompt_suite.sh qwen35base qwen35base prosa
|
|
||||||
#
|
|
||||||
# Environment:
|
|
||||||
# OUT_DIR Zielverzeichnis (Default: ~/llamacppctl_prompt_tests)
|
|
||||||
# CONFIG Config-Datei (Default: llama.cpp.config)
|
|
||||||
# MAX_TOKENS Antwortbudget (Default: 20000 -- Reasoning frisst viel; unter
|
|
||||||
# 16000 brechen Prosa/Reden mitten im Denken ab)
|
|
||||||
# CLI Pfad zum CLI (Default: ./.venv/bin/llamacppctl)
|
|
||||||
# OVERWRITE 1 = vorhandene Ausgaben ueberschreiben (Default: 0 = ueberspringen)
|
|
||||||
#
|
|
||||||
set -uo pipefail
|
|
||||||
|
|
||||||
cd "$(git rev-parse --show-toplevel)" || exit 1
|
|
||||||
|
|
||||||
if [ "$#" -lt 3 ]; then
|
|
||||||
sed -n '2,25p' "$0" | sed 's/^# \{0,1\}//'
|
|
||||||
exit 2
|
|
||||||
fi
|
|
||||||
|
|
||||||
PROFILE=$1; LABEL=$2; shift 2
|
|
||||||
OUT_DIR=${OUT_DIR:-"$HOME/llamacppctl_prompt_tests"}
|
|
||||||
CONFIG=${CONFIG:-llama.cpp.config}
|
|
||||||
MAX_TOKENS=${MAX_TOKENS:-20000}
|
|
||||||
CLI=${CLI:-./.venv/bin/llamacppctl}
|
|
||||||
OVERWRITE=${OVERWRITE:-0}
|
|
||||||
|
|
||||||
if [ ! -x "$CLI" ]; then
|
|
||||||
echo "CLI nicht ausfuehrbar: $CLI" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Ohne laufenden Server produziert jeder Prompt nur eine Fehlermeldung als
|
|
||||||
# "Ergebnis" -- lieber sofort abbrechen als 4 Muelldateien schreiben.
|
|
||||||
if ! "$CLI" --check --config "$CONFIG" --profile "$PROFILE" >/dev/null 2>&1; then
|
|
||||||
echo "Server fuer Profil '$PROFILE' ist nicht erreichbar (--check schlug fehl)." >&2
|
|
||||||
echo "Zuerst starten: $CLI --start --config $CONFIG --profile $PROFILE" >&2
|
|
||||||
exit 5
|
|
||||||
fi
|
|
||||||
|
|
||||||
mkdir -p "$OUT_DIR"
|
|
||||||
fail=0
|
|
||||||
|
|
||||||
for domain in "$@"; do
|
|
||||||
system_prompt="example_system_prompts/system_prompt_${domain}.md"
|
|
||||||
if [ ! -f "$system_prompt" ]; then
|
|
||||||
echo "System-Prompt fehlt: $system_prompt" >&2
|
|
||||||
fail=1
|
|
||||||
continue
|
|
||||||
fi
|
|
||||||
|
|
||||||
for prompt in example_user_prompts/"${domain}"_[0-9][0-9]_*.md; do
|
|
||||||
[ -e "$prompt" ] || continue
|
|
||||||
base=$(basename "$prompt" .md) # z. B. prosa_01_werkstatt
|
|
||||||
number=$(echo "$base" | cut -d_ -f2) # z. B. 01
|
|
||||||
out="$OUT_DIR/${LABEL}_${domain}_${number}.out.txt"
|
|
||||||
|
|
||||||
if [ -e "$out" ] && [ "$OVERWRITE" != "1" ]; then
|
|
||||||
echo "== $base -> vorhanden, uebersprungen (OVERWRITE=1 erzwingt)"
|
|
||||||
continue
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "== $base -> $out"
|
|
||||||
if "$CLI" --chat --config "$CONFIG" --profile "$PROFILE" \
|
|
||||||
--system-file "$system_prompt" \
|
|
||||||
--prompt-file "$prompt" \
|
|
||||||
--stream --max-tokens "$MAX_TOKENS" > "$out.part" 2>"$out.err"; then
|
|
||||||
mv "$out.part" "$out"
|
|
||||||
rm -f "$out.err"
|
|
||||||
printf ' %s Woerter\n' "$(wc -w < "$out")"
|
|
||||||
else
|
|
||||||
echo " FEHLGESCHLAGEN -- siehe $out.err" >&2
|
|
||||||
rm -f "$out.part"
|
|
||||||
fail=1
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
done
|
|
||||||
|
|
||||||
echo
|
|
||||||
if [ "$fail" -ne 0 ]; then
|
|
||||||
echo "Mindestens ein Lauf schlug fehl."
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
echo "Fertig. Auswerten mit: scripts/eval_prompt_tests.py"
|
|
||||||
|
|
@ -77,23 +77,15 @@ def check_exit_code(result: CheckResult) -> int:
|
||||||
return EXIT_HTTP_NOT_READY
|
return EXIT_HTTP_NOT_READY
|
||||||
|
|
||||||
|
|
||||||
def _host_path(cfg: ServerConfig, path_str: str) -> Path:
|
|
||||||
path = Path(path_str)
|
|
||||||
return path if path.is_absolute() else cfg.hf_home / path_str
|
|
||||||
|
|
||||||
|
|
||||||
def validate_model_path(cfg: ServerConfig) -> None:
|
def validate_model_path(cfg: ServerConfig) -> None:
|
||||||
"""Check that the model file — and the vision projector, when one is
|
model_path = Path(cfg.model_path)
|
||||||
configured — exist on the host before the container is (re)created."""
|
if model_path.is_absolute():
|
||||||
target = _host_path(cfg, cfg.model_path)
|
target = model_path
|
||||||
|
else:
|
||||||
|
target = cfg.hf_home / cfg.model_path
|
||||||
if not target.exists():
|
if not target.exists():
|
||||||
raise FileNotFoundError(f"model file not found: {target}")
|
raise FileNotFoundError(f"model file not found: {target}")
|
||||||
|
|
||||||
if cfg.mmproj:
|
|
||||||
projector = _host_path(cfg, cfg.mmproj)
|
|
||||||
if not projector.exists():
|
|
||||||
raise FileNotFoundError(f"mmproj file not found: {projector}")
|
|
||||||
|
|
||||||
|
|
||||||
def print_effective_config(cfg: ServerConfig, prompt_cfg: PromptConfig) -> None:
|
def print_effective_config(cfg: ServerConfig, prompt_cfg: PromptConfig) -> None:
|
||||||
payload = {
|
payload = {
|
||||||
|
|
|
||||||
|
|
@ -20,14 +20,6 @@ def build_parser() -> argparse.ArgumentParser:
|
||||||
action.add_argument("--stop", action="store_true", help="Stop/remove the container")
|
action.add_argument("--stop", action="store_true", help="Stop/remove the container")
|
||||||
action.add_argument("--change", action="store_true", help="Stop, reconfigure, and restart")
|
action.add_argument("--change", action="store_true", help="Stop, reconfigure, and restart")
|
||||||
action.add_argument("--chat", action="store_true", help="Send a prompt to a running server")
|
action.add_argument("--chat", action="store_true", help="Send a prompt to a running server")
|
||||||
# A diagnostic action, not a modifier: combining it with --start used to print
|
|
||||||
# the config and then really start the container. It is mutually exclusive with
|
|
||||||
# the other actions so it can never have a side effect.
|
|
||||||
action.add_argument(
|
|
||||||
"--print-effective-config",
|
|
||||||
action="store_true",
|
|
||||||
help="Print the fully merged configuration as JSON and exit. Touches nothing",
|
|
||||||
)
|
|
||||||
|
|
||||||
p.add_argument("--config", default="llama.cpp.config", help="Path to llama.cpp.config")
|
p.add_argument("--config", default="llama.cpp.config", help="Path to llama.cpp.config")
|
||||||
p.add_argument("--profile", help="Model profile name: [model.<profile>] in config")
|
p.add_argument("--profile", help="Model profile name: [model.<profile>] in config")
|
||||||
|
|
@ -54,18 +46,6 @@ def build_parser() -> argparse.ArgumentParser:
|
||||||
p.add_argument("--image", help="Docker image (e.g. ghcr.io/ggml-org/llama.cpp:server-cuda)")
|
p.add_argument("--image", help="Docker image (e.g. ghcr.io/ggml-org/llama.cpp:server-cuda)")
|
||||||
p.add_argument("--hf-home", help="Host path mounted read-only as model storage")
|
p.add_argument("--hf-home", help="Host path mounted read-only as model storage")
|
||||||
p.add_argument("--model-path", help="Model path, relative to --hf-home unless absolute")
|
p.add_argument("--model-path", help="Model path, relative to --hf-home unless absolute")
|
||||||
p.add_argument(
|
|
||||||
"--mmproj",
|
|
||||||
help="Vision projector (mmproj) GGUF, relative to --hf-home unless absolute. "
|
|
||||||
"Must match the base model's vision tower; enables image input",
|
|
||||||
)
|
|
||||||
p.add_argument(
|
|
||||||
"--no-mmproj-offload",
|
|
||||||
dest="mmproj_offload",
|
|
||||||
action="store_false",
|
|
||||||
default=None,
|
|
||||||
help="Keep the vision projector on the CPU instead of offloading it to the GPU",
|
|
||||||
)
|
|
||||||
p.add_argument("--container-name", help="Docker container name (must be unique per config)")
|
p.add_argument("--container-name", help="Docker container name (must be unique per config)")
|
||||||
p.add_argument("--host-port", type=int, help="Host port to publish")
|
p.add_argument("--host-port", type=int, help="Host port to publish")
|
||||||
p.add_argument("--container-port", type=int, help="Container-internal port")
|
p.add_argument("--container-port", type=int, help="Container-internal port")
|
||||||
|
|
@ -154,6 +134,11 @@ def build_parser() -> argparse.ArgumentParser:
|
||||||
p.add_argument("--log-lines", type=int, default=100)
|
p.add_argument("--log-lines", type=int, default=100)
|
||||||
p.add_argument("--dry-run", action="store_true", help="Show the docker run command, do not execute it")
|
p.add_argument("--dry-run", action="store_true", help="Show the docker run command, do not execute it")
|
||||||
p.add_argument("--force", action="store_true", help="Force stop/change despite inconsistencies")
|
p.add_argument("--force", action="store_true", help="Force stop/change despite inconsistencies")
|
||||||
|
p.add_argument(
|
||||||
|
"--print-effective-config",
|
||||||
|
action="store_true",
|
||||||
|
help="Print the fully merged configuration as JSON and exit (unless combined with an action)",
|
||||||
|
)
|
||||||
|
|
||||||
return p
|
return p
|
||||||
|
|
||||||
|
|
@ -200,6 +185,3 @@ def validate_args(args: argparse.Namespace, parser: argparse.ArgumentParser) ->
|
||||||
|
|
||||||
if args.container_name is not None and not args.container_name.strip():
|
if args.container_name is not None and not args.container_name.strip():
|
||||||
parser.error("--container-name must not be empty")
|
parser.error("--container-name must not be empty")
|
||||||
|
|
||||||
if args.mmproj is not None and not args.mmproj.strip():
|
|
||||||
parser.error("--mmproj must not be empty")
|
|
||||||
|
|
|
||||||
|
|
@ -71,8 +71,6 @@ def builtin_defaults() -> dict:
|
||||||
"host": "0.0.0.0",
|
"host": "0.0.0.0",
|
||||||
"expose": "false", # publish only on loopback unless true
|
"expose": "false", # publish only on loopback unless true
|
||||||
"api_key": "",
|
"api_key": "",
|
||||||
"mmproj": "", # vision projector GGUF; empty => text-only server
|
|
||||||
"mmproj_offload": "true",
|
|
||||||
"health_endpoint": "/health",
|
"health_endpoint": "/health",
|
||||||
"models_endpoint": "/v1/models",
|
"models_endpoint": "/v1/models",
|
||||||
"chat_endpoint": "/v1/chat/completions",
|
"chat_endpoint": "/v1/chat/completions",
|
||||||
|
|
@ -139,8 +137,6 @@ def _apply_cli_overrides(merged: dict, args) -> None:
|
||||||
"lock_file": getattr(args, "lock_file", None),
|
"lock_file": getattr(args, "lock_file", None),
|
||||||
"expose": getattr(args, "expose", None),
|
"expose": getattr(args, "expose", None),
|
||||||
"api_key": getattr(args, "api_key", None),
|
"api_key": getattr(args, "api_key", None),
|
||||||
"mmproj": getattr(args, "mmproj", None),
|
|
||||||
"mmproj_offload": getattr(args, "mmproj_offload", None),
|
|
||||||
}
|
}
|
||||||
for key, value in overrides.items():
|
for key, value in overrides.items():
|
||||||
if value is not None:
|
if value is not None:
|
||||||
|
|
@ -197,8 +193,6 @@ def build_server_config(merged: dict) -> ServerConfig:
|
||||||
host=str(merged["host"]),
|
host=str(merged["host"]),
|
||||||
expose=_to_bool(merged.get("expose", "false"), "expose"),
|
expose=_to_bool(merged.get("expose", "false"), "expose"),
|
||||||
api_key=str(merged.get("api_key", "")),
|
api_key=str(merged.get("api_key", "")),
|
||||||
mmproj=str(merged.get("mmproj") or "").strip(),
|
|
||||||
mmproj_offload=_to_bool(merged.get("mmproj_offload", "true"), "mmproj_offload"),
|
|
||||||
health_endpoint=str(merged["health_endpoint"]),
|
health_endpoint=str(merged["health_endpoint"]),
|
||||||
models_endpoint=str(merged["models_endpoint"]),
|
models_endpoint=str(merged["models_endpoint"]),
|
||||||
chat_endpoint=str(merged["chat_endpoint"]),
|
chat_endpoint=str(merged["chat_endpoint"]),
|
||||||
|
|
|
||||||
|
|
@ -105,21 +105,11 @@ def container_logs(name: str, tail: int = 100) -> str:
|
||||||
tail_logs = container_logs
|
tail_logs = container_logs
|
||||||
|
|
||||||
|
|
||||||
def _resolve_hf_path_in_container(path_str: str) -> str:
|
|
||||||
"""Map a host-side model/projector path to its path inside the container.
|
|
||||||
Absolute paths are passed through; relative ones resolve under the
|
|
||||||
read-only /hf_home mount."""
|
|
||||||
if Path(path_str).is_absolute():
|
|
||||||
return path_str
|
|
||||||
return f"/hf_home/{path_str}"
|
|
||||||
|
|
||||||
|
|
||||||
def _resolve_model_path_in_container(cfg: ServerConfig) -> str:
|
def _resolve_model_path_in_container(cfg: ServerConfig) -> str:
|
||||||
return _resolve_hf_path_in_container(cfg.model_path)
|
model_path = Path(cfg.model_path)
|
||||||
|
if model_path.is_absolute():
|
||||||
|
return str(model_path)
|
||||||
def _resolve_mmproj_path_in_container(cfg: ServerConfig) -> str:
|
return f"/hf_home/{cfg.model_path}"
|
||||||
return _resolve_hf_path_in_container(cfg.mmproj)
|
|
||||||
|
|
||||||
|
|
||||||
def _port_publish(cfg: ServerConfig) -> str:
|
def _port_publish(cfg: ServerConfig) -> str:
|
||||||
|
|
@ -191,10 +181,6 @@ def build_run_command(cfg: ServerConfig) -> list:
|
||||||
cmd.append("--kv-unified")
|
cmd.append("--kv-unified")
|
||||||
if cfg.cont_batching:
|
if cfg.cont_batching:
|
||||||
cmd.append("--cont-batching")
|
cmd.append("--cont-batching")
|
||||||
if cfg.mmproj:
|
|
||||||
cmd += ["--mmproj", _resolve_mmproj_path_in_container(cfg)]
|
|
||||||
if not cfg.mmproj_offload:
|
|
||||||
cmd.append("--no-mmproj-offload")
|
|
||||||
if cfg.api_key:
|
if cfg.api_key:
|
||||||
cmd += ["--api-key", cfg.api_key]
|
cmd += ["--api-key", cfg.api_key]
|
||||||
cmd.extend(cfg.extra_args)
|
cmd.extend(cfg.extra_args)
|
||||||
|
|
|
||||||
|
|
@ -3,7 +3,7 @@
|
||||||
Orchestration order:
|
Orchestration order:
|
||||||
1. Parse CLI arguments (cli.build_parser)
|
1. Parse CLI arguments (cli.build_parser)
|
||||||
2. Run semantic validation (cli.validate_args)
|
2. Run semantic validation (cli.validate_args)
|
||||||
3. Check that docker is available (skipped for --print-effective-config)
|
3. Check that docker is available
|
||||||
4. Resolve prompt sources (prompt_io) under an InputPolicy built from CLI flags
|
4. Resolve prompt sources (prompt_io) under an InputPolicy built from CLI flags
|
||||||
5. Resolve effective server/prompt config (config.resolve_effective_config)
|
5. Resolve effective server/prompt config (config.resolve_effective_config)
|
||||||
6. Dispatch to the requested action (actions.do_*)
|
6. Dispatch to the requested action (actions.do_*)
|
||||||
|
|
@ -55,9 +55,7 @@ def run(argv: Optional[Sequence[str]] = None) -> int:
|
||||||
args = parser.parse_args(argv)
|
args = parser.parse_args(argv)
|
||||||
validate_args(args, parser)
|
validate_args(args, parser)
|
||||||
|
|
||||||
# --print-effective-config is purely diagnostic: it must work on a host
|
if not docker_available():
|
||||||
# without a Docker daemon, and it must never touch a container.
|
|
||||||
if not args.print_effective_config and not docker_available():
|
|
||||||
print("docker is not available on PATH (or the daemon is not reachable)", file=sys.stderr)
|
print("docker is not available on PATH (or the daemon is not reachable)", file=sys.stderr)
|
||||||
return 1
|
return 1
|
||||||
|
|
||||||
|
|
@ -76,6 +74,9 @@ def run(argv: Optional[Sequence[str]] = None) -> int:
|
||||||
|
|
||||||
if args.print_effective_config:
|
if args.print_effective_config:
|
||||||
actions.print_effective_config(server_cfg, prompt_cfg)
|
actions.print_effective_config(server_cfg, prompt_cfg)
|
||||||
|
# An action is always required by argparse; only exit early if the run
|
||||||
|
# is purely diagnostic (no action would actually do anything).
|
||||||
|
if not (args.start or args.check or args.stop or args.change or args.chat):
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
if args.start:
|
if args.start:
|
||||||
|
|
|
||||||
|
|
@ -60,14 +60,6 @@ class ServerConfig:
|
||||||
# as a Bearer token on every request.
|
# as a Bearer token on every request.
|
||||||
api_key: str = ""
|
api_key: str = ""
|
||||||
|
|
||||||
# Optional multimodal projector (vision encoder) GGUF. Resolved relative to
|
|
||||||
# hf_home unless absolute, like model_path. Must match the base model's
|
|
||||||
# vision tower. Empty => text-only server.
|
|
||||||
mmproj: str = ""
|
|
||||||
# llama.cpp offloads the projector to the GPU by default; set False to keep
|
|
||||||
# it on the CPU when VRAM is tight (emits --no-mmproj-offload).
|
|
||||||
mmproj_offload: bool = True
|
|
||||||
|
|
||||||
extra_args: list = field(default_factory=list)
|
extra_args: list = field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -213,34 +213,3 @@ def test_container_lock_force_bypasses_busy(tmp_path, capsys):
|
||||||
entered = True
|
entered = True
|
||||||
assert entered is True
|
assert entered is True
|
||||||
assert "busy lock" in capsys.readouterr().err
|
assert "busy lock" in capsys.readouterr().err
|
||||||
|
|
||||||
|
|
||||||
# --- model / mmproj path validation ---------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_validate_model_path_accepts_existing_model(tmp_path):
|
|
||||||
(tmp_path / "m.gguf").write_bytes(b"x")
|
|
||||||
cfg = make_cfg(hf_home=tmp_path, model_path="m.gguf")
|
|
||||||
actions.validate_model_path(cfg) # must not raise
|
|
||||||
|
|
||||||
|
|
||||||
def test_validate_model_path_rejects_missing_model(tmp_path):
|
|
||||||
cfg = make_cfg(hf_home=tmp_path, model_path="absent.gguf")
|
|
||||||
with pytest.raises(FileNotFoundError, match="model file not found"):
|
|
||||||
actions.validate_model_path(cfg)
|
|
||||||
|
|
||||||
|
|
||||||
def test_validate_model_path_rejects_missing_mmproj(tmp_path):
|
|
||||||
# The model exists but the configured projector does not: --change must fail
|
|
||||||
# here, before the running container is removed.
|
|
||||||
(tmp_path / "m.gguf").write_bytes(b"x")
|
|
||||||
cfg = make_cfg(hf_home=tmp_path, model_path="m.gguf", mmproj="absent-proj.gguf")
|
|
||||||
with pytest.raises(FileNotFoundError, match="mmproj file not found"):
|
|
||||||
actions.validate_model_path(cfg)
|
|
||||||
|
|
||||||
|
|
||||||
def test_validate_model_path_accepts_existing_mmproj(tmp_path):
|
|
||||||
(tmp_path / "m.gguf").write_bytes(b"x")
|
|
||||||
(tmp_path / "proj.gguf").write_bytes(b"x")
|
|
||||||
cfg = make_cfg(hf_home=tmp_path, model_path="m.gguf", mmproj="proj.gguf")
|
|
||||||
actions.validate_model_path(cfg) # must not raise
|
|
||||||
|
|
|
||||||
|
|
@ -147,16 +147,9 @@ def test_dry_run_flag():
|
||||||
assert args.dry_run is True
|
assert args.dry_run is True
|
||||||
|
|
||||||
|
|
||||||
def test_print_effective_config_is_a_standalone_action():
|
def test_print_effective_config_flag():
|
||||||
args = parse(["--print-effective-config"])
|
args = parse(["--start", "--print-effective-config"])
|
||||||
assert args.print_effective_config is True
|
assert args.print_effective_config is True
|
||||||
assert args.start is False
|
|
||||||
|
|
||||||
|
|
||||||
def test_print_effective_config_cannot_be_combined_with_start():
|
|
||||||
# Regression: as a plain flag this printed the config and then really started
|
|
||||||
# the container, silently replacing a running one.
|
|
||||||
parse_error(["--print-effective-config", "--start"])
|
|
||||||
|
|
||||||
|
|
||||||
def test_reasoning_choice_validated():
|
def test_reasoning_choice_validated():
|
||||||
|
|
|
||||||
|
|
@ -55,10 +55,6 @@ model_alias = alt_llm
|
||||||
[model.noname]
|
[model.noname]
|
||||||
host_port = 9002
|
host_port = 9002
|
||||||
|
|
||||||
[model.vision]
|
|
||||||
container_name = test_vision
|
|
||||||
mmproj = qwen3/mmproj.gguf
|
|
||||||
|
|
||||||
[prompt.concise]
|
[prompt.concise]
|
||||||
system_prompt = Be brief.
|
system_prompt = Be brief.
|
||||||
"""
|
"""
|
||||||
|
|
@ -284,57 +280,6 @@ def test_stream_and_read_timeout_from_config(tmp_path):
|
||||||
assert prompt_cfg.connect_timeout == 5.0
|
assert prompt_cfg.connect_timeout == 5.0
|
||||||
|
|
||||||
|
|
||||||
def test_mmproj_empty_by_default(tmp_path):
|
|
||||||
cfg_path = _write_config(tmp_path)
|
|
||||||
server_cfg, _ = resolve_effective_config(_parse(["--start", "--config", str(cfg_path)]))
|
|
||||||
assert server_cfg.mmproj == ""
|
|
||||||
assert server_cfg.mmproj_offload is True # llama.cpp offloads by default
|
|
||||||
|
|
||||||
|
|
||||||
def test_mmproj_from_profile(tmp_path):
|
|
||||||
cfg_path = _write_config(tmp_path)
|
|
||||||
args = _parse(["--start", "--config", str(cfg_path), "--profile", "vision"])
|
|
||||||
server_cfg, _ = resolve_effective_config(args)
|
|
||||||
assert server_cfg.mmproj == "qwen3/mmproj.gguf"
|
|
||||||
assert server_cfg.container_name == "test_vision"
|
|
||||||
|
|
||||||
|
|
||||||
def test_mmproj_cli_overrides_config(tmp_path):
|
|
||||||
cfg_path = _write_config(tmp_path)
|
|
||||||
args = _parse(
|
|
||||||
[
|
|
||||||
"--start",
|
|
||||||
"--config",
|
|
||||||
str(cfg_path),
|
|
||||||
"--profile",
|
|
||||||
"vision",
|
|
||||||
"--mmproj",
|
|
||||||
"other/proj.gguf",
|
|
||||||
"--no-mmproj-offload",
|
|
||||||
]
|
|
||||||
)
|
|
||||||
server_cfg, _ = resolve_effective_config(args)
|
|
||||||
assert server_cfg.mmproj == "other/proj.gguf"
|
|
||||||
assert server_cfg.mmproj_offload is False
|
|
||||||
|
|
||||||
|
|
||||||
def test_mmproj_offload_false_from_config(tmp_path):
|
|
||||||
cfg = CONFIG_BASIC.replace(
|
|
||||||
"[model.vision]\ncontainer_name = test_vision",
|
|
||||||
"[model.vision]\ncontainer_name = test_vision\nmmproj_offload = false",
|
|
||||||
)
|
|
||||||
cfg_path = _write_config(tmp_path, cfg)
|
|
||||||
args = _parse(["--start", "--config", str(cfg_path), "--profile", "vision"])
|
|
||||||
server_cfg, _ = resolve_effective_config(args)
|
|
||||||
assert server_cfg.mmproj_offload is False
|
|
||||||
|
|
||||||
|
|
||||||
def test_blank_mmproj_via_cli_rejected(tmp_path):
|
|
||||||
cfg_path = _write_config(tmp_path)
|
|
||||||
with pytest.raises(SystemExit):
|
|
||||||
_parse(["--start", "--config", str(cfg_path), "--mmproj", " "])
|
|
||||||
|
|
||||||
|
|
||||||
def test_stream_and_read_timeout_cli_overrides_config(tmp_path):
|
def test_stream_and_read_timeout_cli_overrides_config(tmp_path):
|
||||||
cfg = CONFIG_BASIC.replace("poll_interval = 2", "poll_interval = 2\nread_timeout = 900")
|
cfg = CONFIG_BASIC.replace("poll_interval = 2", "poll_interval = 2\nread_timeout = 900")
|
||||||
cfg_path = _write_config(tmp_path, cfg)
|
cfg_path = _write_config(tmp_path, cfg)
|
||||||
|
|
|
||||||
|
|
@ -178,41 +178,6 @@ def test_build_run_command_relative_model_path():
|
||||||
assert cmd[idx + 1] == "/hf_home/qwen3/default.gguf"
|
assert cmd[idx + 1] == "/hf_home/qwen3/default.gguf"
|
||||||
|
|
||||||
|
|
||||||
def test_build_run_command_no_mmproj_by_default():
|
|
||||||
cmd = build_run_command(make_cfg())
|
|
||||||
assert "--mmproj" not in cmd
|
|
||||||
assert "--no-mmproj-offload" not in cmd
|
|
||||||
|
|
||||||
|
|
||||||
def test_build_run_command_mmproj_relative_path_resolves_under_hf_home():
|
|
||||||
cfg = make_cfg(mmproj="qwen3/mmproj.gguf")
|
|
||||||
cmd = build_run_command(cfg)
|
|
||||||
idx = cmd.index("--mmproj")
|
|
||||||
assert cmd[idx + 1] == "/hf_home/qwen3/mmproj.gguf"
|
|
||||||
# offload is llama.cpp's default, so the opt-out flag must stay absent
|
|
||||||
assert "--no-mmproj-offload" not in cmd
|
|
||||||
|
|
||||||
|
|
||||||
def test_build_run_command_mmproj_absolute_path_passed_through():
|
|
||||||
cfg = make_cfg(mmproj="/abs/mmproj.gguf")
|
|
||||||
cmd = build_run_command(cfg)
|
|
||||||
idx = cmd.index("--mmproj")
|
|
||||||
assert cmd[idx + 1] == "/abs/mmproj.gguf"
|
|
||||||
|
|
||||||
|
|
||||||
def test_build_run_command_mmproj_offload_disabled():
|
|
||||||
cfg = make_cfg(mmproj="qwen3/mmproj.gguf", mmproj_offload=False)
|
|
||||||
cmd = build_run_command(cfg)
|
|
||||||
assert "--no-mmproj-offload" in cmd
|
|
||||||
|
|
||||||
|
|
||||||
def test_build_run_command_no_mmproj_offload_needs_mmproj():
|
|
||||||
# Without a projector the offload opt-out is meaningless and must not leak
|
|
||||||
# into the command line (llama.cpp would reject it).
|
|
||||||
cfg = make_cfg(mmproj="", mmproj_offload=False)
|
|
||||||
assert "--no-mmproj-offload" not in build_run_command(cfg)
|
|
||||||
|
|
||||||
|
|
||||||
def test_format_command_for_display_quotes_properly():
|
def test_format_command_for_display_quotes_properly():
|
||||||
cmd = ["docker", "run", "--name", "has space"]
|
cmd = ["docker", "run", "--name", "has space"]
|
||||||
out = format_command_for_display(cmd)
|
out = format_command_for_display(cmd)
|
||||||
|
|
|
||||||
|
|
@ -1,57 +0,0 @@
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
|
|
||||||
|
|
||||||
from llamacppctl import actions, main as main_mod # noqa: E402
|
|
||||||
|
|
||||||
CONFIG = """
|
|
||||||
[default]
|
|
||||||
hf_home = /srv/models
|
|
||||||
model_path = qwen3/default.gguf
|
|
||||||
container_name = test_main
|
|
||||||
"""
|
|
||||||
|
|
||||||
|
|
||||||
def _config(tmp_path: Path) -> Path:
|
|
||||||
path = tmp_path / "llama.cpp.config"
|
|
||||||
path.write_text(CONFIG, encoding="utf-8")
|
|
||||||
return path
|
|
||||||
|
|
||||||
|
|
||||||
def test_print_effective_config_has_no_side_effects(tmp_path, monkeypatch, capsys):
|
|
||||||
"""Regression: --print-effective-config used to fall through into do_start()
|
|
||||||
because argparse always required an action, silently replacing a running
|
|
||||||
container."""
|
|
||||||
started = []
|
|
||||||
monkeypatch.setattr(actions, "do_start", lambda *a, **k: started.append(1))
|
|
||||||
monkeypatch.setattr(actions, "do_change", lambda *a, **k: started.append(1))
|
|
||||||
monkeypatch.setattr(actions, "do_stop", lambda *a, **k: started.append(1))
|
|
||||||
|
|
||||||
rc = main_mod.run(["--print-effective-config", "--config", str(_config(tmp_path))])
|
|
||||||
|
|
||||||
assert rc == 0
|
|
||||||
assert started == []
|
|
||||||
assert '"container_name": "test_main"' in capsys.readouterr().out
|
|
||||||
|
|
||||||
|
|
||||||
def test_print_effective_config_works_without_docker(tmp_path, monkeypatch, capsys):
|
|
||||||
"""It is a pure config check, so it must not require a reachable daemon."""
|
|
||||||
def boom() -> bool:
|
|
||||||
raise AssertionError("docker_available() must not be consulted")
|
|
||||||
|
|
||||||
monkeypatch.setattr(main_mod, "docker_available", boom)
|
|
||||||
|
|
||||||
rc = main_mod.run(["--print-effective-config", "--config", str(_config(tmp_path))])
|
|
||||||
|
|
||||||
assert rc == 0
|
|
||||||
assert '"hf_home": "/srv/models"' in capsys.readouterr().out
|
|
||||||
|
|
||||||
|
|
||||||
def test_start_still_requires_docker(tmp_path, monkeypatch, capsys):
|
|
||||||
monkeypatch.setattr(main_mod, "docker_available", lambda: False)
|
|
||||||
|
|
||||||
rc = main_mod.run(["--start", "--config", str(_config(tmp_path))])
|
|
||||||
|
|
||||||
assert rc == 1
|
|
||||||
assert "docker is not available" in capsys.readouterr().err
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue