Pin images by digest; add update, smoke-test and status scripts
Goal: at every session start, run exactly the application the repo says — and be able to move to a new upstream release deliberately, with a way back. The old setup (v1-latest + pull_policy: always) was unreliable in both directions. `docker compose up -d` silently pulled a new application, including an irreversible SurrealDB schema migration, while a reboot (restart: always) or `docker compose start` kept running the old image without pulling anything. The running version was effectively unpredictable. Note v1-latest tracks releases, not main: the current image (v1.10.0, 2026-06-18) IS the latest release — the repo being "ahead" is unreleased code, which is explicitly not wanted here. So this is about guaranteeing and detecting, not catching up. - docker-compose.yml: both images pinned by digest, pull_policy: missing. - scripts/check_updates.sh: reports running version/digest vs the latest release and registry digest. Changes nothing; meant for the session-start ritual. - scripts/update_stack.sh: resolve new digest -> stop -> back up surreal_data, notebook_data and docker-compose.yml -> repin -> start -> smoke test -> roll back data AND compose file if the smoke test fails. The data backup is the point: the app migrates the DB on startup and an older app cannot read a migrated DB, so a bad update would otherwise be a one-way door. tar runs inside a container because the data dirs are root-owned and the host user cannot restore over them. - scripts/smoke_test.sh: checks what actually breaks here, not just "does it start". Every local adaptation leans on upstream internals and can break silently: the prompts/podcast directory mount masks the image's directory (a new template upstream would be invisible), config/content_core.yaml freezes content-core's defaults (because CCORE_CONFIG_PATH replaces rather than merges), and the env-var workarounds depend on current content-core/esperanto/podcast_creator behaviour. So it verifies those assumptions explicitly, plus API, TTS audio, STT and a real YouTube transcript. Both scripts read the digest from the lfnovo/open_notebook line specifically. A naive "first sha256 in the file" grep matches surrealdb (listed first) — caught while testing: check_updates.sh falsely reported an update, and update_stack.sh would have rewritten the database image instead of the application. Verified: smoke test green against the current stack, and red (exit 1) when failures are injected (dead TTS port, removed template variable). Backup/restore mechanics exercised separately: 44 MB in 3.3s, restore yields 13 DB files and 91 notebook files. The update path itself cannot be exercised end to end until a newer image exists. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
f350e49bd5
commit
4805fe75ed
7 changed files with 434 additions and 7 deletions
57
scripts/check_updates.sh
Executable file
57
scripts/check_updates.sh
Executable file
|
|
@ -0,0 +1,57 @@
|
|||
#!/usr/bin/env bash
|
||||
# Zeigt, ob es ein neues Open-Notebook-Release gibt. Aendert NICHTS.
|
||||
#
|
||||
# Gedacht fuer den Sitzungsstart: Du siehst, worauf du laeufst und ob etwas Neues da
|
||||
# ist — und entscheidest selbst, wann du migrierst (./scripts/update_stack.sh).
|
||||
# Bewusst kein Auto-Update: ein Image-Update migriert die Datenbank und kann unsere
|
||||
# Anpassungen brechen (siehe scripts/smoke_test.sh).
|
||||
set -uo pipefail
|
||||
|
||||
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$REPO"
|
||||
|
||||
TAG="lfnovo/open_notebook:v1-latest"
|
||||
|
||||
# Gezielt aus der open_notebook-Zeile: ein blindes grep auf den ersten Digest der Datei
|
||||
# erwischt surrealdb (steht weiter oben) und vergleicht dann Aepfel mit Birnen.
|
||||
pinned="$(grep -oP 'lfnovo/open_notebook\S*@\Ksha256:[0-9a-f]+' docker-compose.yml | head -1)"
|
||||
running_ver="$(docker compose exec -T open_notebook grep -m1 '^version' /app/pyproject.toml 2>/dev/null |
|
||||
grep -oP '(?<=")[^"]+' || echo '?')"
|
||||
|
||||
echo "Laeuft gerade"
|
||||
echo " Version : ${running_ver}"
|
||||
echo " Digest : ${pinned:-(nicht gepinnt!)}"
|
||||
|
||||
echo
|
||||
echo "Verfuegbar"
|
||||
latest_rel="$(curl -sf -m 10 https://api.github.com/repos/lfnovo/open-notebook/releases/latest |
|
||||
python3 -c "import json,sys;d=json.load(sys.stdin);print(d['tag_name'],'vom',d['published_at'][:10])" 2>/dev/null)"
|
||||
echo " Release : ${latest_rel:-(GitHub nicht erreichbar)}"
|
||||
|
||||
remote="$(docker buildx imagetools inspect "$TAG" --format '{{.Manifest.Digest}}' 2>/dev/null)"
|
||||
if [ -z "$remote" ]; then
|
||||
echo " Image : (Registry nicht erreichbar)"
|
||||
exit 0
|
||||
fi
|
||||
echo " Image : $remote"
|
||||
|
||||
echo
|
||||
if [ "$remote" = "$pinned" ]; then
|
||||
echo "Du laeufst auf dem aktuellen Release. Nichts zu tun."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
cat <<EOF
|
||||
NEUES IMAGE VERFUEGBAR.
|
||||
|
||||
gepinnt : $pinned
|
||||
neu : $remote
|
||||
|
||||
Update mit Backup, Rauchtest und Rollback:
|
||||
./scripts/update_stack.sh
|
||||
|
||||
Vorher lohnt ein Blick in die Release-Notes — vor allem auf Aenderungen an den
|
||||
Podcast-Vorlagen, an content-core und an der Datenbank:
|
||||
https://github.com/lfnovo/open-notebook/releases
|
||||
EOF
|
||||
exit 0
|
||||
161
scripts/smoke_test.sh
Executable file
161
scripts/smoke_test.sh
Executable file
|
|
@ -0,0 +1,161 @@
|
|||
#!/usr/bin/env bash
|
||||
# Rauchtest fuer den laufenden Stack.
|
||||
#
|
||||
# Prueft nicht "laeuft der Container", sondern genau die Dinge, die bei diesem
|
||||
# Deployment schon einmal kaputt waren — inklusive der Annahmen, auf denen unsere
|
||||
# Anpassungen beruhen. Genau die koennen ein Image-Update stillschweigend brechen:
|
||||
# der Bind-Mount ueber prompts/podcast/ verdeckt neue Dateien, config/content_core.yaml
|
||||
# friert alte Defaults ein (CCORE_CONFIG_PATH ersetzt, statt zu mergen), und die
|
||||
# Env-Var-Workarounds haengen an Interna von content-core/esperanto/podcast_creator.
|
||||
#
|
||||
# ./scripts/smoke_test.sh # alles
|
||||
# SKIP_NET=1 ./scripts/smoke_test.sh # ohne Tests, die ins Internet muessen
|
||||
#
|
||||
# Exit 0 = alles gruen. Exit 1 = mindestens ein Test rot.
|
||||
set -uo pipefail
|
||||
|
||||
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$REPO"
|
||||
|
||||
API="http://127.0.0.1:5055"
|
||||
IMAGE="$(grep -oP '(?<=image: )lfnovo/open_notebook\S+' docker-compose.yml | head -1)"
|
||||
PY=/app/.venv/bin/python
|
||||
fails=0
|
||||
|
||||
ok() { printf ' \033[32mOK\033[0m %s\n' "$1"; }
|
||||
fail() { printf ' \033[31mFEHL\033[0m %s\n' "$1"; fails=$((fails + 1)); }
|
||||
sec() { printf '\n%s\n' "$1"; }
|
||||
|
||||
inapp() { docker compose exec -T open_notebook "$@" 2>/dev/null; }
|
||||
|
||||
sec "Grundfunktion"
|
||||
|
||||
if curl -sf -m 10 "$API/api/models/providers" -o /tmp/.st_prov 2>/dev/null; then
|
||||
if grep -q ollama /tmp/.st_prov && grep -q openrouter /tmp/.st_prov; then
|
||||
ok "API erreichbar, Provider ollama + openrouter verfuegbar"
|
||||
else
|
||||
fail "API erreichbar, aber Provider fehlen: $(cat /tmp/.st_prov)"
|
||||
fi
|
||||
else
|
||||
fail "API $API nicht erreichbar"
|
||||
fi
|
||||
rm -f /tmp/.st_prov
|
||||
|
||||
sec "Lokale Sprachdienste (vom Container aus)"
|
||||
|
||||
if inapp "$PY" -c "
|
||||
import httpx,sys
|
||||
r = httpx.post('http://host.docker.internal:8901/audio/speech',
|
||||
json={'model':'chatterbox','voice':'default','input':'Test.','response_format':'wav'}, timeout=180)
|
||||
sys.exit(0 if r.status_code == 200 and len(r.content) > 1000 else 1)"; then
|
||||
ok "TTS antwortet mit Audio (host.docker.internal:8901)"
|
||||
else
|
||||
fail "TTS liefert kein Audio — Dienst tot (ConnectError) oder VRAM voll (HTTP 500)?
|
||||
systemctl --user status open-notebook-tts; journalctl --user -u open-notebook-tts -n 30"
|
||||
fi
|
||||
|
||||
if inapp "$PY" -c "
|
||||
import httpx,sys
|
||||
r = httpx.get('http://host.docker.internal:8902/health', timeout=30)
|
||||
sys.exit(0 if r.status_code == 200 else 1)"; then
|
||||
ok "STT erreichbar (host.docker.internal:8902)"
|
||||
else
|
||||
fail "STT nicht erreichbar — systemctl --user status open-notebook-stt"
|
||||
fi
|
||||
|
||||
sec "Annahmen unserer Anpassungen (brechen leise bei Image-Updates)"
|
||||
|
||||
# 1) prompts/podcast: Verzeichnis-Mount verdeckt das Image-Verzeichnis komplett.
|
||||
# Kommt upstream eine dritte Vorlage dazu, waere sie unsichtbar.
|
||||
img_tpl="$(docker run --rm --entrypoint ls "$IMAGE" /app/prompts/podcast 2>/dev/null | sort | tr '\n' ' ')"
|
||||
our_tpl="$(ls prompts/podcast | sort | tr '\n' ' ')"
|
||||
if [ "$img_tpl" = "$our_tpl" ]; then
|
||||
ok "prompts/podcast: Dateiliste deckt sich mit dem Image ($our_tpl)"
|
||||
else
|
||||
fail "prompts/podcast weicht ab — unser Mount verdeckt Image-Dateien!
|
||||
Image: $img_tpl
|
||||
wir : $our_tpl"
|
||||
fi
|
||||
|
||||
# 2) Unsere Patches in den Vorlagen sind noch drin.
|
||||
if grep -q '{{ language }}' prompts/podcast/outline.jinja prompts/podcast/transcript.jinja &&
|
||||
head -1 prompts/podcast/outline.jinja | grep -q '/no_think'; then
|
||||
ok "Vorlagen enthalten Sprachanweisung ({{ language }}) und /no_think"
|
||||
else
|
||||
fail "Vorlagen-Patches fehlen — Podcasts kaemen wieder auf Englisch / mit kaputtem JSON"
|
||||
fi
|
||||
|
||||
# 3) podcast_creator reicht `language` ueberhaupt noch in die Vorlagen durch.
|
||||
if inapp grep -q "language" /app/.venv/lib/python3.12/site-packages/podcast_creator/nodes.py; then
|
||||
ok "podcast_creator kennt weiterhin ein language-Feld"
|
||||
else
|
||||
fail "podcast_creator reicht kein language mehr durch — Vorlagen-Patch wirkungslos"
|
||||
fi
|
||||
|
||||
# 4) TTS_BATCH_SIZE existiert noch als Stellschraube (sonst wieder 5 parallele Clips -> OOM).
|
||||
if inapp grep -q "TTS_BATCH_SIZE" /app/.venv/lib/python3.12/site-packages/podcast_creator/nodes.py; then
|
||||
ok "podcast_creator liest weiterhin TTS_BATCH_SIZE"
|
||||
else
|
||||
fail "TTS_BATCH_SIZE wird nicht mehr gelesen — Gefahr paralleler Clips (CUDA OOM)"
|
||||
fi
|
||||
|
||||
# 5) content-core sucht YouTube-Untertitel weiterhin ueber preferred_languages.
|
||||
if inapp grep -q "preferred_languages" /app/.venv/lib/python3.12/site-packages/content_core/processors/youtube.py; then
|
||||
ok "content-core nutzt weiterhin preferred_languages (deutsche Untertitel)"
|
||||
else
|
||||
fail "content-core sucht Untertitel anders — config/content_core.yaml pruefen"
|
||||
fi
|
||||
|
||||
# 6) content-core baut das STT-Modell weiterhin ohne base_url (deshalb die Env-Var).
|
||||
if inapp grep -q "OPENAI_COMPATIBLE_BASE_URL" /app/.venv/lib/python3.12/site-packages/esperanto/providers/stt/openai_compatible.py; then
|
||||
ok "esperanto akzeptiert weiterhin OPENAI_COMPATIBLE_BASE_URL_STT (Audio-Transkription)"
|
||||
else
|
||||
fail "esperanto liest die Env-Var nicht mehr — Audio-Quellen fielen auf OpenAI zurueck"
|
||||
fi
|
||||
|
||||
# 7) CCORE_CONFIG_PATH ersetzt die Config komplett. Bringt ein Update neue
|
||||
# Top-Level-Schluessel mit, frieren wir die alten ein — hier faellt das auf.
|
||||
img_keys="$(docker run --rm --entrypoint "$PY" -e CCORE_CONFIG_PATH= "$IMAGE" \
|
||||
-c "from content_core.config import load_config; print(' '.join(sorted(load_config())))" 2>/dev/null)"
|
||||
our_keys="$(inapp "$PY" -c "
|
||||
import yaml; print(' '.join(sorted(yaml.safe_load(open('/app/config/content_core.yaml')))))")"
|
||||
missing="$(comm -23 <(tr ' ' '\n' <<<"$img_keys" | sort -u) <(tr ' ' '\n' <<<"$our_keys" | sort -u) | tr '\n' ' ')"
|
||||
if [ -z "${missing// /}" ] && [ -n "$img_keys" ]; then
|
||||
ok "config/content_core.yaml enthaelt alle Default-Schluessel des Images"
|
||||
else
|
||||
fail "config/content_core.yaml fehlen Schluessel aus dem Image: ${missing:-(Image-Defaults nicht lesbar)}
|
||||
Neu erzeugen — siehe Kopfkommentar in config/content_core.yaml"
|
||||
fi
|
||||
|
||||
# 8) Episode-Profile haben eine Sprache (sonst wieder englische Podcasts).
|
||||
if curl -sf -m 10 "$API/api/episode-profiles" | python3 -c "
|
||||
import json,sys
|
||||
p=json.load(sys.stdin)
|
||||
sys.exit(0 if p and all(x.get('language') for x in p) else 1)" 2>/dev/null; then
|
||||
ok "alle Episode-Profile haben ein language-Feld gesetzt"
|
||||
else
|
||||
fail "mindestens ein Episode-Profil ohne language — Podcast kaeme auf Englisch"
|
||||
fi
|
||||
|
||||
if [ "${SKIP_NET:-0}" != "1" ]; then
|
||||
sec "Ende-zu-Ende (braucht Internet)"
|
||||
if inapp "$PY" -c "
|
||||
import asyncio,sys
|
||||
from content_core import extract_content
|
||||
async def m():
|
||||
r = await extract_content({'url':'https://www.youtube.com/watch?v=hzxiegk9QAg'})
|
||||
sys.exit(0 if len((r.content or '')) > 1000 else 1)
|
||||
asyncio.run(m())"; then
|
||||
ok "YouTube-Quelle liefert Transkript (deutsche Untertitel werden gefunden)"
|
||||
else
|
||||
fail "YouTube-Extraktion liefert keinen Text — CCORE_CONFIG_PATH/Sprachen pruefen"
|
||||
fi
|
||||
fi
|
||||
|
||||
sec "Ergebnis"
|
||||
if [ "$fails" -eq 0 ]; then
|
||||
echo " Alles gruen."
|
||||
exit 0
|
||||
fi
|
||||
echo " $fails Test(s) fehlgeschlagen."
|
||||
exit 1
|
||||
104
scripts/update_stack.sh
Executable file
104
scripts/update_stack.sh
Executable file
|
|
@ -0,0 +1,104 @@
|
|||
#!/usr/bin/env bash
|
||||
# Aktualisiert Open Notebook auf das neueste veroeffentlichte Image — mit Rueckfallpunkt.
|
||||
#
|
||||
# Ablauf:
|
||||
# 1. Neuen Digest aus der Registry holen (nichts zu tun, wenn schon aktuell).
|
||||
# 2. Stack stoppen und Daten sichern (surreal_data + notebook_data + docker-compose.yml).
|
||||
# 3. Digest in docker-compose.yml eintragen, Stack starten.
|
||||
# 4. Rauchtest (scripts/smoke_test.sh).
|
||||
# 5. Rot -> alles zurueckrollen (Daten UND Compose-Datei). Gruen -> fertig.
|
||||
#
|
||||
# Der Rollback der Daten ist der Grund fuer das Backup: Open Notebook migriert die
|
||||
# SurrealDB beim Start, und ein Downgrade der Anwendung kann eine migrierte Datenbank
|
||||
# nicht wieder lesen. Ohne Backup waere ein missgluecktes Update eine Einbahnstrasse.
|
||||
#
|
||||
# ./scripts/update_stack.sh # auf neuestes Image
|
||||
# ./scripts/update_stack.sh --dry-run # nur zeigen, was passieren wuerde
|
||||
set -euo pipefail
|
||||
|
||||
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$REPO"
|
||||
|
||||
TAG="lfnovo/open_notebook:v1-latest"
|
||||
BACKUP_DIR="$REPO/backups"
|
||||
KEEP_BACKUPS=3
|
||||
DRY_RUN=0
|
||||
[ "${1:-}" = "--dry-run" ] && DRY_RUN=1
|
||||
|
||||
say() { printf '\n\033[1m%s\033[0m\n' "$1"; }
|
||||
|
||||
# Gezielt aus der open_notebook-Zeile. Ein blindes grep auf den ersten Digest der Datei
|
||||
# erwischt surrealdb (steht weiter oben) — das sed weiter unten wuerde dann den falschen
|
||||
# Digest ersetzen und die Datenbank austauschen statt der Anwendung.
|
||||
pinned="$(grep -oP 'lfnovo/open_notebook\S*@\Ksha256:[0-9a-f]+' docker-compose.yml | head -1)"
|
||||
[ -n "$pinned" ] || { echo "FEHLER: kein open_notebook-Digest in docker-compose.yml gepinnt." >&2; exit 1; }
|
||||
|
||||
say "1/5 Registry abfragen"
|
||||
remote="$(docker buildx imagetools inspect "$TAG" --format '{{.Manifest.Digest}}')"
|
||||
echo " gepinnt: $pinned"
|
||||
echo " neu : $remote"
|
||||
if [ "$remote" = "$pinned" ]; then
|
||||
echo " Bereits aktuell — nichts zu tun."
|
||||
exit 0
|
||||
fi
|
||||
if [ "$DRY_RUN" = 1 ]; then
|
||||
echo " (--dry-run: hier wuerde das Update starten)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
ts="$(date +%Y%m%d-%H%M%S)"
|
||||
backup="$BACKUP_DIR/$ts"
|
||||
mkdir -p "$backup"
|
||||
|
||||
say "2/5 Stack stoppen und sichern -> backups/$ts"
|
||||
docker compose down
|
||||
cp docker-compose.yml "$backup/docker-compose.yml"
|
||||
# Die Datenverzeichnisse gehoeren root (der Container schreibt als root), deshalb wird
|
||||
# im Container gepackt statt auf dem Host — spart sudo.
|
||||
docker run --rm -v "$REPO:/work" -w /work --entrypoint tar \
|
||||
"lfnovo/open_notebook@$pinned" -czf "backups/$ts/data.tgz" surreal_data notebook_data
|
||||
echo " $(du -sh "$backup/data.tgz" | cut -f1) gesichert"
|
||||
|
||||
restore() {
|
||||
say "ROLLBACK"
|
||||
docker compose down || true
|
||||
cp "$backup/docker-compose.yml" docker-compose.yml
|
||||
docker run --rm -v "$REPO:/work" -w /work --entrypoint sh \
|
||||
"lfnovo/open_notebook@$pinned" -c \
|
||||
"rm -rf surreal_data notebook_data && tar -xzf backups/$ts/data.tgz"
|
||||
docker compose up -d
|
||||
echo " Alter Stand ($pinned) ist wiederhergestellt, Daten aus backups/$ts."
|
||||
echo " Das neue Image bleibt lokal liegen — Ursache pruefen, dann erneut versuchen."
|
||||
}
|
||||
|
||||
say "3/5 Auf neues Image umstellen"
|
||||
sed -i "s|@$pinned|@$remote|" docker-compose.yml
|
||||
docker compose up -d
|
||||
echo " gestartet, warte auf die API..."
|
||||
for _ in $(seq 1 30); do
|
||||
curl -sf -m 3 http://127.0.0.1:5055/api/models/providers >/dev/null 2>&1 && break
|
||||
sleep 5
|
||||
done
|
||||
|
||||
say "4/5 Rauchtest"
|
||||
if ./scripts/smoke_test.sh; then
|
||||
say "5/5 Fertig"
|
||||
newver="$(docker compose exec -T open_notebook grep -m1 '^version' /app/pyproject.toml |
|
||||
grep -oP '(?<=")[^"]+' || echo '?')"
|
||||
echo " Update auf $newver erfolgreich, Rauchtest gruen."
|
||||
echo " Backup: backups/$ts (bleibt liegen)"
|
||||
echo
|
||||
echo " Bitte den neuen Digest committen:"
|
||||
echo " git add docker-compose.yml && git commit -m 'Update Open Notebook auf $newver'"
|
||||
else
|
||||
echo
|
||||
echo " Rauchtest ROT — das Update wird zurueckgenommen."
|
||||
restore
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Alte Backups aufraeumen (Platte ist knapp).
|
||||
ls -1dt "$BACKUP_DIR"/*/ 2>/dev/null | tail -n +$((KEEP_BACKUPS + 1)) | while read -r old; do
|
||||
echo " raeume altes Backup weg: $(basename "$old")"
|
||||
rm -rf "$old"
|
||||
done
|
||||
Loading…
Add table
Add a link
Reference in a new issue