From 5b389fae58f1f85aa1407ea841649e3983963c3b Mon Sep 17 00:00:00 2001 From: dschlueter Date: Sat, 11 Jul 2026 15:29:14 +0200 Subject: [PATCH 1/2] Guard against stopping the stack while a podcast is running MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Stopping the stack kills an in-flight podcast job, but its status stays "running" in the database forever: it never finishes, it blocks prune_podcast_data.py (which deliberately refuses to run while a job is active), and it cannot even be retried — /retry only accepts episodes in state "failed". I walked into this myself: I ran `docker compose down` for the non-root switch without checking for running jobs, and killed a podcast that had already produced 37 clips. update_stack.sh now refuses to start when a podcast job is running, and says why. Verified against a genuinely running job: it aborts before touching the stack. BEDIENUNGSANLEITUNG documents how to recover an existing zombie. The status does not live on the episode but on the linked `command` record (the episode's own job_status is null), so the fix is to set that record to 'failed' via SurrealDB, after which the normal retry works. Co-Authored-By: Claude Opus 4.8 --- BEDIENUNGSANLEITUNG.md | 36 ++++++++++++++++++++++++++++++++++++ scripts/update_stack.sh | 18 ++++++++++++++++++ 2 files changed, 54 insertions(+) diff --git a/BEDIENUNGSANLEITUNG.md b/BEDIENUNGSANLEITUNG.md index f86ed80..f0d2c70 100644 --- a/BEDIENUNGSANLEITUNG.md +++ b/BEDIENUNGSANLEITUNG.md @@ -543,6 +543,42 @@ for t in YouTubeTranscriptApi().list('VIDEO_ID'): print(t.language_code, t.is_ge Kommt hier nichts zurück, hat das Video keine Untertitel — dann hilft nur, das Video separat herunterzuladen und über die STT-Funktion (Audio-Upload) zu transkribieren. +### Podcast hängt ewig auf „running" (Karteileiche nach Container-Neustart) + +Wird der Stack gestoppt (`docker compose down`, Update, Reboot), während ein Podcast läuft, ist der +Job weg — aber in der Datenbank steht er **weiterhin auf `running`**. Er wird nie fertig, blockiert +`prune_podcast_data.py` (das bricht bei laufenden Jobs absichtlich ab) und lässt sich auch nicht +wiederholen: `/retry` akzeptiert nur Episoden im Status `failed`. + +Erkennen: keine neuen Clips im Episodenordner, GPU 2 im Leerlauf (`nvidia-smi`), nichts im Log +(`docker compose logs --since 5m open_notebook`). + +Der Status hängt nicht an der Episode, sondern am verknüpften `command`-Datensatz. Ihn auf +`failed` setzen — danach funktioniert der normale Retry-Knopf: + +```bash +set -a; . ./.env; set +a +# 1. Command-ID der Episode holen +curl -s -X POST http://127.0.0.1:8000/sql -u "$SURREAL_USER:$SURREAL_PASSWORD" \ + -H "surreal-ns: open_notebook" -H "surreal-db: open_notebook" -H "Accept: application/json" \ + -d "SELECT command FROM episode:;" +# 2. Auf failed setzen +curl -s -X POST http://127.0.0.1:8000/sql -u "$SURREAL_USER:$SURREAL_PASSWORD" \ + -H "surreal-ns: open_notebook" -H "surreal-db: open_notebook" -H "Accept: application/json" \ + -d "UPDATE command: SET status = 'failed', error_message = 'Worker beim Neustart beendet';" +# 3. Wiederholen +curl -X POST http://127.0.0.1:5055/api/podcasts/episodes//retry +``` + +**Vorbeugen:** vor `docker compose down` oder `./scripts/update_stack.sh` kurz prüfen, ob ein +Podcast läuft — die Episodenliste in der UI zeigt es, oder: + +```bash +curl -s http://127.0.0.1:5055/api/podcasts/episodes | python3 -c " +import json,sys +print([e['name'] for e in json.load(sys.stdin) if e['job_status']=='running'] or 'kein Job läuft')" +``` + ### Podcast schlägt beim Vertonen fehl („Failed to generate speech") Text und Transkript sind fertig, erst die Sprachausgabe scheitert. Die genaue Fehlermeldung diff --git a/scripts/update_stack.sh b/scripts/update_stack.sh index f987919..ec0c1a9 100755 --- a/scripts/update_stack.sh +++ b/scripts/update_stack.sh @@ -46,6 +46,24 @@ if [ "$DRY_RUN" = 1 ]; then exit 0 fi +# Ein laufender Podcast ueberlebt das Stoppen nicht: der Job ist weg, sein Status bleibt +# aber auf "running" stehen. Er wird nie fertig, blockiert prune_podcast_data.py und laesst +# sich nicht per /retry wiederholen (das will "failed"). Also vorher fragen, nicht nachher +# aufraeumen. +running_jobs="$(curl -sf -m 10 http://127.0.0.1:5055/api/podcasts/episodes 2>/dev/null | + python3 -c " +import json,sys +try: eps = json.load(sys.stdin) +except Exception: sys.exit(0) +print('; '.join(e['name'] for e in eps if e.get('job_status') in ('running','pending','submitted')))" 2>/dev/null)" +if [ -n "${running_jobs// /}" ]; then + echo + echo "ABBRUCH: Es laeuft gerade ein Podcast-Job: $running_jobs" + echo " Das Stoppen wuerde ihn abschiessen und als Karteileiche zuruecklassen." + echo " Bitte warten, bis er fertig ist, und danach erneut starten." + exit 1 +fi + ts="$(date +%Y%m%d-%H%M%S)" backup="$BACKUP_DIR/$ts" mkdir -p "$backup" From a10874c574c05445718f1d1eea09fb82f5f216ca Mon Sep 17 00:00:00 2001 From: dschlueter Date: Sat, 11 Jul 2026 15:56:02 +0200 Subject: [PATCH 2/2] Pin the transcript JSON keys so a stray "sender" cannot kill the podcast MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The retried podcast failed with "Failed to parse ValidatedTranscript ... 2 validation errors ... missing": qwen3.5:27b had emitted "sender" instead of "speaker" in 2 of 14 dialogue entries. Pydantic rejects the object, langchain raises OUTPUT_PARSING_FAILURE and the entire job dies — after the outline and most of the transcript were already done. The template showed the schema as a JSON example but never forbade other key names, so the model was free to drift. transcript.jinja now states the constraint explicitly, right where the schema is defined. This does not make the model deterministic — a parse failure still occurred once on the next run, but podcast_creator's own retry absorbed it and the episode completed: 15:16 min of audio, 53 clips. Before the change the same failure was fatal. Also, since this is the run that proved the non-root switch end to end: the final MP3 is owned by dschlueter, and prune_podcast_data.py cleaned up the leftovers of the two dead runs (5.5 MB) with a plain host-side rmtree. Co-Authored-By: Claude Opus 4.8 --- prompts/podcast/transcript.jinja | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/prompts/podcast/transcript.jinja b/prompts/podcast/transcript.jinja index 3b9b75f..40d787a 100644 --- a/prompts/podcast/transcript.jinja +++ b/prompts/podcast/transcript.jinja @@ -87,6 +87,11 @@ Follow these format requirements strictly: } ``` +CRITICAL KEY NAMES: every object inside "transcript" must use EXACTLY the two keys +"speaker" and "dialogue" — never "sender", "name", "text", "content" or any other +variant, not even for a single entry. One wrong key makes the entire podcast fail +to parse and the whole job is lost. + Formatting instructions: {{ format_instructions}}