dschlueter
03877588e1
Podcast generation failed twice at the audio stage, for two unrelated reasons that look similar from the UI but need opposite fixes. 1) TTS/STT ran as nohup background processes and silently did not survive a reboot. Podcasts then failed with "Failed to generate speech: All connection attempts failed" (httpx.ConnectError) even though outline and transcript had generated fine. Both now run as systemd user units. The unit files are versioned in services/systemd/ (using %h, not a hardcoded home) and installed by scripts/start_services.sh, which also enables linger so they start on boot without a login session. They pin GPU 2 by UUID, not by index: CUDA orders devices "fastest first", so index 2 can resolve to the T600. 2) GPU 2 is shared by three processes (TTS, STT and the separate chatterbox-tts MCP service on :9999), leaving ~12 GB of headroom. podcast_creator sends TTS_BATCH_SIZE (default 5) clips concurrently, and since /audio/speech is a sync FastAPI handler, they generated genuinely in parallel on one shared model. Activation memory multiplied, the TTS process hit 16.7 GB and threw torch.OutOfMemoryError, surfacing as "HTTP 500" from the endpoint. tts_server.py now serializes generation behind a lock (GPU-bound work, so parallelism buys no throughput — it only multiplies peak VRAM) and frees the cache afterwards. TTS_BATCH_SIZE=1 keeps the client from queuing requests in that lock and running into esperanto's 300s TTS timeout; ESPERANTO_TTS_TIMEOUT is raised to 600s as headroom. Verified: 5 concurrent /audio/speech requests all return 200 with GPU 2 peaking at ~11.5 GB (was 16.7 GB for the TTS process alone), and the previously failed episode now completes end to end — 38/38 batches, 10:56 min of audio, zero OOM. Docs record both failure signatures side by side, since ConnectError (server dead) and HTTP 500 (server alive, out of VRAM) have very different remedies. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
57 lines
1.9 KiB
Bash
Executable file
57 lines
1.9 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# Installiert und startet die lokalen TTS/STT-Wrapper-Server (services/tts_server.py,
|
|
# services/stt_server.py) als systemd-User-Dienste auf GPU 2.
|
|
#
|
|
# Frueher liefen die beiden als nohup-Hintergrundprozesse — die ueberleben keinen
|
|
# Reboot, und ein Podcast scheitert dann in der Vertonungsphase mit
|
|
# "Failed to generate speech: All connection attempts failed". Als systemd-Units
|
|
# starten sie automatisch mit (dank enable-linger auch ohne Login).
|
|
#
|
|
# Idempotent: mehrfach ausfuehrbar, aktualisiert die Units und startet sie neu.
|
|
set -euo pipefail
|
|
|
|
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
UNIT_SRC="$REPO/services/systemd"
|
|
UNIT_DST="$HOME/.config/systemd/user"
|
|
UNITS=(open-notebook-tts.service open-notebook-stt.service)
|
|
|
|
mkdir -p "$UNIT_DST"
|
|
for unit in "${UNITS[@]}"; do
|
|
install -m 644 "$UNIT_SRC/$unit" "$UNIT_DST/$unit"
|
|
done
|
|
systemctl --user daemon-reload
|
|
|
|
# Ohne Linger laufen User-Dienste nur waehrend einer aktiven Login-Session und
|
|
# starten nicht beim Boot.
|
|
if [ "$(loginctl show-user "$USER" --property=Linger --value 2>/dev/null)" != "yes" ]; then
|
|
echo "Aktiviere linger (Dienste sollen auch ohne Login laufen)..."
|
|
loginctl enable-linger "$USER"
|
|
fi
|
|
|
|
echo "Starte TTS/STT-Dienste..."
|
|
systemctl --user enable --quiet "${UNITS[@]}"
|
|
systemctl --user restart "${UNITS[@]}"
|
|
|
|
# Modelle werden beim Start geladen (whisper large-v3 braucht ~1 Minute).
|
|
wait_healthy() {
|
|
local name="$1" port="$2"
|
|
for _ in $(seq 1 45); do
|
|
if curl -sf -m 2 "http://127.0.0.1:${port}/health" >/dev/null 2>&1; then
|
|
echo " $name (Port $port) ist bereit."
|
|
return 0
|
|
fi
|
|
sleep 4
|
|
done
|
|
echo "WARNUNG: $name antwortet nach 180s nicht auf /health." >&2
|
|
echo " Logs: journalctl --user -u open-notebook-${name} -n 50" >&2
|
|
return 1
|
|
}
|
|
|
|
rc=0
|
|
wait_healthy tts 8901 || rc=1
|
|
wait_healthy stt 8902 || rc=1
|
|
|
|
if [ "$rc" -eq 0 ]; then
|
|
echo "Fertig. Status: systemctl --user status ${UNITS[*]}"
|
|
fi
|
|
exit "$rc"
|