open_notebook/docker-compose.yml

64 lines
3 KiB
YAML
Raw Normal View History

services:
surrealdb:
image: surrealdb/surrealdb:v2
restart: always
pull_policy: always
command: start --log info --user ${SURREAL_USER:-root} --pass ${SURREAL_PASSWORD:-root} rocksdb:/mydata/mydatabase.db
user: root # Required for bind mounts on Linux
environment:
- SURREAL_EXPERIMENTAL_GRAPHQL=true
ports:
- "127.0.0.1:8000:8000"
volumes:
- ./surreal_data:/mydata
open_notebook:
image: lfnovo/open_notebook:v1-latest
restart: always
pull_policy: always
depends_on:
- surrealdb
ports:
- "127.0.0.1:8502:8502"
- "127.0.0.1:5055:5055"
environment:
- OPEN_NOTEBOOK_ENCRYPTION_KEY=${OPEN_NOTEBOOK_ENCRYPTION_KEY}
- SURREAL_URL=ws://surrealdb:8000/rpc
- SURREAL_USER=${SURREAL_USER:-root}
- SURREAL_PASSWORD=${SURREAL_PASSWORD:-root}
- SURREAL_NAMESPACE=open_notebook
- SURREAL_DATABASE=open_notebook
- OLLAMA_BASE_URL=http://host.docker.internal:11434
- OPENROUTER_API_KEY=${OPENROUTER_API_KEY}
Fix two podcast failures: dead TTS server after reboot + CUDA OOM Podcast generation failed twice at the audio stage, for two unrelated reasons that look similar from the UI but need opposite fixes. 1) TTS/STT ran as nohup background processes and silently did not survive a reboot. Podcasts then failed with "Failed to generate speech: All connection attempts failed" (httpx.ConnectError) even though outline and transcript had generated fine. Both now run as systemd user units. The unit files are versioned in services/systemd/ (using %h, not a hardcoded home) and installed by scripts/start_services.sh, which also enables linger so they start on boot without a login session. They pin GPU 2 by UUID, not by index: CUDA orders devices "fastest first", so index 2 can resolve to the T600. 2) GPU 2 is shared by three processes (TTS, STT and the separate chatterbox-tts MCP service on :9999), leaving ~12 GB of headroom. podcast_creator sends TTS_BATCH_SIZE (default 5) clips concurrently, and since /audio/speech is a sync FastAPI handler, they generated genuinely in parallel on one shared model. Activation memory multiplied, the TTS process hit 16.7 GB and threw torch.OutOfMemoryError, surfacing as "HTTP 500" from the endpoint. tts_server.py now serializes generation behind a lock (GPU-bound work, so parallelism buys no throughput — it only multiplies peak VRAM) and frees the cache afterwards. TTS_BATCH_SIZE=1 keeps the client from queuing requests in that lock and running into esperanto's 300s TTS timeout; ESPERANTO_TTS_TIMEOUT is raised to 600s as headroom. Verified: 5 concurrent /audio/speech requests all return 200 with GPU 2 peaking at ~11.5 GB (was 16.7 GB for the TTS process alone), and the previously failed episode now completes end to end — 38/38 batches, 10:56 min of audio, zero OOM. Docs record both failure signatures side by side, since ConnectError (server dead) and HTTP 500 (server alive, out of VRAM) have very different remedies. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 13:02:32 +02:00
# podcast_creator schickt sonst 5 Audio-Clips gleichzeitig an den TTS-Server.
# Die laufen dort echt parallel auf einer GPU -> Aktivierungsspeicher
# vervielfacht sich -> CUDA OOM (GPU 2 teilt sich TTS, STT und den
# chatterbox-MCP-Dienst). Der TTS-Server serialisiert intern zwar ohnehin,
# aber bei Batch 5 warten 4 Requests im Lock und laufen ins 300s-Timeout.
- TTS_BATCH_SIZE=1
# Kopfraum fuer lange Segmente; Default waere 300s pro Clip.
- ESPERANTO_TTS_TIMEOUT=600
Fix YouTube sources with German transcripts landing empty Adding a German YouTube video created a source with a title but no content — nothing to chat with, nothing to summarise. The failure was silent in the UI; the reason only showed up in the container log as "Failed to get transcript for video <id> after retries: No suitable transcript found". Open Notebook extracts YouTube via content-core, which reads captions with youtube-transcript-api. content_core/processors/youtube.py only looks for the languages listed in preferred_languages, and the built-in default is ["en", "es", "pt"] — no German. All four fallback attempts in _fetch_best_transcript use that same list, so a video carrying only a German transcript raises NoTranscriptFound, get_best_transcript swallows it and returns None, and the source is stored with an empty body. The title still arrives because it is scraped separately, which is what makes this look like a success. The youtube_transcripts.preferred_languages key in the package's own cc_config.yaml appears to be the knob for this, but it is dead code on the default path: load_config() copies only the "extraction" block out of that file, so the key never reaches CONFIG and the hardcoded fallback always wins. The only way to set it is CCORE_CONFIG_PATH. config/content_core.yaml (bind-mounted read-only) therefore sets preferred_languages: ["de", "en", "es", "pt"]. Note CCORE_CONFIG_PATH REPLACES the config wholesale rather than merging it (config.py: return yaml.safe_load(file)). The file is consequently a full dump of the effective default config plus the new key — a minimal override file would silently drop the extraction engines and the model/timeout defaults. Its header comment carries the command that regenerates it from inside the container, for when an image update changes content-core's defaults. Verified end to end on the reported video (hzxiegk9QAg): extraction now yields 21553 characters of German transcript, the source is created through POST /api/sources with that full text, embedding produces 17 chunks in 3.4s, and vector search returns the video as top hit. This only covers caption languages. A video with no captions at all still yields an empty source — there is no download-and-transcribe fallback in the image. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 13:20:14 +02:00
# content-core sucht YouTube-Transkripte sonst nur in en/es/pt — deutsche
# Videos landen dann als LEERE Quelle im Notebook. Siehe config/content_core.yaml.
- CCORE_CONFIG_PATH=/app/config/content_core.yaml
Add audio fallback so videos without captions work Open Notebook only reads existing captions from a video. Without them the source is created empty. yt-dlp lives on the host, not in the image, so the chain can't run inside the app: scripts/add_video_source.py bridges it. Captions present -> the URL goes in as a normal "link" source (fast, no download). No captions -> yt-dlp pulls just the audio track (mono 16 kHz mp3, ~6 MB per 17 min) and uploads it as an "upload" source. Open Notebook then transcribes it itself with its configured speech-to-text model — the local faster-whisper server — and embeds it. The script deliberately does not transcribe: the app's own pipeline already does STT, chunking and embedding. OPENAI_COMPATIBLE_BASE_URL_STT is what makes the upload path work at all. Open Notebook passes its STT model (openai_compatible/faster-whisper-large-v3) into content-core, but content-core builds it with AIFactory.create_speech_to_text(provider, model, {'timeout': ...}) — with no base_url. Esperanto can't locate the local server, content-core falls back to its default (openai/whisper-1) and the job dies with "OpenAI API key not found". That message is misleading: nothing is missing but the URL. The env var supplies it. An OPENAI_API_KEY is deliberately NOT put into the container — that would ship audio to a paid cloud service while a working Whisper sits idle on GPU 2. Caption availability is probed with youtube-transcript-api inside the container, not with yt-dlp. yt-dlp's automatic_captions lists YouTube's ~100 auto-translation targets (German is always among them), which youtube-transcript-api does not accept as transcripts — trusting it would route a Japanese-only video down the caption path and produce an empty source again. The uploaded mp3 is removed afterwards by the script. The API's delete_source=true flag is meant for exactly this but is inert in this content-core version: extract_content drops the field when building the graph state (the returned state carries delete_source=None), so the delete_file node never fires. Verified by running the flag through the real API path and watching the file survive. Files are also named after the video id, since every upload previously landed as audio.mp3 and a second video would have collided with the first. Verified end to end: the reported video (hzxiegk9QAg, 17:14) transcribes to 22069 characters via the local Whisper server (two segment requests, both 200), embeds into 14 chunks, and uploads/ is empty afterwards. An English video (aircAruvnKk) works with automatic language detection. Whisper's text is noticeably cleaner than YouTube's auto-captions ("GitHub" vs "Gitub", real punctuation), so --force-audio is useful even when captions exist. Known gap: the audio path stores the audio file as the source asset, so the original video URL is not recorded on the source (the caption path records it). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 13:49:33 +02:00
# Audio-Quellen (Upload, Video-Tonspur): Open Notebook reicht sein STT-Modell
# (openai_compatible/faster-whisper-large-v3) an content-core weiter, aber
# content-core baut das Modell OHNE base_url (es gibt nur timeout mit). Esperanto
# weiss dann nicht, wo der lokale Whisper-Server steht, faellt auf OpenAI zurueck
# und scheitert mit "OpenAI API key not found". Diese Variable liefert die URL nach.
- OPENAI_COMPATIBLE_BASE_URL_STT=http://host.docker.internal:8902
extra_hosts:
- "host.docker.internal:host-gateway"
volumes:
- ./notebook_data:/app/data
# Podcast-Prompt-Vorlagen mit expliziter Sprachanweisung ({{ language }})
# und abgeschaltetem Thinking-Modus (/no_think), damit Podcasts in der
# eingestellten Sprache erzeugt werden (statt Englisch) und das JSON bei
# langen Segmenten nicht durch riesige Reasoning-Blöcke abgeschnitten wird.
# Verzeichnis-Mount (nicht Einzeldateien), damit Edits ohne Inode-Probleme
# nach einem Container-Neustart greifen. Der Ordner im Image enthält nur
# genau diese zwei Vorlagen.
- ./prompts/podcast:/app/prompts/podcast:ro
Fix YouTube sources with German transcripts landing empty Adding a German YouTube video created a source with a title but no content — nothing to chat with, nothing to summarise. The failure was silent in the UI; the reason only showed up in the container log as "Failed to get transcript for video <id> after retries: No suitable transcript found". Open Notebook extracts YouTube via content-core, which reads captions with youtube-transcript-api. content_core/processors/youtube.py only looks for the languages listed in preferred_languages, and the built-in default is ["en", "es", "pt"] — no German. All four fallback attempts in _fetch_best_transcript use that same list, so a video carrying only a German transcript raises NoTranscriptFound, get_best_transcript swallows it and returns None, and the source is stored with an empty body. The title still arrives because it is scraped separately, which is what makes this look like a success. The youtube_transcripts.preferred_languages key in the package's own cc_config.yaml appears to be the knob for this, but it is dead code on the default path: load_config() copies only the "extraction" block out of that file, so the key never reaches CONFIG and the hardcoded fallback always wins. The only way to set it is CCORE_CONFIG_PATH. config/content_core.yaml (bind-mounted read-only) therefore sets preferred_languages: ["de", "en", "es", "pt"]. Note CCORE_CONFIG_PATH REPLACES the config wholesale rather than merging it (config.py: return yaml.safe_load(file)). The file is consequently a full dump of the effective default config plus the new key — a minimal override file would silently drop the extraction engines and the model/timeout defaults. Its header comment carries the command that regenerates it from inside the container, for when an image update changes content-core's defaults. Verified end to end on the reported video (hzxiegk9QAg): extraction now yields 21553 characters of German transcript, the source is created through POST /api/sources with that full text, embedding produces 17 chunks in 3.4s, and vector search returns the video as top hit. This only covers caption languages. A video with no captions at all still yields an empty source — there is no download-and-transcribe fallback in the image. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 13:20:14 +02:00
# content-core-Konfiguration (u.a. deutsche YouTube-Transkripte), per
# CCORE_CONFIG_PATH oben referenziert.
- ./config:/app/config:ro