dschlueter/open_notebook
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Fix YouTube sources with German transcripts landing empty

Browse source 
Adding a German YouTube video created a source with a title but no content —
nothing to chat with, nothing to summarise. The failure was silent in the UI;
the reason only showed up in the container log as "Failed to get transcript for
video <id> after retries: No suitable transcript found".

Open Notebook extracts YouTube via content-core, which reads captions with
youtube-transcript-api. content_core/processors/youtube.py only looks for the
languages listed in preferred_languages, and the built-in default is
["en", "es", "pt"] — no German. All four fallback attempts in
_fetch_best_transcript use that same list, so a video carrying only a German
transcript raises NoTranscriptFound, get_best_transcript swallows it and returns
None, and the source is stored with an empty body. The title still arrives
because it is scraped separately, which is what makes this look like a success.

The youtube_transcripts.preferred_languages key in the package's own
cc_config.yaml appears to be the knob for this, but it is dead code on the
default path: load_config() copies only the "extraction" block out of that file,
so the key never reaches CONFIG and the hardcoded fallback always wins. The only
way to set it is CCORE_CONFIG_PATH.

config/content_core.yaml (bind-mounted read-only) therefore sets
preferred_languages: ["de", "en", "es", "pt"].

Note CCORE_CONFIG_PATH REPLACES the config wholesale rather than merging it
(config.py: return yaml.safe_load(file)). The file is consequently a full dump of
the effective default config plus the new key — a minimal override file would
silently drop the extraction engines and the model/timeout defaults. Its header
comment carries the command that regenerates it from inside the container, for
when an image update changes content-core's defaults.

Verified end to end on the reported video (hzxiegk9QAg): extraction now yields
21553 characters of German transcript, the source is created through
POST /api/sources with that full text, embedding produces 17 chunks in 3.4s, and
vector search returns the video as top hit.

This only covers caption languages. A video with no captions at all still yields
an empty source — there is no download-and-transcribe fallback in the image.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Dieter Schlüter 2026-07-11 13:20:14 +02:00
commit 2a057d04e1
5 changed files with 144 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

30
CLAUDE.md
View file

@ -89,6 +89,36 @@ the `clips/` of completed episodes. Two guards matter — it aborts if any job i
skips directories touched in the last 60 minutes. **Deletion runs inside the container** (`docker compose
exec … rm -rf`), because the container writes as root and the host user can't remove those directories.
### YouTube sources — German transcripts need `CCORE_CONFIG_PATH`
Open Notebook extracts YouTube via `content-core`, which pulls captions with `youtube-transcript-api`
(no `yt-dlp`/`pytube` in the image). `content_core/processors/youtube.py` only ever looks for the
languages in `preferred_languages`, and the built-in default is `["en", "es", "pt"]` — **no German**.
All four fallback attempts in `_fetch_best_transcript` use that same list, so a video with only a
German transcript raises `NoTranscriptFound`, `get_best_transcript` swallows it and returns `None`,
and the source is created with an **empty body**. The title is still fetched (separate HTTP scrape),
so the failure is quiet: a source that looks fine but has no content, with the real reason only in
the container log (`Failed to get transcript for video <id> after retries`).
The `youtube_transcripts.preferred_languages` key in the package's `cc_config.yaml` looks like the
fix, but it is **dead code** on the default path: `load_config()` copies only the `extraction` block
out of `cc_config.yaml`, so the key never reaches `CONFIG` and the hardcoded `["en","es","pt"]`
fallback always wins. The only way to set it is `CCORE_CONFIG_PATH`.
Fix: `config/content_core.yaml` (bind-mounted read-only, `CCORE_CONFIG_PATH=/app/config/content_core.yaml`
in `docker-compose.yml`) sets `preferred_languages: ["de", "en", "es", "pt"]`.
**`CCORE_CONFIG_PATH` replaces the config wholesale — it does not merge** (`config.py`:
`return yaml.safe_load(file)`). So that file is a full dump of the effective default config plus the
`youtube_transcripts` key, not a small override; a minimal file would silently drop `extraction`
(engine selection) and the model/timeout defaults. Its header comment carries the one-liner that
regenerates it from inside the container — do that after an image update if content-core's defaults
change. The `openai`/`gpt-4o-mini` entries in it are content-core's internal defaults and are unused
here (Open Notebook picks its own models); they're only present because the file must be complete.
Note this only fixes *caption* languages. A video with no captions at all still yields an empty
source — there is no audio-download-and-transcribe fallback wired in.
### Text-to-speech / speech-to-text
Open Notebook's `openai_compatible` provider type talks to any endpoint implementing OpenAI's audio API shape (`POST /audio/speech`, `POST /audio/transcriptions` — see `esperanto.providers.tts.openai_compatible` / `.stt.openai_compatible` inside the app container for the exact contract). Since neither Ollama nor OpenRouter support TTS/STT, two thin wrapper servers in `services/` expose the locally installed tools this way: