Fix YouTube sources with German transcripts landing empty

Adding a German YouTube video created a source with a title but no content —
nothing to chat with, nothing to summarise. The failure was silent in the UI;
the reason only showed up in the container log as "Failed to get transcript for
video <id> after retries: No suitable transcript found".

Open Notebook extracts YouTube via content-core, which reads captions with
youtube-transcript-api. content_core/processors/youtube.py only looks for the
languages listed in preferred_languages, and the built-in default is
["en", "es", "pt"] — no German. All four fallback attempts in
_fetch_best_transcript use that same list, so a video carrying only a German
transcript raises NoTranscriptFound, get_best_transcript swallows it and returns
None, and the source is stored with an empty body. The title still arrives
because it is scraped separately, which is what makes this look like a success.

The youtube_transcripts.preferred_languages key in the package's own
cc_config.yaml appears to be the knob for this, but it is dead code on the
default path: load_config() copies only the "extraction" block out of that file,
so the key never reaches CONFIG and the hardcoded fallback always wins. The only
way to set it is CCORE_CONFIG_PATH.

config/content_core.yaml (bind-mounted read-only) therefore sets
preferred_languages: ["de", "en", "es", "pt"].

Note CCORE_CONFIG_PATH REPLACES the config wholesale rather than merging it
(config.py: return yaml.safe_load(file)). The file is consequently a full dump of
the effective default config plus the new key — a minimal override file would
silently drop the extraction engines and the model/timeout defaults. Its header
comment carries the command that regenerates it from inside the container, for
when an image update changes content-core's defaults.

Verified end to end on the reported video (hzxiegk9QAg): extraction now yields
21553 characters of German transcript, the source is created through
POST /api/sources with that full text, embedding produces 17 chunks in 3.4s, and
vector search returns the video as top hit.

This only covers caption languages. A video with no captions at all still yields
an empty source — there is no download-and-transcribe fallback in the image.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Dieter Schlüter 2026-07-11 13:20:14 +02:00
commit 2a057d04e1
5 changed files with 144 additions and 0 deletions

78
config/content_core.yaml Normal file
View file

@ -0,0 +1,78 @@
# content-core Konfiguration fuer dieses Deployment.
#
# Zweck: YouTube-Quellen mit deutschem Transkript nutzbar machen.
#
# content-core sucht Transkripte nur in "preferred_languages". Der eingebaute
# Standard ist ["en", "es", "pt"] — ohne Deutsch. Ein Video mit ausschliesslich
# deutschem Transkript liefert deshalb NoTranscriptFound, und Open Notebook legt
# die Quelle mit LEEREM Inhalt an (der Titel wird trotzdem geholt, der Fehler
# taucht nur im Log auf).
#
# Der Schluessel youtube_transcripts steht zwar in der cc_config.yaml des Pakets,
# wird von load_config() im Standardfall aber gar nicht eingelesen (dort wird nur
# der Block "extraction" uebernommen) — er ist also wirkungslos. Einziger Weg,
# ihn zu setzen: CCORE_CONFIG_PATH auf eine eigene Datei zeigen lassen.
#
# ACHTUNG: CCORE_CONFIG_PATH ERSETZT die Konfiguration vollstaendig, es wird
# nicht zusammengefuehrt (content_core/config.py: return yaml.safe_load(file)).
# Diese Datei ist deshalb ein vollstaendiger Abzug der effektiven Standardconfig
# plus youtube_transcripts. Bei einem Image-Update mit geaenderten Defaults neu
# erzeugen:
#
# docker compose exec -T open_notebook /app/.venv/bin/python -c "
# from content_core.config import CONFIG; import yaml, copy
# cfg = copy.deepcopy(CONFIG)
# cfg['youtube_transcripts'] = {'preferred_languages': ['de','en','es','pt']}
# print(yaml.safe_dump(cfg, sort_keys=False, allow_unicode=True))" > config/content_core.yaml
#
# Die openai/gpt-4o-mini-Eintraege unten sind content-core-interne Defaults; sie
# werden hier nicht benutzt (Open Notebook waehlt seine Modelle selbst) und
# stehen nur drin, weil die Datei vollstaendig sein muss.
speech_to_text:
provider: openai
model_name: whisper-1
timeout: 3600
default_model:
provider: openai
model_name: gpt-4o-mini
config:
temperature: 0.5
top_p: 1
max_tokens: 2000
timeout: 300
cleanup_model:
provider: openai
model_name: gpt-4o-mini
config:
temperature: 0
max_tokens: 8000
output_format: json
timeout: 600
summary_model:
provider: openai
model_name: gpt-4o-mini
config:
temperature: 0
top_p: 1
max_tokens: 2000
timeout: 300
extraction:
document_engine: auto
url_engine: auto
audio:
concurrency: 3
firecrawl:
api_url: null
docling:
output_format: markdown
pymupdf:
enable_formula_ocr: false
formula_threshold: 3
ocr_fallback: true
youtube_transcripts:
preferred_languages:
- de
- en
- es
- pt