Add audio fallback so videos without captions work

Open Notebook only reads existing captions from a video. Without them the source
is created empty. yt-dlp lives on the host, not in the image, so the chain can't
run inside the app: scripts/add_video_source.py bridges it.

Captions present -> the URL goes in as a normal "link" source (fast, no download).
No captions     -> yt-dlp pulls just the audio track (mono 16 kHz mp3, ~6 MB per
                   17 min) and uploads it as an "upload" source. Open Notebook
                   then transcribes it itself with its configured speech-to-text
                   model — the local faster-whisper server — and embeds it. The
                   script deliberately does not transcribe: the app's own pipeline
                   already does STT, chunking and embedding.

OPENAI_COMPATIBLE_BASE_URL_STT is what makes the upload path work at all. Open
Notebook passes its STT model (openai_compatible/faster-whisper-large-v3) into
content-core, but content-core builds it with
AIFactory.create_speech_to_text(provider, model, {'timeout': ...}) — with no
base_url. Esperanto can't locate the local server, content-core falls back to its
default (openai/whisper-1) and the job dies with "OpenAI API key not found". That
message is misleading: nothing is missing but the URL. The env var supplies it.
An OPENAI_API_KEY is deliberately NOT put into the container — that would ship
audio to a paid cloud service while a working Whisper sits idle on GPU 2.

Caption availability is probed with youtube-transcript-api inside the container,
not with yt-dlp. yt-dlp's automatic_captions lists YouTube's ~100 auto-translation
targets (German is always among them), which youtube-transcript-api does not
accept as transcripts — trusting it would route a Japanese-only video down the
caption path and produce an empty source again.

The uploaded mp3 is removed afterwards by the script. The API's delete_source=true
flag is meant for exactly this but is inert in this content-core version:
extract_content drops the field when building the graph state (the returned state
carries delete_source=None), so the delete_file node never fires. Verified by
running the flag through the real API path and watching the file survive. Files
are also named after the video id, since every upload previously landed as
audio.mp3 and a second video would have collided with the first.

Verified end to end: the reported video (hzxiegk9QAg, 17:14) transcribes to 22069
characters via the local Whisper server (two segment requests, both 200), embeds
into 14 chunks, and uploads/ is empty afterwards. An English video (aircAruvnKk)
works with automatic language detection. Whisper's text is noticeably cleaner than
YouTube's auto-captions ("GitHub" vs "Gitub", real punctuation), so --force-audio
is useful even when captions exist.

Known gap: the audio path stores the audio file as the source asset, so the
original video URL is not recorded on the source (the caption path records it).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Dieter Schlüter 2026-07-11 13:49:33 +02:00
commit f350e49bd5
6 changed files with 340 additions and 2 deletions

View file

@ -116,8 +116,40 @@ regenerates it from inside the container — do that after an image update if co
change. The `openai`/`gpt-4o-mini` entries in it are content-core's internal defaults and are unused
here (Open Notebook picks its own models); they're only present because the file must be complete.
Note this only fixes *caption* languages. A video with no captions at all still yields an empty
source — there is no audio-download-and-transcribe fallback wired in.
Note this only fixes *caption* languages. A video with **no** captions is handled by the audio
fallback below.
### Video without captions — audio fallback (`scripts/add_video_source.py`)
`yt-dlp` exists on the **host** only (`~/miniforge3/bin/yt-dlp`), not in the image, so the chain
can't live inside the app. `scripts/add_video_source.py` bridges it:
- captions available → hands the URL to Open Notebook as a `link` source (fast, no download);
- no captions → `yt-dlp -x` pulls just the audio track (mono 16 kHz mp3, ~6 MB per 17 min) and
uploads it as an `upload` source. Open Notebook then transcribes it **itself** via its configured
speech-to-text model — i.e. the local faster-whisper server — and embeds it. We deliberately do
not transcribe in the script: the app's own pipeline already does STT, chunking and embedding.
Caption availability is probed with `youtube-transcript-api` **inside the container**, not with
`yt-dlp`. `yt-dlp`'s `automatic_captions` lists YouTube's ~100 auto-*translation* targets (German
is always among them), which `youtube-transcript-api` does not accept as transcripts — trusting it
would send a Japanese-only video down the caption path and produce an empty source again.
**`OPENAI_COMPATIBLE_BASE_URL_STT` is what makes the upload path work at all.** Open Notebook passes
its STT model (`openai_compatible`/`faster-whisper-large-v3`) into content-core as
`audio_provider`/`audio_model` (`open_notebook/graphs/source.py`), but content-core builds it with
`AIFactory.create_speech_to_text(provider, model, {'timeout': …})` — **no `base_url`**
(`content_core/processors/audio.py`). Esperanto then can't locate the local server, content-core
silently falls back to its default (`openai`/`whisper-1`) and the job dies with *"OpenAI API key not
found"* — which reads like a missing key but is really a missing URL. The env var supplies it. Do
**not** "fix" this by putting an `OPENAI_API_KEY` into the container: that would send audio to OpenAI
(paid, off-machine) while a working local Whisper sits idle on GPU 2, and it contradicts the
two-provider rule above.
The script uploads with `delete_source=true` and names the file after the video id, so
`notebook_data/uploads/` doesn't accumulate a few MB per video and a second video can't collide with
the previous `audio.mp3`. Known gap: the `upload` path stores the audio file as the source's asset,
so the original video URL is not recorded on the source (the `link` path does record it).
### Text-to-speech / speech-to-text