Add audio fallback so videos without captions work
Open Notebook only reads existing captions from a video. Without them the source
is created empty. yt-dlp lives on the host, not in the image, so the chain can't
run inside the app: scripts/add_video_source.py bridges it.
Captions present -> the URL goes in as a normal "link" source (fast, no download).
No captions -> yt-dlp pulls just the audio track (mono 16 kHz mp3, ~6 MB per
17 min) and uploads it as an "upload" source. Open Notebook
then transcribes it itself with its configured speech-to-text
model — the local faster-whisper server — and embeds it. The
script deliberately does not transcribe: the app's own pipeline
already does STT, chunking and embedding.
OPENAI_COMPATIBLE_BASE_URL_STT is what makes the upload path work at all. Open
Notebook passes its STT model (openai_compatible/faster-whisper-large-v3) into
content-core, but content-core builds it with
AIFactory.create_speech_to_text(provider, model, {'timeout': ...}) — with no
base_url. Esperanto can't locate the local server, content-core falls back to its
default (openai/whisper-1) and the job dies with "OpenAI API key not found". That
message is misleading: nothing is missing but the URL. The env var supplies it.
An OPENAI_API_KEY is deliberately NOT put into the container — that would ship
audio to a paid cloud service while a working Whisper sits idle on GPU 2.
Caption availability is probed with youtube-transcript-api inside the container,
not with yt-dlp. yt-dlp's automatic_captions lists YouTube's ~100 auto-translation
targets (German is always among them), which youtube-transcript-api does not
accept as transcripts — trusting it would route a Japanese-only video down the
caption path and produce an empty source again.
The uploaded mp3 is removed afterwards by the script. The API's delete_source=true
flag is meant for exactly this but is inert in this content-core version:
extract_content drops the field when building the graph state (the returned state
carries delete_source=None), so the delete_file node never fires. Verified by
running the flag through the real API path and watching the file survive. Files
are also named after the video id, since every upload previously landed as
audio.mp3 and a second video would have collided with the first.
Verified end to end: the reported video (hzxiegk9QAg, 17:14) transcribes to 22069
characters via the local Whisper server (two segment requests, both 200), embeds
into 14 chunks, and uploads/ is empty afterwards. An English video (aircAruvnKk)
works with automatic language detection. Whisper's text is noticeably cleaner than
YouTube's auto-captions ("GitHub" vs "Gitub", real punctuation), so --force-audio
is useful even when captions exist.
Known gap: the audio path stores the audio file as the source asset, so the
original video URL is not recorded on the source (the caption path records it).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
2a057d04e1
commit
f350e49bd5
6 changed files with 340 additions and 2 deletions
36
CLAUDE.md
36
CLAUDE.md
|
|
@ -116,8 +116,40 @@ regenerates it from inside the container — do that after an image update if co
|
|||
change. The `openai`/`gpt-4o-mini` entries in it are content-core's internal defaults and are unused
|
||||
here (Open Notebook picks its own models); they're only present because the file must be complete.
|
||||
|
||||
Note this only fixes *caption* languages. A video with no captions at all still yields an empty
|
||||
source — there is no audio-download-and-transcribe fallback wired in.
|
||||
Note this only fixes *caption* languages. A video with **no** captions is handled by the audio
|
||||
fallback below.
|
||||
|
||||
### Video without captions — audio fallback (`scripts/add_video_source.py`)
|
||||
|
||||
`yt-dlp` exists on the **host** only (`~/miniforge3/bin/yt-dlp`), not in the image, so the chain
|
||||
can't live inside the app. `scripts/add_video_source.py` bridges it:
|
||||
|
||||
- captions available → hands the URL to Open Notebook as a `link` source (fast, no download);
|
||||
- no captions → `yt-dlp -x` pulls just the audio track (mono 16 kHz mp3, ~6 MB per 17 min) and
|
||||
uploads it as an `upload` source. Open Notebook then transcribes it **itself** via its configured
|
||||
speech-to-text model — i.e. the local faster-whisper server — and embeds it. We deliberately do
|
||||
not transcribe in the script: the app's own pipeline already does STT, chunking and embedding.
|
||||
|
||||
Caption availability is probed with `youtube-transcript-api` **inside the container**, not with
|
||||
`yt-dlp`. `yt-dlp`'s `automatic_captions` lists YouTube's ~100 auto-*translation* targets (German
|
||||
is always among them), which `youtube-transcript-api` does not accept as transcripts — trusting it
|
||||
would send a Japanese-only video down the caption path and produce an empty source again.
|
||||
|
||||
**`OPENAI_COMPATIBLE_BASE_URL_STT` is what makes the upload path work at all.** Open Notebook passes
|
||||
its STT model (`openai_compatible`/`faster-whisper-large-v3`) into content-core as
|
||||
`audio_provider`/`audio_model` (`open_notebook/graphs/source.py`), but content-core builds it with
|
||||
`AIFactory.create_speech_to_text(provider, model, {'timeout': …})` — **no `base_url`**
|
||||
(`content_core/processors/audio.py`). Esperanto then can't locate the local server, content-core
|
||||
silently falls back to its default (`openai`/`whisper-1`) and the job dies with *"OpenAI API key not
|
||||
found"* — which reads like a missing key but is really a missing URL. The env var supplies it. Do
|
||||
**not** "fix" this by putting an `OPENAI_API_KEY` into the container: that would send audio to OpenAI
|
||||
(paid, off-machine) while a working local Whisper sits idle on GPU 2, and it contradicts the
|
||||
two-provider rule above.
|
||||
|
||||
The script uploads with `delete_source=true` and names the file after the video id, so
|
||||
`notebook_data/uploads/` doesn't accumulate a few MB per video and a second video can't collide with
|
||||
the previous `audio.mp3`. Known gap: the `upload` path stores the audio file as the source's asset,
|
||||
so the original video URL is not recorded on the source (the `link` path does record it).
|
||||
|
||||
### Text-to-speech / speech-to-text
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue