The retried podcast failed with "Failed to parse ValidatedTranscript ... 2
validation errors ... missing": qwen3.5:27b had emitted "sender" instead of
"speaker" in 2 of 14 dialogue entries. Pydantic rejects the object, langchain
raises OUTPUT_PARSING_FAILURE and the entire job dies — after the outline and most
of the transcript were already done.
The template showed the schema as a JSON example but never forbade other key names,
so the model was free to drift. transcript.jinja now states the constraint
explicitly, right where the schema is defined.
This does not make the model deterministic — a parse failure still occurred once on
the next run, but podcast_creator's own retry absorbed it and the episode completed:
15:16 min of audio, 53 clips. Before the change the same failure was fatal.
Also, since this is the run that proved the non-root switch end to end: the final
MP3 is owned by dschlueter, and prune_podcast_data.py cleaned up the leftovers of
the two dead runs (5.5 MB) with a plain host-side rmtree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Stopping the stack kills an in-flight podcast job, but its status stays "running"
in the database forever: it never finishes, it blocks prune_podcast_data.py (which
deliberately refuses to run while a job is active), and it cannot even be retried —
/retry only accepts episodes in state "failed".
I walked into this myself: I ran `docker compose down` for the non-root switch
without checking for running jobs, and killed a podcast that had already produced
37 clips.
update_stack.sh now refuses to start when a podcast job is running, and says why.
Verified against a genuinely running job: it aborts before touching the stack.
BEDIENUNGSANLEITUNG documents how to recover an existing zombie. The status does not
live on the episode but on the linked `command` record (the episode's own job_status
is null), so the fix is to set that record to 'failed' via SurrealDB, after which the
normal retry works.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Everything the containers wrote into the bind mounts (surreal_data, notebook_data)
was owned by root, so the host user could not delete or back up his own podcast
data — prune_podcast_data.py and update_stack.sh had to detour through
`docker compose exec` for every rm and tar.
That was not a requirement, just the default: the open_notebook image declares no
USER, and the compose file even overrode surrealdb's own non-root user (65532)
with `user: root`, under the comment "Required for bind mounts on Linux" — which
is not true.
Both services now run as user: "1000:1000". Two things this needs:
- HOME=/tmp for open_notebook. Without it HOME resolves to "/" for a non-root uid,
uv cannot create /.cache/uv, and api + worker exit 2 at startup. Verified by
running the image as uid 1000 both ways.
- The data directories must be owned by that uid. Existing data was adopted with a
throwaway root container (chown -R), no sudo needed.
Both scripts drop the container detour and operate on the host directly, which is
simpler and now honest. smoke_test.sh gained two checks so a silent regression to
root cannot go unnoticed: the container's uid must match the host user, and no
foreign-owned files may exist under the data directories.
Verified: containers run as uid 1000, new files land as dschlueter and are
deletable without sudo, SurrealDB writes as 1000, a source can be created,
embedded and deleted through the API, and the full smoke test is green.
Note this deviates from what the image expects (it assumes root), so it is exactly
the kind of assumption an update can break — hence the smoke-test checks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Audited README, BEDIENUNGSANLEITUNG and CLAUDE.md against what the repo and the
running stack actually do. Findings, all fixed:
- CLAUDE.md claimed the repo holds "only deployment config (docker-compose.yml,
.env, .gitignore)". It has not been that for a while: services/ is our own
source (TTS/STT servers + systemd units), scripts/ is tooling, and
prompts/podcast/ + config/content_core.yaml are overlays bind-mounted over the
image. Those overlays are the fragile part, so the section now says so and
points at smoke_test.sh.
- CLAUDE.md still described the image as a moving tag and argued against patching
app code "because it would rot against pull_policy: always" — both obsolete
since the digest pin.
- README listed neither yt-dlp (needed by add_video_source.py) nor smoke_test.sh
in the verification step, and section 2.3 still told you to `docker compose pull`
as if the tag moved.
- BEDIENUNGSANLEITUNG had nothing at all about updates — the very thing a user has
to do periodically. New section 8 covers check_updates / update_stack / smoke_test
and, importantly, why updating is not automatic (DB migration is one-way without
a backup; the local adaptations can break silently without crashing).
- Renumbered the ad-hoc "3a. Updates" in README into a real section 4, and the
BEDIENUNGSANLEITUNG sections after the new 8 accordingly.
Verified mechanically: every scripts//services//config//prompts/ path named in the
docs exists, and all internal and cross-document anchors resolve (0 dead links —
one was already broken before this change: README pointed at
BEDIENUNGSANLEITUNG.md#stimmklonung instead of #5-stimmklonung).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Goal: at every session start, run exactly the application the repo says — and be able
to move to a new upstream release deliberately, with a way back.
The old setup (v1-latest + pull_policy: always) was unreliable in both directions.
`docker compose up -d` silently pulled a new application, including an irreversible
SurrealDB schema migration, while a reboot (restart: always) or `docker compose start`
kept running the old image without pulling anything. The running version was effectively
unpredictable.
Note v1-latest tracks releases, not main: the current image (v1.10.0, 2026-06-18) IS the
latest release — the repo being "ahead" is unreleased code, which is explicitly not
wanted here. So this is about guaranteeing and detecting, not catching up.
- docker-compose.yml: both images pinned by digest, pull_policy: missing.
- scripts/check_updates.sh: reports running version/digest vs the latest release and
registry digest. Changes nothing; meant for the session-start ritual.
- scripts/update_stack.sh: resolve new digest -> stop -> back up surreal_data,
notebook_data and docker-compose.yml -> repin -> start -> smoke test -> roll back data
AND compose file if the smoke test fails. The data backup is the point: the app migrates
the DB on startup and an older app cannot read a migrated DB, so a bad update would
otherwise be a one-way door. tar runs inside a container because the data dirs are
root-owned and the host user cannot restore over them.
- scripts/smoke_test.sh: checks what actually breaks here, not just "does it start".
Every local adaptation leans on upstream internals and can break silently: the
prompts/podcast directory mount masks the image's directory (a new template upstream
would be invisible), config/content_core.yaml freezes content-core's defaults (because
CCORE_CONFIG_PATH replaces rather than merges), and the env-var workarounds depend on
current content-core/esperanto/podcast_creator behaviour. So it verifies those
assumptions explicitly, plus API, TTS audio, STT and a real YouTube transcript.
Both scripts read the digest from the lfnovo/open_notebook line specifically. A naive
"first sha256 in the file" grep matches surrealdb (listed first) — caught while testing:
check_updates.sh falsely reported an update, and update_stack.sh would have rewritten the
database image instead of the application.
Verified: smoke test green against the current stack, and red (exit 1) when failures are
injected (dead TTS port, removed template variable). Backup/restore mechanics exercised
separately: 44 MB in 3.3s, restore yields 13 DB files and 91 notebook files. The update
path itself cannot be exercised end to end until a newer image exists.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Open Notebook only reads existing captions from a video. Without them the source
is created empty. yt-dlp lives on the host, not in the image, so the chain can't
run inside the app: scripts/add_video_source.py bridges it.
Captions present -> the URL goes in as a normal "link" source (fast, no download).
No captions -> yt-dlp pulls just the audio track (mono 16 kHz mp3, ~6 MB per
17 min) and uploads it as an "upload" source. Open Notebook
then transcribes it itself with its configured speech-to-text
model — the local faster-whisper server — and embeds it. The
script deliberately does not transcribe: the app's own pipeline
already does STT, chunking and embedding.
OPENAI_COMPATIBLE_BASE_URL_STT is what makes the upload path work at all. Open
Notebook passes its STT model (openai_compatible/faster-whisper-large-v3) into
content-core, but content-core builds it with
AIFactory.create_speech_to_text(provider, model, {'timeout': ...}) — with no
base_url. Esperanto can't locate the local server, content-core falls back to its
default (openai/whisper-1) and the job dies with "OpenAI API key not found". That
message is misleading: nothing is missing but the URL. The env var supplies it.
An OPENAI_API_KEY is deliberately NOT put into the container — that would ship
audio to a paid cloud service while a working Whisper sits idle on GPU 2.
Caption availability is probed with youtube-transcript-api inside the container,
not with yt-dlp. yt-dlp's automatic_captions lists YouTube's ~100 auto-translation
targets (German is always among them), which youtube-transcript-api does not
accept as transcripts — trusting it would route a Japanese-only video down the
caption path and produce an empty source again.
The uploaded mp3 is removed afterwards by the script. The API's delete_source=true
flag is meant for exactly this but is inert in this content-core version:
extract_content drops the field when building the graph state (the returned state
carries delete_source=None), so the delete_file node never fires. Verified by
running the flag through the real API path and watching the file survive. Files
are also named after the video id, since every upload previously landed as
audio.mp3 and a second video would have collided with the first.
Verified end to end: the reported video (hzxiegk9QAg, 17:14) transcribes to 22069
characters via the local Whisper server (two segment requests, both 200), embeds
into 14 chunks, and uploads/ is empty afterwards. An English video (aircAruvnKk)
works with automatic language detection. Whisper's text is noticeably cleaner than
YouTube's auto-captions ("GitHub" vs "Gitub", real punctuation), so --force-audio
is useful even when captions exist.
Known gap: the audio path stores the audio file as the source asset, so the
original video URL is not recorded on the source (the caption path records it).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adding a German YouTube video created a source with a title but no content —
nothing to chat with, nothing to summarise. The failure was silent in the UI;
the reason only showed up in the container log as "Failed to get transcript for
video <id> after retries: No suitable transcript found".
Open Notebook extracts YouTube via content-core, which reads captions with
youtube-transcript-api. content_core/processors/youtube.py only looks for the
languages listed in preferred_languages, and the built-in default is
["en", "es", "pt"] — no German. All four fallback attempts in
_fetch_best_transcript use that same list, so a video carrying only a German
transcript raises NoTranscriptFound, get_best_transcript swallows it and returns
None, and the source is stored with an empty body. The title still arrives
because it is scraped separately, which is what makes this look like a success.
The youtube_transcripts.preferred_languages key in the package's own
cc_config.yaml appears to be the knob for this, but it is dead code on the
default path: load_config() copies only the "extraction" block out of that file,
so the key never reaches CONFIG and the hardcoded fallback always wins. The only
way to set it is CCORE_CONFIG_PATH.
config/content_core.yaml (bind-mounted read-only) therefore sets
preferred_languages: ["de", "en", "es", "pt"].
Note CCORE_CONFIG_PATH REPLACES the config wholesale rather than merging it
(config.py: return yaml.safe_load(file)). The file is consequently a full dump of
the effective default config plus the new key — a minimal override file would
silently drop the extraction engines and the model/timeout defaults. Its header
comment carries the command that regenerates it from inside the container, for
when an image update changes content-core's defaults.
Verified end to end on the reported video (hzxiegk9QAg): extraction now yields
21553 characters of German transcript, the source is created through
POST /api/sources with that full text, embedding produces 17 chunks in 3.4s, and
vector search returns the video as top hit.
This only covers caption languages. A video with no captions at all still yields
an empty source — there is no download-and-transcribe fallback in the image.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Deleting an episode does not delete its data. DELETE /api/podcasts/episodes/{id}
resolves episode.audio_file and unlinks that one MP3 — the enclosing
data/podcasts/episodes/<uuid>/ directory, holding clips/ (one MP3 per dialogue
segment), outline.json and transcript.json, is never touched. Failed and
/retry-replaced runs leak a directory as well. Nothing reaps any of it: this
deployment had 15 directories on disk against 4 episodes in the database.
That is an upstream gap (roughly a shutil.rmtree of audio_path.parent.parent in
that handler), not something configurable here. Deliberately not patched by
bind-mounting a modified router — that would fork app logic into a deployment
repo and rot silently against pull_policy: always. Local cleanup instead.
scripts/prune_podcast_data.py reconciles the episode list from the API against
the directories on disk and removes:
- orphaned directories (no corresponding episode), and
- clips/ of completed episodes, since the clips are intermediate output once
the final MP3 exists.
Dry-run by default; --yes applies, --keep-clips restricts it to orphans.
Two guards, both tested: it aborts when any job is running/pending (a running
job has no audio_file yet, so its working directory is indistinguishable from an
orphan) and it skips directories touched within the last 60 minutes (--min-age).
It also refuses to act if the API is unreachable, rather than guessing.
Deletion runs inside the container (docker compose exec … rm -rf): the container
writes as root, so the host user cannot remove those directories — a plain
host-side rmtree fails with EPERM after the first directory.
Verified on this deployment: freed 13 MB (10 orphaned dirs + clips of 3 episodes,
49 MB -> 36 MB); all four surviving episodes still stream byte-identical MP3s
from /audio afterwards.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Podcast generation failed twice at the audio stage, for two unrelated reasons
that look similar from the UI but need opposite fixes.
1) TTS/STT ran as nohup background processes and silently did not survive a
reboot. Podcasts then failed with "Failed to generate speech: All connection
attempts failed" (httpx.ConnectError) even though outline and transcript had
generated fine.
Both now run as systemd user units. The unit files are versioned in
services/systemd/ (using %h, not a hardcoded home) and installed by
scripts/start_services.sh, which also enables linger so they start on boot
without a login session. They pin GPU 2 by UUID, not by index: CUDA orders
devices "fastest first", so index 2 can resolve to the T600.
2) GPU 2 is shared by three processes (TTS, STT and the separate chatterbox-tts
MCP service on :9999), leaving ~12 GB of headroom. podcast_creator sends
TTS_BATCH_SIZE (default 5) clips concurrently, and since /audio/speech is a
sync FastAPI handler, they generated genuinely in parallel on one shared
model. Activation memory multiplied, the TTS process hit 16.7 GB and threw
torch.OutOfMemoryError, surfacing as "HTTP 500" from the endpoint.
tts_server.py now serializes generation behind a lock (GPU-bound work, so
parallelism buys no throughput — it only multiplies peak VRAM) and frees the
cache afterwards. TTS_BATCH_SIZE=1 keeps the client from queuing requests in
that lock and running into esperanto's 300s TTS timeout; ESPERANTO_TTS_TIMEOUT
is raised to 600s as headroom.
Verified: 5 concurrent /audio/speech requests all return 200 with GPU 2 peaking
at ~11.5 GB (was 16.7 GB for the TTS process alone), and the previously failed
episode now completes end to end — 38/38 batches, 10:56 min of audio, zero OOM.
Docs record both failure signatures side by side, since ConnectError (server
dead) and HTTP 500 (server alive, out of VRAM) have very different remedies.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two root causes, both fixed in the bind-mounted podcast prompt templates
(prompts/podcast/*.jinja):
1. Language: the episode profile's `language` field IS passed to both
podcast_creator templates as {{ language }}, but the stock templates never
reference it — so podcasts came out English (driven only by the English
stock briefings/speakers), then read aloud by the German-locked Chatterbox
TTS = "English with a German accent". Added a CRITICAL LANGUAGE REQUIREMENT
block keyed on {{ language }} to both templates; now `language: "de"`
actually forces German. Verified end-to-end: a full run produced a 44-line
all-German transcript + audio.
2. Invalid json output failures: qwen3.5:27b is a thinking model; on long
segments the <think> block ate the response-token budget and truncated the
JSON. Prepended /no_think to both templates (~3x faster, valid JSON).
Templates are bind-mounted read-only via docker-compose (directory mount, so
edits survive a container restart without inode-staleness). Bundled briefings
and speaker backstories were also translated to German to reduce drift.
Known limitation documented: feeding an entire book (~70k tokens) as podcast
content makes each of the 6 LLM calls take ~3.5 min and is unreliable; use a
shorter source or summary. Confirmed working with concise content.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Consolidates and extends the curl patterns already scattered across other
sections into one reference point, notes the API requires no auth (loopback-
only), and points to the interactive /docs Swagger UI as the best starting
point for exploring endpoints not yet covered here.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- README's "Bekannte Stolpersteine" quick-reference list was missing the
podcast-language pitfall (already documented in CLAUDE.md/BEDIENUNGSANLEITUNG.md).
- The search/ask feature was only mentioned in passing; added a full section
with the working curl examples and the SSE response shape, since it's the
fastest/most accurate way to query long documents.
Everything else cross-checked against live state (credential num_ctx,
default models, running services, episode profile languages) and matched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The three bundled episode profiles (business_analysis, solo_expert,
tech_discussion) shipped with language=null, which podcast_creator treats as
"unspecified" and defaults to English for outline/transcript generation
regardless of source language. Set language="de" on all three.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
README.md gives the precise, reproducible install path (tested the two
scripts against the live instance - both idempotent, correctly detect
already-registered credentials/models). BEDIENUNGSANLEITUNG.md covers daily
usage: notebooks/sources, the three chat context modes (and why "nur
Erkenntnisse" needs a transformation run first), podcasts, voice cloning,
and troubleshooting for the issues actually hit during setup (num_ctx
truncation, embedding GPU contention, ufw).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
esperanto's OllamaLanguageModel silently caps num_ctx at 8192 unless the
credential sets it explicitly, no matter what OLLAMA_CONTEXT_LENGTH or the
model's real context window is. Any chat context beyond that got truncated,
and because qwen3.5:27b is a thinking model, the truncated prompt usually
produced an empty final answer (swallowed by clean_thinking_content) rather
than a visible error - looked like "chat gives no answer" in the UI. Set
num_ctx=98304 on the Ollama chat credential: the largest value that still
keeps the whole model on GPU 1 (131072 spills onto CPU, 5-6x slower).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
qwen3-embedding (repurposed 8B LLM, ~13GB loaded) didn't fit on GPU 1
alongside the resident 17GB chat model, so Ollama silently ran it 44-56%
on CPU. A single embed call took 39-54s, so any real document's chunk
batches blew past esperanto's 60s embedding timeout and got stuck retrying
forever - looked like "adding the source failed" but extraction was fine.
Also note: credential num_ctx doesn't work for embeddings (esperanto sends
it at the wrong JSON level, Ollama ignores it) - a small dedicated
embedding model is the real fix, not context-window tuning.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
male_thorsten/female_kerstin come from Piper's openly-licensed German voice
models, not scraped real-person audio, to avoid cloning someone's voice
without consent. Speaker profiles now use distinct voices per speaker instead
of everyone defaulting to the owner's own cloned voice.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
tts_server.py now resolves `voice` against services/voices/*.wav (gitignored,
personal recordings) and clones via chatterbox's audio_prompt_path, falling
back to "default" (symlinked to ~/chatterbox-tts-cli/my_voice_deutsch_60s.wav)
for unknown names, or to the old built-in-voice-by-language-code behavior if
`voice` is a recognized language. All podcast speaker profiles now point at
voice_id "default" until distinct per-speaker reference clips are added.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Wraps the locally installed chatterbox-tts and faster-whisper packages in thin
FastAPI servers implementing OpenAI's audio API shape, since neither Ollama nor
OpenRouter support speech. Pinned to GPU 2 to avoid contending with Ollama's
resident models on GPU 1. Requires UFW rules for 8901/8902 (same pattern as
the existing 11434 rule) since UFW defaults to deny-incoming.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
