open_notebook/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## What this repo is

This is a local Docker Compose deployment of [Open Notebook](https://github.com/lfnovo/open-notebook) (upstream image `lfnovo/open_notebook`), not a clone of its source. The application's own code is **not** here and its bugs/features belong upstream — this repo only concerns itself with how the stack is run locally.

What *is* here, beyond `docker-compose.yml`/`.env`:

- `services/` — our own source: the TTS/STT wrapper servers (Open Notebook has no local TTS/STT provider) plus their systemd units.
- `scripts/` — setup, maintenance and update tooling (see Commands below).
- `prompts/podcast/`, `config/content_core.yaml` — **overlays** bind-mounted over files inside the image, to fix upstream behaviour we can't configure otherwise (podcast language/reliability, YouTube caption languages). These are the fragile part: they can silently rot when the image is updated, which is what `scripts/smoke_test.sh` exists to catch.

## Commands

```bash
docker compose up -d              # start (images are digest-pinned; no implicit upgrade)
docker compose down                # stop and remove containers (data volumes persist)
docker compose logs -f open_notebook   # follow app logs
docker compose ps                  # container status

./scripts/check_updates.sh         # is there a newer release? (reports only, changes nothing)
./scripts/update_stack.sh          # backup -> update -> smoke test -> rollback on failure
./scripts/smoke_test.sh            # verify the stack still does what we rely on
```

`smoke_test.sh` is the closest thing to a test suite here — run it after touching
`docker-compose.yml`, the bind-mounted overlays, or the host TTS/STT services.

## Image versioning

Images are **pinned by digest**, and updating is an explicit act — never a side effect of
starting the stack.

The previous setup (`v1-latest` + `pull_policy: always`) was unreliable in *both* directions:
`docker compose up -d` silently pulled a new application — including an irreversible SurrealDB
schema migration — while a reboot (`restart: always`) or `docker compose start` kept running the
**old** image without any pull. So the running version was essentially unpredictable.

Note `v1-latest` tracks *releases*, not `main`: the tag lags the repo by weeks. Chasing `main`
would mean building from source and running unreleased code — explicitly not wanted here.

Updating (`scripts/update_stack.sh`): resolve the new digest → stop → back up `surreal_data` +
`notebook_data` + `docker-compose.yml` into `backups/<ts>/` → rewrite the digest → start → run
`smoke_test.sh` → **roll back data and compose file if the smoke test fails**. The data backup is
the point: the app migrates the database on startup, and an older application generally cannot
read a migrated database, so a bad update would otherwise be a one-way door. Backup/restore runs
`tar` **inside a container** because the data directories are root-owned (the container writes as
root) and the host user cannot restore over them.

Both scripts read the digest from the `lfnovo/open_notebook` line specifically — a naive "first
sha256 in the file" grep picks up **surrealdb** (it is listed first), which would make the update
script rewrite the database image instead of the application.

### Why a smoke test, and not just "does it start"

Every local adaptation in this repo leans on upstream internals, and an update can break them
*silently* — the stack comes up fine and misbehaves later:

- the `prompts/podcast/` **directory** mount masks the image's directory: a third template added
  upstream would simply be invisible;
- `config/content_core.yaml` is a full dump of content-core's defaults (because `CCORE_CONFIG_PATH`
  replaces rather than merges), so new upstream defaults would be frozen out;
- the env-var workarounds (`TTS_BATCH_SIZE`, `OPENAI_COMPATIBLE_BASE_URL_STT`,
  `ESPERANTO_TTS_TIMEOUT`) depend on how content-core / esperanto / podcast_creator behave today.

`smoke_test.sh` therefore checks those assumptions explicitly (template file list vs. the image,
`{{ language }}` and `/no_think` still present, `language` still plumbed by podcast_creator,
`TTS_BATCH_SIZE` still read, `preferred_languages` still used, esperanto still honouring
`OPENAI_COMPATIBLE_BASE_URL_STT`, our config still covering all default keys) on top of the obvious
end-to-end ones (API, TTS audio, STT, YouTube transcript). `SKIP_NET=1` skips the internet-dependent
check.

## Architecture

Two services, defined in `docker-compose.yml`:

- **surrealdb** (`surrealdb/surrealdb:v2`) — the database, RocksDB-backed, bound to `127.0.0.1:8000` only (local debugging access, e.g. `surreal sql`). Credentials come from `SURREAL_USER`/`SURREAL_PASSWORD` (default `root`/`root` if unset in `.env`) and are shared with the `open_notebook` service so they stay in sync.
- **open_notebook** (`lfnovo/open_notebook:v1-latest`, digest-pinned — see *Image versioning*) — bundles the web UI (port 8502) and REST API (port 5055) in one image, both bound to `127.0.0.1` only. Connects to `surrealdb` over the internal compose network (`ws://surrealdb:8000/rpc`).

Data persists in `./surreal_data` and `./notebook_data` (bind mounts, gitignored).

### AI providers

This deployment intentionally uses only two AI providers — no OpenAI/Anthropic/Google keys are wired in, even though such keys may exist in the host environment:

- **Ollama**, running natively on the host (not containerized) at `0.0.0.0:11434`, restricted to GPU 1+2 (RTX 3090s) via `CUDA_VISIBLE_DEVICES=1,2` in its systemd unit — GPU 0 (a T600 used for the desktop) is intentionally excluded. The `open_notebook` container reaches it via `OLLAMA_BASE_URL=http://host.docker.internal:11434`, which requires the `extra_hosts: host.docker.internal:host-gateway` entry in the compose file (Linux has no Docker-Desktop magic DNS for this).
- **OpenRouter**, the only cloud provider, configured via `OPENROUTER_API_KEY` (read from the host environment into `.env`).

Neither `surrealdb` nor `open_notebook` do local model inference themselves, so no GPU device reservation is needed in the compose file — all GPU usage happens inside the host's Ollama process.

**Embedding model:** use `nomic-embed-text` (274 MB, 2048 native context), not `qwen3-embedding` (a repurposed 8B-class LLM, ~13GB loaded). With the resident `qwen3.5:27b` chat model already using 17GB of GPU 1's 24GB, `qwen3-embedding` didn't fit and Ollama silently ran it 44-56% on CPU — a single embedding call took 39-54s, so any real document (many chunks, batched) blew past esperanto's 60s embedding timeout (`ESPERANTO_EMBEDDING_TIMEOUT`) and got stuck in an infinite retry loop that looked like "adding the source failed" even though extraction worked fine. Note: the credential's `num_ctx` field (meant to shrink Ollama's context/VRAM footprint) does **not** work for embeddings — esperanto's `OllamaEmbeddingModel` puts it at the top level of the `/api/embed` payload instead of nesting it under `options`, so Ollama silently ignores it. Don't rely on `num_ctx` to fix embedding VRAM contention; use a genuinely small embedding model instead.

A separate local `llama.cpp` server exists on this machine (managed via `~/llamacppctl-server_neu/llamacppctl/`, container name `llama_cpp_server`) but is **not** wired into this stack. It can be added later as an additional OpenAI-compatible provider through the Open Notebook UI (Settings → AI Providers) if needed — note it competes for the same GPU 1/2 VRAM as Ollama, so don't run large models on both simultaneously.

**Chat model context window:** the Ollama credential used for `qwen3.5:27b` (`credential:di8b31l16zikyyb6isyi`) has `num_ctx=98304` set explicitly. Without this, esperanto's `OllamaLanguageModel` silently defaults `num_ctx` to **8192** regardless of the model's real capability or `OLLAMA_CONTEXT_LENGTH` — any chat context (e.g. "full content" mode on a real document) beyond that gets truncated by Ollama, and because `qwen3.5:27b` is a thinking model, the truncated prompt frequently produces an empty final answer (the response is silently swallowed by `clean_thinking_content`) instead of a visible error — looks exactly like "chat gives no answer" from the UI. `98304` is the largest value that still keeps the whole model on GPU 1 (`ollama ps` shows `100% GPU`); `131072` spills ~91% onto CPU and makes a single response take 5-6 minutes instead of ~1-2. Even at 98304, a full-book "Volltext" chat (~70k+ tokens of context) takes ~2-3 minutes — that's inherent to processing that much context on one consumer GPU, not a bug. For large documents, the default short/RAG context mode (only relevant chunks, not the whole document) stays fast; reserve "Volltext" for shorter sources. If `qwen3.5:27b` ever gets swapped for a different Ollama chat model, re-check this credential's `num_ctx` still makes sense for its VRAM footprint.

**Podcast language + reliability — fixed via patched prompt templates (`prompts/podcast/`,
bind-mounted).** Two separate problems, one fix location:

1. *Language.* The episode profile's `language` field (`"de"` → resolved to `"German"`) **is**
   passed into both `podcast_creator` prompt templates as a `{{ language }}` variable
   (`nodes.py`, both `generate_outline_node` and `generate_transcript_node`), but the stock
   templates shipped in the image never reference it — so setting `language` had no effect and
   podcasts came out in English (the stock briefings and speaker backstories are English, which
   is the only language signal the model then sees). English text read aloud by the German-locked
   Chatterbox TTS is what produced the "English with a bad German accent" symptom.
2. *Reliability.* `qwen3.5:27b` is a thinking model. Podcast content is the **entire notebook**
   (`Notebook.get_context()` → every source's full text + all insights; for a whole book that's
   ~70k+ tokens), fed into *each* transcript-segment call. On long segments the `<think>` block
   eats the 5000-token response budget and the JSON gets truncated → `Invalid json output` →
   whole job fails. (Confirmed: `/no_think` cut a segment from 98s to 34s and left valid JSON.)

Fix — both handled in the two templates under `prompts/podcast/`, bind-mounted read-only over
the image's copies via `docker-compose.yml`:
- Prepended `/no_think` as the first line of both templates (Qwen honors it anywhere; harmless
  text for non-Qwen models) — disables thinking, freeing the token budget and ~3x speedup.
- Added a `{% if language %}CRITICAL LANGUAGE REQUIREMENT … in {{ language }}{% endif %}` block
  near the end of both templates, so the already-plumbed `language` field now actually forces the
  output language.

The bundled briefings and speaker backstory/personality were **also** translated to German (belt
and suspenders; reduces English drift from the source content), but the template change is what
makes `language` authoritative. **Editing these templates requires a container restart** to take
effect — they're bind-mounted, and the mount is the directory `prompts/podcast/` (not individual
files, which go stale on inode-replacing edits). If you switch off `qwen3.5:27b`, the `/no_think`
line becomes inert but harmless.

If you create your own episode profile, just set `language` — the template handles the rest.

**Deleting an episode leaks its working directory (upstream gap).** `DELETE /api/podcasts/episodes/{id}`
(`api/routers/podcasts.py`) resolves `episode.audio_file` and calls `audio_path.unlink()` on it — the
final MP3 only. The enclosing `data/podcasts/episodes/<uuid>/` directory, holding `clips/` (one MP3 per
dialogue segment), `outline.json` and `transcript.json`, is never touched, and nothing else reaps it.
Failed and `/retry`-replaced runs leak a directory too (the retry writes a *new* episode row with a new
uuid dir; the old one loses its DB entry and just stays). Upstream this would be roughly
`shutil.rmtree(audio_path.parent.parent)` in that handler. We deliberately do **not** patch it here by
bind-mounting a modified router — that would fork app logic into a deployment repo and rot against
image updates (the same trap the existing overlays already carry). Instead:
`scripts/prune_podcast_data.py` (dry-run by default, `--yes` to
apply) reconciles the episode list from the API against the directories on disk and removes orphans plus
the `clips/` of completed episodes. Two guards matter — it aborts if any job is `running`/`pending`
(a running job has no `audio_file` yet, so its directory is indistinguishable from an orphan) and it
skips directories touched in the last 60 minutes. **Deletion runs inside the container** (`docker compose
exec … rm -rf`), because the container writes as root and the host user can't remove those directories.

### YouTube sources — German transcripts need `CCORE_CONFIG_PATH`

Open Notebook extracts YouTube via `content-core`, which pulls captions with `youtube-transcript-api`
(no `yt-dlp`/`pytube` in the image). `content_core/processors/youtube.py` only ever looks for the
languages in `preferred_languages`, and the built-in default is `["en", "es", "pt"]` — **no German**.
All four fallback attempts in `_fetch_best_transcript` use that same list, so a video with only a
German transcript raises `NoTranscriptFound`, `get_best_transcript` swallows it and returns `None`,
and the source is created with an **empty body**. The title is still fetched (separate HTTP scrape),
so the failure is quiet: a source that looks fine but has no content, with the real reason only in
the container log (`Failed to get transcript for video <id> after retries`).

The `youtube_transcripts.preferred_languages` key in the package's `cc_config.yaml` looks like the
fix, but it is **dead code** on the default path: `load_config()` copies only the `extraction` block
out of `cc_config.yaml`, so the key never reaches `CONFIG` and the hardcoded `["en","es","pt"]`
fallback always wins. The only way to set it is `CCORE_CONFIG_PATH`.

Fix: `config/content_core.yaml` (bind-mounted read-only, `CCORE_CONFIG_PATH=/app/config/content_core.yaml`
in `docker-compose.yml`) sets `preferred_languages: ["de", "en", "es", "pt"]`.

**`CCORE_CONFIG_PATH` replaces the config wholesale — it does not merge** (`config.py`:
`return yaml.safe_load(file)`). So that file is a full dump of the effective default config plus the
`youtube_transcripts` key, not a small override; a minimal file would silently drop `extraction`
(engine selection) and the model/timeout defaults. Its header comment carries the one-liner that
regenerates it from inside the container — do that after an image update if content-core's defaults
change. The `openai`/`gpt-4o-mini` entries in it are content-core's internal defaults and are unused
here (Open Notebook picks its own models); they're only present because the file must be complete.

Note this only fixes *caption* languages. A video with **no** captions is handled by the audio
fallback below.

### Video without captions — audio fallback (`scripts/add_video_source.py`)

`yt-dlp` exists on the **host** only (`~/miniforge3/bin/yt-dlp`), not in the image, so the chain
can't live inside the app. `scripts/add_video_source.py` bridges it:

- captions available → hands the URL to Open Notebook as a `link` source (fast, no download);
- no captions → `yt-dlp -x` pulls just the audio track (mono 16 kHz mp3, ~6 MB per 17 min) and
  uploads it as an `upload` source. Open Notebook then transcribes it **itself** via its configured
  speech-to-text model — i.e. the local faster-whisper server — and embeds it. We deliberately do
  not transcribe in the script: the app's own pipeline already does STT, chunking and embedding.

Caption availability is probed with `youtube-transcript-api` **inside the container**, not with
`yt-dlp`. `yt-dlp`'s `automatic_captions` lists YouTube's ~100 auto-*translation* targets (German
is always among them), which `youtube-transcript-api` does not accept as transcripts — trusting it
would send a Japanese-only video down the caption path and produce an empty source again.

**`OPENAI_COMPATIBLE_BASE_URL_STT` is what makes the upload path work at all.** Open Notebook passes
its STT model (`openai_compatible`/`faster-whisper-large-v3`) into content-core as
`audio_provider`/`audio_model` (`open_notebook/graphs/source.py`), but content-core builds it with
`AIFactory.create_speech_to_text(provider, model, {'timeout': …})` — **no `base_url`**
(`content_core/processors/audio.py`). Esperanto then can't locate the local server, content-core
silently falls back to its default (`openai`/`whisper-1`) and the job dies with *"OpenAI API key not
found"* — which reads like a missing key but is really a missing URL. The env var supplies it. Do
**not** "fix" this by putting an `OPENAI_API_KEY` into the container: that would send audio to OpenAI
(paid, off-machine) while a working local Whisper sits idle on GPU 2, and it contradicts the
two-provider rule above.

The script uploads with `delete_source=true` and names the file after the video id, so
`notebook_data/uploads/` doesn't accumulate a few MB per video and a second video can't collide with
the previous `audio.mp3`. Known gap: the `upload` path stores the audio file as the source's asset,
so the original video URL is not recorded on the source (the `link` path does record it).

### Text-to-speech / speech-to-text

Open Notebook's `openai_compatible` provider type talks to any endpoint implementing OpenAI's audio API shape (`POST /audio/speech`, `POST /audio/transcriptions` — see `esperanto.providers.tts.openai_compatible` / `.stt.openai_compatible` inside the app container for the exact contract). Since neither Ollama nor OpenRouter support TTS/STT, two thin wrapper servers in `services/` expose the locally installed tools this way:

- `services/tts_server.py` — wraps `chatterbox-tts` (via `~/chatterbox-tts-cli/chatterbox_cli_v4.py`), run with the `chatterbox` conda env (`~/miniforge3/envs/chatterbox/bin/python`).
- `services/stt_server.py` — wraps `faster-whisper` (`large-v3` model), run with the base `python3`.

Both run as **systemd user units**. The unit files live in the repo (`services/systemd/*.service`,
using `%h` rather than a hardcoded home) and `scripts/start_services.sh` installs them into
`~/.config/systemd/user/`, enables `linger` and starts them — so they come up on boot without a
login session. Re-run that script after editing a unit; it's idempotent. They log to the journal,
not to `services/logs/` (those files are leftovers from the earlier `nohup` setup):

```bash
systemctl --user status open-notebook-tts open-notebook-stt
systemctl --user restart open-notebook-tts        # after editing tts_server.py or adding voices
journalctl --user -u open-notebook-tts -f
```

They were previously plain `nohup ... &` background processes, which silently did not survive a
reboot — a podcast then failed at the audio stage with `Failed to generate speech: All connection
attempts failed` (`httpx.ConnectError` against `host.docker.internal:8901`) even though outline
and transcript had generated fine. That's the signature of a dead TTS server, not a model problem.

Both units pin GPU 2 **by UUID** (`CUDA_VISIBLE_DEVICES=GPU-83ba6d1f-…`), not by index: GPU 1 is
occupied by Ollama's resident models (`OLLAMA_KEEP_ALIVE=-1` keeps them loaded), so TTS/STT go to
GPU 2 rather than racing Ollama for VRAM. The index form (`CUDA_VISIBLE_DEVICES=2`) is **not**
reliable here — CUDA's default device order is "fastest first", not nvidia-smi's PCI order, so
index 2 can resolve to the T600. Re-check the UUID with
`nvidia-smi --query-gpu=index,name,uuid --format=csv` if the GPUs are ever reseated.

#### GPU 2 is shared — TTS concurrency is capped on purpose

GPU 2 does **not** belong to the TTS server alone. Three processes sit on it: `open-notebook-tts`
(~4 GB idle-loaded), `open-notebook-stt` (faster-whisper `large-v3`, ~4 GB), and the unrelated
`chatterbox-tts.service` MCP server on port 9999 (~3.4 GB). That leaves roughly 12 GB of working
headroom, and TTS inference must fit inside it.

`podcast_creator` requests `TTS_BATCH_SIZE` clips **concurrently** (`asyncio.gather` per batch,
`nodes.py`; upstream default **5**). `tts_server.py`'s `/audio/speech` is a *sync* FastAPI handler,
so FastAPI runs each request in a threadpool thread — five clips therefore generate genuinely in
parallel on one shared model, and the activation memory multiplies. That drove the TTS process to
**16.7 GB** and produced `torch.OutOfMemoryError` inside chatterbox's `s3gen.embed_ref`, surfacing
in the app as `Failed to generate speech: OpenAI-compatible TTS endpoint error: HTTP 500`. Note
this is a *different* failure from the dead-server `ConnectError` above — a 500 means the server is
alive and rejecting the work, so check the TTS journal for the CUDA OOM before assuming a restart
will help.

Two changes keep it bounded, and both matter:

- **`tts_server.py` serializes generation behind `_GPU_LOCK`** (a `threading.Lock` around model load
  + the chunk loop, plus `torch.cuda.empty_cache()` afterwards). Generation is GPU-bound, so running
  clips in parallel buys no throughput on a single card — it only multiplies peak VRAM. This is the
  hard guarantee: it holds no matter which client calls, and no matter what batch size they use.
- **`TTS_BATCH_SIZE=1` in `docker-compose.yml`** so the client sends clips one at a time. Without
  this, the lock still prevents the OOM, but 4 of the 5 batched requests sit queued in it — and with
  ~40-60 s per clip the last one approaches esperanto's **300 s** TTS timeout. `ESPERANTO_TTS_TIMEOUT=600`
  is set alongside it as extra headroom for long segments.

Measured after the fix: 5 concurrent `/audio/speech` requests all return 200, GPU 2 peaks at
~11.5 GB total (vs. 16.7 GB for the TTS process alone before), i.e. ~12 GB of headroom left.
If you add another GPU-resident service to GPU 2, re-check that budget.

Reachability from the `open_notebook` container requires **two things**, both already done: the `extra_hosts: host.docker.internal:host-gateway` entry in `docker-compose.yml`, and UFW rules allowing inbound `8901/tcp` and `8902/tcp` (same pattern as the existing `11434/tcp` rule for Ollama — UFW defaults to deny-incoming, so any new host-side port a container needs to reach must be explicitly opened).

Registered in Open Notebook as `openai_compatible` credentials with `base_url=http://host.docker.internal:8901` (TTS) / `:8902` (STT), and set as `default_text_to_speech_model` / `default_speech_to_text_model`.

#### Voice cloning

`tts_server.py` resolves the incoming `voice` field via `services/voices/*.wav` (gitignored — personal voice recordings): `<name>.wav` becomes a selectable voice `"<name>"`, cloned via chatterbox's `audio_prompt_path`. Unknown/omitted voices fall back to `"default"`; a bare language code (e.g. `"en"`) still works and skips cloning (built-in voice for that language). All cloned voices are generated in German (`CLONE_LANG` in the script) since only German reference clips exist here — add entries to `VOICE_LANG_OVERRIDES` in the script if you add clips in other languages.

Three reference clips exist in `services/voices/` (gitignored, not committed):

- `default.wav` — symlink to `~/chatterbox-tts-cli/my_voice_deutsch_60s.wav`, the owner's own voice.
- `male_thorsten.wav` / `female_kerstin.wav` — synthesized with [Piper](https://github.com/rhasspy/piper) using the CC0-licensed `de_DE-thorsten-medium` and `de_DE-kerstin-low` voice models (downloaded from `huggingface.co/rhasspy/piper-voices`). Deliberately **not** real people's recordings scraped from the internet — Piper's voices are explicitly released for this kind of downstream synthesis/cloning use, unlike e.g. Common Voice (consented for ASR training, not cloning) or random web audio (no consent at all). Regenerate with:
  ```bash
  piper --model de_DE-thorsten-medium.onnx --output_file male_thorsten.wav <<< "some German sentence"
  ```

Podcast speaker profiles are assigned across these three: `business_panel` (Marcus→male_thorsten, Elena→female_kerstin, Johny→default), `tech_experts` (Alex→male_thorsten, Jamie→female_kerstin), `solo_expert`/`test_speaker_local` (→female_kerstin/Anna). To add another distinct voice, drop a new `<name>.wav` (~10-30s, clean single-speaker audio, licensing-checked) into `services/voices/`, restart the TTS server (`systemctl --user restart open-notebook-tts` — voices are scanned once at startup), and set the speaker's `voice_id` to `<name>` via `PUT /api/speaker-profiles/{id}`.

## Secrets

`.env` (gitignored) holds `OPEN_NOTEBOOK_ENCRYPTION_KEY` (encrypts provider credentials stored in SurrealDB), `SURREAL_PASSWORD`, and `OPENROUTER_API_KEY`. `.env.example` is the committed template. Provider API keys can also be entered directly in the Open Notebook UI after first startup — the project's own docs recommend this over env vars for better security since they get encrypted at rest.

## Git

This directory is its own independent git repository (`main` branch) even though the parent home directory is also a git repo — this is intentional, not an accident to fix.