dschlueter/open_notebook
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Add prune_podcast_data.py: clean up podcast data Open Notebook leaves behind

Browse source 
Deleting an episode does not delete its data. DELETE /api/podcasts/episodes/{id}
resolves episode.audio_file and unlinks that one MP3 — the enclosing
data/podcasts/episodes/<uuid>/ directory, holding clips/ (one MP3 per dialogue
segment), outline.json and transcript.json, is never touched. Failed and
/retry-replaced runs leak a directory as well. Nothing reaps any of it: this
deployment had 15 directories on disk against 4 episodes in the database.

That is an upstream gap (roughly a shutil.rmtree of audio_path.parent.parent in
that handler), not something configurable here. Deliberately not patched by
bind-mounting a modified router — that would fork app logic into a deployment
repo and rot silently against pull_policy: always. Local cleanup instead.

scripts/prune_podcast_data.py reconciles the episode list from the API against
the directories on disk and removes:
  - orphaned directories (no corresponding episode), and
  - clips/ of completed episodes, since the clips are intermediate output once
    the final MP3 exists.

Dry-run by default; --yes applies, --keep-clips restricts it to orphans.
Two guards, both tested: it aborts when any job is running/pending (a running
job has no audio_file yet, so its working directory is indistinguishable from an
orphan) and it skips directories touched within the last 60 minutes (--min-age).
It also refuses to act if the API is unreachable, rather than guessing.

Deletion runs inside the container (docker compose exec … rm -rf): the container
writes as root, so the host user cannot remove those directories — a plain
host-side rmtree fails with EPERM after the first directory.

Verified on this deployment: freed 13 MB (10 orphaned dirs + clips of 3 episodes,
49 MB -> 36 MB); all four surviving episodes still stream byte-identical MP3s
from /audio afterwards.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Dieter Schlüter 2026-07-11 13:14:19 +02:00
commit ba98ac4eb4
5 changed files with 268 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

15
CLAUDE.md
View file

@ -74,6 +74,21 @@ line becomes inert but harmless.
If you create your own episode profile, just set `language` — the template handles the rest.
**Deleting an episode leaks its working directory (upstream gap).** `DELETE /api/podcasts/episodes/{id}`
(`api/routers/podcasts.py`) resolves `episode.audio_file` and calls `audio_path.unlink()` on it — the
final MP3 only. The enclosing `data/podcasts/episodes/<uuid>/` directory, holding `clips/` (one MP3 per
dialogue segment), `outline.json` and `transcript.json`, is never touched, and nothing else reaps it.
Failed and `/retry`-replaced runs leak a directory too (the retry writes a *new* episode row with a new
uuid dir; the old one loses its DB entry and just stays). Upstream this would be roughly
`shutil.rmtree(audio_path.parent.parent)` in that handler. We deliberately do **not** patch it here by
bind-mounting a modified router — that would fork app logic into a deployment repo and silently rot
against `pull_policy: always`. Instead: `scripts/prune_podcast_data.py` (dry-run by default, `--yes` to
apply) reconciles the episode list from the API against the directories on disk and removes orphans plus
the `clips/` of completed episodes. Two guards matter — it aborts if any job is `running`/`pending`
(a running job has no `audio_file` yet, so its directory is indistinguishable from an orphan) and it
skips directories touched in the last 60 minutes. **Deletion runs inside the container** (`docker compose
exec … rm -rf`), because the container writes as root and the host user can't remove those directories.
### Text-to-speech / speech-to-text
Open Notebook's `openai_compatible` provider type talks to any endpoint implementing OpenAI's audio API shape (`POST /audio/speech`, `POST /audio/transcriptions` — see `esperanto.providers.tts.openai_compatible` / `.stt.openai_compatible` inside the app container for the exact contract). Since neither Ollama nor OpenRouter support TTS/STT, two thin wrapper servers in `services/` expose the locally installed tools this way: