dschlueter/open_notebook
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Add local TTS/STT wrapper servers (chatterbox, faster-whisper) as openai_compatible providers

Browse source 
Wraps the locally installed chatterbox-tts and faster-whisper packages in thin
FastAPI servers implementing OpenAI's audio API shape, since neither Ollama nor
OpenRouter support speech. Pinned to GPU 2 to avoid contending with Ollama's
resident models on GPU 1. Requires UFW rules for 8901/8902 (same pattern as
the existing 11434 rule) since UFW defaults to deny-incoming.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Dieter Schlüter 2026-07-10 21:41:44 +02:00
commit fb1fbcfabe
4 changed files with 198 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

21
CLAUDE.md
View file

@ -37,6 +37,27 @@ Neither `surrealdb` nor `open_notebook` do local model inference themselves, so
A separate local `llama.cpp` server exists on this machine (managed via `~/llamacppctl-server_neu/llamacppctl/`, container name `llama_cpp_server`) but is **not** wired into this stack. It can be added later as an additional OpenAI-compatible provider through the Open Notebook UI (Settings → AI Providers) if needed — note it competes for the same GPU 1/2 VRAM as Ollama, so don't run large models on both simultaneously.
### Text-to-speech / speech-to-text
Open Notebook's `openai_compatible` provider type talks to any endpoint implementing OpenAI's audio API shape (`POST /audio/speech`, `POST /audio/transcriptions` — see `esperanto.providers.tts.openai_compatible` / `.stt.openai_compatible` inside the app container for the exact contract). Since neither Ollama nor OpenRouter support TTS/STT, two thin wrapper servers in `services/` expose the locally installed tools this way:
- `services/tts_server.py` — wraps `chatterbox-tts` (via `~/chatterbox-tts-cli/chatterbox_cli_v4.py`), run with the `chatterbox` conda env (`~/miniforge3/envs/chatterbox/bin/python`).
- `services/stt_server.py` — wraps `faster-whisper` (`large-v3` model), run with the base `python3`.
Both are plain background processes (`nohup ... &`), **not** systemd units — they don't survive a reboot; restart manually if needed:
```bash
cd ~/open-notebook/services
CUDA_VISIBLE_DEVICES=2 TTS_PORT=8901 nohup ~/miniforge3/envs/chatterbox/bin/python tts_server.py > logs/tts_server.log 2>&1 &
CUDA_VISIBLE_DEVICES=2 STT_PORT=8902 nohup python3 stt_server.py > logs/stt_server.log 2>&1 &
```
They're pinned to `CUDA_VISIBLE_DEVICES=2` deliberately — GPU 1 tends to fill up with Ollama's resident models (`OLLAMA_KEEP_ALIVE=-1` keeps them loaded), so TTS/STT get GPU 2 to themselves rather than risking an OOM race with Ollama.
Reachability from the `open_notebook` container requires **two things**, both already done: the `extra_hosts: host.docker.internal:host-gateway` entry in `docker-compose.yml`, and UFW rules allowing inbound `8901/tcp` and `8902/tcp` (same pattern as the existing `11434/tcp` rule for Ollama — UFW defaults to deny-incoming, so any new host-side port a container needs to reach must be explicitly opened).
Registered in Open Notebook as `openai_compatible` credentials with `base_url=http://host.docker.internal:8901` (TTS) / `:8902` (STT), and set as `default_text_to_speech_model` / `default_speech_to_text_model`.
## Secrets
`.env` (gitignored) holds `OPEN_NOTEBOOK_ENCRYPTION_KEY` (encrypts provider credentials stored in SurrealDB), `SURREAL_PASSWORD`, and `OPENROUTER_API_KEY`. `.env.example` is the committed template. Provider API keys can also be entered directly in the Open Notebook UI after first startup — the project's own docs recommend this over env vars for better security since they get encrypted at rest.