diff --git a/.gitignore b/.gitignore index 68aa1fd..03c8c44 100644 --- a/.gitignore +++ b/.gitignore @@ -2,3 +2,4 @@ surreal_data/ notebook_data/ services/logs/ +services/voices/ diff --git a/CLAUDE.md b/CLAUDE.md index 7f76da7..dcbcef1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -58,6 +58,12 @@ Reachability from the `open_notebook` container requires **two things**, both al Registered in Open Notebook as `openai_compatible` credentials with `base_url=http://host.docker.internal:8901` (TTS) / `:8902` (STT), and set as `default_text_to_speech_model` / `default_speech_to_text_model`. +#### Voice cloning + +`tts_server.py` resolves the incoming `voice` field via `services/voices/*.wav` (gitignored — personal voice recordings): `.wav` becomes a selectable voice `""`, cloned via chatterbox's `audio_prompt_path`. Unknown/omitted voices fall back to `"default"`; a bare language code (e.g. `"en"`) still works and skips cloning (built-in voice for that language). All cloned voices are generated in German (`CLONE_LANG` in the script) since only German reference clips exist here — add entries to `VOICE_LANG_OVERRIDES` in the script if you add clips in other languages. + +`services/voices/default.wav` is a symlink to `~/chatterbox-tts-cli/my_voice_deutsch_60s.wav` (the owner's own voice). All podcast speaker profiles (`business_panel`, `tech_experts`, `solo_expert`, `test_speaker_local`) currently point every speaker at `voice_id: "default"` — there's only one reference clip, so multi-speaker podcasts don't yet sound like distinct people, only the transcript labels differ. To give a speaker their own voice, drop a new `.wav` (~10-30s, clean single-speaker audio) into `services/voices/`, restart `tts_server.py` (voices are scanned once at startup), and set that speaker's `voice_id` to `` via `PUT /api/speaker-profiles/{id}`. + ## Secrets `.env` (gitignored) holds `OPEN_NOTEBOOK_ENCRYPTION_KEY` (encrypts provider credentials stored in SurrealDB), `SURREAL_PASSWORD`, and `OPENROUTER_API_KEY`. `.env.example` is the committed template. Provider API keys can also be entered directly in the Open Notebook UI after first startup — the project's own docs recommend this over env vars for better security since they get encrypted at rest. diff --git a/services/tts_server.py b/services/tts_server.py index 6c9b6ab..b54340a 100644 --- a/services/tts_server.py +++ b/services/tts_server.py @@ -4,12 +4,20 @@ Implements exactly the endpoint esperanto's OpenAICompatibleTextToSpeechModel calls: POST /audio/speech {model, voice, input, response_format} -> raw audio bytes +Voice cloning: `voice` selects a named reference clip from VOICES_DIR (each +.wav becomes voice ""). Unknown/omitted voices fall back to +"default". A recognized language code (see chatterbox_cli_v4.SUPPORTED_LANGS) +is still accepted as `voice` for the old behaviour: no cloning, built-in voice +for that language. All cloned voices are generated in German (CLONE_LANG) — +this deployment only has German reference clips; extend VOICE_LANG_OVERRIDES +below if you add clips in other languages. + Start: ~/miniforge3/envs/chatterbox/bin/python tts_server.py Env vars: TTS_HOST (default 0.0.0.0), TTS_PORT (default 8901) - CUDA_VISIBLE_DEVICES should be set by the caller (e.g. "1,2") to keep GPU 0 free. + CUDA_VISIBLE_DEVICES should be set by the caller (e.g. "2") to keep GPU 0/1 free. """ from __future__ import annotations @@ -27,11 +35,24 @@ import torchaudio as ta from fastapi import FastAPI, HTTPException, Response from pydantic import BaseModel -app = FastAPI(title="Chatterbox TTS (OpenAI-compatible)", version="1.0") +app = FastAPI(title="Chatterbox TTS (OpenAI-compatible, voice cloning)", version="2.0") _DEVICE = tts.get_device(None) _model_cache: dict[str, tuple] = {} +VOICES_DIR = Path(__file__).parent / "voices" +CLONE_LANG = "de" # language used for every cloned reference voice +VOICE_LANG_OVERRIDES: dict[str, str] = {} # e.g. {"john": "en"} if you add an English clip + + +def _discover_voices() -> dict[str, Path]: + if not VOICES_DIR.is_dir(): + return {} + return {p.stem: p for p in sorted(VOICES_DIR.glob("*.wav"))} + + +VOICES = _discover_voices() + def _get_model(lang: str): key = "en" if lang == "en" else "multi" @@ -40,9 +61,20 @@ def _get_model(lang: str): return _model_cache[key] +def _resolve_voice(voice: str) -> tuple[str, str | None]: + """Map the incoming `voice` string to (language, audio_prompt_path|None).""" + if voice in VOICES: + return VOICE_LANG_OVERRIDES.get(voice, CLONE_LANG), str(VOICES[voice]) + if voice in tts.SUPPORTED_LANGS: + return voice, None + if "default" in VOICES: + return CLONE_LANG, str(VOICES["default"]) + return "de", None + + class SpeechRequest(BaseModel): model: str | None = None - voice: str = "de" + voice: str = "default" input: str response_format: str = "mp3" @@ -58,17 +90,32 @@ _FORMAT_CONTENT_TYPE = { @app.get("/health") def health(): - return {"status": "ok", "device": _DEVICE} + return {"status": "ok", "device": _DEVICE, "voices": sorted(VOICES.keys())} @app.get("/audio/voices") def voices(): - return {"voices": [{"id": lang, "name": lang} for lang in sorted(tts.SUPPORTED_LANGS)]} + cloned = [ + { + "id": name, + "name": name, + "gender": "NEUTRAL", + "language_code": VOICE_LANG_OVERRIDES.get(name, CLONE_LANG), + "description": f"Cloned voice from {path.name}", + } + for name, path in VOICES.items() + ] + langs = [ + {"id": lang, "name": lang, "gender": "NEUTRAL", "language_code": lang, + "description": "Built-in voice (no cloning)"} + for lang in sorted(tts.SUPPORTED_LANGS) + ] + return {"voices": cloned + langs} @app.post("/audio/speech") def speech(req: SpeechRequest): - lang = req.voice if req.voice in tts.SUPPORTED_LANGS else "de" + lang, voice_path = _resolve_voice(req.voice) raw = tts.clean_raw_text(req.input) raw_chunks = tts.split_into_sentences(raw, max_len=400) @@ -81,7 +128,7 @@ def speech(req: SpeechRequest): wavs = [] for chunk in chunks: - wavs.append(tts.generate_chunk(model, model_kind, chunk, lang, None)) + wavs.append(tts.generate_chunk(model, model_kind, chunk, lang, voice_path)) final = wavs[0] if len(wavs) == 1 else torch.cat(wavs, dim=-1) with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as wav_tmp: @@ -108,5 +155,6 @@ def speech(req: SpeechRequest): if __name__ == "__main__": import uvicorn + print(f"[chatterbox-tts] Registrierte Stimmen: {sorted(VOICES.keys()) or '(keine, nur Sprachcodes)'}") uvicorn.run(app, host=os.environ.get("TTS_HOST", "0.0.0.0"), port=int(os.environ.get("TTS_PORT", "8901")))