dschlueter/open_notebook
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Document the embedding-timeout root cause and switch to nomic-embed-text

Browse source 
qwen3-embedding (repurposed 8B LLM, ~13GB loaded) didn't fit on GPU 1
alongside the resident 17GB chat model, so Ollama silently ran it 44-56%
on CPU. A single embed call took 39-54s, so any real document's chunk
batches blew past esperanto's 60s embedding timeout and got stuck retrying
forever - looked like "adding the source failed" but extraction was fine.
Also note: credential num_ctx doesn't work for embeddings (esperanto sends
it at the wrong JSON level, Ollama ignores it) - a small dedicated
embedding model is the real fix, not context-window tuning.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Dieter Schlüter 2026-07-10 23:00:34 +02:00
commit 631ed26558
1 changed files with 2 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

2
CLAUDE.md
View file

@ -35,6 +35,8 @@ This deployment intentionally uses only two AI providers — no OpenAI/Anthropic
Neither `surrealdb` nor `open_notebook` do local model inference themselves, so no GPU device reservation is needed in the compose file — all GPU usage happens inside the host's Ollama process. Neither `surrealdb` nor `open_notebook` do local model inference themselves, so no GPU device reservation is needed in the compose file — all GPU usage happens inside the host's Ollama process.
**Embedding model:** use `nomic-embed-text` (274 MB, 2048 native context), not `qwen3-embedding` (a repurposed 8B-class LLM, ~13GB loaded). With the resident `qwen3.5:27b` chat model already using 17GB of GPU 1's 24GB, `qwen3-embedding` didn't fit and Ollama silently ran it 44-56% on CPU — a single embedding call took 39-54s, so any real document (many chunks, batched) blew past esperanto's 60s embedding timeout (`ESPERANTO_EMBEDDING_TIMEOUT`) and got stuck in an infinite retry loop that looked like "adding the source failed" even though extraction worked fine. Note: the credential's `num_ctx` field (meant to shrink Ollama's context/VRAM footprint) does **not** work for embeddings — esperanto's `OllamaEmbeddingModel` puts it at the top level of the `/api/embed` payload instead of nesting it under `options`, so Ollama silently ignores it. Don't rely on `num_ctx` to fix embedding VRAM contention; use a genuinely small embedding model instead.
A separate local `llama.cpp` server exists on this machine (managed via `~/llamacppctl-server_neu/llamacppctl/`, container name `llama_cpp_server`) but is **not** wired into this stack. It can be added later as an additional OpenAI-compatible provider through the Open Notebook UI (Settings → AI Providers) if needed — note it competes for the same GPU 1/2 VRAM as Ollama, so don't run large models on both simultaneously. A separate local `llama.cpp` server exists on this machine (managed via `~/llamacppctl-server_neu/llamacppctl/`, container name `llama_cpp_server`) but is **not** wired into this stack. It can be added later as an additional OpenAI-compatible provider through the Open Notebook UI (Settings → AI Providers) if needed — note it competes for the same GPU 1/2 VRAM as Ollama, so don't run large models on both simultaneously.
### Text-to-speech / speech-to-text ### Text-to-speech / speech-to-text