From 778f820e27fc9ab82e6ce72bb5d183f13325b55c Mon Sep 17 00:00:00 2001 From: dschlueter Date: Fri, 10 Jul 2026 23:28:12 +0200 Subject: [PATCH] Document the num_ctx=8192 default that caused empty chat answers esperanto's OllamaLanguageModel silently caps num_ctx at 8192 unless the credential sets it explicitly, no matter what OLLAMA_CONTEXT_LENGTH or the model's real context window is. Any chat context beyond that got truncated, and because qwen3.5:27b is a thinking model, the truncated prompt usually produced an empty final answer (swallowed by clean_thinking_content) rather than a visible error - looked like "chat gives no answer" in the UI. Set num_ctx=98304 on the Ollama chat credential: the largest value that still keeps the whole model on GPU 1 (131072 spills onto CPU, 5-6x slower). Co-Authored-By: Claude Sonnet 5 --- CLAUDE.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index 4174e30..a01c6f8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -39,6 +39,8 @@ Neither `surrealdb` nor `open_notebook` do local model inference themselves, so A separate local `llama.cpp` server exists on this machine (managed via `~/llamacppctl-server_neu/llamacppctl/`, container name `llama_cpp_server`) but is **not** wired into this stack. It can be added later as an additional OpenAI-compatible provider through the Open Notebook UI (Settings → AI Providers) if needed — note it competes for the same GPU 1/2 VRAM as Ollama, so don't run large models on both simultaneously. +**Chat model context window:** the Ollama credential used for `qwen3.5:27b` (`credential:di8b31l16zikyyb6isyi`) has `num_ctx=98304` set explicitly. Without this, esperanto's `OllamaLanguageModel` silently defaults `num_ctx` to **8192** regardless of the model's real capability or `OLLAMA_CONTEXT_LENGTH` — any chat context (e.g. "full content" mode on a real document) beyond that gets truncated by Ollama, and because `qwen3.5:27b` is a thinking model, the truncated prompt frequently produces an empty final answer (the response is silently swallowed by `clean_thinking_content`) instead of a visible error — looks exactly like "chat gives no answer" from the UI. `98304` is the largest value that still keeps the whole model on GPU 1 (`ollama ps` shows `100% GPU`); `131072` spills ~91% onto CPU and makes a single response take 5-6 minutes instead of ~1-2. Even at 98304, a full-book "Volltext" chat (~70k+ tokens of context) takes ~2-3 minutes — that's inherent to processing that much context on one consumer GPU, not a bug. For large documents, the default short/RAG context mode (only relevant chunks, not the whole document) stays fast; reserve "Volltext" for shorter sources. If `qwen3.5:27b` ever gets swapped for a different Ollama chat model, re-check this credential's `num_ctx` still makes sense for its VRAM footprint. + ### Text-to-speech / speech-to-text Open Notebook's `openai_compatible` provider type talks to any endpoint implementing OpenAI's audio API shape (`POST /audio/speech`, `POST /audio/transcriptions` — see `esperanto.providers.tts.openai_compatible` / `.stt.openai_compatible` inside the app container for the exact contract). Since neither Ollama nor OpenRouter support TTS/STT, two thin wrapper servers in `services/` expose the locally installed tools this way: