Compare commits
8 commits
19b3998948
...
b70d6f364a
| Author | SHA1 | Date | |
|---|---|---|---|
| b70d6f364a | |||
| 48a79da12f | |||
| e753788a23 | |||
| fdf65e8623 | |||
| 6cd6e783aa | |||
| 96dcd87ca8 | |||
| c36b884aab | |||
| 703a695bed |
29 changed files with 2113 additions and 17 deletions
3
.gitignore
vendored
3
.gitignore
vendored
|
|
@ -39,3 +39,6 @@ Ideen/
|
|||
# Local voice models
|
||||
config/voices/*.onnx
|
||||
config/voices/*.onnx.json
|
||||
|
||||
# Eval-Roh-Ergebnisse (Tool-Calling-Harness, instanz-/laufabhängig)
|
||||
eval/tool_calling/results_*.json
|
||||
|
|
|
|||
|
|
@ -874,7 +874,7 @@ default_language = "de"
|
|||
default_input_endpoint = "local-default"
|
||||
default_output_endpoint = "local-default"
|
||||
|
||||
openrouter_llm_model = "mistralai/mistral-small-24b-instruct-2501"
|
||||
openrouter_llm_model = "mistralai/mistral-small-3.2-24b-instruct"
|
||||
llm_fallback = "openrouter"
|
||||
|
||||
# WICHTIG: Piper als TTS-Fallback. Greift, wenn Cartesia eine Sprache nicht kann
|
||||
|
|
@ -1636,7 +1636,7 @@ Priorität der Wartezeit bei 429:
|
|||
Für zuverlässigen Betrieb das **Bezahlmodell** verwenden:
|
||||
|
||||
```toml
|
||||
openrouter_llm_model = "mistralai/mistral-small-24b-instruct-2501"
|
||||
openrouter_llm_model = "mistralai/mistral-small-3.2-24b-instruct"
|
||||
```
|
||||
|
||||
Free-Tier-Modelle (`:free`-Suffix) sind für Produktion ungeeignet — hohe
|
||||
|
|
|
|||
207
Docs/weg2-tool-calling.md
Normal file
207
Docs/weg2-tool-calling.md
Normal file
|
|
@ -0,0 +1,207 @@
|
|||
# Design: Web-Suche per Tool-Calling (Weg 2)
|
||||
|
||||
> Stand: Entwurf. Beschreibt das Konzept für die Frische-/Web-Such-Funktion auf
|
||||
> Basis von agentischem Tool-Calling. Modellwahl und Schwellen sind durch den
|
||||
> Eval-Harness (`eval/tool_calling/`) belegt; Implementierung folgt diesem Dokument.
|
||||
|
||||
## 1. Ziel & Leitidee
|
||||
|
||||
Das zentrale LLM kennt nur Wissen bis zu seinem Trainings-Cutoff. Fragt der Nutzer
|
||||
nach etwas, das sich seither geändert haben könnte (aktuelle Amtsträger, Preise,
|
||||
Wetter, Nachrichten, „lebt X noch", Öffnungszeiten …), soll das Modell **selbst**
|
||||
eine Web-Suche auslösen, die frischen Fakten holen und die Antwort **in der
|
||||
Alexis-Persona** formulieren.
|
||||
|
||||
**Weg 2 = Tool-Calling + Anreichern:** Das Antwortmodell entscheidet via Werkzeug,
|
||||
ob es sucht; die Suche liefert nur Fakten, das Modell formuliert. Vorteil:
|
||||
durchgängige Stimme, Persona/Safety/History bleiben beim Hauptmodell. Bewusst
|
||||
verworfen wurde der separate Vorab-Klassifikator („Weg 1"), weil das gewählte
|
||||
Modell sich zuverlässig genug selbst triggert (siehe §7).
|
||||
|
||||
## 2. Modellwahl (belegt)
|
||||
|
||||
`mistralai/mistral-small-3.2-24b-instruct` über OpenRouter. Eval-Ergebnis
|
||||
(69 Fälle × 5 Läufe): Trigger-Recall **93 %**, Vorrang-Treue **100 %**,
|
||||
Wohlgeformtheit **100 %**, gefährliche Kategorien (Amtsträger/Preise/Nachrichten/
|
||||
Sport/Releases) **100 %**, Wetter inkl. implizitem Ort 98 %. Günstig und klein →
|
||||
niedrige Latenz (im Senioren-Kontext doppelt wertvoll). Details und Vergleich
|
||||
(Flash/Pro/alte Baseline) in der Memory-Notiz bzw. `eval/tool_calling/`.
|
||||
|
||||
> Achtung: Die bisherige TOML-Baseline `mistral-small-24b-instruct-2501` kann
|
||||
> **kein** Tool-Calling (OpenRouter 404 „No endpoints found that support tool use").
|
||||
> Ein Revert darauf würde Weg 2 abschalten.
|
||||
|
||||
## 3. Keystone: Der Tool-Loop lebt *im* LLM-Provider
|
||||
|
||||
Ein neuer Wrapper **`ToolCallingLLM`** implementiert die bestehende
|
||||
`LLMProvider`-Schnittstelle (`complete()`/`stream()`), wickelt aber intern die
|
||||
Agenten-Schleife ab. Der Orchestrator ruft weiter nur `self.llm.stream(...)` und
|
||||
weiß nichts von Tools. Folge: **ein neuer `LLM_REGISTRY`-Eintrag**, keine
|
||||
Kern-Änderung, volle Kompatibilität mit `resolve_route`, Fallback-Ketten und
|
||||
Profilen. Das ist das Leitprinzip „jede Achse austauschbar".
|
||||
|
||||
`ToolCallingLLM` enthält einen **eigenen** tool-fähigen OpenRouter-Client (Keim:
|
||||
`eval/tool_calling/client.py`) — nicht den plain `OpenRouterLLMProvider`, da
|
||||
dessen `complete()` keine `tools` kennt.
|
||||
|
||||
## 4. Datenfluss
|
||||
|
||||
**Kein-Tool-Turn (Normalfall, kein Latenz-Regress):**
|
||||
```
|
||||
stream() → Modell mit tools=[…] → Text-Deltas → 1:1 an on_token/SentenceChunker
|
||||
```
|
||||
|
||||
**Such-Turn (augment):**
|
||||
```
|
||||
(0) Koreferenz-Vorstufe: Pronomen aus History auflösen (gegated)
|
||||
(1) Modell → tool_call("web_search", query)
|
||||
(2) on_tool_start(language) → Filler SOFORT sprechen/anzeigen
|
||||
(3) SonarTool(query) → Fakten (~1–3 s, vom Filler überdeckt)
|
||||
(4) Modell-Runde 2 mit tool-Ergebnis → finale Antwort, gestreamt, in Persona
|
||||
```
|
||||
|
||||
## 5. Komponenten
|
||||
|
||||
### 5.1 `SonarTool`
|
||||
Ruft `perplexity/sonar` über OpenRouter (gleiche Chat-API wie `OpenRouterLLMProvider`).
|
||||
Prompt an Sonar: knapp, Zielsprache, reine Fakten. Rückgabe als `tool`-Message ans
|
||||
Modell. **Citations** als strukturierte Metadaten an die UI-Bubble (nicht ins TTS).
|
||||
Timeout/Fehler → Sentinel „keine aktuellen Daten verfügbar", damit das Modell
|
||||
ehrlich punktet statt zu hängen.
|
||||
|
||||
### 5.2 `ToolCallingLLM`
|
||||
Begrenzte Schleife (max. ~3 Runden gegen Runaway). System-Prompt =
|
||||
Senioren-Persona (`SYSTEM_PROMPT` aus `openrouter.py`) **+** kalibrierte
|
||||
Trigger-Leitlinie **+** Vorrang-Regel („Tool-Ergebnis ist maßgeblich und neuer
|
||||
als dein Wissen; bei Widerspruch folge dem Tool, nicht mischen"). Implementiert
|
||||
`complete()` (für `chat_text`/`translate`) und `stream()`.
|
||||
|
||||
### 5.3 Koreferenz-Vorstufe (in v1)
|
||||
Schließt die einzige nicht-triviale Eval-Restkante (`ctx_alive_nl`:
|
||||
nl + Pronomen-aus-History; benannt + de-Pronomen sind 5/5). Ein
|
||||
dekontextualisierender Rewrite macht die letzte Äußerung mit Hilfe des Verlaufs
|
||||
selbstständig („hij" → „Rutger Hauer"). **Gegated**: nur bei kurzer Folgefrage
|
||||
*mit* Pronomen *und* vorhandener History, damit er nicht jeden Turn kostet.
|
||||
|
||||
### 5.4 Filler / Beruhigung
|
||||
`stream()` erhält einen `on_tool_start(language)`-Callback. Der Orchestrator
|
||||
verdrahtet ihn auf zwei Pools (`app/pipeline/fillers.py`): **OPENING** (sofort beim
|
||||
Such-Start) und **PATIENCE** (bei längerer Recherche nachgeschoben). Beide werden
|
||||
**zufällig** gewählt (schnelles Feedback, nicht roboterhaft) und gehen über
|
||||
`spoken_adapter`/`tts_normalizer`/`tts` → `on_audio` **und** `on_token`.
|
||||
|
||||
**Eine Pflege-Stelle, Englisch:** Die Master-Sätze stehen als englische Listen in
|
||||
`fillers.py` (`OPENING_PHRASES`/`PATIENCE_PHRASES`). Zur Laufzeit werden sie **einmal
|
||||
je Sprache übersetzt und gecacht** (`FillerPhrases`, Übersetzer = OpenRouter-Modell,
|
||||
durchgängig höfliche Anrede „Sie"/„vous"/„usted"…). Die **Standardsprache wird beim
|
||||
Start vorgewärmt** (`warmup.py`) → erster Such-Turn ohne Übersetzungs-Latenz; andere
|
||||
Sprachen werden beim ersten Bedarf einmalig übersetzt. Fällt die Übersetzung aus →
|
||||
Fallback Englisch.
|
||||
|
||||
**Geduld-Schleife:** Beim Tool-Start läuft ein Hintergrund-Task, der alle
|
||||
`FILLER_PATIENCE_INTERVAL` Sekunden (Mindestabstand **10 s** zwischen Opening und
|
||||
Geduldssätzen bzw. zwischen Geduldssätzen) einen zufälligen PATIENCE-Satz
|
||||
nachschiebt. Sobald die Antwort vorliegt (erstes Antwort-Delta), wird die Schleife
|
||||
abgeräumt **und** ein anstehender Filler/Geduldssatz NICHT mehr ausgegeben
|
||||
(`answer_started`-Gate in `_emit_filler`).
|
||||
|
||||
**Ephemeralität (Invariante):** Filler laufen über den `on_filler`-Callback, nicht
|
||||
über den Delta-Stream — sie landen **nicht** in `trace.semantic_response` und
|
||||
**nicht** im gespeicherten History-Turn.
|
||||
|
||||
**Kanal:** Der Orchestrator emittiert Filler über `on_filler`; die WS-Schicht sendet
|
||||
`{type:"filler", text}`. Das Frontend zeigt ihn **transient in der Status-Zeile**
|
||||
(nicht in der Antwort-Bubble) und spricht ihn im **Geräte-TTS-Modus** sofort per
|
||||
`speechSynthesis.speak()` **ohne `cancel`** (mehrere Filler laufen in die Queue; die
|
||||
Antwort `cancelt` später und übernimmt) — abgesichert über `deviceVoicesReady()`/
|
||||
`deviceVoiceReady(lang)` (keine Verschärfung der bekannten Regression). Im Server-TTS-
|
||||
Modus wird der Filler zusätzlich als „Satz null" über die Chunk-Queue gesprochen.
|
||||
|
||||
### 5.5 Konfiguration & Opt-out
|
||||
`web_search_enabled: bool = True` — **global an per Default**, pro Nutzer/Profil
|
||||
**abschaltbar** (Opt-out greift über die bestehende Präzedenz
|
||||
Defaults < Profil < Nutzer-Prefs). Der Schalter ist die nutzerfreundliche
|
||||
Admin-/Profil-Option; intern wählt `build_orchestrator` daraufhin den
|
||||
tool-fähigen vs. den plain Provider — **beide mit demselben konfigurierten
|
||||
Modell** (`openrouter_llm_model`), damit der Modellwechsel an *einer* Stelle bleibt.
|
||||
|
||||
### 5.6 Resilienz & Metriken
|
||||
Fallback: Sonar-Ausfall → honest-punt im Loop; totaler Modellausfall → bestehende
|
||||
`llm_fallback`-Kette. **Metriken** (`app/metrics.py`): Tool-Calls und Sonar-Calls
|
||||
separat zählen — Kostensicht **und** Live-Beobachtung der Trigger-Rate.
|
||||
**Kein** eigenes Sonar-Kontingent in v1 (erst Metrik-Sicht; `quota.py`-Anbindung
|
||||
später bei Bedarf).
|
||||
|
||||
### 5.7 Such-Backstop (heikle Kategorien)
|
||||
Das Modell self-triggert zu ~93 %; die Misses liegen in den Kategorien, in denen
|
||||
eine veraltete Erstantwort **konfident-falsch** ist (Amtsträger, Wohnort lebender
|
||||
Personen, „lebt X noch"). `app/pipeline/search_backstop.py` erkennt solche
|
||||
Phrasierungen per mehrsprachiger Regex (`should_force_search`, Gegenwarts-Schutz:
|
||||
„wer **ist**", nicht „wer **war**" → historische Fragen triggern nicht; offline
|
||||
getestet). Bei Treffer erzwingt `ToolCallingLLM` die Suche **deterministisch**.
|
||||
|
||||
**Wichtige Designentscheidung:** *Nicht* per `tool_choice`-Forcen — das honoriert
|
||||
das Modell nicht zuverlässig (beobachtet: forced choice ignoriert → veraltete
|
||||
Antwort). Stattdessen führt der Wrapper die Suche **selbst direkt** aus und speist
|
||||
das Ergebnis als **System-Kontext** ein (`_inject_forced_search`); das Modell
|
||||
formuliert daraus. System-Kontext (statt synthetischem tool_call) funktioniert mit
|
||||
tool-fähigen **und** tool-unfähigen Modellen (letztere lehnen tool-Messages mit 405
|
||||
ab). Metrik: `search_forced_total`. Recall-optimiert (lieber eine überflüssige
|
||||
Suche als eine konfident falsche Antwort).
|
||||
|
||||
### 5.8 Härtung: tool-unfähiges Modell
|
||||
Wählt jemand (Admin/Preset) ein Modell ohne Tool-fähigen Endpoint, liefert
|
||||
OpenRouter ein 404 „No endpoints found that support tool use". `ToolCallingLLM`
|
||||
erkennt das (`_looks_tool_unsupported`), **merkt es für den Turn** (`_tools_unsupported`),
|
||||
deaktiviert Tools und antwortet **sofort plain** (kein Retry-Hänger, keine stille
|
||||
Stale-Antwort durch die Fallback-Kette). Log-Warnung + Metrik `tool_unsupported_total`.
|
||||
Wichtig: Der **Backstop bleibt wirksam** (er speist Fakten als System-Kontext ein,
|
||||
nicht als tool_call) → heikle Kategorien bleiben auch auf einem tool-unfähigen Modell
|
||||
korrekt; nur nicht-heikle Volatil-Fragen (Wetter/Preise) fallen auf plain zurück.
|
||||
|
||||
## 6. Die zwei harten Stellen (bewusst benannt)
|
||||
|
||||
- **Streaming + Tool-Erkennung.** Beim gestreamten ersten Call kommen
|
||||
`tool_call`-Deltas fragmentiert. Der Wrapper setzt sie zusammen und
|
||||
unterscheidet: Text-Deltas → durchreichen (kein Regress im Normalfall);
|
||||
materialisiert sich ein Tool-Call → Stream verwerfen, Filler, Sonar, zweite
|
||||
Runde streamen. mistral emittiert *entweder* Tool-Call *oder* Text — das macht
|
||||
es handhabbar; das Delta-Zusammensetzen ist die eigentliche Arbeit.
|
||||
- **Filler-Ephemeralität.** Spielt in Bubble *und* TTS, darf aber nie in
|
||||
`semantic_response`/Store/`history`. Der Callback-Weg löst das by-design — beim
|
||||
Implementieren strikt einhalten.
|
||||
|
||||
## 7. Warum nicht Weg 1 (separater Klassifikator)
|
||||
|
||||
Der Eval zeigte: das Modell self-triggert auf den gefährlichen, konfident-
|
||||
veraltbaren Kategorien zu 100 %; die Recall-Lücke (→93 %) liegt nur in
|
||||
low-/medium-harm-Slices (Logistik honest-punt, eine nl-Koreferenz-Kante).
|
||||
Tool-Calling spart den Vorab-Roundtrip und ist zugleich die Basis für den
|
||||
späteren Multitool-Ausbau (weitere Werkzeuge in dieselbe Schleife, Vorrang-Regel
|
||||
wird zur Tool-Rangordnung).
|
||||
|
||||
## 8. Phasenplan
|
||||
|
||||
- **v1:** `SonarTool` + `ToolCallingLLM` (Loop, Persona+Vorrang-Prompt, eigener
|
||||
Tool-Client) + Koreferenz-Vorstufe + Registry-Eintrag + `web_search_enabled`
|
||||
(global an, Opt-out) + Filler für **Server-TTS** + Tool/Sonar-Metriken.
|
||||
schedule_hours-honest-punt akzeptiert.
|
||||
- **Erledigt (nach v1):** Backstop für heikle Kategorien (§5.7), Filler für
|
||||
**Gerät-TTS** (§5.4), **Citations in die UI-Bubble** (`PipelineTrace.citations`
|
||||
→ `on_citations` → WS `semantic`-Event → Quellen-Links unter der Antwort).
|
||||
- **Erledigt:** „no-tool-endpoints"-404-Härtung (§5.8); `schedule_hours`-Nudge
|
||||
(Öffnungszeiten/Fahrpläne jetzt im Backstop); zentrale Datum-/Uhrzeit-Auskunft
|
||||
(`app/core/clock.py` → `now_context()` in alle LLM-System-Prompts injiziert, damit
|
||||
relative Zeitangaben aufgelöst werden).
|
||||
- **Fast-follow (offen):** —
|
||||
- **Später (Multitool):** weitere Tools in dieselbe `ToolCallingLLM`-Schleife
|
||||
(Kalender, Erinnerungen, Medizin-Safety) mit Tool-Rangordnung.
|
||||
|
||||
## 9. Berührte Dateien (Implementierungs-Landkarte)
|
||||
|
||||
- neu: `app/providers/llm/tool_calling.py` (`ToolCallingLLM`), `app/tools/web_search.py` (`SonarTool`), `app/pipeline/decontextualizer.py` (Koreferenz-Vorstufe)
|
||||
- `app/dependencies.py` — `LLM_REGISTRY`-Eintrag; `build_orchestrator` (tool vs. plain je `web_search_enabled`)
|
||||
- `app/core/orchestrator.py` — `on_tool_start`-Callback in `chat_stream`, Filler-Emission (ephemer)
|
||||
- `app/config.py` / `app/runtime_config.py` — `web_search_enabled` (Default true, RUNTIME_SETTABLE + Nutzer-Pref-Opt-out)
|
||||
- `app/metrics.py` — Zähler `tool_calls_total`, `sonar_calls_total`
|
||||
- Filler-Texte je Sprache (Pool)
|
||||
|
|
@ -136,6 +136,12 @@ async def _run_turn(
|
|||
audio_seq += 1
|
||||
await websocket.send_bytes(chunk)
|
||||
|
||||
# Filler (Beruhigungs-/Geduldssätze) als eigener Event: Status-Zeile + Geräte-TTS.
|
||||
on_filler = None
|
||||
if stream or audio_stream:
|
||||
async def on_filler(filler_text):
|
||||
await websocket.send_json({"type": "filler", "text": filler_text})
|
||||
|
||||
try:
|
||||
if stream or audio_stream:
|
||||
trace, audio = await orchestrator.chat_stream(
|
||||
|
|
@ -146,6 +152,7 @@ async def _run_turn(
|
|||
history=llm_context,
|
||||
on_token=on_token,
|
||||
on_audio=on_audio,
|
||||
on_filler=on_filler,
|
||||
text_only=text_only,
|
||||
)
|
||||
else:
|
||||
|
|
@ -169,7 +176,8 @@ async def _run_turn(
|
|||
maybe_schedule_extraction(store, user.id, session_id)
|
||||
|
||||
await websocket.send_json(
|
||||
{"type": "semantic", "text": trace.semantic_response, "spoken": trace.spoken_response}
|
||||
{"type": "semantic", "text": trace.semantic_response,
|
||||
"spoken": trace.spoken_response, "citations": trace.citations}
|
||||
)
|
||||
# Im text_only-Modus kommt kein Audio (Gerät spricht selbst).
|
||||
if not audio_stream and not text_only:
|
||||
|
|
|
|||
|
|
@ -188,6 +188,7 @@ class Settings(BaseSettings):
|
|||
memory_extraction_max: int = 50
|
||||
memory_extraction_provider: str = ""
|
||||
audio_stream_default: bool = True # satzweises TTS als Default (Admin kann abschalten)
|
||||
web_search_enabled: bool = True # Web-Suche via Tool-Calling (Weg 2); global an, pro Nutzer abschaltbar
|
||||
# TTS-Text-Normalisierung: auto|full|light|off. "auto" = piper -> full, Cloud -> light.
|
||||
tts_normalize_level: str = "auto"
|
||||
stt_fallback: str = "" # kommaseparierte Provider-Namen (Fallback-Kette)
|
||||
|
|
|
|||
25
app/core/clock.py
Normal file
25
app/core/clock.py
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
"""Zentrale Datum-/Uhrzeit-Auskunft für alle Programmteile.
|
||||
|
||||
Jedes Modul kann hiermit das aktuelle Datum und die Uhrzeit (lokale Zeitzone)
|
||||
erfragen, um relative Zeitangaben ("heute", "morgen", "in 3 Stunden", "nächsten
|
||||
Montag") in absolute Werte umzurechnen. Wird u. a. in die LLM-System-Prompts
|
||||
injiziert (`now_context()`), damit die Modelle relative Zeitangaben auflösen können.
|
||||
"""
|
||||
from datetime import datetime
|
||||
|
||||
_WEEKDAYS_EN = ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"]
|
||||
|
||||
|
||||
def now() -> datetime:
|
||||
"""Aktuelle lokale, zeitzonenbewusste Zeit."""
|
||||
return datetime.now().astimezone()
|
||||
|
||||
|
||||
def now_context() -> str:
|
||||
"""Knapper, eindeutiger Datum-/Uhrzeit-Hinweis (Englisch) für LLM-System-Prompts."""
|
||||
n = now()
|
||||
tz = n.strftime("%Z") or "local"
|
||||
return (f"The current date and time is {n.strftime('%Y-%m-%d')} "
|
||||
f"({_WEEKDAYS_EN[n.weekday()]}) {n.strftime('%H:%M')} ({tz}). "
|
||||
f"Use this to resolve relative dates and times such as 'today', "
|
||||
f"'tomorrow', 'next Monday', or 'in 3 hours'.")
|
||||
|
|
@ -1,9 +1,15 @@
|
|||
import asyncio
|
||||
import inspect
|
||||
|
||||
from app.schemas import AudioChunk, PipelineTrace
|
||||
from app.pipeline.sentence_chunker import SentenceChunker
|
||||
from app.pipeline.fillers import FillerPhrases
|
||||
from app.metrics import timer, metrics
|
||||
|
||||
# Mindestabstand (Sekunden) zwischen Beruhigungssatz und Geduldssätzen bzw.
|
||||
# zwischen Geduldssätzen, bei noch laufender Recherche.
|
||||
FILLER_PATIENCE_INTERVAL = 10.0
|
||||
|
||||
|
||||
def _stage(name: str):
|
||||
return timer("stage_duration_seconds", {"stage": name})
|
||||
|
|
@ -21,10 +27,22 @@ _LANG_NAMES = {
|
|||
"ru": "русский", "zh": "中文",
|
||||
}
|
||||
|
||||
def _stream_supports(stream_fn, name: str) -> bool:
|
||||
"""Ob stream() ein bestimmtes kwarg (oder **kwargs) akzeptiert — sonst nicht übergeben.
|
||||
|
||||
Hält den Orchestrator kompatibel mit stream()-Implementierungen ohne on_tool_start
|
||||
(Test-Doubles, ältere Provider).
|
||||
"""
|
||||
try:
|
||||
params = inspect.signature(stream_fn).parameters.values()
|
||||
except (TypeError, ValueError):
|
||||
return False
|
||||
return any(p.name == name or p.kind == p.VAR_KEYWORD for p in params)
|
||||
|
||||
|
||||
class Orchestrator:
|
||||
def __init__(self, stt, llm, tts, input_cleaner, spoken_adapter, tts_normalizer,
|
||||
normalize_level: str = "full"):
|
||||
normalize_level: str = "full", fillers=None):
|
||||
self.stt = stt
|
||||
self.llm = llm
|
||||
self.tts = tts
|
||||
|
|
@ -32,6 +50,8 @@ class Orchestrator:
|
|||
self.spoken_adapter = spoken_adapter
|
||||
self.tts_normalizer = tts_normalizer
|
||||
self.normalize_level = normalize_level
|
||||
# Ohne externe Verdrahtung: englische Master-Sätze (kein Crash in Tests).
|
||||
self.fillers = fillers or FillerPhrases()
|
||||
|
||||
async def _emit_to_output(self, audio: bytes, output) -> None:
|
||||
"""Schreibt das synthetisierte Audio durch den gewaehlten Output-Endpunkt.
|
||||
|
|
@ -167,6 +187,7 @@ class Orchestrator:
|
|||
history: list[dict] | None = None,
|
||||
on_token=None,
|
||||
on_audio=None,
|
||||
on_filler=None,
|
||||
text_only: bool = False,
|
||||
):
|
||||
"""Wie chat_text, aber gestreamt.
|
||||
|
|
@ -226,12 +247,72 @@ class Orchestrator:
|
|||
await consumer_task
|
||||
await queue.put(sentence)
|
||||
|
||||
patience_task: asyncio.Task | None = None
|
||||
answer_started = False
|
||||
|
||||
async def _emit_filler(phrase: str) -> None:
|
||||
"""Ephemer: dedizierter Filler-Kanal (Anzeige/Status + Geräte-TTS) +
|
||||
_dispatch (Server-TTS, "Satz null").
|
||||
|
||||
Läuft bewusst NICHT über den Token-Stream -> nie in parts/
|
||||
semantic_response/History. Sobald die Antwort vorliegt, wird ein
|
||||
anstehender Filler/Geduldssatz NICHT mehr ausgegeben.
|
||||
"""
|
||||
if answer_started:
|
||||
return
|
||||
if on_filler is not None:
|
||||
await on_filler(phrase)
|
||||
elif on_token is not None:
|
||||
await on_token(phrase + " ")
|
||||
if chunker is not None:
|
||||
await _dispatch(phrase)
|
||||
|
||||
async def _patience_loop(lang: str | None) -> None:
|
||||
# Bei längerer Recherche zufällige Geduldssätze nachschieben.
|
||||
try:
|
||||
while True:
|
||||
await asyncio.sleep(FILLER_PATIENCE_INTERVAL)
|
||||
await _emit_filler(await self.fillers.patience(lang))
|
||||
except asyncio.CancelledError:
|
||||
return
|
||||
|
||||
async def _on_tool_start(lang: str | None) -> None:
|
||||
nonlocal patience_task
|
||||
await _emit_filler(await self.fillers.opening(lang))
|
||||
if patience_task is None or patience_task.done():
|
||||
patience_task = asyncio.create_task(_patience_loop(lang))
|
||||
|
||||
async def _cancel_patience() -> None:
|
||||
nonlocal patience_task
|
||||
task, patience_task = patience_task, None
|
||||
if task is not None and not task.done():
|
||||
task.cancel()
|
||||
try:
|
||||
await task
|
||||
except (asyncio.CancelledError, Exception):
|
||||
pass
|
||||
|
||||
collected_citations: list[str] = []
|
||||
|
||||
async def _on_citations(urls) -> None:
|
||||
for u in (urls or []):
|
||||
if u not in collected_citations:
|
||||
collected_citations.append(u)
|
||||
|
||||
if queue is not None:
|
||||
consumer_task = asyncio.create_task(_consume())
|
||||
|
||||
try:
|
||||
if stream_fn is not None:
|
||||
async for delta in stream_fn(trace.cleaned_transcript or "", history=history, language=effective_language):
|
||||
stream_kwargs = {"history": history, "language": effective_language}
|
||||
if _stream_supports(stream_fn, "on_tool_start"):
|
||||
stream_kwargs["on_tool_start"] = _on_tool_start
|
||||
if _stream_supports(stream_fn, "on_citations"):
|
||||
stream_kwargs["on_citations"] = _on_citations
|
||||
async for delta in stream_fn(trace.cleaned_transcript or "", **stream_kwargs):
|
||||
if not answer_started:
|
||||
answer_started = True # Antwort liegt vor -> keine weiteren Filler
|
||||
await _cancel_patience()
|
||||
parts.append(delta)
|
||||
if on_token:
|
||||
await on_token(delta)
|
||||
|
|
@ -250,6 +331,7 @@ class Orchestrator:
|
|||
trace.semantic_response = "".join(parts)
|
||||
if not trace.semantic_response:
|
||||
raise RuntimeError("LLM returned an empty response")
|
||||
trace.citations = collected_citations
|
||||
|
||||
trace.spoken_response = await self.spoken_adapter.run(
|
||||
trace.semantic_response,
|
||||
|
|
@ -277,8 +359,9 @@ class Orchestrator:
|
|||
trace.tts_ready_text, voice=voice, language=effective_language
|
||||
)
|
||||
finally:
|
||||
# Bei Fehler/Abbruch den noch laufenden Consumer-Task abräumen,
|
||||
# damit kein verwaister Task zurückbleibt.
|
||||
# Geduldsschleife und Consumer-Task abräumen, damit keine verwaisten
|
||||
# Tasks zurückbleiben (bei Fehler/Abbruch).
|
||||
await _cancel_patience()
|
||||
if consumer_task is not None and not consumer_task.done():
|
||||
consumer_task.cancel()
|
||||
try:
|
||||
|
|
|
|||
|
|
@ -43,3 +43,14 @@ async def warmup_local_models() -> None:
|
|||
logger.info("warmup: faster-whisper-Modell geladen")
|
||||
except Exception:
|
||||
logger.exception("warmup: STT-Vorladen fehlgeschlagen (ignoriert)")
|
||||
|
||||
# Filler-Sätze der Standardsprache vorab übersetzen/cachen, damit der erste
|
||||
# Such-Turn sie sofort (ohne Übersetzungs-Latenz) sprechen kann.
|
||||
try:
|
||||
from app.runtime_config import runtime_settings as rs
|
||||
if rs.web_search_enabled and rs.openrouter_api_key:
|
||||
from app.dependencies import get_fillers
|
||||
await get_fillers().warm(rs.default_language)
|
||||
logger.info("warmup: Filler-Sätze (%s) übersetzt/gecacht", rs.default_language)
|
||||
except Exception:
|
||||
logger.exception("warmup: Filler-Vorwärmen fehlgeschlagen (ignoriert)")
|
||||
|
|
|
|||
|
|
@ -17,6 +17,8 @@ from app.providers.stt.openrouter import OpenRouterSTTProvider
|
|||
from app.providers.stt.faster_whisper import FasterWhisperProvider
|
||||
from app.providers.llm.local_openai_compatible import LocalOpenAICompatibleLLM
|
||||
from app.providers.llm.openrouter import OpenRouterLLMProvider
|
||||
from app.providers.llm.tool_calling import ToolCallingLLM
|
||||
from app.tools.web_search import SonarTool
|
||||
from app.providers.tts.openrouter import OpenRouterTTSProvider
|
||||
from app.providers.tts.cartesia import CartesiaTTSProvider
|
||||
from app.providers.tts.chatterbox import ChatterboxTTSProvider
|
||||
|
|
@ -26,6 +28,8 @@ from app.providers.fallback import (
|
|||
FallbackLLMProvider,
|
||||
FallbackTTSProvider,
|
||||
)
|
||||
from app.pipeline.decontextualizer import Decontextualizer
|
||||
from app.pipeline.fillers import FillerPhrases, make_openrouter_translator
|
||||
from app.pipeline.input_cleaner import InputCleaner
|
||||
from app.pipeline.spoken_response_adapter import SpokenResponseAdapter
|
||||
from app.pipeline.tts_normalizer import TTSNormalizer
|
||||
|
|
@ -45,6 +49,20 @@ def get_store() -> Store:
|
|||
_store = SQLiteStore(settings.db_path)
|
||||
return _store
|
||||
|
||||
|
||||
# Filler-Sätze (Web-Suche): Modul-Singleton mit Übersetzungs-Cache.
|
||||
_fillers: FillerPhrases | None = None
|
||||
|
||||
|
||||
def get_fillers(cfg=None) -> FillerPhrases:
|
||||
global _fillers
|
||||
if _fillers is None:
|
||||
cfg = cfg or runtime_settings
|
||||
_fillers = FillerPhrases(
|
||||
translate_pool=make_openrouter_translator(cfg.openrouter_api_key, cfg.openrouter_llm_model)
|
||||
)
|
||||
return _fillers
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Provider-Registries: Modul austauschbar via Name, ohne Kern-Code zu aendern.
|
||||
# Ein neuer Provider = ein Eintrag. Unbekannter Name -> UnknownComponentError.
|
||||
|
|
@ -56,6 +74,13 @@ STT_REGISTRY = {
|
|||
|
||||
LLM_REGISTRY = {
|
||||
"openrouter": lambda s: OpenRouterLLMProvider(s.openrouter_api_key, s.openrouter_llm_model),
|
||||
# Tool-fähige Variante desselben OpenRouter-Modells (Weg 2: web_search via Sonar).
|
||||
"openrouter-tools": lambda s: ToolCallingLLM(
|
||||
s.openrouter_api_key,
|
||||
s.openrouter_llm_model,
|
||||
tools=[SonarTool(s.openrouter_api_key)],
|
||||
decontextualizer=Decontextualizer(s.openrouter_api_key, s.openrouter_llm_model),
|
||||
),
|
||||
"local-openai-compatible": lambda s: LocalOpenAICompatibleLLM(
|
||||
s.local_llm_base_url,
|
||||
s.local_llm_api_key,
|
||||
|
|
@ -156,8 +181,20 @@ ROUTE_KEYS = (
|
|||
"tts_provider",
|
||||
"language",
|
||||
"voice_gender",
|
||||
"web_search_enabled",
|
||||
)
|
||||
|
||||
|
||||
def _as_bool(value, default: bool = True) -> bool:
|
||||
"""Robuste Bool-Auflösung (Prefs/Overrides können Strings sein)."""
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if isinstance(value, str):
|
||||
return value.strip().lower() in ("1", "true", "yes", "on")
|
||||
if value is None:
|
||||
return default
|
||||
return bool(value)
|
||||
|
||||
# Stimm-Auswahl nach Sprache: (effektive) Sprache → beste Piper-Stimme (gender-agnostisch).
|
||||
LANG_TO_PIPER_VOICE: dict[str, str] = {
|
||||
"de": "de_DE-thorsten-high",
|
||||
|
|
@ -236,6 +273,7 @@ class ResolvedRoute:
|
|||
tts_provider: str
|
||||
language: str
|
||||
voice_gender: str = "any"
|
||||
web_search_enabled: bool = True
|
||||
|
||||
def as_dict(self) -> dict:
|
||||
return {
|
||||
|
|
@ -246,6 +284,7 @@ class ResolvedRoute:
|
|||
"tts_provider": self.tts_provider,
|
||||
"language": self.language,
|
||||
"voice_gender": self.voice_gender,
|
||||
"web_search_enabled": self.web_search_enabled,
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -286,6 +325,7 @@ def resolve_route(
|
|||
"llm_provider": cfg.default_llm_provider,
|
||||
"tts_provider": cfg.default_tts_provider,
|
||||
"language": cfg.default_language,
|
||||
"web_search_enabled": cfg.web_search_enabled,
|
||||
}
|
||||
|
||||
user_prefs = user.prefs if user is not None else {}
|
||||
|
|
@ -298,6 +338,8 @@ def resolve_route(
|
|||
if value is not None:
|
||||
resolved[key] = value
|
||||
|
||||
# web_search_enabled kann als String aus Prefs/Overrides kommen -> robust nach bool.
|
||||
resolved["web_search_enabled"] = _as_bool(resolved["web_search_enabled"], cfg.web_search_enabled)
|
||||
route = ResolvedRoute(**resolved)
|
||||
# Admin-Vorgabe „erlaubte Sprachen pro Nutzer": auf eine erlaubte Sprache klemmen.
|
||||
allowed = [s.strip() for s in str(user_prefs.get("allowed_languages") or "").split(",") if s.strip()]
|
||||
|
|
@ -337,14 +379,21 @@ def _resolve_normalize_level(tts_provider: str, cfg: Settings) -> str:
|
|||
|
||||
def build_orchestrator(route: ResolvedRoute, cfg=None) -> Orchestrator:
|
||||
cfg = cfg or runtime_settings
|
||||
# web_search (Weg 2): tool-fähige Variante desselben OpenRouter-Modells wählen.
|
||||
# Nur wenn der aufgelöste LLM-Provider "openrouter" ist — lokale Modelle können
|
||||
# (über diesen Pfad) kein Tool-Calling. Die llm_fallback-Kette bleibt erhalten.
|
||||
llm_name = route.llm_provider
|
||||
if route.web_search_enabled and llm_name == "openrouter":
|
||||
llm_name = "openrouter-tools"
|
||||
return Orchestrator(
|
||||
stt=_provider_chain(STT_REGISTRY, route.stt_provider, cfg.stt_fallback, "stt", cfg),
|
||||
llm=_provider_chain(LLM_REGISTRY, route.llm_provider, cfg.llm_fallback, "llm", cfg),
|
||||
llm=_provider_chain(LLM_REGISTRY, llm_name, cfg.llm_fallback, "llm", cfg),
|
||||
tts=_provider_chain(TTS_REGISTRY, route.tts_provider, cfg.tts_fallback, "tts", cfg),
|
||||
input_cleaner=InputCleaner(),
|
||||
spoken_adapter=SpokenResponseAdapter(),
|
||||
tts_normalizer=TTSNormalizer(),
|
||||
normalize_level=_resolve_normalize_level(route.tts_provider, cfg),
|
||||
fillers=get_fillers(cfg),
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
84
app/pipeline/decontextualizer.py
Normal file
84
app/pipeline/decontextualizer.py
Normal file
|
|
@ -0,0 +1,84 @@
|
|||
"""Koreferenz-Vorstufe: löst Pronomen der letzten Äußerung anhand des Verlaufs auf.
|
||||
|
||||
Schließt die im Tool-Calling-Eval isolierte Restkante (nl + Pronomen-aus-History,
|
||||
z. B. „Leeft hij nog?" → „Leeft Rutger Hauer nog?"). Bewusst **gegated** (kurze
|
||||
Folgefrage MIT Pronomen UND vorhandener History), damit nicht jeder Turn einen
|
||||
Extra-Call kostet. Siehe Docs/weg2-tool-calling.md §5.3.
|
||||
"""
|
||||
import logging
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
ENDPOINT = "https://openrouter.ai/api/v1/chat/completions"
|
||||
|
||||
# Pronomen/Referenzwörter (lowercase, mehrsprachig) für die Gate-Heuristik.
|
||||
_PRONOUNS = {
|
||||
"er", "sie", "es", "der", "die", "das", "den", "dem", "deren", "dessen",
|
||||
"ihn", "ihm", "ihr", # de
|
||||
"he", "she", "it", "they", "him", "her", "them", "that", "those", "these", # en
|
||||
"hij", "ze", "zij", "het", "die", "dat", "hem", "haar", "hen", "hun", # nl
|
||||
"il", "elle", "ils", "elles", "lui", "celui", "celle", # fr
|
||||
"él", "ella", "ellos", "ese", "esa", "lei", "loro", "quello", # es/it
|
||||
}
|
||||
_MAX_WORDS = 8
|
||||
|
||||
_SYSTEM = (
|
||||
"You are a coreference resolver. Given a short conversation and the user's "
|
||||
"latest message, output ONLY that latest message rewritten so it stands on "
|
||||
"its own: resolve pronouns and references to the concrete names or entities "
|
||||
"mentioned earlier. Keep the original language and meaning. Do NOT answer it; "
|
||||
"only rewrite. If it is already self-contained, output it unchanged."
|
||||
)
|
||||
|
||||
|
||||
def _words(text: str) -> list[str]:
|
||||
return [w for w in "".join(c.lower() if (c.isalpha() or c == " ") else " "
|
||||
for c in text).split() if w]
|
||||
|
||||
|
||||
class Decontextualizer:
|
||||
def __init__(self, api_key: str, model: str, timeout: float = 15.0):
|
||||
self.api_key = (api_key or "").strip()
|
||||
self.model = (model or "").strip()
|
||||
self.timeout = timeout
|
||||
|
||||
def _gated(self, text: str, history) -> bool:
|
||||
"""Nur kurze Folgefragen mit Pronomen und vorhandener History."""
|
||||
if not history or not text:
|
||||
return False
|
||||
words = _words(text)
|
||||
if not words or len(words) > _MAX_WORDS:
|
||||
return False
|
||||
return any(w in _PRONOUNS for w in words)
|
||||
|
||||
def _render(self, history: list[dict]) -> str:
|
||||
lines = []
|
||||
for m in history[-6:]:
|
||||
who = "User" if m.get("role") == "user" else "Assistant"
|
||||
lines.append(f"{who}: {m.get('content', '')}")
|
||||
return "\n".join(lines)
|
||||
|
||||
async def run(self, text: str, history: list[dict] | None = None,
|
||||
language: str | None = None) -> str:
|
||||
if not self.api_key or not self._gated(text, history):
|
||||
return text
|
||||
user = (f"Conversation:\n{self._render(history)}\n\n"
|
||||
f"Latest message: {text}\n\nRewritten self-contained message:")
|
||||
payload = {"model": self.model, "temperature": 0.0,
|
||||
"messages": [{"role": "system", "content": _SYSTEM},
|
||||
{"role": "user", "content": user}]}
|
||||
headers = {"Authorization": f"Bearer {self.api_key}", "Content-Type": "application/json"}
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(self.timeout)) as client:
|
||||
resp = await client.post(ENDPOINT, json=payload, headers=headers)
|
||||
resp.raise_for_status()
|
||||
out = (resp.json()["choices"][0]["message"]["content"] or "").strip()
|
||||
except Exception as exc: # noqa: BLE001 — best effort, bei Fehler Original behalten
|
||||
logger.warning("Decontextualizer fehlgeschlagen: %s", exc)
|
||||
return text
|
||||
# Schutz vor Ausreißern (Erklärungen statt Rewrite): nur Plausibles übernehmen.
|
||||
if not out or len(out) > len(text) + 200:
|
||||
return text
|
||||
return out.strip().strip('"')
|
||||
146
app/pipeline/fillers.py
Normal file
146
app/pipeline/fillers.py
Normal file
|
|
@ -0,0 +1,146 @@
|
|||
"""Senioren-Beruhigungssätze für die Web-Suche — englische Master-Quelle.
|
||||
|
||||
DIES IST DIE EINZIGE STELLE zum Pflegen der Sätze. Zur Laufzeit werden sie einmal
|
||||
je Sprache übersetzt und gecacht (die Standardsprache wird beim Serverstart
|
||||
vorgewärmt → sofortige Ausgabe ohne Verzögerung). Zwei Pools:
|
||||
|
||||
- OPENING: sofort beim Such-Start ("Einen Moment, ich schaue kurz nach.")
|
||||
- PATIENCE: bei längerer Recherche nachgeschoben ("Bitte noch etwas Geduld …")
|
||||
|
||||
Beide werden ZUFÄLLIG gewählt, damit schnelles Feedback kommt und es nicht
|
||||
roboterhaft eintönig klingt. Siehe Docs/weg2-tool-calling.md §5.4.
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
import random
|
||||
import re
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
ENDPOINT = "https://openrouter.ai/api/v1/chat/completions"
|
||||
|
||||
# --- Master-Sätze (Englisch — einzige Quelle der Wahrheit) ------------------
|
||||
OPENING_PHRASES = [
|
||||
"One moment, let me look that up for you.",
|
||||
"Just a second, I'll check that for you right away.",
|
||||
"Let me find that out for you — one moment.",
|
||||
"Good question — let me check the latest on that.",
|
||||
"Hold on just a moment, I'm looking it up.",
|
||||
"Let me quickly find the current information for you.",
|
||||
"One little moment while I check that for you.",
|
||||
"I'll go and look that up for you right now.",
|
||||
"Let me see what the latest is — just a moment.",
|
||||
"Give me a second, I'll find that out for you.",
|
||||
"I'm checking that for you now, one moment please.",
|
||||
"Let me look that up so I can tell you exactly.",
|
||||
]
|
||||
|
||||
PATIENCE_PHRASES = [
|
||||
"Thank you for your patience, I'm almost there.",
|
||||
"Just a little longer, please — I'm still looking.",
|
||||
"Bear with me a moment, I'm nearly done.",
|
||||
"Almost ready — thank you for waiting.",
|
||||
"I'm still checking, just a few more seconds.",
|
||||
"Please hold on a little longer, I'm getting there.",
|
||||
"Nearly finished — thank you for your patience.",
|
||||
"Just a moment more, I want to get this right for you.",
|
||||
"I'm still looking it up, please stay with me.",
|
||||
"It's taking a little longer, but I'm almost done.",
|
||||
"Thank you for waiting — I'll have it for you shortly.",
|
||||
"Hang on just a bit longer, I'm nearly there.",
|
||||
"I haven't forgotten you — I'm still checking.",
|
||||
"Almost there now, thank you for being so patient.",
|
||||
"Just finishing up — one more moment, please.",
|
||||
"I'm making sure I get it right for you — nearly done.",
|
||||
]
|
||||
|
||||
_LANG_DISPLAY = {
|
||||
"de": "German", "en": "English", "fr": "French", "es": "Spanish",
|
||||
"it": "Italian", "nl": "Dutch", "pt": "Portuguese", "pl": "Polish",
|
||||
"ar": "Arabic", "ru": "Russian", "zh": "Chinese",
|
||||
}
|
||||
|
||||
|
||||
def _parse_numbered(text: str, n: int) -> list[str] | None:
|
||||
"""Liest eine nummerierte Liste; liefert genau n Zeilen oder None (→ Fallback)."""
|
||||
out = []
|
||||
for raw in (text or "").splitlines():
|
||||
s = re.sub(r"^\s*\d+[.)]\s*", "", raw.strip()).strip().strip('"').strip("„“”")
|
||||
if s:
|
||||
out.append(s)
|
||||
return out[:n] if len(out) >= n else None
|
||||
|
||||
|
||||
def make_openrouter_translator(api_key: str, model: str):
|
||||
"""Liefert eine async translate_pool(phrases, language) -> list[str] | None."""
|
||||
api_key = (api_key or "").strip()
|
||||
model = (model or "").strip()
|
||||
|
||||
async def translate_pool(phrases: list[str], language: str) -> list[str] | None:
|
||||
if not api_key or not model:
|
||||
return None
|
||||
target = _LANG_DISPLAY.get(language, language)
|
||||
numbered = "\n".join(f"{i + 1}. {p}" for i, p in enumerate(phrases))
|
||||
prompt = (
|
||||
f"Translate each numbered phrase into {target}. These are short, warm, "
|
||||
f"reassuring things a voice assistant says to an elderly person while it "
|
||||
f"looks something up. Keep them natural and spoken. ALWAYS use the polite, "
|
||||
f"formal form of address (German 'Sie', not 'du'; French 'vous'; Spanish "
|
||||
f"'usted'; Italian 'Lei'; Dutch 'u') consistently. Same count, same order, "
|
||||
f"no quotes, no extra words. Output ONLY a numbered list of the translations."
|
||||
f"\n\n{numbered}"
|
||||
)
|
||||
payload = {"model": model, "temperature": 0.4,
|
||||
"messages": [{"role": "user", "content": prompt}]}
|
||||
headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(30.0)) as client:
|
||||
resp = await client.post(ENDPOINT, json=payload, headers=headers)
|
||||
resp.raise_for_status()
|
||||
text = resp.json()["choices"][0]["message"]["content"]
|
||||
return _parse_numbered(text, len(phrases))
|
||||
|
||||
return translate_pool
|
||||
|
||||
|
||||
class FillerPhrases:
|
||||
"""Zufällige, in die Zielsprache übersetzte Beruhigungssätze (einmal je Sprache gecacht)."""
|
||||
|
||||
def __init__(self, translate_pool=None):
|
||||
self._translate_pool = translate_pool
|
||||
self._cache: dict[str, dict] = {
|
||||
"en": {"opening": OPENING_PHRASES, "patience": PATIENCE_PHRASES}}
|
||||
self._locks: dict[str, asyncio.Lock] = {}
|
||||
|
||||
async def warm(self, language: str | None) -> None:
|
||||
"""Vorab übersetzen (z. B. Standardsprache beim Serverstart)."""
|
||||
await self._ensure(language)
|
||||
|
||||
async def _ensure(self, language: str | None) -> str:
|
||||
lang = (language or "en").lower()
|
||||
if lang in self._cache:
|
||||
return lang
|
||||
if self._translate_pool is None:
|
||||
return "en" # ohne Übersetzer: englische Master-Sätze
|
||||
lock = self._locks.setdefault(lang, asyncio.Lock())
|
||||
async with lock:
|
||||
if lang in self._cache:
|
||||
return lang
|
||||
opening = patience = None
|
||||
try:
|
||||
opening = await self._translate_pool(OPENING_PHRASES, lang)
|
||||
patience = await self._translate_pool(PATIENCE_PHRASES, lang)
|
||||
except Exception as exc: # noqa: BLE001 — best effort, Fallback Englisch
|
||||
logger.warning("Filler-Übersetzung (%s) fehlgeschlagen: %s", lang, exc)
|
||||
self._cache[lang] = {"opening": opening or OPENING_PHRASES,
|
||||
"patience": patience or PATIENCE_PHRASES}
|
||||
return lang
|
||||
|
||||
async def opening(self, language: str | None) -> str:
|
||||
lang = await self._ensure(language)
|
||||
return random.choice(self._cache[lang]["opening"])
|
||||
|
||||
async def patience(self, language: str | None) -> str:
|
||||
lang = await self._ensure(language)
|
||||
return random.choice(self._cache[lang]["patience"])
|
||||
66
app/pipeline/search_backstop.py
Normal file
66
app/pipeline/search_backstop.py
Normal file
|
|
@ -0,0 +1,66 @@
|
|||
"""Deterministischer Such-Backstop für die heiklen Kategorien.
|
||||
|
||||
Das Modell self-triggert die Web-Suche im Eval zu ~93 %. Die verbleibenden Misses
|
||||
liegen ausgerechnet in den Kategorien, in denen eine veraltete Erstantwort
|
||||
**konfident-falsch** ist (Amtsträger, Wohnort lebender Personen, „lebt X noch").
|
||||
Für klar erkennbare Phrasierungen dieser Kategorien erzwingt der Backstop den
|
||||
web_search-Aufruf (`tool_choice`) — deterministisch, ohne Zusatz-Call.
|
||||
|
||||
Bewusst auf Recall optimiert (lieber eine überflüssige Suche als eine konfident
|
||||
falsche Antwort); gelegentliches Über-Triggern ist akzeptabel. Mehrsprachig
|
||||
(de/en/nl im Kern, etwas fr/es). Offline testbar (reine Regex).
|
||||
"""
|
||||
import re
|
||||
|
||||
# Amts-/Rollenbegriffe, deren Inhaber wechseln.
|
||||
_OFFICE = (
|
||||
r"(?:bundeskanzler(?:in)?|kanzler(?:in)?|pr[äa]sident(?:in)?|president|papst|pope|"
|
||||
r"ministerpr[äa]sident(?:in)?|minister-?president|premierminister|premier|"
|
||||
r"regierungschef(?:in)?|regeringsleider|kanselier|au[ßs]enminister(?:in)?|"
|
||||
r"k[öo]nig(?:in)?|queen|king|kaiser|paus|premier ?minister)"
|
||||
)
|
||||
# Gegenwarts-/Aktualitäts-Marker.
|
||||
_NOW = r"(?:aktuell\w*|derzeit\w*|momentan|heute|jetzt|gerade|amtierend\w*|"\
|
||||
r"current\w*|now|tegenwoordig|nu|huidige|actuel\w*|actual\w*)"
|
||||
|
||||
_PATTERNS = [
|
||||
# Amtsträger: "wer ist <Amt>", "aktueller <Amt>", "<Amt> ... aktuell/jetzt".
|
||||
# Bewusst Gegenwart (wer IST, nicht wer WAR) -> historische Fragen triggern nicht.
|
||||
re.compile(rf"\b(?:wer (?:ist|sind)|who (?:is|are)|wie hei[sß]t|wie is|wie heet)\b"
|
||||
rf".{{0,30}}\b{_OFFICE}\b", re.I),
|
||||
re.compile(rf"\b{_NOW}\s+{_OFFICE}\b", re.I),
|
||||
re.compile(rf"\b{_OFFICE}\b.{{0,30}}\b{_NOW}\b", re.I),
|
||||
# Wohnort lebender Person
|
||||
re.compile(r"\bwo\s+(?:wohnt|lebt|residiert)\b", re.I),
|
||||
re.compile(r"\bwhere\s+(?:does|do|is|are)\b.{0,40}\blive[sd]?\b", re.I),
|
||||
re.compile(r"\bwaar\s+woont\b", re.I),
|
||||
re.compile(r"\bo[ùu]\s+(?:habite|vit)\b", re.I),
|
||||
re.compile(r"\bd[óo]nde\s+vive\b", re.I),
|
||||
# Lebt X noch / still alive
|
||||
re.compile(r"\b(?:lebt|leeft)\b.{0,40}\b(?:noch|nog)\b", re.I),
|
||||
re.compile(r"\bnoch\s+am\s+leben\b", re.I),
|
||||
re.compile(r"\bstill\s+alive\b", re.I),
|
||||
re.compile(r"\b(?:encore|toujours)\s+(?:en\s+vie|vivant)\b", re.I),
|
||||
re.compile(r"\b(?:sigue|todav[íi]a)\b.{0,20}\bviv[oa]\b", re.I),
|
||||
|
||||
# Öffnungszeiten / „hat X geöffnet" (Gegenwart/Logistik)
|
||||
re.compile(r"\b(?:öffnungszeiten|opening hours|openingstijden|horaires? d.ouverture)\b", re.I),
|
||||
re.compile(r"\b(?:geöffnet|geschlossen)\b", re.I),
|
||||
re.compile(r"\bnoch\s+(?:offen|auf|geöffnet)\b", re.I),
|
||||
re.compile(r"\bis\b.{0,30}\bopen\b", re.I),
|
||||
re.compile(r"\b(?:geopend|open)\b.{0,12}\b(?:vandaag|nu)\b", re.I),
|
||||
re.compile(r"\bnotdienst\b", re.I),
|
||||
# Fahrpläne / nächste Abfahrt
|
||||
re.compile(r"\b(?:wann|wanneer|hoe laat|when|what time|à quelle heure)\b.{0,40}"
|
||||
r"\b(?:bus|z[üu]ge?|bahn|tram|s-?bahn|u-?bahn|train|trein|metro|f[äa]hre|veerboot|flug|flight)\b", re.I),
|
||||
re.compile(r"\b(?:nächste[rn]?|next|volgende|prochain\w*)\b.{0,15}"
|
||||
r"\b(?:bus|zug|bahn|train|trein|tram|f[äa]hre|metro)\b", re.I),
|
||||
re.compile(r"\b(?:abfahrt\w*|departure|vertrek\w*)\b", re.I),
|
||||
]
|
||||
|
||||
|
||||
def should_force_search(text: str | None) -> bool:
|
||||
"""True, wenn der Text klar eine heikle, veränderliche Tatsache abfragt."""
|
||||
if not text:
|
||||
return False
|
||||
return any(p.search(text) for p in _PATTERNS)
|
||||
|
|
@ -65,13 +65,14 @@ class FallbackLLMProvider(_Chain):
|
|||
self._on_error(name)
|
||||
raise last_exc
|
||||
|
||||
async def stream(self, text, history=None, session_id=None, language=None) -> AsyncIterator[str]:
|
||||
async def stream(self, text, history=None, session_id=None, language=None,
|
||||
**kwargs) -> AsyncIterator[str]:
|
||||
last_exc = None
|
||||
for index, (name, provider) in enumerate(self.entries):
|
||||
produced = False
|
||||
try:
|
||||
async for delta in provider.stream(
|
||||
text, history=history, session_id=session_id, language=language
|
||||
text, history=history, session_id=session_id, language=language, **kwargs
|
||||
):
|
||||
produced = True
|
||||
yield delta
|
||||
|
|
|
|||
|
|
@ -70,9 +70,11 @@ class LLMProvider(ABC):
|
|||
history: list[dict] | None = None,
|
||||
session_id: str | None = None,
|
||||
language: str | None = None,
|
||||
**kwargs,
|
||||
) -> AsyncIterator[str]:
|
||||
"""Token-Stream. Default: kein echtes Streaming -> komplette Antwort als ein Chunk.
|
||||
|
||||
Provider mit SSE-Unterstuetzung ueberschreiben diese Methode.
|
||||
Provider mit SSE-Unterstuetzung ueberschreiben diese Methode. Unbekannte
|
||||
kwargs (z. B. on_tool_start) werden ignoriert.
|
||||
"""
|
||||
yield await self.complete(text, history=history, session_id=session_id, language=language)
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ from collections.abc import AsyncIterator
|
|||
|
||||
import httpx
|
||||
|
||||
from app.core import clock
|
||||
from app.providers.llm.base import (
|
||||
LLMProvider,
|
||||
lang_instruction,
|
||||
|
|
@ -41,6 +42,7 @@ class LocalOpenAICompatibleLLM(LLMProvider):
|
|||
system_parts: list[str] = []
|
||||
if self.system_prompt:
|
||||
system_parts.append(self.system_prompt)
|
||||
system_parts.append(clock.now_context()) # aktuelles Datum/Uhrzeit (relative Angaben auflösen)
|
||||
rest: list[dict] = []
|
||||
for msg in history or []:
|
||||
if msg.get("role") == "system":
|
||||
|
|
@ -102,6 +104,7 @@ class LocalOpenAICompatibleLLM(LLMProvider):
|
|||
history: list[dict] | None = None,
|
||||
session_id: str | None = None,
|
||||
language: str | None = None,
|
||||
**kwargs, # z. B. on_tool_start — hier ignoriert (kein Tool-Calling)
|
||||
) -> AsyncIterator[str]:
|
||||
async with httpx.AsyncClient(timeout=120) as client:
|
||||
async with client.stream(
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@ from collections.abc import AsyncIterator
|
|||
|
||||
import httpx
|
||||
|
||||
from app.core import clock
|
||||
from app.providers.llm.base import (
|
||||
LLMProvider,
|
||||
lang_instruction,
|
||||
|
|
@ -100,10 +101,10 @@ class OpenRouterLLMProvider(LLMProvider):
|
|||
if not text or not text.strip():
|
||||
raise ValueError("LLM input text is empty")
|
||||
|
||||
system_content = SYSTEM_PROMPT
|
||||
system_content = f"{SYSTEM_PROMPT}\n\n{clock.now_context()}"
|
||||
instr = lang_instruction(language)
|
||||
if instr:
|
||||
system_content = f"{SYSTEM_PROMPT}\n\n{instr}"
|
||||
system_content = f"{system_content}\n\n{instr}"
|
||||
|
||||
messages = [{"role": "system", "content": system_content}]
|
||||
if history:
|
||||
|
|
@ -175,6 +176,7 @@ class OpenRouterLLMProvider(LLMProvider):
|
|||
history: list[dict] | None = None,
|
||||
session_id: str | None = None,
|
||||
language: str | None = None,
|
||||
**kwargs, # z. B. on_tool_start — hier ignoriert (kein Tool-Calling)
|
||||
) -> AsyncIterator[str]:
|
||||
payload = {
|
||||
"model": self.model,
|
||||
|
|
|
|||
373
app/providers/llm/tool_calling.py
Normal file
373
app/providers/llm/tool_calling.py
Normal file
|
|
@ -0,0 +1,373 @@
|
|||
"""ToolCallingLLM: agentischer Wrapper um ein tool-fähiges OpenRouter-Modell.
|
||||
|
||||
Implementiert die LLMProvider-Schnittstelle und wickelt intern die Tool-Schleife
|
||||
ab (Modell entscheidet via web_search, ob es sucht; Werkzeug liefert Fakten;
|
||||
Modell formuliert die finale Antwort in Persona). Der Orchestrator ruft nur
|
||||
`complete()`/`stream()` und weiß nichts von Tools. Siehe Docs/weg2-tool-calling.md.
|
||||
|
||||
Schritt 2: nur `complete()` (nicht gestreamt). Der ABC-Default `stream()` fällt
|
||||
auf `complete()` zurück, sodass der Loop schon durch den Orchestrator nutzbar ist;
|
||||
echtes Streaming + Filler folgen in Schritt 4.
|
||||
"""
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
from collections.abc import AsyncIterator
|
||||
|
||||
import httpx
|
||||
|
||||
from app.core import clock
|
||||
from app.metrics import metrics
|
||||
from app.pipeline.search_backstop import should_force_search
|
||||
from app.providers.llm.base import LLMProvider, lang_instruction, with_lang_reminder
|
||||
from app.providers.llm.openrouter import SYSTEM_PROMPT as PERSONA_PROMPT
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
ENDPOINT = "https://openrouter.ai/api/v1/chat/completions"
|
||||
_RETRY_STATUS = {404, 429, 500, 502, 503}
|
||||
# OpenRouter-404, wenn das Modell keinen Tool-fähigen Endpoint hat.
|
||||
_TOOL_UNSUPPORTED_HINT = "support tool use"
|
||||
|
||||
|
||||
class _ToolsUnsupported(Exception):
|
||||
"""Strukturelles 404: das gewählte Modell kann (über OpenRouter) kein Tool-Calling."""
|
||||
|
||||
|
||||
def _looks_tool_unsupported(status: int, body: str, has_tools: bool) -> bool:
|
||||
return has_tools and status == 404 and _TOOL_UNSUPPORTED_HINT in (body or "").lower()
|
||||
|
||||
# Tool-Schema inkl. Trigger-Kategorien — wortgleich zur im Eval validierten Fassung.
|
||||
WEB_SEARCH_TOOL = {
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "web_search",
|
||||
"description": (
|
||||
"Look up real-world information that may be newer than your "
|
||||
"knowledge or may have changed since your last update. Call it "
|
||||
"whenever the true answer could plausibly have changed, INCLUDING "
|
||||
"when no exact place or date is named. This covers: who currently "
|
||||
"holds an office; where a living person now lives; whether someone "
|
||||
"is still alive; current prices, rates or crypto; current weather "
|
||||
"or outdoor conditions (even phrased as 'is it cold/raining right "
|
||||
"now', using the user's location); sports results and standings; "
|
||||
"the latest version or model of a product; recent news; and "
|
||||
"time-sensitive logistics such as opening hours, schedules, and "
|
||||
"public-transport or train/bus departure times. Do NOT use it for "
|
||||
"timeless knowledge, opinions, jokes, small talk, the current "
|
||||
"clock time or today's date (you already have those), anything "
|
||||
"you can derive yourself, or anything about the user themselves."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"query": {
|
||||
"type": "string",
|
||||
"description": "Concise search query, in the user's language or English.",
|
||||
},
|
||||
},
|
||||
"required": ["query"],
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
# Trigger- + Vorrang-Anweisung. cutoff ist ein bewusst KONSERVATIVER Anker (eher
|
||||
# zu früh als zu spät) — das verschiebt im Zweifel Richtung „suchen", die sichere
|
||||
# Seite (höherer Recall). Im Eval mit genau dieser Formulierung validiert.
|
||||
_TRIGGER_AND_PRIORITY = (
|
||||
"Your knowledge was last updated around {cutoff}. "
|
||||
"When a question could require information newer than {cutoff} that you "
|
||||
"cannot reason out yourself, call web_search before answering. Never call "
|
||||
"web_search merely to find the current time or today's date — you already "
|
||||
"have them. Tool results are authoritative and newer than your memory: if a "
|
||||
"tool result conflicts with what you believe, follow the tool and never "
|
||||
"contradict it."
|
||||
)
|
||||
|
||||
|
||||
def _system_prompt(cutoff: str, language: str | None) -> str:
|
||||
parts = [PERSONA_PROMPT, clock.now_context(), _TRIGGER_AND_PRIORITY.format(cutoff=cutoff)]
|
||||
instr = lang_instruction(language)
|
||||
if instr:
|
||||
parts.append(instr)
|
||||
return "\n\n".join(parts)
|
||||
|
||||
|
||||
class _StreamAcc:
|
||||
"""Sammelt einen gestreamten Modell-Schritt: Text-Buffer + (fragmentierte) tool_calls.
|
||||
|
||||
Beim Streaming kommen tool_calls in Bruchstücken (je `index`: id/name einmal,
|
||||
`arguments` über viele Deltas). Hier zusammengesetzt; Text wird zugleich
|
||||
durchgereicht (siehe _stream_chat).
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self.text: list[str] = []
|
||||
self._tc: dict[int, dict] = {}
|
||||
|
||||
def add_text(self, s: str) -> None:
|
||||
self.text.append(s)
|
||||
|
||||
def add_tool_fragments(self, frags: list[dict]) -> None:
|
||||
for tc in frags:
|
||||
idx = tc.get("index", 0)
|
||||
entry = self._tc.setdefault(idx, {"id": "", "name": "", "arguments": ""})
|
||||
if tc.get("id"):
|
||||
entry["id"] = tc["id"]
|
||||
fn = tc.get("function") or {}
|
||||
if fn.get("name"):
|
||||
entry["name"] = fn["name"]
|
||||
if fn.get("arguments"):
|
||||
entry["arguments"] += fn["arguments"]
|
||||
|
||||
@property
|
||||
def tool_calls(self) -> list[dict]:
|
||||
return [
|
||||
{"id": e["id"] or f"call_{idx}", "type": "function",
|
||||
"function": {"name": e["name"], "arguments": e["arguments"]}}
|
||||
for idx, e in sorted(self._tc.items())
|
||||
]
|
||||
|
||||
def assistant_message(self) -> dict:
|
||||
"""Assistant-Message zum Re-Threading der tool_calls an das Modell."""
|
||||
return {"role": "assistant", "content": "".join(self.text) or None,
|
||||
"tool_calls": self.tool_calls}
|
||||
|
||||
|
||||
class ToolCallingLLM(LLMProvider):
|
||||
def __init__(self, api_key: str, model: str, tools: list,
|
||||
knowledge_cutoff: str = "fall 2024", max_rounds: int = 3,
|
||||
max_retries: int = 4, temperature: float = 0.3,
|
||||
decontextualizer=None):
|
||||
self.api_key = (api_key or "").strip()
|
||||
self.model = (model or "").strip()
|
||||
self.tools = {t.name: t for t in tools}
|
||||
self.cutoff = knowledge_cutoff
|
||||
self.max_rounds = max(1, max_rounds)
|
||||
self.max_retries = max(1, max_retries)
|
||||
self.temperature = temperature
|
||||
self.decontextualizer = decontextualizer
|
||||
# Wird gesetzt, sobald das Modell ein "no tool endpoints"-404 liefert →
|
||||
# für den Rest des Turns ohne Tools (kein Retry-Hänger, keine stille Stale).
|
||||
self._tools_unsupported = False
|
||||
|
||||
def _mark_tools_unsupported(self, body: str) -> None:
|
||||
if not self._tools_unsupported:
|
||||
logger.warning("Modell %r kann kein Tool-Calling → web_search für diesen Turn "
|
||||
"deaktiviert (plain). Backstop nutzt weiter direkte Suche. %s",
|
||||
self.model, (body or "")[:160])
|
||||
metrics.inc("tool_unsupported_total")
|
||||
self._tools_unsupported = True
|
||||
|
||||
async def _resolve_text(self, text: str, history, language) -> str:
|
||||
"""Koreferenz-Vorstufe (gegated) — löst Pronomen aus der History auf."""
|
||||
if self.decontextualizer is not None and history:
|
||||
return await self.decontextualizer.run(text, history, language)
|
||||
return text
|
||||
|
||||
def _initial_messages(self, text: str, history, language) -> list[dict]:
|
||||
if not self.api_key:
|
||||
raise ValueError("OPENROUTER_API_KEY is empty")
|
||||
if not self.model:
|
||||
raise ValueError("ToolCallingLLM model is empty")
|
||||
if not text or not text.strip():
|
||||
raise ValueError("LLM input text is empty")
|
||||
messages = [{"role": "system", "content": _system_prompt(self.cutoff, language)}]
|
||||
if history:
|
||||
messages.extend(history)
|
||||
messages.append({"role": "user", "content": with_lang_reminder(text.strip(), language)})
|
||||
return messages
|
||||
|
||||
async def _chat(self, messages: list[dict], allow_tools: bool = True) -> dict:
|
||||
"""Ein Modell-Aufruf; liefert die Assistant-Message (mit/ohne tool_calls)."""
|
||||
payload = {"model": self.model, "messages": messages, "temperature": self.temperature}
|
||||
if allow_tools and not self._tools_unsupported:
|
||||
payload["tools"] = [WEB_SEARCH_TOOL]
|
||||
payload["tool_choice"] = "auto"
|
||||
timeout = httpx.Timeout(connect=10.0, read=120.0, write=30.0, pool=10.0)
|
||||
headers = {"Authorization": f"Bearer {self.api_key}", "Content-Type": "application/json"}
|
||||
last_error: Exception | None = None
|
||||
|
||||
for attempt in range(1, self.max_retries + 1):
|
||||
async with httpx.AsyncClient(timeout=timeout) as client:
|
||||
resp = await client.post(ENDPOINT, json=payload, headers=headers)
|
||||
# Modell ohne Tool-Endpoint: einmalig merken, Tools entfernen, sofort
|
||||
# ohne Tools erneut (kein Retry-Hänger, keine stille Stale).
|
||||
if _looks_tool_unsupported(resp.status_code, resp.text, "tools" in payload):
|
||||
self._mark_tools_unsupported(resp.text)
|
||||
payload.pop("tools", None)
|
||||
payload.pop("tool_choice", None)
|
||||
continue
|
||||
if resp.status_code in _RETRY_STATUS and attempt < self.max_retries:
|
||||
last_error = RuntimeError(f"OpenRouter {resp.status_code}: {resp.text[:200]}")
|
||||
await asyncio.sleep(min(2.0 * attempt, 30.0))
|
||||
continue
|
||||
resp.raise_for_status()
|
||||
try:
|
||||
data = resp.json()
|
||||
except Exception: # noqa: BLE001 — non-JSON-200 kommt transient vor
|
||||
data = None
|
||||
# Transiente Fehler kommen teils als HTTP 200 mit {"error":...} oder
|
||||
# gar nicht-JSON-Body -> als retrybar behandeln.
|
||||
if not isinstance(data, dict) or "choices" not in data:
|
||||
if attempt < self.max_retries:
|
||||
last_error = RuntimeError(f"Ungültige/leere Antwort: {resp.text[:200]}")
|
||||
await asyncio.sleep(min(2.0 * attempt, 30.0))
|
||||
continue
|
||||
raise RuntimeError(f"Ungültige OpenRouter-Antwort: {resp.text[:300]}")
|
||||
return data["choices"][0]["message"]
|
||||
|
||||
raise last_error or RuntimeError("ToolCallingLLM: alle Versuche fehlgeschlagen")
|
||||
|
||||
async def _run_tool(self, tool_call: dict, language: str | None) -> tuple[str, list]:
|
||||
"""Führt einen Tool-Aufruf aus; liefert (tool-Message-Inhalt, Citations)."""
|
||||
fn = tool_call.get("function", {})
|
||||
name = fn.get("name", "")
|
||||
metrics.inc("tool_calls_total", {"tool": name or "unknown"})
|
||||
tool = self.tools.get(name)
|
||||
if tool is None:
|
||||
logger.warning("Unbekanntes Tool angefragt: %r", name)
|
||||
return f"ERROR: unknown tool {name!r}.", []
|
||||
try:
|
||||
args = json.loads(fn.get("arguments") or "{}")
|
||||
except json.JSONDecodeError:
|
||||
args = {}
|
||||
query = str(args.get("query", "")).strip()
|
||||
result = await tool.run(query, language=language)
|
||||
return result.text, list(getattr(result, "citations", []) or [])
|
||||
|
||||
async def _inject_forced_search(self, messages: list[dict], query: str,
|
||||
language: str | None) -> list:
|
||||
"""Backstop: heikle Kategorie → Suche DETERMINISTISCH ausführen und das
|
||||
Ergebnis als **System-Kontext** einspeisen. Bewusst KEIN synthetischer
|
||||
tool_call: System-Kontext funktioniert mit tool-fähigen UND tool-unfähigen
|
||||
Modellen (letztere lehnen tool-Messages mit 405 ab). Verlässlicher als
|
||||
`tool_choice`-Forcen (das honoriert das Modell nicht zuverlässig).
|
||||
Liefert die Citations zurück."""
|
||||
tool = self.tools.get("web_search")
|
||||
if tool is None:
|
||||
return []
|
||||
metrics.inc("tool_calls_total", {"tool": "web_search"})
|
||||
result = await tool.run(query, language=language)
|
||||
messages.append({
|
||||
"role": "system",
|
||||
"content": ("Aktuelle Web-Suchergebnisse zur Nutzerfrage (maßgeblich, neuer als "
|
||||
"dein Trainingswissen):\n" + result.text +
|
||||
"\n\nBeantworte die Frage des Nutzers auf Basis dieser Ergebnisse."),
|
||||
})
|
||||
return list(getattr(result, "citations", []) or [])
|
||||
|
||||
async def complete(self, text: str, history: list[dict] | None = None,
|
||||
session_id: str | None = None, language: str | None = None) -> str:
|
||||
text = await self._resolve_text(text, history, language)
|
||||
messages = self._initial_messages(text, history, language)
|
||||
if should_force_search(text):
|
||||
metrics.inc("search_forced_total")
|
||||
await self._inject_forced_search(messages, text, language)
|
||||
|
||||
for _round in range(self.max_rounds):
|
||||
msg = await self._chat(messages, allow_tools=True)
|
||||
tool_calls = msg.get("tool_calls")
|
||||
if not tool_calls:
|
||||
content = (msg.get("content") or "").strip()
|
||||
if content:
|
||||
return content
|
||||
break # leer ohne Tool-Call -> finale Runde erzwingen
|
||||
messages.append(msg) # Assistant-Message mit tool_calls (unverändert zurück)
|
||||
for tc in tool_calls:
|
||||
result_text, _cites = await self._run_tool(tc, language)
|
||||
messages.append({"role": "tool", "tool_call_id": tc.get("id", ""),
|
||||
"content": result_text})
|
||||
|
||||
# max_rounds erschöpft (oder leer): finale Antwort ohne weitere Tools erzwingen.
|
||||
final = await self._chat(messages, allow_tools=False)
|
||||
content = (final.get("content") or "").strip()
|
||||
if not content:
|
||||
raise RuntimeError("ToolCallingLLM returned an empty response")
|
||||
return content
|
||||
|
||||
@staticmethod
|
||||
async def _emit_citations(cites: list, on_citations) -> None:
|
||||
if cites and on_citations is not None:
|
||||
await on_citations(cites)
|
||||
|
||||
async def stream(self, text: str, history: list[dict] | None = None,
|
||||
session_id: str | None = None, language: str | None = None,
|
||||
on_tool_start=None, on_citations=None, **kwargs) -> AsyncIterator[str]:
|
||||
"""Gestreamter Loop: Text-Deltas durchreichen; bei tool_call Sonar+nächste Runde.
|
||||
|
||||
Kein Latenz-Regress im Normalfall (kein Tool): die Antwort streamt direkt
|
||||
durch. `on_tool_start(language)` (optional) feuert, bevor ein Tool läuft —
|
||||
Aufhänger für den ephemeren Filler (Schritt 4b).
|
||||
"""
|
||||
text = await self._resolve_text(text, history, language)
|
||||
messages = self._initial_messages(text, history, language)
|
||||
if should_force_search(text):
|
||||
metrics.inc("search_forced_total")
|
||||
if on_tool_start is not None:
|
||||
await on_tool_start(language) # Filler deckt die erzwungene Suche
|
||||
cites = await self._inject_forced_search(messages, text, language)
|
||||
await self._emit_citations(cites, on_citations)
|
||||
|
||||
for _round in range(self.max_rounds):
|
||||
acc = _StreamAcc()
|
||||
try:
|
||||
async for piece in self._stream_chat(messages, allow_tools=True, acc=acc):
|
||||
yield piece
|
||||
except _ToolsUnsupported:
|
||||
# Modell ohne Tool-Endpoint -> Runde plain (ohne Tools) wiederholen.
|
||||
acc = _StreamAcc()
|
||||
async for piece in self._stream_chat(messages, allow_tools=False, acc=acc):
|
||||
yield piece
|
||||
if not acc.tool_calls:
|
||||
return # war Text -> fertig (durchgereicht)
|
||||
if on_tool_start is not None:
|
||||
await on_tool_start(language) # 4b: Filler sofort sprechen/anzeigen
|
||||
messages.append(acc.assistant_message())
|
||||
for tc in acc.tool_calls:
|
||||
result_text, cites = await self._run_tool(tc, language)
|
||||
await self._emit_citations(cites, on_citations)
|
||||
messages.append({"role": "tool", "tool_call_id": tc["id"],
|
||||
"content": result_text})
|
||||
|
||||
# max_rounds erschöpft: finale Runde ohne weitere Tools, gestreamt.
|
||||
acc = _StreamAcc()
|
||||
async for piece in self._stream_chat(messages, allow_tools=False, acc=acc):
|
||||
yield piece
|
||||
|
||||
async def _stream_chat(self, messages: list[dict], allow_tools: bool,
|
||||
acc: _StreamAcc) -> AsyncIterator[str]:
|
||||
"""Ein gestreamter Modell-Call. Yieldet Text-Deltas; füllt acc (Text + tool_calls)."""
|
||||
payload = {"model": self.model, "messages": messages,
|
||||
"temperature": self.temperature, "stream": True}
|
||||
if allow_tools and not self._tools_unsupported:
|
||||
payload["tools"] = [WEB_SEARCH_TOOL]
|
||||
payload["tool_choice"] = "auto"
|
||||
timeout = httpx.Timeout(connect=10.0, read=120.0, write=30.0, pool=10.0)
|
||||
headers = {"Authorization": f"Bearer {self.api_key}", "Content-Type": "application/json"}
|
||||
|
||||
async with httpx.AsyncClient(timeout=timeout) as client:
|
||||
async with client.stream("POST", ENDPOINT, json=payload, headers=headers) as resp:
|
||||
if resp.status_code >= 400:
|
||||
body = (await resp.aread()).decode(errors="replace")
|
||||
# Modell ohne Tool-Endpoint: merken + signalisieren (Caller wiederholt ohne Tools).
|
||||
if _looks_tool_unsupported(resp.status_code, body, "tools" in payload):
|
||||
self._mark_tools_unsupported(body)
|
||||
raise _ToolsUnsupported()
|
||||
raise RuntimeError(f"OpenRouter {resp.status_code}: {body[:200]}")
|
||||
async for line in resp.aiter_lines():
|
||||
if not line.startswith("data:"):
|
||||
continue
|
||||
data = line[len("data:"):].strip()
|
||||
if not data or data == "[DONE]":
|
||||
continue
|
||||
try:
|
||||
delta = json.loads(data)["choices"][0]["delta"]
|
||||
except (ValueError, KeyError, IndexError, TypeError):
|
||||
continue
|
||||
content = delta.get("content")
|
||||
if content:
|
||||
acc.add_text(content)
|
||||
yield content
|
||||
if delta.get("tool_calls"):
|
||||
acc.add_tool_fragments(delta["tool_calls"])
|
||||
|
|
@ -30,6 +30,7 @@ RUNTIME_SETTABLE: dict[str, tuple[str, str, str]] = {
|
|||
"local_llm_max_tokens": ("Max. Tokens (lokal)", "int", "0 = kein Limit"),
|
||||
"tts_normalize_level": ("TTS-Normalisierung", "str", "auto | full | light | off"),
|
||||
"audio_stream_default": ("Audio-Streaming Standard", "bool", "true | false"),
|
||||
"web_search_enabled": ("Web-Suche (Standard)", "bool", "true | false — global an, pro Nutzer abschaltbar"),
|
||||
"memory_extraction_enabled": ("Erinnerungs-Extraktion", "bool", "true | false"),
|
||||
"memory_extraction_every_n_turns": ("Extraktion alle N Turns", "int", "z.B. 3"),
|
||||
"daily_request_limit": ("Tageskontingent (global)", "int", "0 = unbegrenzt"),
|
||||
|
|
|
|||
|
|
@ -34,6 +34,7 @@ class PipelineTrace(BaseModel):
|
|||
semantic_response: str | None = None
|
||||
spoken_response: str | None = None
|
||||
tts_ready_text: str | None = None
|
||||
citations: list[str] = [] # Quellen-URLs der Web-Suche (für die UI-Bubble)
|
||||
|
||||
|
||||
class SpeakRequest(BaseModel):
|
||||
|
|
|
|||
0
app/tools/__init__.py
Normal file
0
app/tools/__init__.py
Normal file
121
app/tools/web_search.py
Normal file
121
app/tools/web_search.py
Normal file
|
|
@ -0,0 +1,121 @@
|
|||
"""web_search-Werkzeug: holt tagesaktuelle Fakten über perplexity/sonar (OpenRouter).
|
||||
|
||||
Reiner Executor — die Tool-Schema-/Trigger-Entscheidung liegt beim ToolCallingLLM
|
||||
(siehe Docs/weg2-tool-calling.md). Liefert knappe Fakten als tool-Message-Inhalt
|
||||
plus Citations als UI-Metadaten (nicht ins TTS). Bei Timeout/Fehler einen
|
||||
honest-punt-Sentinel, damit das Modell ehrlich antwortet statt zu hängen.
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
import httpx
|
||||
|
||||
from app.metrics import metrics
|
||||
from app.providers.llm.base import lang_instruction
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
ENDPOINT = "https://openrouter.ai/api/v1/chat/completions"
|
||||
_RETRY_STATUS = {429, 500, 502, 503}
|
||||
|
||||
# Klare Anweisung an das MODELL (nicht den Nutzer), wenn keine Daten kamen —
|
||||
# das Antwortmodell formuliert daraus den ehrlichen Hinweis in Persona/Sprache.
|
||||
_NO_DATA = ("NO_CURRENT_DATA: the web search returned no usable result. "
|
||||
"Tell the user briefly that you could not retrieve current information.")
|
||||
|
||||
_SONAR_SYSTEM = (
|
||||
"You are a real-time web search assistant. Answer with current, factual "
|
||||
"information only, concisely in 1-3 sentences. Plain text only: no markdown, "
|
||||
"no lists, no citation markers like [1]."
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ToolResult:
|
||||
text: str # tool-Message-Inhalt fürs Modell
|
||||
citations: list[str] = field(default_factory=list) # UI-Metadaten, nicht ins TTS
|
||||
ok: bool = True # False = honest-punt-Sentinel
|
||||
|
||||
|
||||
class SonarTool:
|
||||
"""Web-Such-Executor (perplexity/sonar via OpenRouter)."""
|
||||
|
||||
name = "web_search"
|
||||
|
||||
def __init__(self, api_key: str, model: str = "perplexity/sonar",
|
||||
timeout: float = 20.0, max_retries: int = 2):
|
||||
self.api_key = (api_key or "").strip()
|
||||
self.model = model.strip()
|
||||
self.timeout = timeout
|
||||
self.max_retries = max(1, max_retries)
|
||||
|
||||
def _messages(self, query: str, language: str | None) -> list[dict]:
|
||||
system = _SONAR_SYSTEM
|
||||
instr = lang_instruction(language)
|
||||
if instr:
|
||||
system = f"{_SONAR_SYSTEM} {instr}"
|
||||
return [{"role": "system", "content": system},
|
||||
{"role": "user", "content": query.strip()}]
|
||||
|
||||
async def run(self, query: str, language: str | None = None) -> ToolResult:
|
||||
result = await self._search(query, language)
|
||||
metrics.inc("sonar_calls_total", {"status": "ok" if result.ok else "error"})
|
||||
return result
|
||||
|
||||
async def _search(self, query: str, language: str | None = None) -> ToolResult:
|
||||
if not query or not query.strip():
|
||||
return ToolResult(text=_NO_DATA, ok=False)
|
||||
if not self.api_key:
|
||||
logger.warning("SonarTool ohne API-Key")
|
||||
return ToolResult(text=_NO_DATA, ok=False)
|
||||
|
||||
payload = {"model": self.model, "messages": self._messages(query, language)}
|
||||
headers = {"Authorization": f"Bearer {self.api_key}", "Content-Type": "application/json"}
|
||||
timeout = httpx.Timeout(self.timeout)
|
||||
|
||||
for attempt in range(1, self.max_retries + 1):
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=timeout) as client:
|
||||
resp = await client.post(ENDPOINT, json=payload, headers=headers)
|
||||
if resp.status_code in _RETRY_STATUS and attempt < self.max_retries:
|
||||
await asyncio.sleep(1.5 * attempt)
|
||||
continue
|
||||
resp.raise_for_status()
|
||||
except (httpx.TimeoutException, httpx.HTTPError) as exc:
|
||||
logger.warning("Sonar-Aufruf fehlgeschlagen (Versuch %d): %s", attempt, exc)
|
||||
if attempt < self.max_retries:
|
||||
await asyncio.sleep(1.0 * attempt)
|
||||
continue
|
||||
return ToolResult(text=_NO_DATA, ok=False)
|
||||
|
||||
data = resp.json()
|
||||
try:
|
||||
content = data["choices"][0]["message"]["content"]
|
||||
except (KeyError, IndexError, TypeError):
|
||||
logger.warning("Unerwartete Sonar-Antwort: %s", str(data)[:200])
|
||||
return ToolResult(text=_NO_DATA, ok=False)
|
||||
text = (content or "").strip()
|
||||
if not text:
|
||||
return ToolResult(text=_NO_DATA, ok=False)
|
||||
return ToolResult(text=text, citations=_extract_citations(data), ok=True)
|
||||
|
||||
return ToolResult(text=_NO_DATA, ok=False)
|
||||
|
||||
|
||||
def _extract_citations(data: dict) -> list[str]:
|
||||
"""Citations aus OpenRouter/Perplexity-Antwort: top-level `citations` oder annotations."""
|
||||
cites = data.get("citations")
|
||||
if isinstance(cites, list) and cites:
|
||||
return [c for c in cites if isinstance(c, str)]
|
||||
try:
|
||||
ann = data["choices"][0]["message"].get("annotations") or []
|
||||
except (KeyError, IndexError, TypeError):
|
||||
ann = []
|
||||
urls = []
|
||||
for a in ann:
|
||||
if isinstance(a, dict) and a.get("type") == "url_citation":
|
||||
url = (a.get("url_citation") or {}).get("url")
|
||||
if url:
|
||||
urls.append(url)
|
||||
return urls
|
||||
|
|
@ -513,6 +513,32 @@ function addMessage(role, text, meta = {}) {
|
|||
return div;
|
||||
}
|
||||
|
||||
// Quellen-Links der Web-Suche unter die Antwort-Bubble (max. 4, idempotent).
|
||||
function renderCitations(bubble, urls) {
|
||||
if (!bubble || !Array.isArray(urls) || !urls.length) return;
|
||||
let box = bubble._citationsEl;
|
||||
if (!box) {
|
||||
box = document.createElement("div");
|
||||
box.className = "citations mt-1 text-xs text-slate-500 dark:text-slate-400 "
|
||||
+ "flex flex-wrap gap-x-2 gap-y-0.5";
|
||||
bubble._citationsEl = box;
|
||||
bubble.appendChild(box);
|
||||
}
|
||||
box.innerHTML = "";
|
||||
const label = document.createElement("span");
|
||||
label.textContent = "Quellen:";
|
||||
box.appendChild(label);
|
||||
urls.slice(0, 4).forEach((u) => {
|
||||
let host;
|
||||
try { host = new URL(u).hostname.replace(/^www\./, ""); } catch { host = u; }
|
||||
const a = document.createElement("a");
|
||||
a.href = u; a.target = "_blank"; a.rel = "noopener noreferrer";
|
||||
a.textContent = host;
|
||||
a.className = "underline hover:text-slate-700 dark:hover:text-white";
|
||||
box.appendChild(a);
|
||||
});
|
||||
}
|
||||
|
||||
// ---------- Vorlesen / Replay ----------
|
||||
function setReplayPlaying(btn, on) {
|
||||
if (!btn) return;
|
||||
|
|
@ -679,6 +705,16 @@ function runTurn(path, onopen) {
|
|||
case "ack":
|
||||
routeLang = (msg.route && msg.route.language) || null;
|
||||
break;
|
||||
case "filler":
|
||||
// Beruhigungs-/Geduldssatz: transient in der Status-Zeile (nicht in der
|
||||
// Antwort-Bubble) und im Geräte-Modus sofort sprechen — OHNE cancel, damit
|
||||
// mehrere Filler in die SpeechSynthesis-Queue laufen; die Antwort (semantic)
|
||||
// canceled später und übernimmt.
|
||||
if (msg.text) statusEl.textContent = msg.text;
|
||||
if (msg.text && isDeviceMode() && deviceVoicesReady() && deviceVoiceReady(turnLang())) {
|
||||
try { speechSynthesis.speak(_makeUtterance(msg.text, turnLang())); } catch (e) {}
|
||||
}
|
||||
break;
|
||||
case "audio":
|
||||
// Ankuendigung: der naechste Binaer-Frame ist ein satzweiser TTS-Chunk.
|
||||
expectChunk = true;
|
||||
|
|
@ -694,6 +730,7 @@ function runTurn(path, onopen) {
|
|||
case "semantic":
|
||||
if (!answerEl) answerEl = addMessage("assistant", "", { lang: turnLang() });
|
||||
if (msg.text) answerEl._textEl.textContent = msg.text;
|
||||
if (msg.citations) renderCitations(answerEl, msg.citations);
|
||||
// Sprache final festhalten (für Replay) und im Geräte-Modus lokal vorlesen.
|
||||
// Für TTS das bereinigte "spoken" (ohne Markdown/Emojis) bevorzugen, anzeigen
|
||||
// bleibt der Originaltext.
|
||||
|
|
|
|||
|
|
@ -10,9 +10,9 @@
|
|||
# Nur diese LLM-Schlüssel werden ausgewertet:
|
||||
# default_llm_provider, openrouter_llm_model, local_llm_model
|
||||
presets:
|
||||
- label: "Mistral Small 24B — Standard"
|
||||
- label: "Mistral Small 3.2 24B — Standard"
|
||||
default: true
|
||||
note: "Standard aus der Konfiguration (OpenRouter)"
|
||||
note: "Standard aus der Konfiguration (OpenRouter) — tool-fähig für die Web-Suche"
|
||||
|
||||
- label: "DeepSeek V4 Flash (OpenRouter)"
|
||||
set:
|
||||
|
|
|
|||
492
eval/tool_calling/cases.yaml
Normal file
492
eval/tool_calling/cases.yaml
Normal file
|
|
@ -0,0 +1,492 @@
|
|||
# Eval-Datensatz: Tool-Calling-Trigger für die Frische-/Web-Such-Funktion.
|
||||
#
|
||||
# Felder: id, utterance, lang, expect (search|no_search), category,
|
||||
# optional: history, injected_result, expect_fact, forbid_fact, note.
|
||||
#
|
||||
# Vorrang-Fälle (mit injected_result) sind eine Teilmenge der search-Fälle und
|
||||
# werden doppelt genutzt: Pass 1 misst Trigger-Recall, Pass 2 die Vorrang-Treue.
|
||||
|
||||
# ============================================================
|
||||
# A) MUSS SUCHEN (expect: search)
|
||||
# ============================================================
|
||||
|
||||
# --- officeholder: aktuelle Amtsträger (KEIN Zeit-Stichwort!) ---
|
||||
- id: chancellor_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: officeholder
|
||||
utterance: "Wer ist Bundeskanzler?"
|
||||
- id: uspres_en
|
||||
lang: en
|
||||
expect: search
|
||||
category: officeholder
|
||||
utterance: "Who is the president of the United States?"
|
||||
- id: pm_nl
|
||||
lang: nl
|
||||
expect: search
|
||||
category: officeholder
|
||||
utterance: "Wie is de minister-president van Nederland?"
|
||||
- id: pope_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: officeholder
|
||||
utterance: "Wer ist gerade Papst?"
|
||||
|
||||
# --- residence_living_person: Wohnort lebender Person (DER Trap-Fall) ---
|
||||
- id: heino_home_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: residence_living_person
|
||||
utterance: "Wo wohnt Heino?"
|
||||
note: "Fühlt sich statisch an, ist es aber nicht (Umzug Kitzbühel)."
|
||||
- id: player_club_en
|
||||
lang: en
|
||||
expect: search
|
||||
category: residence_living_person
|
||||
utterance: "Which club does Robert Lewandowski play for now?"
|
||||
- id: singer_home_nl
|
||||
lang: nl
|
||||
expect: search
|
||||
category: residence_living_person
|
||||
utterance: "Waar woont André Rieu tegenwoordig?"
|
||||
|
||||
# --- alive_status: lebt/aktiv? ---
|
||||
- id: alive_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: alive_status
|
||||
utterance: "Lebt Heino noch?"
|
||||
- id: alive_en
|
||||
lang: en
|
||||
expect: search
|
||||
category: alive_status
|
||||
utterance: "Is Brigitte Bardot still alive?"
|
||||
|
||||
# --- weather: Echtzeit ---
|
||||
- id: weather_paris_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: weather
|
||||
utterance: "Wie ist das Wetter in Paris?"
|
||||
- id: weather_rain_nl
|
||||
lang: nl
|
||||
expect: search
|
||||
category: weather
|
||||
utterance: "Gaat het vandaag regenen in Amsterdam?"
|
||||
- id: weather_cold_en
|
||||
lang: en
|
||||
expect: search
|
||||
category: weather
|
||||
utterance: "Is it cold outside right now?"
|
||||
note: "Braucht Ort aus Nutzerprofil — Trigger muss trotzdem feuern."
|
||||
|
||||
# --- price: Kurse/Preise ---
|
||||
- id: stock_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: price
|
||||
utterance: "Was kostet die Apple-Aktie?"
|
||||
- id: gold_nl
|
||||
lang: nl
|
||||
expect: search
|
||||
category: price
|
||||
utterance: "Wat is de goudprijs op dit moment?"
|
||||
- id: fuel_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: price
|
||||
utterance: "Was kostet gerade ein Liter Diesel?"
|
||||
|
||||
# --- latest_release: neueste Version/Produkt ---
|
||||
- id: iphone_en
|
||||
lang: en
|
||||
expect: search
|
||||
category: latest_release
|
||||
utterance: "What is the newest iPhone?"
|
||||
- id: release_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: latest_release
|
||||
utterance: "Welches ist das aktuellste Samsung-Galaxy-Handy?"
|
||||
|
||||
# --- sports_result: Ergebnisse/Tabellen ---
|
||||
- id: match_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: sports_result
|
||||
utterance: "Wie hat Bayern München am Wochenende gespielt?"
|
||||
- id: standings_en
|
||||
lang: en
|
||||
expect: search
|
||||
category: sports_result
|
||||
utterance: "Who is leading the Premier League?"
|
||||
|
||||
# --- news_event: jüngere Ereignisse ---
|
||||
- id: news_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: news_event
|
||||
utterance: "Was ist gerade die wichtigste Nachricht aus Berlin?"
|
||||
- id: news_fr
|
||||
lang: fr
|
||||
expect: search
|
||||
category: news_event
|
||||
utterance: "Que s'est-il passé récemment en France ?"
|
||||
|
||||
# --- schedule_hours: volatile Öffnungs-/Fahrzeiten ---
|
||||
- id: hours_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: schedule_hours
|
||||
utterance: "Hat die Apotheke am Markt heute geöffnet?"
|
||||
- id: train_nl
|
||||
lang: nl
|
||||
expect: search
|
||||
category: schedule_hours
|
||||
utterance: "Hoe laat gaat de volgende trein naar Utrecht?"
|
||||
|
||||
# ============================================================
|
||||
# B) MUSS NICHT SUCHEN (expect: no_search)
|
||||
# ============================================================
|
||||
|
||||
# --- timeless: zeitloses Wissen ---
|
||||
- id: capital_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: timeless
|
||||
utterance: "Was ist die Hauptstadt von Frankreich?"
|
||||
- id: faust_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: timeless
|
||||
utterance: "Wer hat den Faust geschrieben?"
|
||||
- id: boil_en
|
||||
lang: en
|
||||
expect: no_search
|
||||
category: timeless
|
||||
utterance: "At what temperature does water boil?"
|
||||
- id: ww2_nl
|
||||
lang: nl
|
||||
expect: no_search
|
||||
category: timeless
|
||||
utterance: "Wie won de Tweede Wereldoorlog?"
|
||||
- id: math_es
|
||||
lang: es
|
||||
expect: no_search
|
||||
category: timeless
|
||||
utterance: "¿Cuánto es ciento veinte dividido entre cuatro?"
|
||||
|
||||
# --- derivable: ableitbar, kein Web nötig ---
|
||||
- id: heino_age_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: derivable
|
||||
utterance: "Wie alt ist Heino?"
|
||||
note: "Aus Geburtsjahr + heutigem Datum ableitbar."
|
||||
- id: weekday_en
|
||||
lang: en
|
||||
expect: no_search
|
||||
category: derivable
|
||||
utterance: "What day of the week is Christmas this year?"
|
||||
|
||||
# --- static_fact: feststehende Personendaten ---
|
||||
- id: heino_birthday_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: static_fact
|
||||
utterance: "Wann hat Heino Geburtstag?"
|
||||
|
||||
# --- opinion / Vorschlag ---
|
||||
- id: recipe_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: opinion
|
||||
utterance: "Kannst du mir ein einfaches Rezept für Kartoffelsuppe sagen?"
|
||||
- id: advice_nl
|
||||
lang: nl
|
||||
expect: no_search
|
||||
category: opinion
|
||||
utterance: "Welke bloemen kan ik het beste in de schaduw planten?"
|
||||
|
||||
# --- smalltalk / sozial ---
|
||||
- id: howareyou_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: smalltalk
|
||||
utterance: "Wie geht es dir heute?"
|
||||
note: "Keyword-Köder 'heute' — darf NICHT triggern."
|
||||
- id: chat_en
|
||||
lang: en
|
||||
expect: no_search
|
||||
category: smalltalk
|
||||
utterance: "I'm a bit bored, let's talk for a while."
|
||||
|
||||
# --- emotional / Unterstützung ---
|
||||
- id: lonely_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: emotional
|
||||
utterance: "Ich fühle mich heute sehr einsam."
|
||||
- id: worry_nl
|
||||
lang: nl
|
||||
expect: no_search
|
||||
category: emotional
|
||||
utterance: "Ik maak me zorgen en kan niet slapen."
|
||||
|
||||
# --- joke / Unterhaltung ---
|
||||
- id: joke_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: joke
|
||||
utterance: "Erzähl mir bitte einen Witz."
|
||||
|
||||
# --- procedural_timeless ---
|
||||
- id: howto_en
|
||||
lang: en
|
||||
expect: no_search
|
||||
category: procedural_timeless
|
||||
utterance: "How do I cook a soft-boiled egg?"
|
||||
|
||||
# --- personal_memory: betrifft den Nutzer selbst (Store, nicht Web) ---
|
||||
- id: mymed_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: personal_memory
|
||||
utterance: "Welche Medikamente nehme ich morgens?"
|
||||
note: "Aus Nutzerprofil/Erinnerungen, nicht aus dem Web."
|
||||
- id: myname_nl
|
||||
lang: nl
|
||||
expect: no_search
|
||||
category: personal_memory
|
||||
utterance: "Hoe heet mijn dochter ook alweer?"
|
||||
|
||||
# ============================================================
|
||||
# C) ADVERSARIAL (Köder & Grenzfälle — höchster Testwert)
|
||||
# ============================================================
|
||||
|
||||
# --- "aktuell"-Köder, aber persönlich ---
|
||||
- id: trap_fav_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: personal_memory
|
||||
utterance: "Was ist gerade mein Lieblingslied?"
|
||||
note: "'gerade' ist Köder; Antwort aus Erinnerungen."
|
||||
|
||||
# --- aktuell, ABER nicht aus dem Web (Systemfunktion) ---
|
||||
- id: time_now_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: system_clock
|
||||
utterance: "Wie spät ist es?"
|
||||
note: "Echtzeit, aber Systemuhr — kein Web. Grenzfall bewusst no_search."
|
||||
- id: date_today_en
|
||||
lang: en
|
||||
expect: no_search
|
||||
category: system_clock
|
||||
utterance: "What's today's date?"
|
||||
|
||||
# --- "könnte sich theoretisch ändern", praktisch nie ---
|
||||
- id: trap_capital_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: timeless
|
||||
utterance: "Ist Berlin immer noch die Hauptstadt Deutschlands?"
|
||||
note: "Suggeriert Volatilität, ist aber stabil — darf nicht übertriggern."
|
||||
|
||||
# ============================================================
|
||||
# D) FOLGEFRAGEN MIT HISTORY (Kontext-Trigger)
|
||||
# ============================================================
|
||||
|
||||
- id: ctx_heino_home_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: residence_living_person
|
||||
history:
|
||||
- role: user
|
||||
content: "Erzähl mir etwas über den Sänger Heino."
|
||||
- role: assistant
|
||||
content: "Heino ist ein deutscher Schlagersänger, bekannt für seine dunkle Brille."
|
||||
utterance: "Und wo wohnt der eigentlich?"
|
||||
note: "Referent nur aus History — Trigger muss trotzdem feuern."
|
||||
|
||||
- id: ctx_alive_nl
|
||||
lang: nl
|
||||
expect: search
|
||||
category: alive_status
|
||||
history:
|
||||
- role: user
|
||||
content: "Ken je de acteur Rutger Hauer?"
|
||||
- role: assistant
|
||||
content: "Ja, een bekende Nederlandse acteur."
|
||||
utterance: "Leeft hij eigenlijk nog?"
|
||||
|
||||
- id: ctx_followup_static_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: static_fact
|
||||
history:
|
||||
- role: user
|
||||
content: "Wer hat den Faust geschrieben?"
|
||||
- role: assistant
|
||||
content: "Johann Wolfgang von Goethe."
|
||||
utterance: "Und wann wurde der geboren?"
|
||||
note: "Folgefrage, aber zeitlos — darf nicht triggern."
|
||||
|
||||
# ============================================================
|
||||
# E) VORRANG-TESTS (injiziertes Faktum widerspricht dem Training)
|
||||
# ============================================================
|
||||
|
||||
- id: prio_chancellor_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: officeholder
|
||||
utterance: "Wer ist Bundeskanzler?"
|
||||
injected_result: "Bundeskanzler ist Friedrich Merz (CDU), im Amt seit Mai 2025."
|
||||
expect_fact: "Merz"
|
||||
forbid_fact: "Scholz"
|
||||
|
||||
- id: prio_pope_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: officeholder
|
||||
utterance: "Wer ist gerade Papst?"
|
||||
injected_result: "Papst ist Leo XIV., gewählt im Mai 2025."
|
||||
expect_fact: "Leo"
|
||||
forbid_fact: "Franziskus"
|
||||
note: "Sehr jung (nach Cutoff) — harter Vorrang-Test."
|
||||
|
||||
- id: prio_uspres_en
|
||||
lang: en
|
||||
expect: search
|
||||
category: officeholder
|
||||
utterance: "Who is the US president?"
|
||||
injected_result: "Donald Trump is the US president, inaugurated January 2025."
|
||||
expect_fact: "Trump"
|
||||
forbid_fact: "Biden"
|
||||
|
||||
- id: prio_heino_home_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: residence_living_person
|
||||
utterance: "Wo wohnt Heino?"
|
||||
injected_result: "Heino lebt inzwischen dauerhaft in Kitzbühel, Österreich."
|
||||
expect_fact: "Kitzbühel"
|
||||
forbid_fact: "Münstereifel"
|
||||
|
||||
- id: prio_iphone_en
|
||||
lang: en
|
||||
expect: search
|
||||
category: latest_release
|
||||
utterance: "What is the newest iPhone?"
|
||||
injected_result: "The newest model is the iPhone 17, released September 2025."
|
||||
expect_fact: "17"
|
||||
forbid_fact: "16"
|
||||
|
||||
# ============================================================
|
||||
# F) AUFSTOCKUNG schwacher Kategorien (für belastbare Kategorie-Raten)
|
||||
# Bewusst variantenreich: andere Entitäten/Sprachen, explizit + implizit.
|
||||
# ============================================================
|
||||
|
||||
# --- schedule_hours (war nur 2 Fälle) ---
|
||||
- id: bakery_hours_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: schedule_hours
|
||||
utterance: "Wann macht die Bäckerei morgen früh auf?"
|
||||
- id: bus_nl
|
||||
lang: nl
|
||||
expect: search
|
||||
category: schedule_hours
|
||||
utterance: "Wanneer vertrekt de volgende bus naar het centrum?"
|
||||
- id: museum_hours_en
|
||||
lang: en
|
||||
expect: search
|
||||
category: schedule_hours
|
||||
utterance: "What are the opening hours of the British Museum today?"
|
||||
- id: doctor_hours_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: schedule_hours
|
||||
utterance: "Hat die Arztpraxis heute Nachmittag noch geöffnet?"
|
||||
- id: pharmacy_night_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: schedule_hours
|
||||
utterance: "Welche Apotheke hat heute Nacht Notdienst?"
|
||||
|
||||
# --- weather (war 3 Fälle) — Fokus implizit/ortlos/ohne Wort 'Wetter' ---
|
||||
- id: forecast_munich_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: weather
|
||||
utterance: "Wie warm wird es morgen in München?"
|
||||
- id: weather_london_en
|
||||
lang: en
|
||||
expect: search
|
||||
category: weather
|
||||
utterance: "What's the weather going to be like in London tomorrow?"
|
||||
- id: umbrella_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: weather
|
||||
utterance: "Soll ich heute einen Regenschirm mitnehmen?"
|
||||
note: "Impliziter Ort, kein Wort 'Wetter' — der harte Fall."
|
||||
- id: jacket_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: weather
|
||||
utterance: "Brauche ich heute draußen eine Jacke?"
|
||||
note: "Impliziter Ort, kein Wort 'Wetter'."
|
||||
- id: umbrella_nl
|
||||
lang: nl
|
||||
expect: search
|
||||
category: weather
|
||||
utterance: "Moet ik vandaag een paraplu meenemen?"
|
||||
note: "Implizit, nl."
|
||||
|
||||
# --- residence_living_person (war ~4 Fälle) ---
|
||||
- id: becker_home_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: residence_living_person
|
||||
utterance: "Wo lebt Boris Becker inzwischen?"
|
||||
- id: madonna_home_en
|
||||
lang: en
|
||||
expect: search
|
||||
category: residence_living_person
|
||||
utterance: "Where does Madonna live these days?"
|
||||
- id: ctx_residence_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: residence_living_person
|
||||
history:
|
||||
- role: user
|
||||
content: "Kennst du den Moderator Thomas Gottschalk?"
|
||||
- role: assistant
|
||||
content: "Ja, ein bekannter deutscher Fernsehmoderator."
|
||||
utterance: "Wo wohnt der eigentlich inzwischen?"
|
||||
note: "Pronomen-Folgefrage (de) — Gegenstück zur nl-Pronomen-Lücke."
|
||||
|
||||
# --- alive_status (war 3 Fälle) — benannt vs. Pronomen, mehrsprachig ---
|
||||
- id: jagger_en
|
||||
lang: en
|
||||
expect: search
|
||||
category: alive_status
|
||||
utterance: "Is Mick Jagger still alive?"
|
||||
- id: alive_nl2
|
||||
lang: nl
|
||||
expect: search
|
||||
category: alive_status
|
||||
utterance: "Leeft Willeke Alberti nog?"
|
||||
note: "Benannt-nl — isoliert Sprache von Pronomen (vgl. ctx_alive_nl)."
|
||||
- id: ctx_alive_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: alive_status
|
||||
history:
|
||||
- role: user
|
||||
content: "Erzähl mir vom Sänger Udo Lindenberg."
|
||||
- role: assistant
|
||||
content: "Ein bekannter deutscher Rockmusiker mit Hut und Sonnenbrille."
|
||||
utterance: "Lebt der eigentlich noch?"
|
||||
note: "Pronomen-Folgefrage (de)."
|
||||
105
eval/tool_calling/client.py
Normal file
105
eval/tool_calling/client.py
Normal file
|
|
@ -0,0 +1,105 @@
|
|||
"""Minimaler OpenAI-kompatibler Tool-Calling-Client (OpenRouter).
|
||||
|
||||
Bewusst eigenständig gehalten: misst die Tool-Call-Fähigkeit des Modells und
|
||||
wird zugleich der Keim der echten Weg-2-Tool-Schicht. Gleicher Endpunkt/Auth
|
||||
wie app/providers/llm/openrouter.py, nur ohne dessen Streaming/Persona-Ballast.
|
||||
"""
|
||||
import asyncio
|
||||
import json
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
import httpx
|
||||
|
||||
ENDPOINT = "https://openrouter.ai/api/v1/chat/completions"
|
||||
_RETRY_STATUS = {429, 500, 502, 503}
|
||||
|
||||
|
||||
@dataclass
|
||||
class ToolCall:
|
||||
"""Ein einzelner Tool-Aufruf des Modells, inkl. Wohlgeformtheits-Urteil."""
|
||||
id: str
|
||||
name: str
|
||||
raw_arguments: str
|
||||
arguments: dict = field(default_factory=dict)
|
||||
well_formed: bool = False
|
||||
|
||||
@classmethod
|
||||
def parse(cls, tc: dict, expected_name: str = "web_search") -> "ToolCall":
|
||||
fn = tc.get("function", {})
|
||||
raw = fn.get("arguments", "") or ""
|
||||
try:
|
||||
args = json.loads(raw) if raw else {}
|
||||
except json.JSONDecodeError:
|
||||
args = {}
|
||||
well_formed = (
|
||||
fn.get("name") == expected_name
|
||||
and isinstance(args, dict)
|
||||
and bool(str(args.get("query", "")).strip())
|
||||
)
|
||||
return cls(id=tc.get("id", ""), name=fn.get("name", ""),
|
||||
raw_arguments=raw, arguments=args, well_formed=well_formed)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ModelTurn:
|
||||
tool_calls: list[ToolCall]
|
||||
text: str | None
|
||||
finish_reason: str
|
||||
raw_message: dict # Original-Assistant-Message zum Re-Threading in Pass 2
|
||||
|
||||
@property
|
||||
def searched(self) -> bool:
|
||||
return bool(self.tool_calls)
|
||||
|
||||
@property
|
||||
def query(self) -> str | None:
|
||||
return self.tool_calls[0].arguments.get("query") if self.tool_calls else None
|
||||
|
||||
|
||||
class ToolCallingClient:
|
||||
def __init__(self, api_key: str, model: str, temperature: float = 0.3,
|
||||
max_retries: int = 4):
|
||||
self.api_key = (api_key or "").strip()
|
||||
self.model = model.strip()
|
||||
self.temperature = temperature
|
||||
self.max_retries = max(1, max_retries)
|
||||
|
||||
async def respond(self, messages: list[dict], tools: list[dict],
|
||||
tool_choice: str = "auto") -> ModelTurn:
|
||||
payload = {
|
||||
"model": self.model,
|
||||
"messages": messages,
|
||||
"tools": tools,
|
||||
"tool_choice": tool_choice,
|
||||
"temperature": self.temperature,
|
||||
}
|
||||
timeout = httpx.Timeout(connect=10.0, read=120.0, write=30.0, pool=10.0)
|
||||
headers = {"Authorization": f"Bearer {self.api_key}",
|
||||
"Content-Type": "application/json"}
|
||||
last_error: Exception | None = None
|
||||
|
||||
for attempt in range(1, self.max_retries + 1):
|
||||
async with httpx.AsyncClient(timeout=timeout) as c:
|
||||
resp = await c.post(ENDPOINT, json=payload, headers=headers)
|
||||
if resp.status_code in _RETRY_STATUS and attempt < self.max_retries:
|
||||
last_error = RuntimeError(f"OpenRouter {resp.status_code}: {resp.text[:200]}")
|
||||
await asyncio.sleep(min(2.0 * attempt, 30.0))
|
||||
continue
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
# OpenRouter liefert transiente Provider-/Rate-Fehler teils als HTTP 200
|
||||
# mit {"error": ...} statt {"choices": ...} — als retrybar behandeln.
|
||||
if "choices" not in data:
|
||||
if attempt < self.max_retries:
|
||||
last_error = RuntimeError(f"Antwort ohne 'choices': {str(data)[:200]}")
|
||||
await asyncio.sleep(min(2.0 * attempt, 30.0))
|
||||
continue
|
||||
raise RuntimeError(f"Antwort ohne 'choices' nach Retries: {str(data)[:300]}")
|
||||
choice = data["choices"][0]
|
||||
msg = choice["message"]
|
||||
calls = [ToolCall.parse(tc) for tc in (msg.get("tool_calls") or [])]
|
||||
return ModelTurn(tool_calls=calls, text=msg.get("content"),
|
||||
finish_reason=choice.get("finish_reason", ""),
|
||||
raw_message=msg)
|
||||
|
||||
raise last_error or RuntimeError("OpenRouter: alle Versuche fehlgeschlagen")
|
||||
161
eval/tool_calling/run_eval.py
Normal file
161
eval/tool_calling/run_eval.py
Normal file
|
|
@ -0,0 +1,161 @@
|
|||
"""Runner für den Tool-Calling-Eval.
|
||||
|
||||
Lädt cases.yaml, fragt das Modell je Fall N× (Pass 1: Tool-Call-Entscheidung;
|
||||
Pass 2 nur bei Vorrang-Fällen: injiziertes Ergebnis, finale Antwort prüfen),
|
||||
und gibt einen Report aus, der die Weg-2-Tauglichkeit/Modellwahl entscheidet.
|
||||
|
||||
Echte OpenRouter-Calls — gehört zur `make smoke`-Familie, NICHT in `make test`.
|
||||
|
||||
Aufruf:
|
||||
OPENROUTER_API_KEY=... python eval/tool_calling/run_eval.py [MODELL] [LÄUFE]
|
||||
"""
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import statistics
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
from datetime import date
|
||||
|
||||
import yaml
|
||||
|
||||
from client import ToolCallingClient
|
||||
from tool import WEB_SEARCH_TOOL, build_messages
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
CASES_PATH = os.path.join(HERE, "cases.yaml")
|
||||
|
||||
# Wissens-Cutoff des ANTWORTENDEN Modells (nicht des Klassifikators) + heutiges Datum.
|
||||
CUTOFF = "fall 2024"
|
||||
TODAY = date.today().isoformat()
|
||||
|
||||
# Tauglichkeits-Schwellen (siehe Entscheidungsregel im Plan).
|
||||
THRESHOLDS = {"recall": 0.95, "precision": 0.80, "well_formed": 0.99, "priority": 0.95}
|
||||
|
||||
# Marker, die das verbotene (veraltete) Faktum als VERGANGENHEIT ausweisen.
|
||||
# Erwähnt das Modell den alten Wert nur historisch ("früher war ..."), ist das
|
||||
# kein Vorrang-Fehler — es folgt ja dem Tool-Ergebnis.
|
||||
_PAST_MARKERS = (
|
||||
"früher", "ehemalig", "vorher", "davor", "zuvor", "bis ", " war ", "wurde ",
|
||||
"previously", "former", "used to", "until ", "no longer", "past",
|
||||
)
|
||||
|
||||
|
||||
def priority_ok(answer: str, expect_fact: str, forbid_fact: str) -> bool:
|
||||
"""Folgt die Antwort dem Tool? Aktuelles Faktum da; veraltetes höchstens historisch."""
|
||||
low = answer.lower()
|
||||
if expect_fact.lower() not in low:
|
||||
return False
|
||||
if forbid_fact.lower() not in low:
|
||||
return True
|
||||
return any(m in low for m in _PAST_MARKERS)
|
||||
|
||||
|
||||
async def run_case(client: ToolCallingClient, case: dict, runs: int,
|
||||
sem: asyncio.Semaphore) -> dict:
|
||||
"""Führt einen Fall N× aus und sammelt die Roh-Ergebnisse je Lauf."""
|
||||
base = build_messages(case, CUTOFF, TODAY)
|
||||
laeufe = []
|
||||
for _ in range(runs):
|
||||
try:
|
||||
async with sem:
|
||||
t1 = await client.respond(base, [WEB_SEARCH_TOOL])
|
||||
except Exception as exc:
|
||||
# Ein einzelner fehlerhafter Lauf darf nicht den ganzen Batch killen.
|
||||
laeufe.append({"searched": None, "error": str(exc)[:200],
|
||||
"query": None, "well_formed": None, "prio_ok": None, "answer": None})
|
||||
continue
|
||||
rec = {
|
||||
"searched": t1.searched,
|
||||
"query": t1.query,
|
||||
"well_formed": all(tc.well_formed for tc in t1.tool_calls) if t1.searched else None,
|
||||
"prio_ok": None,
|
||||
"answer": None,
|
||||
}
|
||||
# Pass 2: nur bei Vorrang-Fällen und nur wenn überhaupt gesucht wurde.
|
||||
if case.get("injected_result") and t1.searched:
|
||||
tc = t1.tool_calls[0]
|
||||
msgs = base + [t1.raw_message, {
|
||||
"role": "tool",
|
||||
"tool_call_id": tc.id,
|
||||
"content": case["injected_result"],
|
||||
}]
|
||||
async with sem:
|
||||
t2 = await client.respond(msgs, [WEB_SEARCH_TOOL], tool_choice="none")
|
||||
ans = (t2.text or "")
|
||||
rec["answer"] = ans
|
||||
rec["prio_ok"] = priority_ok(ans, case["expect_fact"], case["forbid_fact"])
|
||||
laeufe.append(rec)
|
||||
return {"case": case, "runs": laeufe}
|
||||
|
||||
|
||||
async def main(model: str, runs: int, conc: int = 4) -> None:
|
||||
cases = yaml.safe_load(open(CASES_PATH, encoding="utf-8"))
|
||||
key = os.environ.get("OPENROUTER_API_KEY")
|
||||
if not key:
|
||||
sys.exit("OPENROUTER_API_KEY fehlt in der Umgebung.")
|
||||
client = ToolCallingClient(key, model)
|
||||
sem = asyncio.Semaphore(conc)
|
||||
results = await asyncio.gather(*(run_case(client, c, runs, sem) for c in cases))
|
||||
report(model, runs, results)
|
||||
dump_path = os.path.join(HERE, f"results_{model.replace('/', '_')}.json")
|
||||
with open(dump_path, "w", encoding="utf-8") as f:
|
||||
json.dump(results, f, ensure_ascii=False, indent=2)
|
||||
print(f"\nRoh-Ergebnisse: {dump_path}")
|
||||
|
||||
|
||||
def report(model: str, runs: int, results: list[dict]) -> None:
|
||||
recall, precision, well_formed, priority = [], [], [], []
|
||||
per_cat: dict[str, list[float]] = defaultdict(list)
|
||||
offenders: list[tuple[str, str, float]] = []
|
||||
|
||||
errored = 0
|
||||
for res in results:
|
||||
case, laeufe = res["case"], res["runs"]
|
||||
want_search = case["expect"] == "search"
|
||||
valid = [r for r in laeufe if r["searched"] is not None]
|
||||
errored += len(laeufe) - len(valid)
|
||||
if not valid:
|
||||
continue # ganzer Fall fehlerhaft -> nicht werten
|
||||
searched_rate = statistics.mean(r["searched"] for r in valid)
|
||||
correct_rate = searched_rate if want_search else 1.0 - searched_rate
|
||||
(recall if want_search else precision).append(correct_rate)
|
||||
per_cat[case["category"]].append(correct_rate)
|
||||
|
||||
wfs = [r["well_formed"] for r in laeufe if r["well_formed"] is not None]
|
||||
if wfs:
|
||||
well_formed.append(statistics.mean(wfs))
|
||||
prios = [r["prio_ok"] for r in laeufe if r["prio_ok"] is not None]
|
||||
if prios:
|
||||
priority.append(statistics.mean(prios))
|
||||
if correct_rate < 0.8:
|
||||
offenders.append((case["id"], case["category"], round(correct_rate, 2)))
|
||||
|
||||
def pct(xs: list[float]) -> str:
|
||||
return f"{100 * statistics.mean(xs):.0f}%" if xs else "n/a"
|
||||
|
||||
def flag(xs: list[float], key: str) -> str:
|
||||
if not xs:
|
||||
return ""
|
||||
return " OK" if statistics.mean(xs) >= THRESHOLDS[key] else " ⚠"
|
||||
|
||||
print(f"\n=== {model} ({runs} Läufe/Fall, heute={TODAY}) ===")
|
||||
if errored:
|
||||
print(f"(Hinweis: {errored} Einzelläufe mit Fehler übersprungen)")
|
||||
print(f"Trigger-Recall (muss-suchen) {pct(recall):>5}{flag(recall, 'recall')} (Ziel ≥95%)")
|
||||
print(f"Trigger-Precision (nicht-suchen) {pct(precision):>5}{flag(precision, 'precision')} (Ziel ≥80%)")
|
||||
print(f"Wohlgeformtheit {pct(well_formed):>5}{flag(well_formed, 'well_formed')} (Ziel ≥99%)")
|
||||
print(f"Vorrang-Treue {pct(priority):>5}{flag(priority, 'priority')} (Ziel ≥95%)")
|
||||
print("\npro Kategorie:")
|
||||
for cat, xs in sorted(per_cat.items()):
|
||||
print(f" {cat:26} {pct(xs)}")
|
||||
if offenders:
|
||||
print("\nSchwächste Fälle (<80%):")
|
||||
for oid, cat, rate in sorted(offenders, key=lambda x: x[2]):
|
||||
print(f" {oid:24} {cat:26} {rate}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
arg_model = sys.argv[1] if len(sys.argv) > 1 else "deepseek/deepseek-v4-flash"
|
||||
arg_runs = int(sys.argv[2]) if len(sys.argv) > 2 else 5
|
||||
asyncio.run(main(arg_model, arg_runs))
|
||||
58
eval/tool_calling/tool.py
Normal file
58
eval/tool_calling/tool.py
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
"""web_search-Tool-Definition + Trigger-Prompt.
|
||||
|
||||
Der Beschreibungs-/Prompt-Text ist derselbe kalibrierte „braucht das Wissen
|
||||
neuer als dein Cutoff"-Text, der später produktiv die Trigger-Leitlinie bildet.
|
||||
Der Harness misst damit Modell UND Prompt gemeinsam.
|
||||
"""
|
||||
|
||||
WEB_SEARCH_TOOL = {
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "web_search",
|
||||
"description": (
|
||||
"Look up real-world information that may be newer than your "
|
||||
"knowledge or may have changed since your last update. Call it "
|
||||
"whenever the true answer could plausibly have changed, INCLUDING "
|
||||
"when no exact place or date is named. This covers: who currently "
|
||||
"holds an office; where a living person now lives; whether someone "
|
||||
"is still alive; current prices, rates or crypto; current weather "
|
||||
"or outdoor conditions (even phrased as 'is it cold/raining right "
|
||||
"now', using the user's location); sports results and standings; "
|
||||
"the latest version or model of a product; recent news; and "
|
||||
"time-sensitive logistics such as opening hours, schedules, and "
|
||||
"public-transport or train/bus departure times. Do NOT use it for "
|
||||
"timeless knowledge, opinions, jokes, small talk, the current "
|
||||
"clock time or today's date (you already have those), anything "
|
||||
"you can derive yourself, or anything about the user themselves."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"query": {
|
||||
"type": "string",
|
||||
"description": "Concise search query, in the user's language or English.",
|
||||
},
|
||||
},
|
||||
"required": ["query"],
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
SYSTEM_PROMPT = (
|
||||
"You are a voice assistant. Your knowledge was last updated around "
|
||||
"{cutoff}. Today is {today}. When a question could require information "
|
||||
"newer than {cutoff} that you cannot reason out yourself, call web_search "
|
||||
"before answering. Never call web_search merely to find the current time "
|
||||
"or today's date — you already have them. Tool results are authoritative "
|
||||
"and newer than your memory: if a tool result conflicts with what you "
|
||||
"believe, follow the tool and never contradict it."
|
||||
)
|
||||
|
||||
|
||||
def build_messages(case: dict, cutoff: str, today: str) -> list[dict]:
|
||||
"""Baut die Message-Liste: System-Prompt + (optionale) History + Nutzeräußerung."""
|
||||
msgs = [{"role": "system",
|
||||
"content": SYSTEM_PROMPT.format(cutoff=cutoff, today=today)}]
|
||||
msgs += case.get("history", [])
|
||||
msgs.append({"role": "user", "content": case["utterance"]})
|
||||
return msgs
|
||||
|
|
@ -24,7 +24,10 @@ def test_single_system_message_when_history_has_system():
|
|||
assert msgs[-1] == {"role": "user", "content": "Wie geht es dir?"}
|
||||
|
||||
|
||||
def test_no_system_message_without_prompt_or_history():
|
||||
def test_clock_system_message_without_prompt_or_history():
|
||||
# Datum/Uhrzeit-Kontext wird immer als (einzige) fuehrende System-Nachricht gesetzt.
|
||||
llm = _llm(system_prompt="")
|
||||
msgs = llm._build_messages("Hallo", None)
|
||||
assert msgs == [{"role": "user", "content": "Hallo"}]
|
||||
assert msgs[0]["role"] == "system"
|
||||
assert "current date and time" in msgs[0]["content"].lower()
|
||||
assert msgs[1:] == [{"role": "user", "content": "Hallo"}]
|
||||
|
|
|
|||
53
tests/test_search_backstop.py
Normal file
53
tests/test_search_backstop.py
Normal file
|
|
@ -0,0 +1,53 @@
|
|||
"""Offline-Tests für den deterministischen Such-Backstop (heikle Kategorien)."""
|
||||
import pytest
|
||||
|
||||
from app.pipeline.search_backstop import should_force_search
|
||||
|
||||
FORCE = [
|
||||
"Wer ist Bundeskanzler?",
|
||||
"Wer ist aktuell Bundeskanzler?",
|
||||
"Who is the president of the United States?",
|
||||
"Wie heißt der Papst?",
|
||||
"Wie is de huidige minister-president van Nederland?",
|
||||
"Wer ist gerade Papst?",
|
||||
"Wo wohnt Heino?",
|
||||
"Wo lebt Heino heute?",
|
||||
"Where does Madonna live these days?",
|
||||
"Waar woont André Rieu tegenwoordig?",
|
||||
"Lebt Heino noch?",
|
||||
"Is Brigitte Bardot still alive?",
|
||||
"Leeft Willeke Alberti nog?",
|
||||
# schedule_hours / Logistik
|
||||
"Hat die Arztpraxis heute Nachmittag noch geöffnet?",
|
||||
"Welche Apotheke hat heute Nacht Notdienst?",
|
||||
"What are the opening hours of the British Museum?",
|
||||
"Wann fährt der nächste Bus ins Zentrum?",
|
||||
"Wanneer vertrekt de volgende trein naar Utrecht?",
|
||||
"Wann geht der nächste Zug nach München?",
|
||||
]
|
||||
|
||||
NO_FORCE = [
|
||||
"Was ist die Hauptstadt von Frankreich?",
|
||||
"Erzähl mir einen Witz.",
|
||||
"Wie alt ist Heino?",
|
||||
"Wer war der erste Bundeskanzler?", # historisch (Vergangenheit)
|
||||
"Wann hat Heino Geburtstag?",
|
||||
"Wie geht es dir heute?",
|
||||
"Kannst du mir ein Rezept für Kartoffelsuppe geben?",
|
||||
"Wer hat den Faust geschrieben?",
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("text", FORCE)
|
||||
def test_forces_on_heikle_kategorien(text):
|
||||
assert should_force_search(text) is True, text
|
||||
|
||||
|
||||
@pytest.mark.parametrize("text", NO_FORCE)
|
||||
def test_no_force_on_harmless(text):
|
||||
assert should_force_search(text) is False, text
|
||||
|
||||
|
||||
def test_empty_and_none():
|
||||
assert should_force_search(None) is False
|
||||
assert should_force_search("") is False
|
||||
Loading…
Add table
Add a link
Reference in a new issue