feat(llm): Web-Suche per Tool-Calling (Weg 2) — Sonar + ToolCallingLLM
Frische-/Web-Such-Funktion: Das zentrale Modell entscheidet selbst via web_search-Tool, ob es tagesaktuelle Fakten braucht, holt sie über perplexity/sonar und formuliert die Antwort in Persona (Augment). - SonarTool (app/tools/web_search.py): Fakten via perplexity/sonar, Citations als Metadaten, honest-punt-Sentinel bei Fehler/Timeout. - ToolCallingLLM (app/providers/llm/tool_calling.py): agentischer Loop als LLMProvider; complete() + gestreamtes stream() mit SSE-Tool-Assembler; Persona- + Trigger- + Vorrang-Prompt (Tool-Ergebnis schlaegt Gedaechtnis). - Verdrahtung: Registry-Eintrag openrouter-tools; web_search_enabled (global an, pro Nutzer/Profil abschaltbar) via Route-Layering; build_orchestrator waehlt tool-faehig vs. plain, Fallback-Kette erhalten. - Filler: ephemerer Beruhigungssatz beim Tool-Start (sofort angezeigt UND gesprochen als Satz null), nie in semantic_response/History; on_tool_start defensiv durch die stream()-Kette gefaedelt (kein Bruch bestehender Provider). - Modellwechsel: Standard auf mistralai/mistral-small-3.2-24b-instruct (tool-faehig; im Eval einziger Recall-Gate-Passer). 2501 ist tool-unfaehig. - Eval-Harness (eval/tool_calling/): Datensatz + Runner zur Modellauswahl. Doc: Docs/weg2-tool-calling.md. Tests: 290 gruen. Offen (Schritt 5): Koreferenz-Vorstufe (nl-Pronomen) + Metrik-Zaehler. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
19b3998948
commit
703a695bed
19 changed files with 1470 additions and 9 deletions
492
eval/tool_calling/cases.yaml
Normal file
492
eval/tool_calling/cases.yaml
Normal file
|
|
@ -0,0 +1,492 @@
|
|||
# Eval-Datensatz: Tool-Calling-Trigger für die Frische-/Web-Such-Funktion.
|
||||
#
|
||||
# Felder: id, utterance, lang, expect (search|no_search), category,
|
||||
# optional: history, injected_result, expect_fact, forbid_fact, note.
|
||||
#
|
||||
# Vorrang-Fälle (mit injected_result) sind eine Teilmenge der search-Fälle und
|
||||
# werden doppelt genutzt: Pass 1 misst Trigger-Recall, Pass 2 die Vorrang-Treue.
|
||||
|
||||
# ============================================================
|
||||
# A) MUSS SUCHEN (expect: search)
|
||||
# ============================================================
|
||||
|
||||
# --- officeholder: aktuelle Amtsträger (KEIN Zeit-Stichwort!) ---
|
||||
- id: chancellor_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: officeholder
|
||||
utterance: "Wer ist Bundeskanzler?"
|
||||
- id: uspres_en
|
||||
lang: en
|
||||
expect: search
|
||||
category: officeholder
|
||||
utterance: "Who is the president of the United States?"
|
||||
- id: pm_nl
|
||||
lang: nl
|
||||
expect: search
|
||||
category: officeholder
|
||||
utterance: "Wie is de minister-president van Nederland?"
|
||||
- id: pope_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: officeholder
|
||||
utterance: "Wer ist gerade Papst?"
|
||||
|
||||
# --- residence_living_person: Wohnort lebender Person (DER Trap-Fall) ---
|
||||
- id: heino_home_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: residence_living_person
|
||||
utterance: "Wo wohnt Heino?"
|
||||
note: "Fühlt sich statisch an, ist es aber nicht (Umzug Kitzbühel)."
|
||||
- id: player_club_en
|
||||
lang: en
|
||||
expect: search
|
||||
category: residence_living_person
|
||||
utterance: "Which club does Robert Lewandowski play for now?"
|
||||
- id: singer_home_nl
|
||||
lang: nl
|
||||
expect: search
|
||||
category: residence_living_person
|
||||
utterance: "Waar woont André Rieu tegenwoordig?"
|
||||
|
||||
# --- alive_status: lebt/aktiv? ---
|
||||
- id: alive_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: alive_status
|
||||
utterance: "Lebt Heino noch?"
|
||||
- id: alive_en
|
||||
lang: en
|
||||
expect: search
|
||||
category: alive_status
|
||||
utterance: "Is Brigitte Bardot still alive?"
|
||||
|
||||
# --- weather: Echtzeit ---
|
||||
- id: weather_paris_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: weather
|
||||
utterance: "Wie ist das Wetter in Paris?"
|
||||
- id: weather_rain_nl
|
||||
lang: nl
|
||||
expect: search
|
||||
category: weather
|
||||
utterance: "Gaat het vandaag regenen in Amsterdam?"
|
||||
- id: weather_cold_en
|
||||
lang: en
|
||||
expect: search
|
||||
category: weather
|
||||
utterance: "Is it cold outside right now?"
|
||||
note: "Braucht Ort aus Nutzerprofil — Trigger muss trotzdem feuern."
|
||||
|
||||
# --- price: Kurse/Preise ---
|
||||
- id: stock_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: price
|
||||
utterance: "Was kostet die Apple-Aktie?"
|
||||
- id: gold_nl
|
||||
lang: nl
|
||||
expect: search
|
||||
category: price
|
||||
utterance: "Wat is de goudprijs op dit moment?"
|
||||
- id: fuel_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: price
|
||||
utterance: "Was kostet gerade ein Liter Diesel?"
|
||||
|
||||
# --- latest_release: neueste Version/Produkt ---
|
||||
- id: iphone_en
|
||||
lang: en
|
||||
expect: search
|
||||
category: latest_release
|
||||
utterance: "What is the newest iPhone?"
|
||||
- id: release_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: latest_release
|
||||
utterance: "Welches ist das aktuellste Samsung-Galaxy-Handy?"
|
||||
|
||||
# --- sports_result: Ergebnisse/Tabellen ---
|
||||
- id: match_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: sports_result
|
||||
utterance: "Wie hat Bayern München am Wochenende gespielt?"
|
||||
- id: standings_en
|
||||
lang: en
|
||||
expect: search
|
||||
category: sports_result
|
||||
utterance: "Who is leading the Premier League?"
|
||||
|
||||
# --- news_event: jüngere Ereignisse ---
|
||||
- id: news_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: news_event
|
||||
utterance: "Was ist gerade die wichtigste Nachricht aus Berlin?"
|
||||
- id: news_fr
|
||||
lang: fr
|
||||
expect: search
|
||||
category: news_event
|
||||
utterance: "Que s'est-il passé récemment en France ?"
|
||||
|
||||
# --- schedule_hours: volatile Öffnungs-/Fahrzeiten ---
|
||||
- id: hours_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: schedule_hours
|
||||
utterance: "Hat die Apotheke am Markt heute geöffnet?"
|
||||
- id: train_nl
|
||||
lang: nl
|
||||
expect: search
|
||||
category: schedule_hours
|
||||
utterance: "Hoe laat gaat de volgende trein naar Utrecht?"
|
||||
|
||||
# ============================================================
|
||||
# B) MUSS NICHT SUCHEN (expect: no_search)
|
||||
# ============================================================
|
||||
|
||||
# --- timeless: zeitloses Wissen ---
|
||||
- id: capital_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: timeless
|
||||
utterance: "Was ist die Hauptstadt von Frankreich?"
|
||||
- id: faust_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: timeless
|
||||
utterance: "Wer hat den Faust geschrieben?"
|
||||
- id: boil_en
|
||||
lang: en
|
||||
expect: no_search
|
||||
category: timeless
|
||||
utterance: "At what temperature does water boil?"
|
||||
- id: ww2_nl
|
||||
lang: nl
|
||||
expect: no_search
|
||||
category: timeless
|
||||
utterance: "Wie won de Tweede Wereldoorlog?"
|
||||
- id: math_es
|
||||
lang: es
|
||||
expect: no_search
|
||||
category: timeless
|
||||
utterance: "¿Cuánto es ciento veinte dividido entre cuatro?"
|
||||
|
||||
# --- derivable: ableitbar, kein Web nötig ---
|
||||
- id: heino_age_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: derivable
|
||||
utterance: "Wie alt ist Heino?"
|
||||
note: "Aus Geburtsjahr + heutigem Datum ableitbar."
|
||||
- id: weekday_en
|
||||
lang: en
|
||||
expect: no_search
|
||||
category: derivable
|
||||
utterance: "What day of the week is Christmas this year?"
|
||||
|
||||
# --- static_fact: feststehende Personendaten ---
|
||||
- id: heino_birthday_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: static_fact
|
||||
utterance: "Wann hat Heino Geburtstag?"
|
||||
|
||||
# --- opinion / Vorschlag ---
|
||||
- id: recipe_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: opinion
|
||||
utterance: "Kannst du mir ein einfaches Rezept für Kartoffelsuppe sagen?"
|
||||
- id: advice_nl
|
||||
lang: nl
|
||||
expect: no_search
|
||||
category: opinion
|
||||
utterance: "Welke bloemen kan ik het beste in de schaduw planten?"
|
||||
|
||||
# --- smalltalk / sozial ---
|
||||
- id: howareyou_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: smalltalk
|
||||
utterance: "Wie geht es dir heute?"
|
||||
note: "Keyword-Köder 'heute' — darf NICHT triggern."
|
||||
- id: chat_en
|
||||
lang: en
|
||||
expect: no_search
|
||||
category: smalltalk
|
||||
utterance: "I'm a bit bored, let's talk for a while."
|
||||
|
||||
# --- emotional / Unterstützung ---
|
||||
- id: lonely_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: emotional
|
||||
utterance: "Ich fühle mich heute sehr einsam."
|
||||
- id: worry_nl
|
||||
lang: nl
|
||||
expect: no_search
|
||||
category: emotional
|
||||
utterance: "Ik maak me zorgen en kan niet slapen."
|
||||
|
||||
# --- joke / Unterhaltung ---
|
||||
- id: joke_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: joke
|
||||
utterance: "Erzähl mir bitte einen Witz."
|
||||
|
||||
# --- procedural_timeless ---
|
||||
- id: howto_en
|
||||
lang: en
|
||||
expect: no_search
|
||||
category: procedural_timeless
|
||||
utterance: "How do I cook a soft-boiled egg?"
|
||||
|
||||
# --- personal_memory: betrifft den Nutzer selbst (Store, nicht Web) ---
|
||||
- id: mymed_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: personal_memory
|
||||
utterance: "Welche Medikamente nehme ich morgens?"
|
||||
note: "Aus Nutzerprofil/Erinnerungen, nicht aus dem Web."
|
||||
- id: myname_nl
|
||||
lang: nl
|
||||
expect: no_search
|
||||
category: personal_memory
|
||||
utterance: "Hoe heet mijn dochter ook alweer?"
|
||||
|
||||
# ============================================================
|
||||
# C) ADVERSARIAL (Köder & Grenzfälle — höchster Testwert)
|
||||
# ============================================================
|
||||
|
||||
# --- "aktuell"-Köder, aber persönlich ---
|
||||
- id: trap_fav_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: personal_memory
|
||||
utterance: "Was ist gerade mein Lieblingslied?"
|
||||
note: "'gerade' ist Köder; Antwort aus Erinnerungen."
|
||||
|
||||
# --- aktuell, ABER nicht aus dem Web (Systemfunktion) ---
|
||||
- id: time_now_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: system_clock
|
||||
utterance: "Wie spät ist es?"
|
||||
note: "Echtzeit, aber Systemuhr — kein Web. Grenzfall bewusst no_search."
|
||||
- id: date_today_en
|
||||
lang: en
|
||||
expect: no_search
|
||||
category: system_clock
|
||||
utterance: "What's today's date?"
|
||||
|
||||
# --- "könnte sich theoretisch ändern", praktisch nie ---
|
||||
- id: trap_capital_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: timeless
|
||||
utterance: "Ist Berlin immer noch die Hauptstadt Deutschlands?"
|
||||
note: "Suggeriert Volatilität, ist aber stabil — darf nicht übertriggern."
|
||||
|
||||
# ============================================================
|
||||
# D) FOLGEFRAGEN MIT HISTORY (Kontext-Trigger)
|
||||
# ============================================================
|
||||
|
||||
- id: ctx_heino_home_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: residence_living_person
|
||||
history:
|
||||
- role: user
|
||||
content: "Erzähl mir etwas über den Sänger Heino."
|
||||
- role: assistant
|
||||
content: "Heino ist ein deutscher Schlagersänger, bekannt für seine dunkle Brille."
|
||||
utterance: "Und wo wohnt der eigentlich?"
|
||||
note: "Referent nur aus History — Trigger muss trotzdem feuern."
|
||||
|
||||
- id: ctx_alive_nl
|
||||
lang: nl
|
||||
expect: search
|
||||
category: alive_status
|
||||
history:
|
||||
- role: user
|
||||
content: "Ken je de acteur Rutger Hauer?"
|
||||
- role: assistant
|
||||
content: "Ja, een bekende Nederlandse acteur."
|
||||
utterance: "Leeft hij eigenlijk nog?"
|
||||
|
||||
- id: ctx_followup_static_de
|
||||
lang: de
|
||||
expect: no_search
|
||||
category: static_fact
|
||||
history:
|
||||
- role: user
|
||||
content: "Wer hat den Faust geschrieben?"
|
||||
- role: assistant
|
||||
content: "Johann Wolfgang von Goethe."
|
||||
utterance: "Und wann wurde der geboren?"
|
||||
note: "Folgefrage, aber zeitlos — darf nicht triggern."
|
||||
|
||||
# ============================================================
|
||||
# E) VORRANG-TESTS (injiziertes Faktum widerspricht dem Training)
|
||||
# ============================================================
|
||||
|
||||
- id: prio_chancellor_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: officeholder
|
||||
utterance: "Wer ist Bundeskanzler?"
|
||||
injected_result: "Bundeskanzler ist Friedrich Merz (CDU), im Amt seit Mai 2025."
|
||||
expect_fact: "Merz"
|
||||
forbid_fact: "Scholz"
|
||||
|
||||
- id: prio_pope_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: officeholder
|
||||
utterance: "Wer ist gerade Papst?"
|
||||
injected_result: "Papst ist Leo XIV., gewählt im Mai 2025."
|
||||
expect_fact: "Leo"
|
||||
forbid_fact: "Franziskus"
|
||||
note: "Sehr jung (nach Cutoff) — harter Vorrang-Test."
|
||||
|
||||
- id: prio_uspres_en
|
||||
lang: en
|
||||
expect: search
|
||||
category: officeholder
|
||||
utterance: "Who is the US president?"
|
||||
injected_result: "Donald Trump is the US president, inaugurated January 2025."
|
||||
expect_fact: "Trump"
|
||||
forbid_fact: "Biden"
|
||||
|
||||
- id: prio_heino_home_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: residence_living_person
|
||||
utterance: "Wo wohnt Heino?"
|
||||
injected_result: "Heino lebt inzwischen dauerhaft in Kitzbühel, Österreich."
|
||||
expect_fact: "Kitzbühel"
|
||||
forbid_fact: "Münstereifel"
|
||||
|
||||
- id: prio_iphone_en
|
||||
lang: en
|
||||
expect: search
|
||||
category: latest_release
|
||||
utterance: "What is the newest iPhone?"
|
||||
injected_result: "The newest model is the iPhone 17, released September 2025."
|
||||
expect_fact: "17"
|
||||
forbid_fact: "16"
|
||||
|
||||
# ============================================================
|
||||
# F) AUFSTOCKUNG schwacher Kategorien (für belastbare Kategorie-Raten)
|
||||
# Bewusst variantenreich: andere Entitäten/Sprachen, explizit + implizit.
|
||||
# ============================================================
|
||||
|
||||
# --- schedule_hours (war nur 2 Fälle) ---
|
||||
- id: bakery_hours_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: schedule_hours
|
||||
utterance: "Wann macht die Bäckerei morgen früh auf?"
|
||||
- id: bus_nl
|
||||
lang: nl
|
||||
expect: search
|
||||
category: schedule_hours
|
||||
utterance: "Wanneer vertrekt de volgende bus naar het centrum?"
|
||||
- id: museum_hours_en
|
||||
lang: en
|
||||
expect: search
|
||||
category: schedule_hours
|
||||
utterance: "What are the opening hours of the British Museum today?"
|
||||
- id: doctor_hours_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: schedule_hours
|
||||
utterance: "Hat die Arztpraxis heute Nachmittag noch geöffnet?"
|
||||
- id: pharmacy_night_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: schedule_hours
|
||||
utterance: "Welche Apotheke hat heute Nacht Notdienst?"
|
||||
|
||||
# --- weather (war 3 Fälle) — Fokus implizit/ortlos/ohne Wort 'Wetter' ---
|
||||
- id: forecast_munich_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: weather
|
||||
utterance: "Wie warm wird es morgen in München?"
|
||||
- id: weather_london_en
|
||||
lang: en
|
||||
expect: search
|
||||
category: weather
|
||||
utterance: "What's the weather going to be like in London tomorrow?"
|
||||
- id: umbrella_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: weather
|
||||
utterance: "Soll ich heute einen Regenschirm mitnehmen?"
|
||||
note: "Impliziter Ort, kein Wort 'Wetter' — der harte Fall."
|
||||
- id: jacket_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: weather
|
||||
utterance: "Brauche ich heute draußen eine Jacke?"
|
||||
note: "Impliziter Ort, kein Wort 'Wetter'."
|
||||
- id: umbrella_nl
|
||||
lang: nl
|
||||
expect: search
|
||||
category: weather
|
||||
utterance: "Moet ik vandaag een paraplu meenemen?"
|
||||
note: "Implizit, nl."
|
||||
|
||||
# --- residence_living_person (war ~4 Fälle) ---
|
||||
- id: becker_home_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: residence_living_person
|
||||
utterance: "Wo lebt Boris Becker inzwischen?"
|
||||
- id: madonna_home_en
|
||||
lang: en
|
||||
expect: search
|
||||
category: residence_living_person
|
||||
utterance: "Where does Madonna live these days?"
|
||||
- id: ctx_residence_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: residence_living_person
|
||||
history:
|
||||
- role: user
|
||||
content: "Kennst du den Moderator Thomas Gottschalk?"
|
||||
- role: assistant
|
||||
content: "Ja, ein bekannter deutscher Fernsehmoderator."
|
||||
utterance: "Wo wohnt der eigentlich inzwischen?"
|
||||
note: "Pronomen-Folgefrage (de) — Gegenstück zur nl-Pronomen-Lücke."
|
||||
|
||||
# --- alive_status (war 3 Fälle) — benannt vs. Pronomen, mehrsprachig ---
|
||||
- id: jagger_en
|
||||
lang: en
|
||||
expect: search
|
||||
category: alive_status
|
||||
utterance: "Is Mick Jagger still alive?"
|
||||
- id: alive_nl2
|
||||
lang: nl
|
||||
expect: search
|
||||
category: alive_status
|
||||
utterance: "Leeft Willeke Alberti nog?"
|
||||
note: "Benannt-nl — isoliert Sprache von Pronomen (vgl. ctx_alive_nl)."
|
||||
- id: ctx_alive_de
|
||||
lang: de
|
||||
expect: search
|
||||
category: alive_status
|
||||
history:
|
||||
- role: user
|
||||
content: "Erzähl mir vom Sänger Udo Lindenberg."
|
||||
- role: assistant
|
||||
content: "Ein bekannter deutscher Rockmusiker mit Hut und Sonnenbrille."
|
||||
utterance: "Lebt der eigentlich noch?"
|
||||
note: "Pronomen-Folgefrage (de)."
|
||||
105
eval/tool_calling/client.py
Normal file
105
eval/tool_calling/client.py
Normal file
|
|
@ -0,0 +1,105 @@
|
|||
"""Minimaler OpenAI-kompatibler Tool-Calling-Client (OpenRouter).
|
||||
|
||||
Bewusst eigenständig gehalten: misst die Tool-Call-Fähigkeit des Modells und
|
||||
wird zugleich der Keim der echten Weg-2-Tool-Schicht. Gleicher Endpunkt/Auth
|
||||
wie app/providers/llm/openrouter.py, nur ohne dessen Streaming/Persona-Ballast.
|
||||
"""
|
||||
import asyncio
|
||||
import json
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
import httpx
|
||||
|
||||
ENDPOINT = "https://openrouter.ai/api/v1/chat/completions"
|
||||
_RETRY_STATUS = {429, 500, 502, 503}
|
||||
|
||||
|
||||
@dataclass
|
||||
class ToolCall:
|
||||
"""Ein einzelner Tool-Aufruf des Modells, inkl. Wohlgeformtheits-Urteil."""
|
||||
id: str
|
||||
name: str
|
||||
raw_arguments: str
|
||||
arguments: dict = field(default_factory=dict)
|
||||
well_formed: bool = False
|
||||
|
||||
@classmethod
|
||||
def parse(cls, tc: dict, expected_name: str = "web_search") -> "ToolCall":
|
||||
fn = tc.get("function", {})
|
||||
raw = fn.get("arguments", "") or ""
|
||||
try:
|
||||
args = json.loads(raw) if raw else {}
|
||||
except json.JSONDecodeError:
|
||||
args = {}
|
||||
well_formed = (
|
||||
fn.get("name") == expected_name
|
||||
and isinstance(args, dict)
|
||||
and bool(str(args.get("query", "")).strip())
|
||||
)
|
||||
return cls(id=tc.get("id", ""), name=fn.get("name", ""),
|
||||
raw_arguments=raw, arguments=args, well_formed=well_formed)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ModelTurn:
|
||||
tool_calls: list[ToolCall]
|
||||
text: str | None
|
||||
finish_reason: str
|
||||
raw_message: dict # Original-Assistant-Message zum Re-Threading in Pass 2
|
||||
|
||||
@property
|
||||
def searched(self) -> bool:
|
||||
return bool(self.tool_calls)
|
||||
|
||||
@property
|
||||
def query(self) -> str | None:
|
||||
return self.tool_calls[0].arguments.get("query") if self.tool_calls else None
|
||||
|
||||
|
||||
class ToolCallingClient:
|
||||
def __init__(self, api_key: str, model: str, temperature: float = 0.3,
|
||||
max_retries: int = 4):
|
||||
self.api_key = (api_key or "").strip()
|
||||
self.model = model.strip()
|
||||
self.temperature = temperature
|
||||
self.max_retries = max(1, max_retries)
|
||||
|
||||
async def respond(self, messages: list[dict], tools: list[dict],
|
||||
tool_choice: str = "auto") -> ModelTurn:
|
||||
payload = {
|
||||
"model": self.model,
|
||||
"messages": messages,
|
||||
"tools": tools,
|
||||
"tool_choice": tool_choice,
|
||||
"temperature": self.temperature,
|
||||
}
|
||||
timeout = httpx.Timeout(connect=10.0, read=120.0, write=30.0, pool=10.0)
|
||||
headers = {"Authorization": f"Bearer {self.api_key}",
|
||||
"Content-Type": "application/json"}
|
||||
last_error: Exception | None = None
|
||||
|
||||
for attempt in range(1, self.max_retries + 1):
|
||||
async with httpx.AsyncClient(timeout=timeout) as c:
|
||||
resp = await c.post(ENDPOINT, json=payload, headers=headers)
|
||||
if resp.status_code in _RETRY_STATUS and attempt < self.max_retries:
|
||||
last_error = RuntimeError(f"OpenRouter {resp.status_code}: {resp.text[:200]}")
|
||||
await asyncio.sleep(min(2.0 * attempt, 30.0))
|
||||
continue
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
# OpenRouter liefert transiente Provider-/Rate-Fehler teils als HTTP 200
|
||||
# mit {"error": ...} statt {"choices": ...} — als retrybar behandeln.
|
||||
if "choices" not in data:
|
||||
if attempt < self.max_retries:
|
||||
last_error = RuntimeError(f"Antwort ohne 'choices': {str(data)[:200]}")
|
||||
await asyncio.sleep(min(2.0 * attempt, 30.0))
|
||||
continue
|
||||
raise RuntimeError(f"Antwort ohne 'choices' nach Retries: {str(data)[:300]}")
|
||||
choice = data["choices"][0]
|
||||
msg = choice["message"]
|
||||
calls = [ToolCall.parse(tc) for tc in (msg.get("tool_calls") or [])]
|
||||
return ModelTurn(tool_calls=calls, text=msg.get("content"),
|
||||
finish_reason=choice.get("finish_reason", ""),
|
||||
raw_message=msg)
|
||||
|
||||
raise last_error or RuntimeError("OpenRouter: alle Versuche fehlgeschlagen")
|
||||
161
eval/tool_calling/run_eval.py
Normal file
161
eval/tool_calling/run_eval.py
Normal file
|
|
@ -0,0 +1,161 @@
|
|||
"""Runner für den Tool-Calling-Eval.
|
||||
|
||||
Lädt cases.yaml, fragt das Modell je Fall N× (Pass 1: Tool-Call-Entscheidung;
|
||||
Pass 2 nur bei Vorrang-Fällen: injiziertes Ergebnis, finale Antwort prüfen),
|
||||
und gibt einen Report aus, der die Weg-2-Tauglichkeit/Modellwahl entscheidet.
|
||||
|
||||
Echte OpenRouter-Calls — gehört zur `make smoke`-Familie, NICHT in `make test`.
|
||||
|
||||
Aufruf:
|
||||
OPENROUTER_API_KEY=... python eval/tool_calling/run_eval.py [MODELL] [LÄUFE]
|
||||
"""
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import statistics
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
from datetime import date
|
||||
|
||||
import yaml
|
||||
|
||||
from client import ToolCallingClient
|
||||
from tool import WEB_SEARCH_TOOL, build_messages
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
CASES_PATH = os.path.join(HERE, "cases.yaml")
|
||||
|
||||
# Wissens-Cutoff des ANTWORTENDEN Modells (nicht des Klassifikators) + heutiges Datum.
|
||||
CUTOFF = "fall 2024"
|
||||
TODAY = date.today().isoformat()
|
||||
|
||||
# Tauglichkeits-Schwellen (siehe Entscheidungsregel im Plan).
|
||||
THRESHOLDS = {"recall": 0.95, "precision": 0.80, "well_formed": 0.99, "priority": 0.95}
|
||||
|
||||
# Marker, die das verbotene (veraltete) Faktum als VERGANGENHEIT ausweisen.
|
||||
# Erwähnt das Modell den alten Wert nur historisch ("früher war ..."), ist das
|
||||
# kein Vorrang-Fehler — es folgt ja dem Tool-Ergebnis.
|
||||
_PAST_MARKERS = (
|
||||
"früher", "ehemalig", "vorher", "davor", "zuvor", "bis ", " war ", "wurde ",
|
||||
"previously", "former", "used to", "until ", "no longer", "past",
|
||||
)
|
||||
|
||||
|
||||
def priority_ok(answer: str, expect_fact: str, forbid_fact: str) -> bool:
|
||||
"""Folgt die Antwort dem Tool? Aktuelles Faktum da; veraltetes höchstens historisch."""
|
||||
low = answer.lower()
|
||||
if expect_fact.lower() not in low:
|
||||
return False
|
||||
if forbid_fact.lower() not in low:
|
||||
return True
|
||||
return any(m in low for m in _PAST_MARKERS)
|
||||
|
||||
|
||||
async def run_case(client: ToolCallingClient, case: dict, runs: int,
|
||||
sem: asyncio.Semaphore) -> dict:
|
||||
"""Führt einen Fall N× aus und sammelt die Roh-Ergebnisse je Lauf."""
|
||||
base = build_messages(case, CUTOFF, TODAY)
|
||||
laeufe = []
|
||||
for _ in range(runs):
|
||||
try:
|
||||
async with sem:
|
||||
t1 = await client.respond(base, [WEB_SEARCH_TOOL])
|
||||
except Exception as exc:
|
||||
# Ein einzelner fehlerhafter Lauf darf nicht den ganzen Batch killen.
|
||||
laeufe.append({"searched": None, "error": str(exc)[:200],
|
||||
"query": None, "well_formed": None, "prio_ok": None, "answer": None})
|
||||
continue
|
||||
rec = {
|
||||
"searched": t1.searched,
|
||||
"query": t1.query,
|
||||
"well_formed": all(tc.well_formed for tc in t1.tool_calls) if t1.searched else None,
|
||||
"prio_ok": None,
|
||||
"answer": None,
|
||||
}
|
||||
# Pass 2: nur bei Vorrang-Fällen und nur wenn überhaupt gesucht wurde.
|
||||
if case.get("injected_result") and t1.searched:
|
||||
tc = t1.tool_calls[0]
|
||||
msgs = base + [t1.raw_message, {
|
||||
"role": "tool",
|
||||
"tool_call_id": tc.id,
|
||||
"content": case["injected_result"],
|
||||
}]
|
||||
async with sem:
|
||||
t2 = await client.respond(msgs, [WEB_SEARCH_TOOL], tool_choice="none")
|
||||
ans = (t2.text or "")
|
||||
rec["answer"] = ans
|
||||
rec["prio_ok"] = priority_ok(ans, case["expect_fact"], case["forbid_fact"])
|
||||
laeufe.append(rec)
|
||||
return {"case": case, "runs": laeufe}
|
||||
|
||||
|
||||
async def main(model: str, runs: int, conc: int = 4) -> None:
|
||||
cases = yaml.safe_load(open(CASES_PATH, encoding="utf-8"))
|
||||
key = os.environ.get("OPENROUTER_API_KEY")
|
||||
if not key:
|
||||
sys.exit("OPENROUTER_API_KEY fehlt in der Umgebung.")
|
||||
client = ToolCallingClient(key, model)
|
||||
sem = asyncio.Semaphore(conc)
|
||||
results = await asyncio.gather(*(run_case(client, c, runs, sem) for c in cases))
|
||||
report(model, runs, results)
|
||||
dump_path = os.path.join(HERE, f"results_{model.replace('/', '_')}.json")
|
||||
with open(dump_path, "w", encoding="utf-8") as f:
|
||||
json.dump(results, f, ensure_ascii=False, indent=2)
|
||||
print(f"\nRoh-Ergebnisse: {dump_path}")
|
||||
|
||||
|
||||
def report(model: str, runs: int, results: list[dict]) -> None:
|
||||
recall, precision, well_formed, priority = [], [], [], []
|
||||
per_cat: dict[str, list[float]] = defaultdict(list)
|
||||
offenders: list[tuple[str, str, float]] = []
|
||||
|
||||
errored = 0
|
||||
for res in results:
|
||||
case, laeufe = res["case"], res["runs"]
|
||||
want_search = case["expect"] == "search"
|
||||
valid = [r for r in laeufe if r["searched"] is not None]
|
||||
errored += len(laeufe) - len(valid)
|
||||
if not valid:
|
||||
continue # ganzer Fall fehlerhaft -> nicht werten
|
||||
searched_rate = statistics.mean(r["searched"] for r in valid)
|
||||
correct_rate = searched_rate if want_search else 1.0 - searched_rate
|
||||
(recall if want_search else precision).append(correct_rate)
|
||||
per_cat[case["category"]].append(correct_rate)
|
||||
|
||||
wfs = [r["well_formed"] for r in laeufe if r["well_formed"] is not None]
|
||||
if wfs:
|
||||
well_formed.append(statistics.mean(wfs))
|
||||
prios = [r["prio_ok"] for r in laeufe if r["prio_ok"] is not None]
|
||||
if prios:
|
||||
priority.append(statistics.mean(prios))
|
||||
if correct_rate < 0.8:
|
||||
offenders.append((case["id"], case["category"], round(correct_rate, 2)))
|
||||
|
||||
def pct(xs: list[float]) -> str:
|
||||
return f"{100 * statistics.mean(xs):.0f}%" if xs else "n/a"
|
||||
|
||||
def flag(xs: list[float], key: str) -> str:
|
||||
if not xs:
|
||||
return ""
|
||||
return " OK" if statistics.mean(xs) >= THRESHOLDS[key] else " ⚠"
|
||||
|
||||
print(f"\n=== {model} ({runs} Läufe/Fall, heute={TODAY}) ===")
|
||||
if errored:
|
||||
print(f"(Hinweis: {errored} Einzelläufe mit Fehler übersprungen)")
|
||||
print(f"Trigger-Recall (muss-suchen) {pct(recall):>5}{flag(recall, 'recall')} (Ziel ≥95%)")
|
||||
print(f"Trigger-Precision (nicht-suchen) {pct(precision):>5}{flag(precision, 'precision')} (Ziel ≥80%)")
|
||||
print(f"Wohlgeformtheit {pct(well_formed):>5}{flag(well_formed, 'well_formed')} (Ziel ≥99%)")
|
||||
print(f"Vorrang-Treue {pct(priority):>5}{flag(priority, 'priority')} (Ziel ≥95%)")
|
||||
print("\npro Kategorie:")
|
||||
for cat, xs in sorted(per_cat.items()):
|
||||
print(f" {cat:26} {pct(xs)}")
|
||||
if offenders:
|
||||
print("\nSchwächste Fälle (<80%):")
|
||||
for oid, cat, rate in sorted(offenders, key=lambda x: x[2]):
|
||||
print(f" {oid:24} {cat:26} {rate}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
arg_model = sys.argv[1] if len(sys.argv) > 1 else "deepseek/deepseek-v4-flash"
|
||||
arg_runs = int(sys.argv[2]) if len(sys.argv) > 2 else 5
|
||||
asyncio.run(main(arg_model, arg_runs))
|
||||
58
eval/tool_calling/tool.py
Normal file
58
eval/tool_calling/tool.py
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
"""web_search-Tool-Definition + Trigger-Prompt.
|
||||
|
||||
Der Beschreibungs-/Prompt-Text ist derselbe kalibrierte „braucht das Wissen
|
||||
neuer als dein Cutoff"-Text, der später produktiv die Trigger-Leitlinie bildet.
|
||||
Der Harness misst damit Modell UND Prompt gemeinsam.
|
||||
"""
|
||||
|
||||
WEB_SEARCH_TOOL = {
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "web_search",
|
||||
"description": (
|
||||
"Look up real-world information that may be newer than your "
|
||||
"knowledge or may have changed since your last update. Call it "
|
||||
"whenever the true answer could plausibly have changed, INCLUDING "
|
||||
"when no exact place or date is named. This covers: who currently "
|
||||
"holds an office; where a living person now lives; whether someone "
|
||||
"is still alive; current prices, rates or crypto; current weather "
|
||||
"or outdoor conditions (even phrased as 'is it cold/raining right "
|
||||
"now', using the user's location); sports results and standings; "
|
||||
"the latest version or model of a product; recent news; and "
|
||||
"time-sensitive logistics such as opening hours, schedules, and "
|
||||
"public-transport or train/bus departure times. Do NOT use it for "
|
||||
"timeless knowledge, opinions, jokes, small talk, the current "
|
||||
"clock time or today's date (you already have those), anything "
|
||||
"you can derive yourself, or anything about the user themselves."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"query": {
|
||||
"type": "string",
|
||||
"description": "Concise search query, in the user's language or English.",
|
||||
},
|
||||
},
|
||||
"required": ["query"],
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
SYSTEM_PROMPT = (
|
||||
"You are a voice assistant. Your knowledge was last updated around "
|
||||
"{cutoff}. Today is {today}. When a question could require information "
|
||||
"newer than {cutoff} that you cannot reason out yourself, call web_search "
|
||||
"before answering. Never call web_search merely to find the current time "
|
||||
"or today's date — you already have them. Tool results are authoritative "
|
||||
"and newer than your memory: if a tool result conflicts with what you "
|
||||
"believe, follow the tool and never contradict it."
|
||||
)
|
||||
|
||||
|
||||
def build_messages(case: dict, cutoff: str, today: str) -> list[dict]:
|
||||
"""Baut die Message-Liste: System-Prompt + (optionale) History + Nutzeräußerung."""
|
||||
msgs = [{"role": "system",
|
||||
"content": SYSTEM_PROMPT.format(cutoff=cutoff, today=today)}]
|
||||
msgs += case.get("history", [])
|
||||
msgs.append({"role": "user", "content": case["utterance"]})
|
||||
return msgs
|
||||
Loading…
Add table
Add a link
Reference in a new issue