feat: adaptiver Modell-Router (Circuit-Breaker) + parallele KI-Analyse
Der reale Erst-Scan dauerte ~10 min, weil pro Kandidat sequenziell erst die rate-limited Free-Modelle (429) probiert wurden. Zwei Hebel beheben das: 1. ModelRouter: programmatische Telemetrie statt LLM-Orchestrator. Pro Modell Erfolg/Latenz; ausgefallene/rate-limited Modelle bekommen per Circuit-Breaker einen Cooldown und werden übersprungen, das schnellste gesunde Modell zuerst. Failsafe-Pruning toter Slugs über OpenRouter /models (24h-Cache). State persistent in data/ai_router_state.json. 2. Parallelisierung: ThreadPoolExecutor mit konfigurierbarer max_concurrency (Default 8, I/O-gebunden → an API-Rate-Limits gebunden, nicht an CPU-Kerne). Klassifikationen laufen nebenläufig, Merge im Hauptthread (keine Locks), findings deterministisch sortiert. Fingerprint-Gruppierung: identischer Inhalt wird nur einmal klassifiziert, Funde aber für alle URLs emittiert. Realtest bredelar.info: Frisch-Scan von ~10 min auf 1:01 min; Breaker öffnete 14× (Free-Modelle übersprungen), alle Verdikte von gemini-2.5-flash-lite. Config: max_concurrency, breaker_failure_threshold, breaker_cooldown_seconds, refresh_models. Tests: 251 grün (+12: Router-Reihung/Breaker/Pruning-failsafe, parallel==sequenziell, Dedup, deterministische Reihenfolge). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
31730403b3
commit
125d80e7f9
7 changed files with 415 additions and 65 deletions
|
|
@ -264,6 +264,34 @@ class BaselineManager:
|
|||
json.dumps(ledger, indent=2, ensure_ascii=False), encoding="utf-8"
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# AI model-router state (circuit-breaker telemetry + /models catalog)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
@property
|
||||
def _ai_router_state_path(self) -> Path:
|
||||
return self.data_dir / "ai_router_state.json"
|
||||
|
||||
def load_ai_router_state(self) -> dict:
|
||||
"""Load the model-router state. Returns a safe default if none/corrupt."""
|
||||
default = {"models": {}, "catalog": {"slugs": [], "fetched_at": None}}
|
||||
if not self._ai_router_state_path.exists():
|
||||
return default
|
||||
try:
|
||||
data = json.loads(self._ai_router_state_path.read_text(encoding="utf-8"))
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
return default
|
||||
data.setdefault("models", {})
|
||||
data.setdefault("catalog", {"slugs": [], "fetched_at": None})
|
||||
return data
|
||||
|
||||
def save_ai_router_state(self, state: dict) -> None:
|
||||
"""Persist the model-router state."""
|
||||
self.data_dir.mkdir(parents=True, exist_ok=True)
|
||||
self._ai_router_state_path.write_text(
|
||||
json.dumps(state, indent=2, ensure_ascii=False), encoding="utf-8"
|
||||
)
|
||||
|
||||
def dismiss_ai_entries(self, fingerprints: list[str] | None = None) -> int:
|
||||
"""
|
||||
Mark ledger entries as dismissed (false-positive acknowledgement).
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue