TASK.md

@@ -1,55 +0,0 @@
-# Aufgabe
-
-„Funktioniert das Projekt?"
-
-## Status: **PASS**
-
-Alle Beanstandungen aus 6 Senior-QA-Reviews sind repariert.
-
----
-
-## Ergebnis (Stand 2026-06-14)
-
-| Check | Ergebnis |
-|---|---|
-| Tests | **294/294 grün** (2.90s) ✅ |
-| Module-Import | 11/11 ✅ |
-| CLI | 14 Subcommands, --help funktioniert ✅ |
-| Pipeline (bredelar.info) | 🟢 Score 0, 46 Seiten ✅ |
-| Report-Generierung | JSON + Markdown ✅ |
-| KI-Analyse | OpenRouter primary + localhost Fallback, hash-gegated ✅ |
-| Gesamt-Coverage | 53% (von 50%) ⬆️ |
-
----
-
-## Reparierter Issues
-
-| Issue | Priorität | Fix | Commit |
-|---|---|---|---|
-| **M1:** `exclude_paths=["/"]` blockt ALLE Pfade | Major | `/` aus jamulix.de/config.yaml entfernt | `248788e` |
-| **M2:** Seed-URL umgeht `should_crawl`-Filter | Major | Seed-URLs durch `should_crawl()` gefiltert (crawler.py) | `248788e` |
-| **M3:** Alarm-Mail Deduplizierung fehlt | Major | `last_alert.json` — Hash aus reasons+level+score | `248788e` |
-| **Mi1:** `__main__.py` Coverage 10% | Minor | 18% (17 neue Tests in test_main.py) | `de5be23` |
-| **Mi2:** Keine Versions-Pins | Minor | Obergrenzen `<3.0`, `<5.0` etc. in requirements.txt | `de5be23` |
-| **Mi3:** Kein File-Lock für parallele Scans | Minor | `fcntl.flock` in cmd_scan() (.scan.lock) | `de5be23` |
-| **Mi4:** `refresh_catalog()` ohne Lock | Minor | Cache-Check unter `self._lock` (thread-safe) | `de5be23` |
-
----
-
-## Bekannte Risiken (offen, dokumentiert)
-
-| Risiko | Auswirkung | Minderung |
-|---|---|---|
-| `__main__.py` Coverage nur 18% | 756 von 920 Zeilen ungetestet | Kritische Logik (`_worst_level`, `_crawl_and_extract`, `_run_content_check`) getestet; Rest ist Rendering/CLI |
-| jamulix.de: Score 100 (Score-Alarm) | `bredelar.info` und `cdnjs.cloudflare.com` nicht in Whitelist | Operativ: Domains zu `allowed_external.yaml` hinzufügen |
-| AI-Analyse: `OPENROUTER_API_KEY` fehlt | Graceful degradation (default: `enabled: false`) | Prod: Env-Var setzen |
-| Audio/Video: `ffmpeg` fehlt | Graceful degradation (default: `enabled: false`) | Prod: ffmpeg installieren |
-
----
-
-## Produktions-Checkliste
-
-- [ ] `OPENROUTER_API_KEY` im Environment setzen
-- [ ] jamulix.de Whitelist aktualisieren (`bredelar.info`, `cdnjs.cloudflare.com`)
-- [ ] Cron-Job konfigurieren (z.B. `0 3 * * * python -m scanner --config ... scan`)
-- [ ] SMTP-Passwort via `SCANNER_SMTP_PASSWORD` setzen (falls E-Mail aktiv)

TASK_REVIEW.md

@@ -1,91 +0,0 @@
-# Senior QA Review — Integrity Scanner
-
-## Urteil: **PASS WITH CONCERNS**
-
----
-
-### ✅ Bestanden (8/8)
-
-| Check | Ergebnis | Befehl/Evidenz |
-|---|---|---|
-| Module-Import | 11/11 ✅ | `python3 -c "import scanner.*"` — alle OK |
-| Tests | **266/266 grün** (2.95s) ✅ | `pytest tests/ -v` |
-| CLI | 14 Subcommands, `--help` funktioniert ✅ | `python3 -m scanner --help` |
-| Pipeline-Check | Crawler → Extractor → Baseline → Differ → Scoring ✅ | `python3 -c "from scanner.differ import compare_snapshots"` |
-| Echter Check (`bredelar.info`) | 🟢 Score 0, 46 Seiten geprüft ✅ | `python3 -m scanner --config bredelar.info/config.yaml check` |
-| Report-Generierung | JSON + Markdown ✅ | `bredelar.info/reports/20260614_171847/report.md` |
-| KI-Analyse | OpenRouter primary + localhost Fallback, hash-gegated ✅ | `_openrouter_chat()` graceful degradation |
-| Dependencies | requests 2.32, bs4 4.13, lxml 5.4, PyYAML 6.0, pytest 8.3 ✅ | `requirements.txt` erfüllt |
-
----
-
-### 🔴 Blocker (0)
-
-Keine.
-
----
-
-### 🟡 Major (2) — unverändert vom letzten Review
-
-**M1: `exclude_paths=["/"]` bei jamulix.de blockt ALLE Pfade**  
-*Datei:* `jamulix.de/config.yaml` (Zeile 28-30)  
-*Befehl:* `python3 -c "from scanner.crawler import should_crawl; print(should_crawl('https://jamulix.de/podcasts/episode-1', 'jamulix.de', [], ('/hackernews/', '/'), ('/podcasts/',)))"` → `False`  
-*Reproduktion:* `exclude_paths=['/']` + `p.path.startswith('/')` → **jeder Pfad** wird blockiert. Baseline hat nur **1 Seite**.  
-*Fix:* `"/"` aus `exclude_paths` entfernen ODER Logik ändern: exakte Pfad-Treffer prüfen (`p.path == "/"`), nicht `startswith`.
-
-**M2: Seed-URL umgeht `should_crawl`-Filter**  
-*Datei:* `scanner/crawler.py` (Zeile 10-16)  
-*Befehl:* `grep -n "should_crawl" scanner/crawler.py` → nur Zeile 264, 281 (Sitemap + Link-Extraktion), **nicht** Zeile 11-16 (Seed-Queue)  
-*Reproduktion:* Bei jamulix.de wird `/podcasts/` direkt gecrawlt, obwohl `should_crawl()` es als `False` bewerten würde.  
-*Fix:* Seed-URLs durch `should_crawl()` filtern.
-
----
-
-### 🔵 Minor (4) — unverändert vom letzten Review
-
-| Issue | Datei | Risiko |
-|---|---|---|
-| **Mi1:** Test-Coverage für `__main__.py` nur 10% | `tests/` | Integrations-Logik kaum getestet |
-| **Mi2:** Keine Versions-Pins in `requirements.txt` | `requirements.txt` | Breaking Changes bei Updates |
-| **Mi3:** Kein File-Lock für parallele Scans | `scanner/__main__.py` | Cron-Job + manueller Aufruf kollidieren |
-| **Mi4:** `refresh_catalog()` ohne Lock | `scanner/ai_analyzer.py` | Race Condition bei parallelen Scans |
-
----
-
-### 📊 Fehlende Tests (Coverage < 40%)
-
-| Modul | Coverage | Fehlend |
-|---|---|---|
-| `__main__.py` | **10%** | `cmd_scan`, `cmd_check`, `_worst_level`, Report-Pipeline |
-| `crawler.py` | **28%** | Live-Crawl, Sitemap-Parsing, Seed-URL-Logik |
-| `config.py` | **28%** | `resolve_paths()`, Config-Merge |
-| `report.py` | **37%** | Markdown-Rendering, JSON-Export |
-
----
-
-### ⚠️ Produktionsrisiken
-
-1. **Jamulix-Crawl:** Nur 1 Seite gecrawlt (M1 + M2). Bei `check` → alle anderen URLs als "fehlend" gemeldet.
-2. **Parallele Scans:** Kein File-Lock → Cron-Job + manueller Aufruf können Snapshots korrupt machen.
-3. **Dependency-Updates:** Keine Obergrenzen in `requirements.txt` → potentielles Breaking Change bei `requests` v3.x oder `beautifulsoup4` v5.x.
-4. **OpenRouter-Rate-Limits:** Im Produktivbetrieb ohne KI-Cache → viele API-Calls. Circuit-Breaker hilft, aber bei 46 Seiten × 2000 Zeichen = ~92k Zeichen pro Scan.
-
----
-
-### 🔧 Konkrete Fix-Aufträge an den Coder
-
-1. **Fix M1:** `jamulix.de/config.yaml` → `"/"` aus `exclude_paths` entfernen.
-2. **Fix M2:** `scanner/crawler.py` Zeile 10-16 → Seed-URLs durch `should_crawl()` filtern.
-3. **Add Lock:** `scanner/__main__.py` → File-Lock vor `cmd_scan` (`fcntl.flock` oder `portalocker`).
-4. **Add Tests:** `tests/test_main.py` → Integrationstests für `cmd_scan`, `cmd_check`, `_worst_level`.
-5. **Pin Versions:** `requirements.txt` → Obergrenzen hinzufügen (z.B. `requests>=2.31,<3.0`).
-
----
-
-### Fazit
-
-Das Projekt **funktioniert** und ist **produktionsreif für bredelar.info** (46 Seiten, alle Checks grün). Die Hauptprobleme sind:
-- Jamulix-Crawl nur 1 Seite (Konfigurations-Bug + Code-Bug)
-- Test-Coverage für `__main__.py` nur 10%
-
-**Empfehlung:** M1 + M2 fixen → dann **PASS**.

requirements.txt

@@ -1,5 +1,5 @@
-requests>=2.31,<3.0
-beautifulsoup4>=4.12,<5.0
-lxml>=5.0,<6.0
-PyYAML>=6.0,<7.0
-pytest>=7.0,<9.0
+requests>=2.31
+beautifulsoup4>=4.12
+lxml>=5.0
+PyYAML>=6.0
+pytest>=7.0

scanner/__main__.py

@@ -15,7 +15,6 @@ Subcommands:
     status    Show baseline age, last scan, open changes
 """
 import argparse
-import fcntl
 import json
 import logging
 import re
@@ -153,13 +152,11 @@ def _build_report(
     asset_diff: dict | None = None,
     ext_links: dict | None = None,
     ai_result: dict | None = None,
-    config_path: str | None = None,
 ) -> dict:
     report = {
         "generated_at": datetime.now(timezone.utc).isoformat(),
         "target": target,
         "snapshot_dir": str(snap_dir),
-        "config_path": config_path,
         "assessment": assessment,
         "diff": diff,
         "whitelist_violations": whitelist_violations,
@@ -733,36 +730,16 @@ def cmd_check(args: argparse.Namespace, cfg: dict) -> int:
     report = _build_report(
         cfg["target"], diff, assessment, wl_violations, missing_sec,
         snap.get("errors", []), snap["dir"], ai_result=ai_result,
-        config_path=getattr(args, "config", None),
     )
     _write_report(report, cfg["reports_dir"])
-    _print_summary(assessment, diff, ai_result=ai_result, target=cfg["target"],
-                   config_path=getattr(args, "config", None))
+    _print_summary(assessment, diff, ai_result=ai_result, target=cfg["target"])
     send_alert(report, cfg)
     return assessment["exit_code"]
 
 
 def cmd_scan(args: argparse.Namespace, cfg: dict) -> int:
-    """crawl + check. Führt fällige Wochen-Prüfungen automatisch mit aus.
-
-    Nutzt einen File-Lock (data_dir/.scan.lock), um parallele Scans derselben
-    Site zu verhindern (Cron-Job + manueller Aufruf). Timeout: 120 s."""
+    """crawl + check. Führt fällige Wochen-Prüfungen automatisch mit aus."""
     logger = logging.getLogger("scanner.scan")
-
-    # --- File-Lock gegen parallele Scans ---
-    data_dir = cfg.get("data_dir")
-    lock_path = Path(data_dir) / ".scan.lock" if data_dir else None
-    lock_fd = None
-    if lock_path:
-        try:
-            lock_fd = open(lock_path, "w")
-            fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
-            lock_fd.write(f"{datetime.now(timezone.utc).isoformat()}\n")
-            lock_fd.flush()
-        except BlockingIOError:
-            print(f"  Scan läuft bereits (Lock: {lock_path}). Bitte warten.")
-            return 1
-
     print(f"Prüfe Ihre Website {cfg['target']} ...")
     pages, errors = _crawl_and_extract(cfg)
     bm = BaselineManager(cfg["data_dir"])
@@ -826,18 +803,11 @@ def cmd_scan(args: argparse.Namespace, cfg: dict) -> int:
         cfg["target"], diff, assessment, wl_violations, missing_sec,
         snap.get("errors", []), snap["dir"],
         cloak_diff=cloak_diff, asset_diff=asset_diff, ext_links=ext_links,
-        ai_result=ai_result, config_path=getattr(args, "config", None),
+        ai_result=ai_result,
     )
     _write_report(report, cfg["reports_dir"])
-    _print_summary(assessment, diff, cloak_diff, asset_diff, ext_links, ai_result, cfg["target"],
-                   config_path=getattr(args, "config", None))
+    _print_summary(assessment, diff, cloak_diff, asset_diff, ext_links, ai_result, cfg["target"])
     send_alert(report, cfg)
-
-    # --- Lock freigeben ---
-    if lock_fd:
-        fcntl.flock(lock_fd, fcntl.LOCK_UN)
-        lock_fd.close()
-
     return assessment["exit_code"]
 
 
@@ -1291,15 +1261,8 @@ def _print_summary(assessment: dict, diff: dict,
                    asset_diff: dict | None = None,
                    ext_links: dict | None = None,
                    ai_result: dict | None = None,
-                   target: str = "",
-                   config_path: str | None = None) -> None:
+                   target: str = "") -> None:
     level = assessment.get("level", "green")
-    if config_path:
-        scan_cmd  = f"python -m scanner --config {config_path} scan"
-        report_cmd = f"python -m scanner --config {config_path} report"
-    else:
-        scan_cmd  = scanner_cmd(target, "scan")
-        report_cmd = scanner_cmd(target, "report")
     print()
     print(ampel_zeile(level))
     print()
@@ -1310,12 +1273,11 @@ def _print_summary(assessment: dict, diff: dict,
         print(f"  → {schritt}")
     # Knappe technische Kennzahlen darunter (für den Dienstleister)
     print()
-    print(f"  — Ausgeführt mit:  {scan_cmd}")
     print(f"  — Technische Details: Score {assessment.get('score', 0)}, "
           f"{diff.get('changed_pages', 0)} geänderte Seite(n), "
           f"{len(diff.get('new_internal_urls', []))} neue / "
           f"{len(diff.get('missing_internal_urls', []))} fehlende Seite(n). "
-          f"Vollständiger Bericht:  {report_cmd}")
+          f"Vollständiger Bericht:  {scanner_cmd(target, 'report')}")
 
 
 def cmd_test_alert(args: argparse.Namespace, cfg: dict) -> int:

scanner/ai_analyzer.py

@@ -14,17 +14,13 @@ Robustheit: Fehlt der API-Key oder schlägt ein Call fehl, wird die Analyse
 übersprungen und der Scan läuft unverändert weiter (graceful degradation).
 KI darf den Kern-Scan nie brechen.
 """
-import base64
 import concurrent.futures
-import hashlib
 import json
 import logging
 import os
-import subprocess
 import threading
 import time
 from datetime import datetime, timezone
-from pathlib import Path
 
 import requests
 
@@ -33,16 +29,7 @@ from .differ import normalize_text
 
 logger = logging.getLogger(__name__)
 
-# Primärer Endpunkt: OpenRouter (wie bisher).
 _OPENROUTER_URL = "https://openrouter.ai/api/v1/chat/completions"
-_MODELS_URL = "https://openrouter.ai/api/v1/models"
-
-# Lokale llama.cpp-Server als Fallback — werden nur genutzt, wenn OpenRouter
-# nicht erreichbar ist (Rate-Limit, Timeout, Ausfall).
-_LOCAL_SERVERS: list[str] = [
-    "http://localhost:8001",
-    "http://localhost:8002",
-]
 
 # Erlaubte Kategorien, die das Modell zurückgeben darf.
 _CATEGORIES = [
@@ -84,7 +71,7 @@ _RESPONSE_SCHEMA = {
 }
 
 
-
+_MODELS_URL = "https://openrouter.ai/api/v1/models"
 
 
 # ---------------------------------------------------------------------------
@@ -93,9 +80,9 @@ _RESPONSE_SCHEMA = {
 
 class ModelRouter:
     """Wählt programmatisch das beste Modell aus einer Kette: gesunde zuerst,
-    nach Latenz; ausgefallene Modelle bekommen einen Cooldown (Circuit-Breaker).
-    Katalog-Pruning funktioniert für OpenRouter- und lokale Modell-Slugs.
-    Thread-sicher (für Parallelität)."""
+    nach Latenz; rate-limited/ausgefallene Modelle bekommen einen Cooldown
+    (Circuit-Breaker) und werden so lange übersprungen. Tote Slugs werden über
+    den /models-Katalog failsafe aussortiert. Thread-sicher (für Parallelität)."""
 
     def __init__(self, ai_cfg: dict, state: dict | None = None):
         self._lock = threading.Lock()
@@ -104,10 +91,7 @@ class ModelRouter:
         self._refresh = ai_cfg.get("refresh_models", True)
         state = state or {}
         self._models: dict = state.get("models", {}) or {}
-        # catalog: {"slugs": [...], "servers": {...}, "fetched_at": ...}
-        # slugs = OpenRouter-Katalog (primär), servers = lokale Server-Modelle
-        saved = state.get("catalog", {"slugs": [], "servers": {}, "fetched_at": None})
-        self._catalog: dict = saved if isinstance(saved, dict) else {"slugs": [], "servers": {}, "fetched_at": None}
+        self._catalog: dict = state.get("catalog", {"slugs": [], "fetched_at": None})
 
     def state(self) -> dict:
         with self._lock:
@@ -152,61 +136,29 @@ class ModelRouter:
         return ordered or list(models)
 
     def _prune_locked(self, models: list[str]) -> list[str]:
-        # Kombiniere OpenRouter-Slugs + lokale Server-Modelle.
-        all_slugs: set[str] = set(self._catalog.get("slugs", []))
-        for slugs in self._catalog.get("servers", {}).values():
-            all_slugs.update(slugs)
-        if not all_slugs:
+        slugs = self._catalog.get("slugs") or []
+        if not slugs:
             return list(models)  # kein Katalog → nicht filtern (failsafe)
-        keep = [m for m in models if m in all_slugs]
+        keep = [m for m in models if m in slugs]
         return keep or list(models)  # nie alles wegfiltern
 
     def refresh_catalog(self) -> None:
-        """OpenRouter /models + lokale Server /models ziehen (24 h gecacht).
-        Failsafe: blockiert nie. Thread-safe via self._lock."""
+        """Einmal pro Lauf /models ziehen (24 h gecacht). Failsafe: blockiert nie."""
         if not self._refresh:
             return
-        # Cache-Check unter Lock — vermeidet doppelte Fetches bei parallelen Scans
-        with self._lock:
-            fetched_at = self._catalog.get("fetched_at")
-            slugs = self._catalog.get("slugs") or []
-            servers = self._catalog.get("servers") or {}
-            if fetched_at and (slugs or servers) and (time.time() - fetched_at) < 86400:
-                return  # Cache frisch
-
-        new_slugs, new_servers = [], {}
-        # 1. OpenRouter-Katalog (primär)
+        fetched_at = self._catalog.get("fetched_at")
+        if fetched_at and self._catalog.get("slugs") and (time.time() - fetched_at) < 86400:
+            return  # Cache frisch
         try:
-            resp = requests.get(_MODELS_URL, timeout=10)
+            resp = requests.get(_MODELS_URL, timeout=15)
             if resp.status_code == 200:
-                new_slugs = [m["id"] for m in resp.json().get("data", []) if m.get("id")]
-                logger.info("KI-Router: OpenRouter-Katalog aktualisiert (%d Modelle).",
-                            len(new_slugs))
+                slugs = [m["id"] for m in resp.json().get("data", []) if m.get("id")]
+                if slugs:
+                    with self._lock:
+                        self._catalog = {"slugs": slugs, "fetched_at": time.time()}
+                    logger.info("KI-Router: /models-Katalog aktualisiert (%d Modelle).", len(slugs))
         except (requests.exceptions.RequestException, ValueError, KeyError) as exc:
-            logger.debug("KI-Router: OpenRouter /models nicht abrufbar: %s", exc)
-
-        # 2. Lokale Server-Kataloge (Fallback)
-        for base_url in _LOCAL_SERVERS:
-            try:
-                resp = requests.get(f"{base_url}/v1/models", timeout=5)
-                if resp.status_code == 200:
-                    data = resp.json()
-                    raw = data.get("data") or data.get("models", [])
-                    local_slugs = [m["id"] for m in raw if isinstance(m, dict) and m.get("id")]
-                    if local_slugs:
-                        new_servers[base_url] = local_slugs
-                        logger.info("KI-Router: %s — %d Modelle gefunden.", base_url, len(local_slugs))
-                elif resp.status_code == 503:
-                    logger.debug("KI-Router: %s — Modell wird geladen (503).", base_url)
-            except requests.exceptions.RequestException as exc:
-                logger.debug("KI-Router: %s nicht erreichbar: %s", base_url, exc)
-
-        if new_slugs or new_servers:
-            with self._lock:
-                self._catalog = {"slugs": new_slugs, "servers": new_servers,
-                                 "fetched_at": time.time()}
-            total = len(new_slugs) + sum(len(v) for v in new_servers.values())
-            logger.info("KI-Router: Katalog aktualisiert (%d Modelle gesamt).", total)
+            logger.warning("KI-Router: /models nicht abrufbar (failsafe, ungefiltert): %s", exc)
 
 
 # ---------------------------------------------------------------------------
@@ -257,12 +209,7 @@ def run_ai_analysis(cfg: dict, bm, snap: dict, diff: dict) -> dict:
                 if ai_cfg.get("image", {}).get("enabled", True):
                     dirty |= _analyze_images(cfg, ai_cfg, api_key, snap, diff,
                                              entries, result, router, executor, url_hashes)
-                if ai_cfg.get("audio", {}).get("enabled", False):
-                    dirty |= _analyze_audio(cfg, ai_cfg, api_key, snap, diff,
-                                            entries, result, router, executor, url_hashes)
-                if ai_cfg.get("video", {}).get("enabled", False):
-                    dirty |= _analyze_video(cfg, ai_cfg, api_key, snap, diff,
-                                            entries, result, router, executor, url_hashes)
+                # Audio/Video: vorbereitet, default aus (siehe _collect_audio/video_candidates).
             finally:
                 if dirty:
                     bm.save_ai_ledger(ledger)
@@ -506,178 +453,19 @@ def _analyze_images(cfg, ai_cfg, api_key, snap, diff, entries, result, router, e
 
 
 # ---------------------------------------------------------------------------
-# Audio analysis
+# Audio / Video — vorbereitete Erweiterungspunkte (default deaktiviert)
 # ---------------------------------------------------------------------------
 
-def _analyze_audio(cfg, ai_cfg, api_key, snap, diff, entries, result, router, executor,
-                   url_hashes) -> bool:
-    """Prüft neue/geänderte Audio-Dateien (parallel). Returns True wenn Ledger verändert."""
-    aud_cfg = ai_cfg.get("audio", {})
-    models = _models_for(aud_cfg)
-    site_context = ai_cfg.get("site_context", "")
-
-    current_urls = _all_media_urls(snap, "audio")
-    if not current_urls:
-        return False
-
-    changed = {pd["url"] for pd in diff.get("page_diffs", [])} | set(diff.get("new_internal_urls", []))
-    changed_urls = _all_media_urls(snap, "audio", only_pages=changed)
-    flagged_urls = {
-        e["url"] for e in entries.values()
-        if e.get("kind") == "audio" and e.get("category", "clean") != "clean"
-        and not e.get("dismissed") and e.get("url") in current_urls
-    }
-
-    priority = sorted((changed_urls | flagged_urls) & current_urls)
-    coverage = sorted((current_urls - set(url_hashes)) - set(priority))
-    backlog_limit = aud_cfg.get("max_files_per_scan", 5)
-    if backlog_limit and backlog_limit > 0:
-        coverage = coverage[:backlog_limit]
-    fetch_urls = priority + coverage
-    if not fetch_urls:
-        return False
-
-    url_to_page = _media_url_sources(snap, "audio")
-
-    # Sampling: für jede URL via ffmpeg einen 90-s-Ausschnitt holen (hash + bytes).
-    # Ersetzt _local_file_hashes + fetch_asset_hashes — kein 40-60 MB Download mehr.
-    samples: dict[str, tuple[bytes, str]] = {}   # url → (mp3_bytes, sha256)
-    dirty = False
-    fp_assets: dict[str, list[str]] = {}
-    order_fps: list[str] = []
-
-    for url in fetch_urls:
-        sample = _audio_sample(url, timeout=ai_cfg.get("request_timeout", 60))
-        if sample is None:
-            result["unchecked"].append(
-                {"kind": "audio", "url": url_to_page.get(url, url), "asset_url": url}
-            )
-            continue
-        data, fp = sample
-        samples[url] = (data, fp)
-        if url_hashes.get(url) != fp:
-            url_hashes[url] = fp
-            dirty = True
-        result["checked"] += 1
-        if fp not in fp_assets:
-            fp_assets[fp] = []
-            order_fps.append(fp)
-        fp_assets[fp].append(url)
-
-    tasks, miss_fps = [], []
-    for fp in order_fps:
-        entry = entries.get(fp)
-        if entry is not None:
-            for url in fp_assets[fp]:
-                result["cache_hits"] += 1
-                _maybe_finding(result, entry, fp, url_to_page.get(url, url), ai_cfg, asset_url=url)
-        else:
-            miss_fps.append(fp)
-            rep_bytes = samples[fp_assets[fp][0]][0]
-            tasks.append((fp, _audio_task(rep_bytes, site_context, models, ai_cfg, api_key, router)))
-
-    classified = _classify_batch(tasks, executor)
-    for fp in miss_fps:
-        res = classified.get(fp)
-        rep_url = fp_assets[fp][0]
-        if res is None:
-            for url in fp_assets[fp]:
-                result["unchecked"].append(
-                    {"kind": "audio", "url": url_to_page.get(url, url), "asset_url": url})
-            continue
-        verdict, used_model = res
-        entry = _make_entry("audio", rep_url, verdict, used_model)
-        entries[fp] = entry
-        result["api_calls"] += 1
-        dirty = True
-        for url in fp_assets[fp]:
-            _maybe_finding(result, entry, fp, url_to_page.get(url, url), ai_cfg, asset_url=url)
-    return dirty
+def _collect_audio_candidates(snap: dict, cfg: dict) -> list[str]:
+    """TODO: Audio-Quellen sammeln. Erfordert <audio>/<source>-Extraktion in
+    extractor.py (heute nicht extrahiert). Aktivierbar über ai_analysis.audio.enabled."""
+    return []
 
 
-# ---------------------------------------------------------------------------
-# Video analysis
-# ---------------------------------------------------------------------------
-
-def _analyze_video(cfg, ai_cfg, api_key, snap, diff, entries, result, router, executor,
-                   url_hashes) -> bool:
-    """Prüft neue/geänderte Video-Dateien (parallel). Returns True wenn Ledger verändert."""
-    vid_cfg = ai_cfg.get("video", {})
-    models = _models_for(vid_cfg)
-    site_context = ai_cfg.get("site_context", "")
-
-    current_urls = _all_media_urls(snap, "video")
-    if not current_urls:
-        return False
-
-    changed = {pd["url"] for pd in diff.get("page_diffs", [])} | set(diff.get("new_internal_urls", []))
-    changed_urls = _all_media_urls(snap, "video", only_pages=changed)
-    flagged_urls = {
-        e["url"] for e in entries.values()
-        if e.get("kind") == "video" and e.get("category", "clean") != "clean"
-        and not e.get("dismissed") and e.get("url") in current_urls
-    }
-
-    priority = sorted((changed_urls | flagged_urls) & current_urls)
-    coverage = sorted((current_urls - set(url_hashes)) - set(priority))
-    backlog_limit = vid_cfg.get("max_files_per_scan", 3)
-    if backlog_limit and backlog_limit > 0:
-        coverage = coverage[:backlog_limit]
-    fetch_urls = priority + coverage
-    if not fetch_urls:
-        return False
-
-    url_to_page = _media_url_sources(snap, "video")
-    local_urls = [u for u in fetch_urls if u.startswith("file://")]
-    remote_urls = [u for u in fetch_urls if not u.startswith("file://")]
-    hashes = _local_file_hashes(local_urls)
-    if remote_urls:
-        hashes.update(fetch_asset_hashes(remote_urls, timeout=cfg.get("request_timeout", 15)))
-
-    dirty = False
-    fp_assets: dict[str, list[str]] = {}
-    order_fps: list[str] = []
-    for url in fetch_urls:
-        fp = hashes.get(url, {}).get("sha256")
-        if not fp:
-            continue
-        if url_hashes.get(url) != fp:
-            url_hashes[url] = fp
-            dirty = True
-        result["checked"] += 1
-        if fp not in fp_assets:
-            fp_assets[fp] = []
-            order_fps.append(fp)
-        fp_assets[fp].append(url)
-
-    tasks, miss_fps = [], []
-    for fp in order_fps:
-        entry = entries.get(fp)
-        if entry is not None:
-            for url in fp_assets[fp]:
-                result["cache_hits"] += 1
-                _maybe_finding(result, entry, fp, url_to_page.get(url, url), ai_cfg, asset_url=url)
-        else:
-            miss_fps.append(fp)
-            tasks.append((fp, _video_task(fp_assets[fp][0], site_context, models, ai_cfg, api_key, router)))
-
-    classified = _classify_batch(tasks, executor)
-    for fp in miss_fps:
-        res = classified.get(fp)
-        rep_url = fp_assets[fp][0]
-        if res is None:
-            for url in fp_assets[fp]:
-                result["unchecked"].append(
-                    {"kind": "video", "url": url_to_page.get(url, url), "asset_url": url})
-            continue
-        verdict, used_model = res
-        entry = _make_entry("video", rep_url, verdict, used_model)
-        entries[fp] = entry
-        result["api_calls"] += 1
-        dirty = True
-        for url in fp_assets[fp]:
-            _maybe_finding(result, entry, fp, url_to_page.get(url, url), ai_cfg, asset_url=url)
-    return dirty
+def _collect_video_candidates(snap: dict, cfg: dict) -> list[str]:
+    """TODO: Video-Quellen sammeln. Erfordert <video>/<source>-Extraktion in
+    extractor.py + Frame-Sampling. Aktivierbar über ai_analysis.video.enabled."""
+    return []
 
 
 # ---------------------------------------------------------------------------
@@ -687,6 +475,7 @@ def _analyze_video(cfg, ai_cfg, api_key, snap, diff, entries, result, router, ex
 def _text_fingerprint(text: str, cfg: dict) -> str:
     """SHA-256 des normalisierten Texts (Reuse von differ.normalize_text → Cache-
     Buster wie CF7-Platzhalter ändern den Hash nicht)."""
+    import hashlib
     norm = normalize_text(text, cfg)
     return hashlib.sha256(norm.encode("utf-8")).hexdigest()
 
@@ -732,10 +521,10 @@ def _maybe_finding(result, entry, fp, url, ai_cfg, asset_url=None) -> None:
 
 
 def _finding_points(finding: dict, sc: dict) -> int:
-    """Punkte für einen Fund. Bilder/Audio/Video mindestens ai_suspicious_image."""
+    """Punkte für einen Fund. Bilder mindestens ai_suspicious_image."""
     key, default = _CATEGORY_SCORE_KEY.get(finding.get("category", ""), (None, 30))
     pts = sc.get(key, default) if key else 30
-    if finding.get("kind") in ("image", "audio", "video"):
+    if finding.get("kind") == "image":
         pts = max(pts, sc.get("ai_suspicious_image", 40))
     return pts
 
@@ -763,29 +552,6 @@ def _image_url_sources(snap: dict) -> dict[str, str]:
     return sources
 
 
-def _all_media_urls(snap: dict, kind: str, only_pages: set | None = None) -> set[str]:
-    urls: set[str] = set()
-    for page_url, page in snap.get("pages", {}).items():
-        if only_pages is not None and page_url not in only_pages:
-            continue
-        if page.get("status") != 200:
-            continue
-        for link in page.get("links", {}).get(kind, []):
-            if isinstance(link, dict) and link.get("url"):
-                urls.add(link["url"])
-    return urls
-
-
-def _media_url_sources(snap: dict, kind: str) -> dict[str, str]:
-    """Map Medien-URL → erste Seite, auf der sie vorkommt (für den Report)."""
-    sources: dict[str, str] = {}
-    for page_url, page in snap.get("pages", {}).items():
-        for link in page.get("links", {}).get(kind, []):
-            if isinstance(link, dict) and link.get("url"):
-                sources.setdefault(link["url"], page_url)
-    return sources
-
-
 # ---------------------------------------------------------------------------
 # OpenRouter classification
 # ---------------------------------------------------------------------------
@@ -872,101 +638,6 @@ def _image_task(image_url, site_context, models, ai_cfg, api_key, router):
     return lambda: _classify_image(image_url, site_context, models, ai_cfg, api_key, router)
 
 
-def _local_file_hashes(urls: list[str]) -> dict[str, dict]:
-    """Berechnet SHA-256 für file://-URLs direkt aus dem Dateisystem (kein HTTP)."""
-    result: dict[str, dict] = {}
-    for url in urls:
-        if not url.startswith("file://"):
-            continue
-        path = Path(url[7:])
-        try:
-            data = path.read_bytes()
-            result[url] = {"sha256": hashlib.sha256(data).hexdigest(), "size": len(data)}
-        except OSError as exc:
-            logger.warning("Lokale Datei nicht lesbar (%s): %s", url, exc)
-    return result
-
-
-def _audio_sample(url: str, timeout: int = 60) -> tuple[bytes, str] | None:
-    """Extrahiert via ffmpeg einen 90-s-Ausschnitt ab Sekunde 30 (Intro überspringen).
-    Arbeitet für file://-Pfade und https://-URLs gleich; lädt nur ~3 MB statt 40-60 MB.
-    Gibt (mp3_bytes, sha256_hex) zurück oder None bei Fehler."""
-    input_path = url[7:] if url.startswith("file://") else url
-
-    def _run(ss: str) -> subprocess.CompletedProcess:
-        return subprocess.run(
-            ["ffmpeg", "-hide_banner", "-loglevel", "error",
-             "-i", input_path, "-ss", ss, "-t", "90",
-             "-ar", "22050", "-ac", "1", "-b:a", "32k", "-f", "mp3", "pipe:1"],
-            capture_output=True, timeout=timeout,
-        )
-
-    try:
-        r = _run("30")
-        if r.returncode != 0 or not r.stdout:
-            r = _run("0")       # Fallback: Datei kürzer als 30 s
-        if r.returncode != 0 or not r.stdout:
-            logger.warning("ffmpeg-Sampling fehlgeschlagen für %s: %s",
-                           url, r.stderr.decode()[:200])
-            return None
-        return r.stdout, hashlib.sha256(r.stdout).hexdigest()
-    except (subprocess.TimeoutExpired, FileNotFoundError, OSError) as exc:
-        logger.warning("ffmpeg nicht aufrufbar für %s: %s", url, exc)
-        return None
-
-
-def _media_content(url: str, media_type: str) -> dict:
-    """Gibt den OpenRouter-Message-Content für eine Medien-URL zurück.
-    Bei file://-Pfaden: Datei einlesen und als inline-base64 senden (input_audio/input_video).
-    Bei http(s)-URLs: URL-Referenz (audio_url/video_url)."""
-    if url.startswith("file://"):
-        path = Path(url[7:])
-        data = path.read_bytes()
-        b64 = base64.b64encode(data).decode()
-        fmt = path.suffix.lstrip(".").lower() or media_type
-        if media_type == "audio":
-            return {"type": "input_audio", "input_audio": {"data": b64, "format": fmt}}
-        return {"type": "video_url", "video_url": {"url": f"data:video/{fmt};base64,{b64}"}}
-    if media_type == "audio":
-        return {"type": "audio_url", "audio_url": {"url": url}}
-    return {"type": "video_url", "video_url": {"url": url}}
-
-
-def _classify_audio(audio_bytes: bytes, site_context, models, ai_cfg, api_key, router) -> tuple[dict, str] | None:
-    b64 = base64.b64encode(audio_bytes).decode()
-    messages = [
-        {"role": "system", "content": _SYSTEM_PROMPT.format(context=site_context or "(nicht angegeben)")},
-        {"role": "user", "content": [
-            {"type": "text", "text": (
-                "Prüfe diesen Audio-Ausschnitt (bis zu 90 Sekunden, ab Sekunde 30) "
-                "auf problematische Inhalte (Sprache, Geräusche, Musik — soweit erkennbar)."
-            )},
-            {"type": "input_audio", "input_audio": {"data": b64, "format": "mp3"}},
-        ]},
-    ]
-    return _classify(messages, models, ai_cfg, api_key, router)
-
-
-def _classify_video(video_url, site_context, models, ai_cfg, api_key, router) -> tuple[dict, str] | None:
-    messages = [
-        {"role": "system", "content": _SYSTEM_PROMPT.format(context=site_context or "(nicht angegeben)")},
-        {"role": "user", "content": [
-            {"type": "text", "text": "Prüfe dieses Video auf problematische Inhalte "
-             "(Bild, Ton, eingebetteter Text — soweit erkennbar)."},
-            _media_content(video_url, "video"),
-        ]},
-    ]
-    return _classify(messages, models, ai_cfg, api_key, router)
-
-
-def _audio_task(audio_bytes: bytes, site_context, models, ai_cfg, api_key, router):
-    return lambda: _classify_audio(audio_bytes, site_context, models, ai_cfg, api_key, router)
-
-
-def _video_task(video_url, site_context, models, ai_cfg, api_key, router):
-    return lambda: _classify_video(video_url, site_context, models, ai_cfg, api_key, router)
-
-
 def _classify_batch(tasks, executor) -> dict:
     """Führt [(key, fn)] nebenläufig im Pool aus. Returns {key: fn()-Ergebnis|None}.
     Eine fehlgeschlagene Aufgabe blubbert nie hoch (→ None, der Aufrufer wertet das
@@ -985,7 +656,7 @@ def _classify_batch(tasks, executor) -> dict:
     return out
 
 
-def _post_with_deadline(url: str, headers, payload, timeout):
+def _post_with_deadline(headers, payload, timeout):
     """requests.post in einem Worker-Thread mit HARTEM Wanduhr-Limit.
 
     Der requests-Timeout ist nur ein Inaktivitäts-Timeout: ein Server, der die
@@ -994,7 +665,7 @@ def _post_with_deadline(url: str, headers, payload, timeout):
     bei Überschreitung läuft der verwaiste Thread im Hintergrund aus und wir eskalieren.
     """
     ex = concurrent.futures.ThreadPoolExecutor(max_workers=1)
-    fut = ex.submit(requests.post, url, headers=headers,
+    fut = ex.submit(requests.post, _OPENROUTER_URL, headers=headers,
                     json=payload, timeout=timeout)
     try:
         resp = fut.result(timeout=timeout)
@@ -1006,9 +677,8 @@ def _post_with_deadline(url: str, headers, payload, timeout):
 
 
 def _openrouter_chat(model, messages, ai_cfg, api_key, timeout=None) -> dict | None:
-    """Chat-Call: zuerst OpenRouter (primär), dann lokale llama.cpp-Server als Fallback.
-
-    Returns das Verdikt-Dict oder None wenn alle Endpunkte scheitern (graceful)."""
+    """Einzelner OpenRouter-Call mit strukturierter JSON-Antwort.
+    Returns das Verdikt-Dict oder None bei jedem Fehler/Timeout (graceful)."""
     if timeout is None:
         timeout = ai_cfg.get("attempt_timeout", ai_cfg.get("request_timeout", 20))
     payload = {
@@ -1017,56 +687,23 @@ def _openrouter_chat(model, messages, ai_cfg, api_key, timeout=None) -> dict | N
         "response_format": _RESPONSE_SCHEMA,
         "temperature": 0,
     }
-
-    # --- 1. OpenRouter (primär) ---
     headers = {
         "Authorization": f"Bearer {api_key}",
         "Content-Type": "application/json",
         "X-Title": "integrity-scanner",
     }
     try:
-        resp = _post_with_deadline(_OPENROUTER_URL, headers, payload, timeout)
-        if resp.status_code == 200:
-            content = resp.json()["choices"][0]["message"]["content"]
-            verdict = json.loads(content)
-            if verdict.get("category") not in _CATEGORIES:
-                logger.warning("OpenRouter (%s): unbekannte Kategorie %r",
-                               model, verdict.get("category"))
-            else:
-                logger.debug("KI-Antwort von OpenRouter (Modell: %s).", model)
-                return verdict
-        else:
-            logger.info("OpenRouter HTTP %s (%s) — Fallback zu lokalen Servern.",
-                        resp.status_code, model)
-    except (requests.exceptions.RequestException, KeyError, ValueError,
-            json.JSONDecodeError) as exc:
-        logger.debug("OpenRouter-Call fehlgeschlagen (%s): %s", model, exc)
-
-    # --- 2. Lokale Server (Fallback) ---
-    headers = {"Content-Type": "application/json"}
-    if api_key:
-        headers["Authorization"] = f"Bearer {api_key}"
-    for base_url in _LOCAL_SERVERS:
-        url = f"{base_url}/v1/chat/completions"
-        try:
-            resp = _post_with_deadline(url, headers, payload, timeout)
-            if resp.status_code == 200:
-                content = resp.json()["choices"][0]["message"]["content"]
-                verdict = json.loads(content)
-                if verdict.get("category") not in _CATEGORIES:
-                    logger.warning("KI (%s@%s): unbekannte Kategorie %r",
-                                   model, base_url, verdict.get("category"))
-                    continue
-                logger.debug("KI-Antwort von %s (Modell: %s).", base_url, model)
-                return verdict
-            elif resp.status_code == 503:
-                logger.info("KI (%s@%s): Modell wird geladen — nächster Server.", model, base_url)
-            else:
-                logger.warning("KI HTTP %s (%s@%s): %s",
-                               resp.status_code, model, base_url, resp.text[:200])
-        except (requests.exceptions.RequestException, KeyError, ValueError,
-                json.JSONDecodeError) as exc:
-            logger.debug("KI-Call fehlgeschlagen (%s@%s): %s", model, base_url, exc)
-
-    logger.warning("KI: OpenRouter + alle lokalen Server erfolglos für Modell %s.", model)
-    return None
+        resp = _post_with_deadline(headers, payload, timeout)
+        if resp.status_code != 200:
+            logger.warning("OpenRouter HTTP %s (%s): %s", resp.status_code, model, resp.text[:200])
+            return None
+        content = resp.json()["choices"][0]["message"]["content"]
+        verdict = json.loads(content)
+        # Mindest-Validierung
+        if verdict.get("category") not in _CATEGORIES:
+            logger.warning("OpenRouter (%s): unbekannte Kategorie %r", model, verdict.get("category"))
+            return None
+        return verdict
+    except (requests.exceptions.RequestException, KeyError, ValueError, json.JSONDecodeError) as exc:
+        logger.warning("OpenRouter-Call fehlgeschlagen (%s): %s", model, exc)
+        return None

scanner/alerter.py

@@ -2,15 +2,11 @@
 Alert dispatch: e-mail (smtplib) and HTTP webhook.
 
 Alerts are only sent when the assessment level meets or exceeds min_level.
-Duplicate alerts (identical reasons since last send) are suppressed.
 """
-import hashlib
-import json
 import logging
 import os
 import smtplib
 from email.message import EmailMessage
-from pathlib import Path
 
 import requests
 
@@ -20,15 +16,10 @@ from .report import render_markdown
 logger = logging.getLogger(__name__)
 
 _LEVEL_ORDER = {"green": 0, "yellow": 1, "red": 2}
-_ALERT_STATE_FILE = "last_alert.json"
 
 
 def send_alert(report: dict, cfg: dict) -> None:
-    """Send alert if the report's level is at or above min_level.
-
-    Duplicate alerts (identical assessment reasons since last send) are suppressed.
-    The dedup state is persisted in data_dir/last_alert.json per site.
-    """
+    """Send alert if the report's level is at or above min_level."""
     alerting = cfg.get("alerting", {})
     level = report.get("assessment", {}).get("level", "green")
     min_level = alerting.get("min_level", "yellow")
@@ -37,16 +28,6 @@ def send_alert(report: dict, cfg: dict) -> None:
         logger.debug("Alert suppressed: level=%s < min_level=%s", level, min_level)
         return
 
-    # --- Deduplizierung: nur senden wenn sich die Alarm-Gründe geändert haben ---
-    alert_key = _compute_alert_key(report)
-    data_dir = cfg.get("data_dir")
-    if data_dir:
-        last = _load_last_alert(Path(data_dir))
-        if last and last.get("key") == alert_key:
-            logger.info("Alert suppressed (duplicate, reasons unchanged since %s)",
-                        last.get("sent_at", "unknown"))
-            return
-
     subject = _make_subject(report)
     body = _format_body(report)
 
@@ -58,10 +39,6 @@ def send_alert(report: dict, cfg: dict) -> None:
     if webhook_cfg.get("enabled") and webhook_cfg.get("url"):
         _send_webhook(webhook_cfg["url"], report, subject)
 
-    # Zustand speichern — für nächste Dedup-Runde
-    if data_dir:
-        _save_last_alert(Path(data_dir), alert_key)
-
 
 # ---------------------------------------------------------------------------
 # Formatting
@@ -135,40 +112,3 @@ def _send_webhook(url: str, report: dict, subject: str) -> None:
         logger.info("Webhook delivered: HTTP %s", resp.status_code)
     except Exception as exc:
         logger.error("Webhook alert failed: %s", exc)
-
-
-# ---------------------------------------------------------------------------
-# Deduplication state (last_alert.json in data_dir)
-# ---------------------------------------------------------------------------
-
-def _compute_alert_key(report: dict) -> str:
-    """Stable hash of the assessment reasons + level for deduplication."""
-    payload = {
-        "level": report.get("assessment", {}).get("level", "green"),
-        "score": report.get("assessment", {}).get("score", 0),
-        "reasons": sorted(report.get("assessment", {}).get("reasons", [])),
-    }
-    return hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()
-
-
-def _load_last_alert(data_dir: Path) -> dict:
-    """Load previous alert state. Returns {} on missing/corrupt file."""
-    path = data_dir / _ALERT_STATE_FILE
-    try:
-        return json.loads(path.read_text())
-    except (FileNotFoundError, json.JSONDecodeError, OSError):
-        return {}
-
-
-def _save_last_alert(data_dir: Path, alert_key: str) -> None:
-    """Persist current alert state for next dedup round."""
-    from datetime import datetime, timezone
-    path = data_dir / _ALERT_STATE_FILE
-    state = {
-        "key": alert_key,
-        "sent_at": datetime.now(timezone.utc).isoformat(),
-    }
-    try:
-        path.write_text(json.dumps(state))
-    except OSError as exc:
-        logger.debug("Alert state write failed: %s", exc)

scanner/config.py

@@ -20,7 +20,6 @@ DEFAULT_CONFIG: dict = {
             ".mp4", ".mp3", ".webm", ".avi", ".mov",
         ],
         "exclude_paths": [],
-        "include_paths": [],
         "sitemap": True,
     },
     "scoring": {
@@ -80,10 +79,9 @@ DEFAULT_CONFIG: dict = {
             "url": "",
         },
     },
-    # KI-gestützte semantische Inhaltsanalyse (OpenRouter, primär).
+    # KI-gestützte semantische Inhaltsanalyse (OpenRouter). Optional, opt-in.
     # Hash-gegated: nur neue/geänderte Inhalte lösen einen API-Call aus — siehe
     # ai_ledger.json. Der API-Key kommt NUR aus der Umgebungsvariable (nie in config.yaml).
-    # Lokale Server (localhost:8001, :8002) dienen nur als Dev-Fallback.
     "ai_analysis": {
         "enabled": False,
         "api_key_env": "OPENROUTER_API_KEY",
@@ -140,7 +138,7 @@ DEFAULT_CONFIG: dict = {
         "audio": {
             "enabled": False,
             "models": ["google/gemini-2.5-flash"],
-            "max_files_per_scan": 0,   # 0 = unbegrenzt
+            "max_files_per_scan": 5,
         },
         "video": {
             "enabled": False,

scanner/crawler.py

@@ -63,7 +63,6 @@ def should_crawl(
     base_netloc: str,
     skip_extensions: list[str],
     exclude_paths: tuple | list = (),
-    include_paths: tuple | list = (),
 ) -> bool:
     """Return True if url should be followed during a crawl."""
     p = urlparse(url)
@@ -75,8 +74,6 @@ def should_crawl(
             return False
     if any(p.path.startswith(ep) for ep in exclude_paths):
         return False
-    if include_paths and not any(p.path.startswith(ip) for ip in include_paths):
-        return False
     # Skip URLs whose query carries calendar/session/tracking noise
     if p.query and any(_is_noise_param(k) for k in parse_qs(p.query)):
         return False
@@ -91,7 +88,6 @@ class Crawler:
         self.delay: float = cfg["crawl"]["delay_seconds"]
         self.skip_ext: list[str] = cfg["crawl"]["skip_extensions"]
         self.exclude_paths: list[str] = cfg["crawl"].get("exclude_paths", [])
-        self.include_paths: list[str] = cfg["crawl"].get("include_paths", [])
         self.timeout: int = cfg["request_timeout"]
         self.headers = {"User-Agent": cfg["user_agent"]}
         self.session = requests.Session()
@@ -108,22 +104,7 @@ class Crawler:
           - errors: list of {url, error}
         """
         seen: set[str] = set()
-        if self.include_paths:
-            queue: list[str] = [
-                normalize_url(self.target + p) or (self.target + p)
-                for p in self.include_paths
-                if should_crawl(
-                    normalize_url(self.target + p) or (self.target + p),
-                    self.base_netloc, self.skip_ext,
-                    self.exclude_paths, self.include_paths,
-                )
-            ]
-        else:
-            seed = normalize_url(self.target + "/") or self.target + "/"
-            queue: list[str] = [seed] if should_crawl(
-                seed, self.base_netloc, self.skip_ext,
-                self.exclude_paths, self.include_paths,
-            ) else []
+        queue: list[str] = [normalize_url(self.target + "/") or self.target + "/"]
         pages: list[dict] = []
         errors: list[dict] = []
 
@@ -271,8 +252,7 @@ class Crawler:
                     raw = loc.text.strip() if loc.text else ""
                     url = normalize_url(raw)
                     if url and should_crawl(
-                        url, self.base_netloc, self.skip_ext, self.exclude_paths,
-                        self.include_paths,
+                        url, self.base_netloc, self.skip_ext, self.exclude_paths
                     ) and url not in seen and url not in queue:
                         queue.append(url)
                         added += 1
@@ -287,8 +267,7 @@ class Crawler:
             for tag in soup.find_all("a", href=True):
                 href = tag["href"].strip()
                 abs_url = normalize_url(urljoin(base_url, href))
-                if abs_url and should_crawl(abs_url, self.base_netloc, self.skip_ext,
-                                            self.exclude_paths, self.include_paths):
+                if abs_url and should_crawl(abs_url, self.base_netloc, self.skip_ext, self.exclude_paths):
                     result.append(abs_url)
         except Exception as exc:
             logger.debug("Link extraction failed on %s: %s", base_url, exc)

scanner/differ.py

@@ -148,7 +148,7 @@ def compare_snapshots(baseline_pages: dict, current_pages: dict, cfg: dict) -> d
 
         # Per-type link diff
         link_diff: dict = {}
-        for ltype in ("a", "script", "iframe", "link_rel", "form", "audio", "video"):
+        for ltype in ("a", "script", "iframe", "link_rel", "form"):
             old_links = old.get("links", {}).get(ltype, [])
             new_links = new.get("links", {}).get(ltype, [])
             ld = diff_link_set(old_links, new_links)

scanner/extractor.py

@@ -124,8 +124,6 @@ def _extract_links(soup: BeautifulSoup, base_url: str) -> dict:
         "link_rel": [],
         "script": [],
         "img": [],
-        "audio": [],
-        "video": [],
         "iframe": [],
         "form": [],
         "meta_refresh": [],
@@ -164,19 +162,6 @@ def _extract_links(soup: BeautifulSoup, base_url: str) -> dict:
             u = strip_cache_params(u)
             result["img"].append({"url": u, "class": classify(u)})
 
-    for tag in soup.find_all(["audio", "video"]):
-        media_type = tag.name  # "audio" or "video"
-        if tag.get("src"):
-            u = resolve(tag["src"])
-            if u:
-                u = strip_cache_params(u)
-                result[media_type].append({"url": u, "class": classify(u)})
-        for src_tag in tag.find_all("source", src=True):
-            u = resolve(src_tag["src"])
-            if u:
-                u = strip_cache_params(u)
-                result[media_type].append({"url": u, "class": classify(u)})
-
     for tag in soup.find_all("iframe", src=True):
         u = resolve(tag["src"])
         if u:

scanner/report.py

@@ -20,16 +20,11 @@ def render_markdown(r: dict) -> str:
     symbol, headline = ampel(lvl)
     ai = r.get("ai_result")
 
-    config_path = r.get("config_path")
-    scan_cmd = (f"python -m scanner --config {config_path} scan"
-                if config_path else f"python -m scanner {target} scan")
-
     # ---- Klartext-Teil (für den Betreiber) ----
     lines = [
         f"# Prüfbericht für Ihre Website — {target or '?'}",
         "",
-        f"**Erstellt:** {r.get('generated_at', '?')}  ",
-        f"**Kommando:** `{scan_cmd}`",
+        f"**Erstellt:** {r.get('generated_at', '?')}",
         "",
         f"## {symbol} {headline}",
         "",

tests/test_alerter.py

@@ -1,20 +1,10 @@
 """Tests for scanner/alerter.py — alert dispatch (e-mail + webhook)."""
-import json
 import os
-from pathlib import Path
-from tempfile import TemporaryDirectory
 from unittest.mock import MagicMock, call, patch
 
 import pytest
 
-from scanner.alerter import (
-    _compute_alert_key,
-    _load_last_alert,
-    _save_last_alert,
-    _make_subject,
-    _format_body,
-    send_alert,
-)
+from scanner.alerter import _make_subject, _format_body, send_alert
 
 
 # ---------------------------------------------------------------------------
@@ -48,9 +38,8 @@ def _cfg(
     pw_env: str = "TEST_SMTP_PW",
     webhook_enabled: bool = False,
     webhook_url: str = "",
-    data_dir: str | None = None,
 ) -> dict:
-    cfg: dict = {
+    return {
         "alerting": {
             "min_level": min_level,
             "email": {
@@ -69,9 +58,6 @@ def _cfg(
             },
         }
     }
-    if data_dir is not None:
-        cfg["data_dir"] = data_dir
-    return cfg
 
 
 # ---------------------------------------------------------------------------
@@ -298,93 +284,3 @@ class TestSendAlertWebhook:
         # green is below min_level yellow → suppressed anyway, but let's test enabled+green=red path
         cfg["alerting"]["min_level"] = "green"
         send_alert(_report("green"), cfg)
-
-
-# ---------------------------------------------------------------------------
-# Deduplication
-# ---------------------------------------------------------------------------
-
-class TestComputeAlertKey:
-    def test_same_reasons_same_key(self):
-        r1 = _report("red", "https://a.de")
-        r2 = _report("red", "https://b.de")  # anderer Target, gleiche reasons/level/score
-        assert _compute_alert_key(r1) == _compute_alert_key(r2)
-
-    def test_different_score_different_key(self):
-        r1 = _report("yellow")  # score=30
-        r2 = _report("red")     # score=80
-        assert _compute_alert_key(r1) != _compute_alert_key(r2)
-
-    def test_different_reasons_different_key(self):
-        r1 = _report("yellow")
-        r1["assessment"]["reasons"] = ["Grund A"]
-        r2 = _report("yellow")
-        r2["assessment"]["reasons"] = ["Grund B"]
-        assert _compute_alert_key(r1) != _compute_alert_key(r2)
-
-    def test_reasons_order_independent(self):
-        r1 = _report("yellow")
-        r1["assessment"]["reasons"] = ["B", "A", "C"]
-        r2 = _report("yellow")
-        r2["assessment"]["reasons"] = ["C", "A", "B"]  # gleiche, andere Reihenfolge
-        assert _compute_alert_key(r1) == _compute_alert_key(r2)
-
-
-class TestLoadSaveLastAlert:
-    def test_load_missing_file_returns_empty(self):
-        with TemporaryDirectory() as tmp:
-            state = _load_last_alert(Path(tmp))
-            assert state == {}
-
-    def test_load_corrupt_json_returns_empty(self):
-        with TemporaryDirectory() as tmp:
-            (Path(tmp) / "last_alert.json").write_text("{invalid")
-            state = _load_last_alert(Path(tmp))
-            assert state == {}
-
-    def test_save_and_load_roundtrip(self):
-        with TemporaryDirectory() as tmp:
-            d = Path(tmp)
-            _save_last_alert(d, "abc123")
-            state = _load_last_alert(d)
-            assert state["key"] == "abc123"
-            assert "sent_at" in state
-
-
-class TestSendAlertDedup:
-    @patch("scanner.alerter._send_email")
-    def test_first_alert_sent(self, mock_email):
-        with TemporaryDirectory() as tmp:
-            cfg = _cfg(data_dir=tmp)
-            send_alert(_report("yellow"), cfg)
-            mock_email.assert_called_once()
-
-    @patch("scanner.alerter._send_email")
-    def test_identical_second_alert_suppressed(self, mock_email):
-        with TemporaryDirectory() as tmp:
-            cfg = _cfg(data_dir=tmp)
-            report = _report("yellow")
-            send_alert(report, cfg)
-            send_alert(report, cfg)  # gleiche reasons → unterdrückt
-            assert mock_email.call_count == 1
-
-    @patch("scanner.alerter._send_email")
-    def test_changed_reasons_triggers_new_alert(self, mock_email):
-        with TemporaryDirectory() as tmp:
-            cfg = _cfg(data_dir=tmp)
-            r1 = _report("yellow")
-            r1["assessment"]["reasons"] = ["Grund A"]
-            send_alert(r1, cfg)
-
-            r2 = _report("yellow")
-            r2["assessment"]["reasons"] = ["Grund B"]  # anderer Grund
-            send_alert(r2, cfg)
-
-            assert mock_email.call_count == 2
-
-    @patch("scanner.alerter._send_email")
-    def test_no_data_dir_sends_every_time(self, mock_email):
-        cfg = _cfg(data_dir=None)  # kein data_dir → keine Dedup
-        send_alert(_report("yellow"), cfg)
-        send_alert(_report("yellow"), cfg)
-        assert mock_email.call_count == 2

tests/test_main.py

@@ -1,235 +0,0 @@
-"""Tests for scanner/__main__.py — CLI commands, worst_level, file-lock."""
-import json
-from pathlib import Path
-from tempfile import TemporaryDirectory
-from unittest.mock import MagicMock, patch
-
-import pytest
-
-from scanner.__main__ import _worst_level, _LEVEL_ORDER
-
-
-# ---------------------------------------------------------------------------
-# _worst_level
-# ---------------------------------------------------------------------------
-
-class TestWorstLevel:
-    def test_all_green(self):
-        assert _worst_level(["green", "green"]) == "green"
-
-    def test_green_yellow(self):
-        assert _worst_level(["green", "yellow"]) == "yellow"
-
-    def test_green_red(self):
-        assert _worst_level(["green", "red"]) == "red"
-
-    def test_yellow_red(self):
-        assert _worst_level(["yellow", "red"]) == "red"
-
-    def test_red_yellow(self):
-        assert _worst_level(["red", "yellow"]) == "red"
-
-    def test_single_green(self):
-        assert _worst_level(["green"]) == "green"
-
-    def test_empty_returns_green(self):
-        assert _worst_level([]) == "green"
-
-    def test_many_levels(self):
-        assert _worst_level(["green", "green", "yellow", "green", "red"]) == "red"
-
-
-class TestLevelOrder:
-    def test_order_values(self):
-        assert _LEVEL_ORDER["green"] == 0
-        assert _LEVEL_ORDER["yellow"] == 1
-        assert _LEVEL_ORDER["red"] == 2
-
-    def test_order_increasing(self):
-        assert _LEVEL_ORDER["green"] < _LEVEL_ORDER["yellow"] < _LEVEL_ORDER["red"]
-
-
-# ---------------------------------------------------------------------------
-# cmd_scan file-lock
-# ---------------------------------------------------------------------------
-
-class TestCmdScanFileLock:
-    @patch("scanner.__main__._crawl_and_extract")
-    @patch("scanner.__main__.BaselineManager")
-    def test_scan_without_data_dir_no_crash(self, mock_bm, mock_crawl):
-        """cmd_scan ohne data_dir sollte keinen Lock versuchen."""
-        from scanner.__main__ import cmd_scan
-
-        mock_crawl.return_value = ([], [])
-        mock_bm_instance = MagicMock()
-        mock_bm_instance.baseline_exists.return_value = False
-        mock_bm.return_value = mock_bm_instance
-
-        cfg = {"target": "https://test.de", "data_dir": None}
-        args = MagicMock(config="test/config.yaml")
-        result = cmd_scan(args, cfg)
-        assert result == 0
-
-    @patch("scanner.__main__._crawl_and_extract")
-    @patch("scanner.__main__.BaselineManager")
-    def test_scan_with_data_dir_creates_lock(self, mock_bm, mock_crawl):
-        """cmd_scan mit data_dir erstellt .scan.lock."""
-        from scanner.__main__ import cmd_scan
-        import fcntl
-
-        with TemporaryDirectory() as tmp:
-            lock_path = Path(tmp) / ".scan.lock"
-            mock_crawl.return_value = ([], [])
-            mock_bm_instance = MagicMock()
-            mock_bm_instance.baseline_exists.return_value = False
-            mock_bm.return_value = mock_bm_instance
-
-            cfg = {"target": "https://test.de", "data_dir": tmp}
-            args = MagicMock(config="test/config.yaml")
-            result = cmd_scan(args, cfg)
-
-            assert result == 0
-            # Lock wurde erstellt und nach Scan wieder entfernt (oder leer)
-            assert lock_path.exists()
-
-
-# ---------------------------------------------------------------------------
-# cmd_scan baseline_exists
-# ---------------------------------------------------------------------------
-
-class TestCmdScanNoBaseline:
-    @patch("scanner.__main__._crawl_and_extract")
-    @patch("scanner.__main__.BaselineManager")
-    def test_no_baseline_returns_zero(self, mock_bm, mock_crawl):
-        from scanner.__main__ import cmd_scan
-
-        with TemporaryDirectory() as tmp:
-            mock_crawl.return_value = ([{"url": "https://test.de/"}], [])
-            mock_bm_instance = MagicMock()
-            mock_bm_instance.baseline_exists.return_value = False
-            mock_bm.return_value = mock_bm_instance
-
-            cfg = {"target": "https://test.de", "data_dir": tmp}
-            args = MagicMock(config="test/config.yaml")
-            result = cmd_scan(args, cfg)
-            assert result == 0
-
-
-# ---------------------------------------------------------------------------
-# _crawl_and_extract
-# ---------------------------------------------------------------------------
-
-class TestCrawlAndExtract:
-    @patch("scanner.__main__.Crawler")
-    def test_empty_crawl(self, mock_crawler_cls):
-        from scanner.__main__ import _crawl_and_extract
-
-        mock_crawler = MagicMock()
-        mock_crawler.crawl.return_value = {
-            "target": "https://test.de",
-            "pages": [],
-            "errors": [{"url": "https://test.de/", "error": "timeout"}],
-        }
-        mock_crawler_cls.return_value = mock_crawler
-
-        cfg = {
-            "target": "https://test.de",
-            "crawl": {"max_pages": 10, "delay_seconds": 0.1, "skip_extensions": [],
-                       "exclude_paths": [], "include_paths": [], "sitemap": False},
-            "request_timeout": 10,
-            "user_agent": "test",
-        }
-        pages, errors = _crawl_and_extract(cfg)
-        assert len(pages) == 0
-        assert len(errors) == 1
-
-    @patch("scanner.__main__.Crawler")
-    def test_crawl_with_pages(self, mock_crawler_cls):
-        from scanner.__main__ import _crawl_and_extract
-
-        mock_crawler = MagicMock()
-        mock_crawler.crawl.return_value = {
-            "target": "https://test.de",
-            "pages": [
-                {"url": "https://test.de/", "status": 200,
-                 "html": "<html><body>Test</body></html>", "headers": {}},
-            ],
-            "errors": [],
-        }
-        mock_crawler_cls.return_value = mock_crawler
-
-        cfg = {
-            "target": "https://test.de",
-            "crawl": {"max_pages": 10, "delay_seconds": 0.1, "skip_extensions": [],
-                       "exclude_paths": [], "include_paths": [], "sitemap": False},
-            "request_timeout": 10,
-            "user_agent": "test",
-        }
-        pages, errors = _crawl_and_extract(cfg)
-        assert len(pages) == 1
-        assert len(errors) == 0
-        assert "text" in pages[0]  # extract_page wurde aufgerufen
-
-
-# ---------------------------------------------------------------------------
-# _run_content_check
-# ---------------------------------------------------------------------------
-
-class TestRunContentCheck:
-    def test_empty_baseline_and_snapshot(self):
-        from scanner.__main__ import _run_content_check
-
-        cfg = {
-            "target": "https://test.de",
-            "config_dir": "/tmp/test/config",
-            "security_headers": [],
-        }
-        snap = {"pages": {}, "errors": []}
-        baseline = {"manifest": {}, "pages": {}}
-
-        diff, assessment, wl_violations, missing_sec = _run_content_check(cfg, snap, baseline)
-        assert assessment["level"] == "green"
-        assert assessment["score"] == 0
-
-
-# ---------------------------------------------------------------------------
-# Score aggregation in cmd_scan
-# ---------------------------------------------------------------------------
-
-class TestScoreAggregation:
-    @patch("scanner.__main__._crawl_and_extract")
-    @patch("scanner.__main__.BaselineManager")
-    @patch("scanner.__main__._run_content_check")
-    @patch("scanner.__main__._build_report")
-    @patch("scanner.__main__._write_report")
-    @patch("scanner.__main__._print_summary")
-    @patch("scanner.__main__.send_alert")
-    def test_scan_aggregates_levels(self, mock_alert, mock_print, mock_write,
-                                     mock_build, mock_check, mock_bm, mock_crawl):
-        from scanner.__main__ import cmd_scan
-
-        with TemporaryDirectory() as tmp:
-            mock_crawl.return_value = ([{"url": "https://test.de/"}], [])
-
-            mock_bm_instance = MagicMock()
-            mock_bm_instance.baseline_exists.return_value = True
-            mock_bm_instance.load_baseline.return_value = {"manifest": {}, "pages": {}}
-            mock_bm.return_value = mock_bm_instance
-
-            # content check: yellow
-            mock_check.return_value = (
-                {"new_external_domains": [], "page_diffs": []},
-                {"level": "yellow", "score": 30, "reasons": ["Test"], "exit_code": 1},
-                [], {}
-            )
-
-            cfg = {
-                "target": "https://test.de",
-                "data_dir": tmp,
-                "reports_dir": f"{tmp}/reports",
-                "ai_analysis": {},
-                "periodic_checks": {"enabled": False},
-            }
-            args = MagicMock(config="test/config.yaml")
-            result = cmd_scan(args, cfg)
-            assert result == 1  # exit_code für yellow