fix: Bugfixes, toten Code entfernt, Cache-Normalisierung & Session

Echte Bugs:
- extractor: JSON-LD-Crash bei Nicht-Dict-Items (isinstance-Guard)
- baseline/__main__: Crawl-Fehler ins Snapshot-Manifest -> erscheinen im Report
- __main__: Whitelist-/Header-/Webshell-Checks nur noch auf status==200
- crawler: Noise-Param-Regex auf Voll-Key (view=/value= nicht mehr verworfen)
- differ/__main__: unerwartetes JSON-LD via eigenem Kanal, auch auf
  unveraenderten Seiten erkannt, kein Re-Alert auf bereits genehmigte Typen

Aufgeraeumt:
- checker: check_internal_paths, find_broken_pages, canonical_is_hijacked,
  check_jsonld_types entfernt (nicht verdrahtet)
- allowed_paths.yaml-Logik und tote Scoring-Keys entfernt
- tote Imports entfernt

Neuer aktiver Schutz:
- Webshell-/Backdoor-Dateinamen-Check verdrahtet (suspicious_filename: 60).
  Regex wortgrenzen-verankert gegen Fehlalarm auf PSEMailerAntispam.js

Effizienz/Struktur:
- crawler nutzt requests.Session (keep-alive)
- Cache-Buster-Normalisierung an einer Stelle (Extraktion) -> stabile
  Snapshots, differ wieder reiner Set-Diff

Tests: 111 gruen (neu: test_checker.py + Regressionstests)
CLAUDE.md aktualisiert

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

scanner/checker.py

@@ -11,8 +11,11 @@ from urllib.parse import urlparse
 
 import yaml
 
+# Webshell / backdoor filename markers. Word boundaries are essential:
+# without them "mailer" matches the legitimate "PSEMailerAntispam.js" and
+# "gate" matches "navigate", producing false positives on every scan.
 _SUSPICIOUS_FILENAME_RE = re.compile(
-    r"(shell|backdoor|cmd|upload|gate|mailer|wso|c99|r57|b374k)",
+    r"\b(shell|backdoor|webshell|c99|r57|wso|b374k|cmd|reverse_shell|bindshell)\b",
     re.I,
 )
 
@@ -87,30 +90,6 @@ def check_links_against_whitelist(
     return violations
 
 
-def check_internal_paths(
-    page: dict,
-    allowed_paths: set[str],
-) -> list[dict]:
-    """
-    Flag internal links to paths not in the allowed_paths whitelist.
-    If allowed_paths is empty, the check is skipped.
-    """
-    if not allowed_paths:
-        return []
-    violations: list[dict] = []
-    for link in page.get("links", {}).get("a", []):
-        if link.get("class") != "internal":
-            continue
-        path = urlparse(link.get("url", "")).path
-        if path and path not in allowed_paths:
-            violations.append({
-                "type": "unlisted_internal_path",
-                "url": link.get("url", ""),
-                "path": path,
-            })
-    return violations
-
-
 # ---------------------------------------------------------------------------
 # Security headers
 # ---------------------------------------------------------------------------
@@ -122,38 +101,6 @@ def check_security_headers(response_headers: dict, required: list[str]) -> list[
     return [h for h in required if h.lower() not in present]
 
 
-# ---------------------------------------------------------------------------
-# Broken links
-# ---------------------------------------------------------------------------
-
-def find_broken_pages(pages: dict) -> list[dict]:
-    """Return all pages whose HTTP status is >= 400 (broken / server error)."""
-    return [
-        {"url": p["url"], "status": p.get("status", 0)}
-        for p in pages.values()
-        if p.get("status", 0) >= 400
-    ]
-
-
-# ---------------------------------------------------------------------------
-# Canonical hijack
-# ---------------------------------------------------------------------------
-
-def canonical_is_hijacked(page: dict, base_netloc: str) -> bool:
-    """
-    Return True if the canonical URL points to a different domain.
-    Canonical appearing in both <link rel=canonical> and metadata is checked.
-    """
-    canonical = (
-        page.get("links", {}).get("canonical")
-        or page.get("metadata", {}).get("canonical")
-    )
-    if not canonical:
-        return False
-    netloc = urlparse(canonical).netloc
-    return bool(netloc) and netloc != base_netloc
-
-
 # ---------------------------------------------------------------------------
 # Suspicious filenames
 # ---------------------------------------------------------------------------
@@ -162,16 +109,3 @@ def is_suspicious_url(url: str) -> bool:
     """Flag URLs with names typical for webshells / backdoors."""
     filename = urlparse(url).path.rsplit("/", 1)[-1]
     return bool(_SUSPICIOUS_FILENAME_RE.search(filename))
-
-
-# ---------------------------------------------------------------------------
-# JSON-LD type check
-# ---------------------------------------------------------------------------
-
-def check_jsonld_types(page: dict, allowed_types: list[str]) -> list[dict]:
-    """Return JSON-LD entries whose @type is not in the allowed list."""
-    allowed = set(allowed_types)
-    return [
-        j for j in page.get("jsonld", [])
-        if j.get("type") not in allowed
-    ]