fix: Bugfixes, toten Code entfernt, Cache-Normalisierung & Session

Echte Bugs:
- extractor: JSON-LD-Crash bei Nicht-Dict-Items (isinstance-Guard)
- baseline/__main__: Crawl-Fehler ins Snapshot-Manifest -> erscheinen im Report
- __main__: Whitelist-/Header-/Webshell-Checks nur noch auf status==200
- crawler: Noise-Param-Regex auf Voll-Key (view=/value= nicht mehr verworfen)
- differ/__main__: unerwartetes JSON-LD via eigenem Kanal, auch auf
  unveraenderten Seiten erkannt, kein Re-Alert auf bereits genehmigte Typen

Aufgeraeumt:
- checker: check_internal_paths, find_broken_pages, canonical_is_hijacked,
  check_jsonld_types entfernt (nicht verdrahtet)
- allowed_paths.yaml-Logik und tote Scoring-Keys entfernt
- tote Imports entfernt

Neuer aktiver Schutz:
- Webshell-/Backdoor-Dateinamen-Check verdrahtet (suspicious_filename: 60).
  Regex wortgrenzen-verankert gegen Fehlalarm auf PSEMailerAntispam.js

Effizienz/Struktur:
- crawler nutzt requests.Session (keep-alive)
- Cache-Buster-Normalisierung an einer Stelle (Extraktion) -> stabile
  Snapshots, differ wieder reiner Set-Diff

Tests: 111 gruen (neu: test_checker.py + Regressionstests)
CLAUDE.md aktualisiert

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CLAUDE.md

@@ -46,11 +46,11 @@ Crawler → Extractor → BaselineManager (snapshot)
 ### Module responsibilities
 
 - **`config.py`** — loads `config.yaml`, deep-merges onto `DEFAULT_CONFIG`. All paths (data_dir, reports_dir, logs_dir, config_dir) are resolved to absolute paths by `resolve_paths()` before being passed to other modules.
-- **`crawler.py`** — breadth-first crawl using `requests`. `normalize_url()` is the canonical URL form used everywhere (strips fragment, lowercases host, removes noise query params, sorts remaining params). `should_crawl()` filters out binary assets and tracking URLs.
-- **`extractor.py`** — parses HTML with bs4+lxml. Returns a single dict per page with keys: `text`, `links`, `metadata`, `hidden_content`, `inline_scripts`, `jsonld`, `comment_links`. The `links` dict has typed sub-lists (`a`, `link_rel`, `script`, `img`, `iframe`, `form`, `meta_refresh`, `canonical`).
-- **`baseline.py`** — `BaselineManager` handles two separate stores: `data/snapshots/YYYYMMDD_HHMMSS/` (raw crawl output) and `data/baseline/` (approved reference). Each page is stored as `<url_key(url)>.json`. `approve_*` methods are the only path into the baseline.
-- **`differ.py`** — `compare_snapshots(baseline_pages, current_pages, cfg)` returns a flat diff dict. `score_diff(diff, cfg)` maps it to `{score, level, reasons, exit_code}`. Thresholds: yellow ≥ 20, red ≥ 60 (configurable). A single new external domain = 50 pts (yellow); combined with hidden content = 90 pts (red).
-- **`checker.py`** — stateless helper functions for whitelist enforcement, security-header auditing, canonical hijack detection. Called from `__main__.cmd_check`, not from `differ`.
+- **`crawler.py`** — breadth-first crawl using a shared `requests.Session` (keep-alive). `normalize_url()` is the canonical URL form used everywhere (strips fragment, lowercases host, removes noise query params via whole-key matching in `_is_noise_param`, sorts remaining params). `should_crawl()` filters out binary assets and tracking URLs.
+- **`extractor.py`** — parses HTML with bs4+lxml. Returns a single dict per page with keys: `text`, `links`, `metadata`, `hidden_content`, `inline_scripts`, `jsonld`, `comment_links`. The `links` dict has typed sub-lists (`a`, `link_rel`, `script`, `img`, `iframe`, `form`, `meta_refresh`, `canonical`). `strip_cache_params()` removes cache-buster query params (e.g. `style.css?v=123`) from asset URLs (`link_rel`/`script`/`img`) **at extraction time**, so snapshots store stable URLs — this is the single source of truth for cache-buster normalisation.
+- **`baseline.py`** — `BaselineManager` handles two separate stores: `data/snapshots/YYYYMMDD_HHMMSS/` (raw crawl output) and `data/baseline/` (approved reference). Each page is stored as `<url_key(url)>.json`. Snapshot `manifest.json` also records crawl `errors`. `approve_*` methods are the only path into the baseline.
+- **`differ.py`** — `compare_snapshots(baseline_pages, current_pages, cfg)` returns a flat diff dict (incl. `unexpected_jsonld`, `new_broken_urls`, `known_broken_urls`). `score_diff(diff, cfg)` maps it to `{score, level, reasons, exit_code}`. Thresholds: yellow ≥ 20, red ≥ 60 (configurable). Unexpected JSON-LD `@type` is detected independently of page diffs (flags only types not allowed AND not already in the baseline for that URL). A single new external domain = 50 pts (yellow); combined with hidden content = 90 pts (red).
+- **`checker.py`** — stateless helpers for whitelist enforcement (`check_links_against_whitelist`), security-header auditing (`check_security_headers`, informational only — not scored), and webshell/backdoor filename detection (`is_suspicious_url`, word-boundary anchored to avoid false positives). Called from `__main__.cmd_check` only on `status == 200` pages.
 - **`alerter.py`** — sends alert only when `level >= min_level`. SMTP password is read from the env var named in `smtp_password_env` (default: `SCANNER_SMTP_PASSWORD`).
 - **`__main__.py`** — wires everything together. `cmd_scan` = `cmd_crawl` (crawl + save snapshot) + `cmd_check` (load baseline + snapshot, diff, score, report, alert). Exit codes: 0=green, 1=yellow, 2=red.
 
@@ -62,7 +62,6 @@ data/baseline/pages/<16hex>.json     # one file per approved URL
 data/snapshots/YYYYMMDD_HHMMSS/      # one dir per crawl run
 reports/YYYYMMDD_HHMMSS/report.{json,md}
 config/allowed_external.yaml         # set of permitted external domains
-config/allowed_paths.yaml            # set of permitted internal paths (optional)
 config/ignore_rules.yaml             # normalization selectors + regex patterns
 ```
 
@@ -70,6 +69,7 @@ config/ignore_rules.yaml             # normalization selectors + regex patterns
 
 | Event | Points |
 |---|---|
+| Suspicious filename (webshell/backdoor link) | 60 |
 | New external domain | 50 |
 | Hidden content (CSS-hidden element with links/text) | 40 |
 | Meta-Refresh | 40 |
@@ -80,4 +80,6 @@ config/ignore_rules.yaml             # normalization selectors + regex patterns
 | Comment-embedded link | 25 |
 | Large text addition (>200 chars) | 20 |
 | Missing internal URL | 20 |
-| Missing security header | 5 |
+| New broken link (4xx/5xx, not in baseline) | 20 |
+
+Security headers are audited and reported but **not** scored. Known broken links (already in the baseline) are reported but not scored.