feat: exclude_paths — skip dynamic areas (RSS feeds, news archives) from scan

Adds crawl.exclude_paths config option (list of URL path prefixes).
Matching URLs are excluded at two levels:
- Crawler: not fetched at all (saves HTTP requests)
- Differ: filtered from both baseline and current before comparison,
  so legacy baseline entries under excluded paths don't appear as
  "missing URLs" after the option is added retroactively

Example config:
  crawl:
    exclude_paths: ["/rss/", "/feed/", "/news/archiv/"]

8 new tests (176 total).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Your Name 2026-06-12 13:14:06 +02:00
commit 3a44cfb515
6 changed files with 93 additions and 2 deletions

View file

@ -14,6 +14,12 @@ crawl:
max_pages: 200
delay_seconds: 1.0
respect_robots_txt: false
# Pfade, die vom Crawl und vom Vergleich komplett ausgeschlossen werden.
# Nützlich für Bereiche mit gewollt häufig wechselnden Inhalten (RSS-Feeds, News-Archive).
# Beispiel: ["/rss/", "/feed/", "/news/archiv/"]
# Alle URLs, deren Pfad mit einem dieser Einträge beginnt, werden ignoriert.
exclude_paths: []
skip_extensions:
- ".jpg"
- ".jpeg"