feat: exclude_paths — skip dynamic areas (RSS feeds, news archives) from scan

Adds crawl.exclude_paths config option (list of URL path prefixes).
Matching URLs are excluded at two levels:
- Crawler: not fetched at all (saves HTTP requests)
- Differ: filtered from both baseline and current before comparison,
  so legacy baseline entries under excluded paths don't appear as
  "missing URLs" after the option is added retroactively

Example config:
  crawl:
    exclude_paths: ["/rss/", "/feed/", "/news/archiv/"]

8 new tests (176 total).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Your Name 2026-06-12 13:14:06 +02:00
commit 3a44cfb515
6 changed files with 93 additions and 2 deletions

View file

@ -19,6 +19,7 @@ DEFAULT_CONFIG: dict = {
".pdf", ".zip", ".woff", ".woff2", ".ttf", ".otf", ".eot",
".mp4", ".mp3", ".webm", ".avi", ".mov",
],
"exclude_paths": [],
},
"scoring": {
"new_external_domain": 50,