feat: exclude_paths — skip dynamic areas (RSS feeds, news archives) from scan
Adds crawl.exclude_paths config option (list of URL path prefixes).
Matching URLs are excluded at two levels:
- Crawler: not fetched at all (saves HTTP requests)
- Differ: filtered from both baseline and current before comparison,
so legacy baseline entries under excluded paths don't appear as
"missing URLs" after the option is added retroactively
Example config:
crawl:
exclude_paths: ["/rss/", "/feed/", "/news/archiv/"]
8 new tests (176 total).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
b01344dcd8
commit
3a44cfb515
6 changed files with 93 additions and 2 deletions
|
|
@ -14,6 +14,12 @@ crawl:
|
|||
max_pages: 200
|
||||
delay_seconds: 1.0
|
||||
respect_robots_txt: false
|
||||
# Pfade, die vom Crawl und vom Vergleich komplett ausgeschlossen werden.
|
||||
# Nützlich für Bereiche mit gewollt häufig wechselnden Inhalten (RSS-Feeds, News-Archive).
|
||||
# Beispiel: ["/rss/", "/feed/", "/news/archiv/"]
|
||||
# Alle URLs, deren Pfad mit einem dieser Einträge beginnt, werden ignoriert.
|
||||
exclude_paths: []
|
||||
|
||||
skip_extensions:
|
||||
- ".jpg"
|
||||
- ".jpeg"
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue