feat: exclude_paths — skip dynamic areas (RSS feeds, news archives) from scan
Adds crawl.exclude_paths config option (list of URL path prefixes).
Matching URLs are excluded at two levels:
- Crawler: not fetched at all (saves HTTP requests)
- Differ: filtered from both baseline and current before comparison,
so legacy baseline entries under excluded paths don't appear as
"missing URLs" after the option is added retroactively
Example config:
crawl:
exclude_paths: ["/rss/", "/feed/", "/news/archiv/"]
8 new tests (176 total).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
b01344dcd8
commit
3a44cfb515
6 changed files with 93 additions and 2 deletions
|
|
@ -19,6 +19,7 @@ DEFAULT_CONFIG: dict = {
|
|||
".pdf", ".zip", ".woff", ".woff2", ".ttf", ".otf", ".eot",
|
||||
".mp4", ".mp3", ".webm", ".avi", ".mov",
|
||||
],
|
||||
"exclude_paths": [],
|
||||
},
|
||||
"scoring": {
|
||||
"new_external_domain": 50,
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue