3a44cfb515
Adds crawl.exclude_paths config option (list of URL path prefixes).
Matching URLs are excluded at two levels:
- Crawler: not fetched at all (saves HTTP requests)
- Differ: filtered from both baseline and current before comparison,
so legacy baseline entries under excluded paths don't appear as
"missing URLs" after the option is added retroactively
Example config:
crawl:
exclude_paths: ["/rss/", "/feed/", "/news/archiv/"]
8 new tests (176 total).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_assets.py | ||
| test_baseline.py | ||
| test_checker.py | ||
| test_cloak.py | ||
| test_crawler.py | ||
| test_differ.py | ||
| test_ext_links.py | ||
| test_extractor.py | ||
| test_plain.py | ||
| test_state.py | ||