dschlueter/integrity_scanner_fuer_statische_Webseiten
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

feat: sitemap seeding — discover orphan pages not reachable via links

Browse source 
Reads robots.txt Sitemap directive and fetches sitemap.xml / sitemap_index.xml
before crawling. All listed URLs are added to the crawl queue so pages
without any inbound links are still scanned. Configurable via crawl.sitemap.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Dieter Schlüter 2026-06-12 17:58:52 +02:00
commit 9801ce6fc4
3 changed files with 81 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

1
scanner/config.py
View file

@ -20,6 +20,7 @@ DEFAULT_CONFIG: dict = {
".mp4", ".mp3", ".webm", ".avi", ".mov",
],
"exclude_paths": [],
"sitemap": True,
},
"scoring": {
"new_external_domain": 50,