Dieter Schlüter 9801ce6fc4 feat: sitemap seeding — discover orphan pages not reachable via links ... Reads robots.txt Sitemap directive and fetches sitemap.xml / sitemap_index.xml before crawling. All listed URLs are added to the crawl queue so pages without any inbound links are still scanned. Configurable via crawl.sitemap. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>		2026-06-12 17:58:52 +02:00
..
config	refactor: rename example.com template to meine-seite.de	2026-06-12 15:43:44 +02:00
config.yaml	feat: sitemap seeding — discover orphan pages not reachable via links	2026-06-12 17:58:52 +02:00