f964e41201
Some servers return 403 for the scanner's bot user-agent but serve the page normally to browsers. On a 403 response, the crawler now retries with a browser-like UA; if the server then returns 2xx, the page is crawled normally (HTML extracted, status recorded as 2xx). Mirrors the same fix already applied to check_external_links. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| __main__.py | ||
| alerter.py | ||
| baseline.py | ||
| checker.py | ||
| config.py | ||
| crawler.py | ||
| differ.py | ||
| extractor.py | ||
| plain.py | ||