Some servers return 403 for the scanner's bot user-agent but serve
the page normally to browsers. On a 403 response, the crawler now
retries with a browser-like UA; if the server then returns 2xx, the
page is crawled normally (HTML extracted, status recorded as 2xx).
Mirrors the same fix already applied to check_external_links.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
f964e41201