fix: retry external link check with browser UA on 403

Servers that block bot user-agents return 403 even for live URLs.
On a 403 response, the checker now retries with a browser-like UA;
if the server then responds with 2xx/3xx, the link is counted as OK.
This eliminates false positives from bot-blocking (e.g. bergbauspuren-bredelar.de).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Your Name 2026-06-12 11:40:27 +02:00
commit 22ed1570a4
2 changed files with 30 additions and 0 deletions

View file

@ -76,6 +76,19 @@ class TestCheckExternalLinks:
assert len(results) == 3
assert inst.head.call_count == 3
def test_403_retries_with_browser_ua_and_succeeds(self):
"""Bot-blocking 403 → retry with browser UA → 200 = link is OK."""
with patch("scanner.checker.requests.Session") as MockSession:
inst = MockSession.return_value
ok_resp = _mock_response(200, "https://example.com/")
inst.head.side_effect = [_mock_response(403), ok_resp]
inst.close = MagicMock()
results = check_external_links(["https://example.com/"], delay=0)
assert results["https://example.com/"]["status"] == 200
assert inst.head.call_count == 2
def test_empty_list_returns_empty(self):
with patch("scanner.checker.requests.Session"):
results = check_external_links([], delay=0)