fix: retry external link check with browser UA on 403
Servers that block bot user-agents return 403 even for live URLs. On a 403 response, the checker now retries with a browser-like UA; if the server then responds with 2xx/3xx, the link is counted as OK. This eliminates false positives from bot-blocking (e.g. bergbauspuren-bredelar.de). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
bed20db8bd
commit
22ed1570a4
2 changed files with 30 additions and 0 deletions
|
|
@ -76,6 +76,19 @@ class TestCheckExternalLinks:
|
|||
assert len(results) == 3
|
||||
assert inst.head.call_count == 3
|
||||
|
||||
def test_403_retries_with_browser_ua_and_succeeds(self):
|
||||
"""Bot-blocking 403 → retry with browser UA → 200 = link is OK."""
|
||||
with patch("scanner.checker.requests.Session") as MockSession:
|
||||
inst = MockSession.return_value
|
||||
ok_resp = _mock_response(200, "https://example.com/")
|
||||
inst.head.side_effect = [_mock_response(403), ok_resp]
|
||||
inst.close = MagicMock()
|
||||
|
||||
results = check_external_links(["https://example.com/"], delay=0)
|
||||
|
||||
assert results["https://example.com/"]["status"] == 200
|
||||
assert inst.head.call_count == 2
|
||||
|
||||
def test_empty_list_returns_empty(self):
|
||||
with patch("scanner.checker.requests.Session"):
|
||||
results = check_external_links([], delay=0)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue