dschlueter/integrity_scanner_fuer_statische_Webseiten
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

fix: retry external link check with browser UA on 403

Browse source 
Servers that block bot user-agents return 403 even for live URLs.
On a 403 response, the checker now retries with a browser-like UA;
if the server then responds with 2xx/3xx, the link is counted as OK.
This eliminates false positives from bot-blocking (e.g. bergbauspuren-bredelar.de).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Your Name 2026-06-12 11:40:27 +02:00
commit 22ed1570a4
2 changed files with 30 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

17
scanner/checker.py
View file

@ -16,6 +16,11 @@ import yaml
logger = logging.getLogger(__name__)
_BROWSER_UA = (
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
)
# Webshell / backdoor filename markers. Word boundaries are essential:
# without them "mailer" matches the legitimate "PSEMailerAntispam.js" and
# "gate" matches "navigate", producing false positives on every scan.
@ -138,6 +143,18 @@ def check_external_links(
if resp.status_code == 405:
resp = session.get(url, timeout=timeout, allow_redirects=True, stream=True)
resp.close()
# 403 often means bot-blocking, not a broken link — retry with browser UA
if resp.status_code == 403:
try:
r2 = session.head(url, timeout=timeout, allow_redirects=True,
headers={"User-Agent": _BROWSER_UA})
if r2.status_code == 405:
r2 = session.get(url, timeout=timeout, allow_redirects=True,
stream=True, headers={"User-Agent": _BROWSER_UA})
r2.close()
resp = r2
except requests.exceptions.RequestException:
pass # keep original 403
results[url] = {"status": resp.status_code, "final_url": resp.url, "error": None}
except requests.exceptions.RequestException as exc:
results[url] = {"status": None, "final_url": None, "error": str(exc)}