fix: Bugfixes, toten Code entfernt, Cache-Normalisierung & Session
Echte Bugs: - extractor: JSON-LD-Crash bei Nicht-Dict-Items (isinstance-Guard) - baseline/__main__: Crawl-Fehler ins Snapshot-Manifest -> erscheinen im Report - __main__: Whitelist-/Header-/Webshell-Checks nur noch auf status==200 - crawler: Noise-Param-Regex auf Voll-Key (view=/value= nicht mehr verworfen) - differ/__main__: unerwartetes JSON-LD via eigenem Kanal, auch auf unveraenderten Seiten erkannt, kein Re-Alert auf bereits genehmigte Typen Aufgeraeumt: - checker: check_internal_paths, find_broken_pages, canonical_is_hijacked, check_jsonld_types entfernt (nicht verdrahtet) - allowed_paths.yaml-Logik und tote Scoring-Keys entfernt - tote Imports entfernt Neuer aktiver Schutz: - Webshell-/Backdoor-Dateinamen-Check verdrahtet (suspicious_filename: 60). Regex wortgrenzen-verankert gegen Fehlalarm auf PSEMailerAntispam.js Effizienz/Struktur: - crawler nutzt requests.Session (keep-alive) - Cache-Buster-Normalisierung an einer Stelle (Extraktion) -> stabile Snapshots, differ wieder reiner Set-Diff Tests: 111 gruen (neu: test_checker.py + Regressionstests) CLAUDE.md aktualisiert Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
b3ef3a90d4
commit
68243a97b9
13 changed files with 323 additions and 153 deletions
|
|
@ -31,9 +31,11 @@ class BaselineManager:
|
|||
# Snapshot operations
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def save_snapshot(self, pages: list[dict]) -> Path:
|
||||
def save_snapshot(self, pages: list[dict], errors: list[dict] | None = None) -> Path:
|
||||
"""
|
||||
Persist a crawl+extraction result as a timestamped snapshot.
|
||||
Crawl errors (network failures) are recorded in the manifest so they
|
||||
survive into the comparison/report stage.
|
||||
Returns the snapshot directory path.
|
||||
"""
|
||||
ts = _utc_stamp()
|
||||
|
|
@ -45,6 +47,7 @@ class BaselineManager:
|
|||
"timestamp": ts,
|
||||
"page_count": len(pages),
|
||||
"urls": [p["url"] for p in pages],
|
||||
"errors": errors or [],
|
||||
}
|
||||
(snap_dir / "manifest.json").write_text(
|
||||
json.dumps(manifest, indent=2, ensure_ascii=False), encoding="utf-8"
|
||||
|
|
@ -71,7 +74,7 @@ class BaselineManager:
|
|||
def load_snapshot(self, snap_dir: Optional[Path] = None) -> dict:
|
||||
"""
|
||||
Load a snapshot (default: latest).
|
||||
Returns {manifest, pages: {url: page_dict}, dir: Path}.
|
||||
Returns {manifest, pages: {url: page_dict}, errors: list, dir: Path}.
|
||||
"""
|
||||
if snap_dir is None:
|
||||
snap_dir = self.latest_snapshot_dir()
|
||||
|
|
@ -85,7 +88,12 @@ class BaselineManager:
|
|||
page = json.loads(f.read_text(encoding="utf-8"))
|
||||
pages[page["url"]] = page
|
||||
|
||||
return {"manifest": manifest, "pages": pages, "dir": snap_dir}
|
||||
return {
|
||||
"manifest": manifest,
|
||||
"pages": pages,
|
||||
"errors": manifest.get("errors", []),
|
||||
"dir": snap_dir,
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Baseline operations
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue