fix: restliche Befunde der unabhängigen QA (QA2-09 bis QA2-19)
- QA2-09: Formeln in HTML-Tabellen als MathML/Formelbild statt $…$; Inline-Noten dort als HTML-Verweis (Markdown wird in HTML-Blöcken nicht gerendert) - QA2-10: PDF-Weg Markdown → HTML → PDF in README, Bedienungsanleitung und Buch-README. Beim direkten Weg Markdown → LaTeX verwirft Pandoc HTML-Tabellen und <a id>-Ziele. Geprüft: C-Buch 405 Seiten, 34 Tabellen, 728/755 Links. - QA2-11: Hinweis code-language-hint bei einsprachigen Büchern - QA2-12: --strict bricht bei nicht dekodierbarem XHTML ab (Exit 3); Hilfetext präzisiert - QA2-13: verdächtig große Medien werden übersprungen (Fehler media-skipped, Platzhalter) statt die ganze Konvertierung abzubrechen - QA2-14: Formelklasse am <img> selbst wird erkannt - QA2-15: wiederholte Diagnosen werden gezählt (count, ×n); MathML in HTML-Tabellen zählt im Formelbericht mit - QA2-16: Unicode-Buchstaben in Ankernamen bleiben erhalten - QA2-17: keine leeren Links (#) auf den eigenen Kapitelanfang; keine namespaced Attribute (epub:type → type=) in HTML-Tabellen - QA2-18: nicht definierte Entitäten werden gemeldet (undefined-entity) - QA2-19: Bild-Platzhalter ohne Kursivschrift (benachbarte ergaben **) Regressionstests: tests/test_qa2_open.py (11 Tests, vorher alle rot). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
daa99415b5
commit
109612db28
12 changed files with 247 additions and 49 deletions
|
|
@ -161,7 +161,9 @@ kannst du sie für alle übrigen Codeblöcke vorgeben:
|
|||
python3 epub2md.py python-buch.epub output/python-buch --code-language python
|
||||
```
|
||||
|
||||
Das hilft Editoren beim Einfärben des Codes. Nachteil: Auch Programmausgaben und Shell-Befehle im Buch
|
||||
Hat das Programm viele Codeblöcke ohne Sprache gefunden, aber alle erkannten in derselben Sprache,
|
||||
schlägt es diese Option im Bericht von sich aus vor (Hinweis `code-language-hint`). Das hilft Editoren
|
||||
beim Einfärben des Codes. Nachteil: Auch Programmausgaben und Shell-Befehle im Buch
|
||||
werden dann als Python markiert.
|
||||
|
||||
Zu lange Codezeilen sind im gedruckten Buch oft umbrochen und die Fortsetzung mit `↪` markiert. Das
|
||||
|
|
@ -231,11 +233,16 @@ python3 epub2md_merge.py output/meinbuch
|
|||
cd output/meinbuch
|
||||
pandoc meinbuch.md --toc -s -o meinbuch.html # Webseite
|
||||
pandoc meinbuch.md --toc -o meinbuch.docx # Word
|
||||
pandoc meinbuch.md --pdf-engine=xelatex -o meinbuch.pdf # PDF (LaTeX nötig)
|
||||
pandoc meinbuch.md -s -o meinbuch.html && \
|
||||
pandoc meinbuch.html --toc --pdf-engine=xelatex -o meinbuch.pdf # PDF (LaTeX nötig)
|
||||
```
|
||||
|
||||
Bei Büchern mit Formeln für HTML zusätzlich `--mathml` angeben, sonst bleiben Matrizen als TeX-Text stehen.
|
||||
|
||||
Für ein PDF immer den Umweg über HTML nehmen, wie oben gezeigt. Beim direkten Weg von Markdown nach PDF
|
||||
lässt Pandoc komplexe Tabellen weg, und Querverweise im PDF funktionieren nicht. Die `README.md` im
|
||||
Ausgabeordner enthält die passenden Befehle für dein Buch.
|
||||
|
||||
### Für ein lokales Sprachmodell (RAG)
|
||||
|
||||
- Die einzelnen Kapiteldateien eignen sich als Abschnitte. Die Metadaten im Datenkopf (Buch, Kapitel,
|
||||
|
|
@ -285,9 +292,13 @@ stehen gesammelt in `diagnostics/warnings.md`.
|
|||
| `noteref-unresolved` | Fußnotenzeichen ohne zugehörige Fußnote | Fehler der Vorlage |
|
||||
| `note-unreferenced` | Fußnote, auf die nichts verweist | Bleibt an ihrer Stelle im Text |
|
||||
| `decode-error` | Zeichen im E-Book waren beschädigt | Betroffene Zeichen erscheinen als `<60>` |
|
||||
| `undefined-entity` | Das E-Book verwendet Sonderzeichen-Namen (`&name;`), die nirgends definiert sind | Diese Zeichen fehlen in der Ausgabe; Fehler der Vorlage |
|
||||
| `media-skipped` (Fehler) | Ein Bild/Medium war verdächtig groß oder stark komprimiert und wurde aus Sicherheitsgründen nicht übernommen | An seiner Stelle steht `[Bild nicht übernommen: …]`; mit `--strict` bricht das Programm ab |
|
||||
| `no-toc` | E-Book ohne Inhaltsverzeichnis | Kapiteltitel werden aus dem Inhalt abgeleitet |
|
||||
| `broken-output-link` | Interne Prüfung fand einen kaputten Link in der Ausgabe | Bitte melden, das sollte nicht vorkommen |
|
||||
|
||||
Steht hinter einer Meldung `(×5)`, ist dieselbe Meldung an derselben Stelle mehrfach aufgetreten.
|
||||
|
||||
Fehler, nach denen trotzdem weitergearbeitet wurde, etwa eine fehlende Kapiteldatei im E-Book, stehen
|
||||
ebenfalls dort. Mit `--strict` bricht das Programm bei solchen kritischen Fehlern stattdessen ab, ohne
|
||||
eine Ausgabe zu schreiben.
|
||||
|
|
|
|||
14
README.md
14
README.md
|
|
@ -222,7 +222,9 @@ weil CommonMark keine `[^…]`-Syntax kennt.
|
|||
- **Layouttabellen / komplexe Tabellen** (`colspan`, `rowspan`, Blockinhalte in Zellen, verschachtelte
|
||||
Tabellen, senkrechte Striche `|` in Code, Formeln oder Link-Zielen – GFM würde die Zeile dort in
|
||||
zusätzliche Zellen zerlegen) bleiben als bereinigtes HTML im Markdown erhalten, mit Warnung. Einfache Tabellen werden
|
||||
GFM-Pipe-Tabellen. Einzellige Layouttabellen werden als normaler Text ausgegeben.
|
||||
GFM-Pipe-Tabellen. Einzellige Layouttabellen werden als normaler Text ausgegeben. Innerhalb solcher
|
||||
HTML-Tabellen rendern Markdown-Programme kein Markdown. Formeln stehen dort deshalb als MathML oder als
|
||||
Formelbild, Hervorhebungen als HTML-Tags, Fußnotenverweise als HTML-Links (auch bei `--footnotes inline`).
|
||||
- **Syntax-Highlighting:** Farb-Spans, `<strong>`/`<em>` in Listings und Zeilennummern werden entfernt.
|
||||
Einrückung, Tabs, `<`, `>`, `&`, `$`, `|`, Backslashes und Backticks bleiben exakt erhalten. Die Sprache
|
||||
kommt aus Klassen/Attributen (`language-python`, `data-lang`, …) oder starken Inhaltssignalen
|
||||
|
|
@ -293,9 +295,17 @@ pandoc programmierbuch.md --toc -o programmierbuch.epub
|
|||
|
||||
cd output/mathematikbuch
|
||||
pandoc mathematikbuch.md --mathml -s -o mathematikbuch.html
|
||||
pandoc mathematikbuch.md --pdf-engine=xelatex -o mathematikbuch.pdf
|
||||
# PDF über den Zwischenschritt HTML (siehe unten)
|
||||
pandoc mathematikbuch.html --toc --pdf-engine=xelatex -o mathematikbuch.pdf
|
||||
```
|
||||
|
||||
**PDF immer über HTML erzeugen.** Beim direkten Weg Markdown → LaTeX verwirft Pandoc allen rohen HTML-Code.
|
||||
Dabei gehen HTML-Tabellen samt Inhalt verloren, und `<a id>`-Sprungziele fehlen, sodass interne Links im PDF
|
||||
ins Leere zeigen. Liest Pandoc dagegen die HTML-Fassung, macht es daraus echte Tabellen und Sprungziele.
|
||||
Getestet mit dem C-Buch (405 Seiten, 34 Tabellen, 728 von 755 internen Links funktionieren; die übrigen
|
||||
stehen in Codeblöcken innerhalb von Tabellen). Die `README.md` jedes konvertierten Buches enthält den
|
||||
passenden Befehl.
|
||||
|
||||
Alternativ direkt aus den Kapiteldateien:
|
||||
|
||||
```bash
|
||||
|
|
|
|||
|
|
@ -86,7 +86,8 @@ def build_parser() -> argparse.ArgumentParser:
|
|||
help="Druck-Umbruchzeichen (↪/➥) in Codeblöcken nicht entfernen; Standard: fortgesetzte "
|
||||
"Zeilen werden wieder zur ursprünglichen Codezeile zusammengefügt")
|
||||
p.add_argument("--strict", action="store_true",
|
||||
help="Bei kritischen Fehlern (fehlende Spine-Dateien, unsichere Pfade, kaputtes XHTML) abbrechen")
|
||||
help="Bei kritischen Fehlern abbrechen: fehlende/unlesbare Spine-Dokumente, nicht dekodierbares "
|
||||
"XHTML, unsichere ZIP-Pfade, übersprungene (verdächtige) Medien")
|
||||
p.add_argument("--force", action="store_true",
|
||||
help="Bestehendes Ausgabeverzeichnis ersetzen – nur wenn es eine frühere epub2md-Ausgabe ist")
|
||||
p.add_argument("--verbose", "-v", action="store_true", help="Verarbeitungsschritte und Warnungen ausgeben")
|
||||
|
|
|
|||
|
|
@ -17,14 +17,16 @@ class Diagnostics:
|
|||
self.unsupported: Counter[str] = Counter()
|
||||
self.unsupported_examples: dict[str, str] = {}
|
||||
self.unresolved_links: list[dict[str, str]] = []
|
||||
self._seen: set[tuple[str, str, str]] = set()
|
||||
self._seen: dict[tuple[str, str, str], Diagnostic] = {}
|
||||
|
||||
def add(self, level: str, code: str, message: str, source: str = "", chapter: str = "") -> None:
|
||||
key = (code, message, source)
|
||||
if key in self._seen:
|
||||
self._seen[key].count += 1
|
||||
return
|
||||
self._seen.add(key)
|
||||
self.items.append(Diagnostic(level, code, message, source, chapter))
|
||||
item = Diagnostic(level, code, message, source, chapter)
|
||||
self._seen[key] = item
|
||||
self.items.append(item)
|
||||
if self.verbose:
|
||||
print(f"[{level}] {message}" + (f" ({source})" if source else ""), file=sys.stderr)
|
||||
|
||||
|
|
@ -54,4 +56,5 @@ class Diagnostics:
|
|||
print(f"… {message}", file=sys.stderr)
|
||||
|
||||
def count(self, level: str) -> int:
|
||||
return sum(1 for d in self.items if d.level == level)
|
||||
"""Number of occurrences (repeated identical diagnostics included)."""
|
||||
return sum(d.count for d in self.items if d.level == level)
|
||||
|
|
|
|||
|
|
@ -43,13 +43,15 @@ def _secure_xml_parser(recover: bool = True) -> etree.XMLParser:
|
|||
_KEEP_ENTITIES = {"amp", "lt", "gt", "quot", "apos"}
|
||||
|
||||
|
||||
def _replace_named_entities(text: str) -> str:
|
||||
def _replace_named_entities(text: str, unknown: list[str] | None = None) -> str:
|
||||
def repl(m: re.Match[str]) -> str:
|
||||
name = m.group(1)
|
||||
if name in _KEEP_ENTITIES:
|
||||
return m.group(0)
|
||||
cp = html.entities.name2codepoint.get(name)
|
||||
if cp is None:
|
||||
if unknown is not None and name not in unknown:
|
||||
unknown.append(name)
|
||||
return m.group(0)
|
||||
return f"&#{cp};"
|
||||
|
||||
|
|
@ -75,11 +77,10 @@ def decode_markup(data: bytes, source: str, diag: Diagnostics) -> str:
|
|||
except (UnicodeDecodeError, LookupError):
|
||||
text = data.decode("utf-8", errors="replace")
|
||||
n = text.count("<EFBFBD>")
|
||||
diag.warn(
|
||||
"decode-error",
|
||||
f"Ungültige Zeichenkodierung ({encoding}); {n} Zeichen durch U+FFFD ersetzt.",
|
||||
source,
|
||||
)
|
||||
message = f"Ungültige Zeichenkodierung ({encoding}); {n} Zeichen durch U+FFFD ersetzt."
|
||||
if diag.strict:
|
||||
diag.error("decode-error", message, source, critical=True)
|
||||
diag.warn("decode-error", message, source)
|
||||
return text
|
||||
|
||||
|
||||
|
|
@ -87,7 +88,12 @@ def parse_xml(data: bytes, source: str, diag: Diagnostics, html_fallback: bool =
|
|||
"""Parse XML securely; optionally fall back to the HTML parser for tag soup."""
|
||||
text = decode_markup(data, source, diag)
|
||||
text = re.sub(r"^\s*<\?xml[^>]*\?>", "", text)
|
||||
text = _replace_named_entities(text)
|
||||
unknown: list[str] = []
|
||||
text = _replace_named_entities(text, unknown)
|
||||
if unknown:
|
||||
# never resolved (XXE protection) and therefore dropped by the parser – say so instead of losing them silently
|
||||
diag.warn("undefined-entity", "Nicht definierte Entitäten entfernt: " + ", ".join(f"&{n};" for n in unknown[:10])
|
||||
+ (" …" if len(unknown) > 10 else ""), source)
|
||||
root = None
|
||||
try:
|
||||
root = etree.fromstring(text, _secure_xml_parser())
|
||||
|
|
@ -165,14 +171,24 @@ class EpubReader:
|
|||
def exists(self, path: str) -> bool:
|
||||
return path in self.names
|
||||
|
||||
def entry_problem(self, path: str) -> str | None:
|
||||
"""Reason why an entry must not be read (too large / suspicious compression ratio), else None."""
|
||||
info = self.names.get(path)
|
||||
if info is None:
|
||||
return None
|
||||
if info.file_size > MAX_MEMBER_SIZE or (
|
||||
info.compress_size and info.file_size > 50 * 1024 * 1024 and info.file_size / info.compress_size > MAX_RATIO
|
||||
):
|
||||
return "ZIP-Eintrag zu groß oder verdächtig stark komprimiert"
|
||||
return None
|
||||
|
||||
def read(self, path: str) -> bytes:
|
||||
info = self.names.get(path)
|
||||
if info is None:
|
||||
raise KeyError(path)
|
||||
if info.file_size > MAX_MEMBER_SIZE or (
|
||||
info.compress_size and info.file_size > 50 * 1024 * 1024 and info.file_size / info.compress_size > MAX_RATIO
|
||||
):
|
||||
raise ContainerError(f"ZIP-Eintrag zu groß oder verdächtig stark komprimiert: {path}")
|
||||
problem = self.entry_problem(path)
|
||||
if problem:
|
||||
raise ContainerError(f"{problem}: {path}")
|
||||
try:
|
||||
with self.zip.open(info) as fh:
|
||||
data = fh.read(MAX_MEMBER_SIZE + 1)
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ from __future__ import annotations
|
|||
|
||||
import html
|
||||
import re
|
||||
from collections.abc import Sequence
|
||||
from collections.abc import Callable, Sequence
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
from urllib.parse import unquote
|
||||
|
|
@ -28,6 +28,7 @@ from .codeblocks import (
|
|||
is_inline_code,
|
||||
)
|
||||
from .diagnostics import Diagnostics
|
||||
from .errors import ContainerError
|
||||
from .footnotes import NoteIndex
|
||||
from .mathml import Unsupported, is_display_tex, looks_like_tex, mathml_to_tex, strip_tex_delimiters
|
||||
from .media import MediaRegistry
|
||||
|
|
@ -227,7 +228,9 @@ def serialize_mathml(math: Any) -> str:
|
|||
|
||||
class Renderer:
|
||||
def __init__(self, options: RenderOptions, resolver: LinkResolver, media: MediaRegistry,
|
||||
notes: NoteIndex, diag: Diagnostics, zip_names: set[str]) -> None:
|
||||
notes: NoteIndex, diag: Diagnostics, zip_names: set[str],
|
||||
entry_problem: Callable[[str], str | None] | None = None) -> None:
|
||||
self.entry_problem = entry_problem or (lambda path: None)
|
||||
self.opt = options
|
||||
self.zip_names = zip_names
|
||||
self.in_table_html = False
|
||||
|
|
@ -902,7 +905,7 @@ class Renderer:
|
|||
num = int(note.label.split("-")[1])
|
||||
if self.ch:
|
||||
self.ch.stats.footnotes += 1
|
||||
if mode == "inline" and not self.in_note:
|
||||
if mode == "inline" and not self.in_note and not self.in_table_html:
|
||||
text = self._note_inline_text(note)
|
||||
if text is not None:
|
||||
label = "Anmerkung" if self.opt.language.startswith("de") else "Note"
|
||||
|
|
@ -1015,6 +1018,11 @@ class Renderer:
|
|||
if not self._exists(path):
|
||||
self.diag.warn("image-missing", f"Bild {src} nicht im EPUB gefunden.", self._source(el), self.file)
|
||||
return [("t", md_escape(f"[Bild fehlt: {alt or src}]"))]
|
||||
problem = self.entry_problem(path)
|
||||
if problem:
|
||||
self.diag.error("media-skipped", f"Medium {path} nicht übernommen: {problem}", self._source(el), self.file,
|
||||
critical=True, exc=ContainerError)
|
||||
return [("t", md_escape(f"[Bild nicht übernommen: {alt or path.rsplit('/', 1)[-1]}]"))]
|
||||
if self._is_formula_image(el, alt, path):
|
||||
return self.formula_image(el, alt, path)
|
||||
if not alt and self.figure_caption:
|
||||
|
|
@ -1031,7 +1039,7 @@ class Renderer:
|
|||
else:
|
||||
self.ch.stats.images += 1
|
||||
if not self.opt.extract_media:
|
||||
return [("r", f"*\\[Bild: {self._alt(alt)} – {md_escape(path)}\\]*")]
|
||||
return [("r", f"\\[Bild: {self._alt(alt)} – {md_escape(path)}\\] ")]
|
||||
mf = self.media.register(path, role=role, hint=alt[:40] if self.media.strategy == "relative" and role != "formula" else "")
|
||||
if self.in_table_html:
|
||||
return [("r", f'<img src="{html.escape(url_path(rel_link(self.file, mf.output)))}" alt="{html.escape(alt)}"/>')]
|
||||
|
|
@ -1044,13 +1052,13 @@ class Renderer:
|
|||
def _is_formula_image(self, el: Any, alt: str, path: str) -> bool:
|
||||
if looks_like_tex(alt):
|
||||
return True
|
||||
for anc in _ancestors(el, 4):
|
||||
for anc in [el] + _ancestors(el, 4):
|
||||
if any(FORMULA_CLASS.search(c) for c in classes(anc)) or {"equation"} & set(epub_types(anc)):
|
||||
return True
|
||||
return False
|
||||
|
||||
def _formula_display(self, el: Any, alt: str) -> bool:
|
||||
for anc in _ancestors(el, 5):
|
||||
for anc in [el] + _ancestors(el, 5):
|
||||
cls = classes(anc)
|
||||
if any(INLINE_FORMULA_CLASS.search(c) for c in cls):
|
||||
return False
|
||||
|
|
@ -1062,7 +1070,7 @@ class Renderer:
|
|||
display = self._formula_display(el, alt)
|
||||
tex = strip_tex_delimiters(alt) if looks_like_tex(alt) else ""
|
||||
source = self._source(el)
|
||||
if tex and self.opt.math in ("auto", "tex"):
|
||||
if tex and self.opt.math in ("auto", "tex") and not self.in_table_html:
|
||||
self.consumed_formula_images.add(path)
|
||||
self.formulas.append({"source": source, "chapter": self.file, "origin": "image-alt-tex",
|
||||
"representation": "tex", "display": str(display).lower()})
|
||||
|
|
@ -1183,24 +1191,19 @@ class Renderer:
|
|||
def _html(self, el: Any, in_pre: bool, keep: bool = False) -> str:
|
||||
name = local(el.tag)
|
||||
if name == "math":
|
||||
if self.opt.math in ("auto", "tex"):
|
||||
try:
|
||||
tex = mathml_to_tex(el)
|
||||
return html.escape(f"\\({tex}\\)" if self.opt.math == "tex" else f"${tex}$", quote=False)
|
||||
except Unsupported:
|
||||
pass
|
||||
# Markdown math ($…$, \(…\)) is not rendered inside an HTML block; MathML is.
|
||||
display = el.get("display") == "block"
|
||||
if self.ch:
|
||||
self.ch.stats.math_as_mathml += 1
|
||||
if display:
|
||||
self.ch.stats.math_block += 1
|
||||
else:
|
||||
self.ch.stats.math_inline += 1
|
||||
self.formulas.append({"source": self._source(el), "chapter": self.file, "origin": "mathml",
|
||||
"representation": "mathml-in-html-table", "display": str(display).lower()})
|
||||
return serialize_mathml(el)
|
||||
if name in ("img", "svg", "image"):
|
||||
if name == "img" and self._is_formula_image(el, el.get("alt") or "", ""):
|
||||
alt = el.get("alt") or ""
|
||||
if looks_like_tex(alt) and self.opt.math in ("auto", "tex"):
|
||||
path = resolve_href(self.doc, el.get("src") or "")
|
||||
if path:
|
||||
self.consumed_formula_images.add(path)
|
||||
tex = strip_tex_delimiters(alt)
|
||||
return html.escape(f"\\({tex}\\)" if self.opt.math == "tex" else f"${tex}$", quote=False)
|
||||
pieces = self.inline(el)
|
||||
return join_inline(pieces)
|
||||
return join_inline(self.inline(el))
|
||||
if name == "a" and self.notes.extracted and self.notes.note_for_ref(el) is not None:
|
||||
return join_inline(self.link(el))
|
||||
text_fn = (lambda t: html.escape(t, quote=False).replace("\n", " ")) if in_pre else (lambda t: html.escape(WS.sub(" ", t), quote=False))
|
||||
|
|
@ -1224,6 +1227,8 @@ class Renderer:
|
|||
return anchor + content
|
||||
attrs = []
|
||||
for k, v in el.attrib.items():
|
||||
if k.startswith("{"):
|
||||
continue # namespaced (epub:type, xml:lang …) – not valid HTML
|
||||
key = k.rsplit("}", 1)[-1]
|
||||
if key in HTML_KEEP_ATTRS:
|
||||
attrs.append(f'{key}="{html.escape(v)}"')
|
||||
|
|
|
|||
|
|
@ -15,6 +15,7 @@ INLINE_CODE = re.compile(r"(?<![\\`])(`+)(?!`)((?:[^`\n]|\n(?![ \t]*\n)|`(?!\1))
|
|||
MD_LINK = re.compile(r"(?<!\\)!?\[((?:[^\[\]\\\n]|\\.|\[(?:[^\[\]\\\n]|\\.)*\]|\n(?![ \t]*\n))*)\]\((<[^>]*>|[^)\s]+)(?:\s+\"[^\"]*\")?\)")
|
||||
HTML_ATTR = re.compile(r"""\b(?:href|src)="([^"]*)\"""")
|
||||
ANCHOR = re.compile(r"""\bid="([^"]+)\"""")
|
||||
PANDOC_ANCHOR = re.compile(r"\{#([^}\s]+)[^}]*\}") # []{#id} spans and {#id} heading attributes
|
||||
HEADING = re.compile(r"^#{1,6}\s+(.*?)\s*#*\s*$", re.M)
|
||||
|
||||
|
||||
|
|
@ -33,7 +34,7 @@ def gfm_slug(heading: str) -> str:
|
|||
|
||||
def anchors_in(text: str) -> set[str]:
|
||||
body = strip_code(text)
|
||||
names = set(ANCHOR.findall(body))
|
||||
names = set(ANCHOR.findall(body)) | set(PANDOC_ANCHOR.findall(body))
|
||||
names.update(gfm_slug(h) for h in HEADING.findall(body))
|
||||
return names
|
||||
|
||||
|
|
|
|||
|
|
@ -82,6 +82,7 @@ class Diagnostic:
|
|||
message: str
|
||||
source: str = ""
|
||||
chapter: str = ""
|
||||
count: int = 1 # identical diagnostics (code, message, source) are counted, not repeated
|
||||
|
||||
def as_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
|
|
@ -90,6 +91,7 @@ class Diagnostic:
|
|||
"message": self.message,
|
||||
"source": self.source,
|
||||
"chapter": self.chapter,
|
||||
"count": self.count,
|
||||
}
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -236,7 +236,7 @@ def _render(reader: EpubReader, package: Any, chapters: list[Chapter], resolver:
|
|||
RenderOptions(format=opts.format, footnotes=opts.footnotes, math=opts.math,
|
||||
extract_media=opts.extract_media, keep_html=opts.keep_html, language=language,
|
||||
code_language=opts.code_language, keep_wrap_markers=opts.keep_wrap_markers),
|
||||
resolver, media, notes, diag, set(reader.names),
|
||||
resolver, media, notes, diag, set(reader.names), reader.entry_problem,
|
||||
)
|
||||
width = max(3, len(str(len(chapters) + 1)))
|
||||
notes_title = "Anmerkungen" if language.startswith("de") else "Notes"
|
||||
|
|
@ -248,6 +248,8 @@ def _render(reader: EpubReader, package: Any, chapters: list[Chapter], resolver:
|
|||
ch.stats.word_count = _word_count(ch.markdown)
|
||||
_check_chapter_size(ch, opts.chapter_level, diag)
|
||||
_report_wrap_markers(renderer, diag)
|
||||
if not opts.code_language:
|
||||
_suggest_code_language(renderer, diag)
|
||||
endnotes: Chapter | None = None
|
||||
endnotes_md = ""
|
||||
if opts.footnotes == "end-of-book" and renderer.book_notes:
|
||||
|
|
@ -256,10 +258,25 @@ def _render(reader: EpubReader, package: Any, chapters: list[Chapter], resolver:
|
|||
endnotes.filename = renderer.endnotes_file
|
||||
endnotes.stats.footnotes = len(renderer.book_notes)
|
||||
endnotes.stats.word_count = _word_count(endnotes_md)
|
||||
skipped = _collect_remaining_media(reader, package, media, renderer, opts)
|
||||
skipped = _collect_remaining_media(reader, package, media, renderer, opts, diag)
|
||||
return _Rendered(renderer, media, cover_out, endnotes, endnotes_md, skipped)
|
||||
|
||||
|
||||
def _suggest_code_language(renderer: Renderer, diag: Diagnostics) -> None:
|
||||
"""Book-level hint: many unlabeled blocks while every recognized block has the same language."""
|
||||
blocks = renderer.code_blocks
|
||||
detected = [b["language"] for b in blocks if b["language"]]
|
||||
unlabeled = len(blocks) - len(detected)
|
||||
if len(blocks) < 10 or len(detected) < 5 or unlabeled < 0.3 * len(blocks):
|
||||
return
|
||||
lang = max(set(detected), key=detected.count)
|
||||
share = detected.count(lang) / len(detected)
|
||||
if share >= 0.9:
|
||||
diag.info("code-language-hint",
|
||||
f"{unlabeled} von {len(blocks)} Codeblöcken ohne erkannte Sprache; {round(share * 100)} % der erkannten "
|
||||
f"Blöcke sind {lang}. Bei einem einsprachigen Buch kennzeichnet --code-language {lang} die übrigen Blöcke.")
|
||||
|
||||
|
||||
def _report_wrap_markers(renderer: Renderer, diag: Diagnostics) -> None:
|
||||
wraps: dict[str, int] = {}
|
||||
for block in renderer.code_blocks:
|
||||
|
|
@ -271,7 +288,7 @@ def _report_wrap_markers(renderer: Renderer, diag: Diagnostics) -> None:
|
|||
|
||||
|
||||
def _collect_remaining_media(reader: EpubReader, package: Any, media: MediaRegistry, renderer: Renderer,
|
||||
opts: Options) -> int:
|
||||
opts: Options, diag: Diagnostics) -> int:
|
||||
"""Register fonts, CSS, audio and unreferenced images; returns the number of formula images replaced by TeX."""
|
||||
skipped_formula_images = 0
|
||||
if opts.extract_media:
|
||||
|
|
@ -281,6 +298,11 @@ def _collect_remaining_media(reader: EpubReader, package: Any, media: MediaRegis
|
|||
if item.media_type in ("application/xhtml+xml", "text/html", "application/x-dtbncx+xml",
|
||||
"application/oebps-package+xml") or item.path == package.opf_path:
|
||||
continue
|
||||
problem = reader.entry_problem(item.path)
|
||||
if problem:
|
||||
diag.error("media-skipped", f"Medium {item.path} nicht übernommen: {problem}", item.path,
|
||||
critical=True, exc=ContainerError)
|
||||
continue
|
||||
if item.path in renderer.consumed_formula_images:
|
||||
skipped_formula_images += 1
|
||||
continue
|
||||
|
|
|
|||
|
|
@ -16,7 +16,7 @@ EXTERNAL_SCHEMES = {"http", "https", "ftp", "mailto", "tel", "urn", "doi", "news
|
|||
|
||||
|
||||
def anchor_name(raw: str) -> str:
|
||||
name = re.sub(r"[^A-Za-z0-9_.:-]", "-", raw).strip("-")
|
||||
name = re.sub(r"[^\w.:-]", "-", raw).strip("-") # \w keeps Unicode letters (über, été)
|
||||
if not name or not name[0].isalpha():
|
||||
name = "a-" + name
|
||||
return name
|
||||
|
|
@ -136,5 +136,7 @@ class LinkResolver:
|
|||
file = self.chapters[self.doc_start[path]].filename
|
||||
rel = rel_link(from_file, file)
|
||||
status = "ok" if not frag else "anchor-missing"
|
||||
return (url_path(rel) or "#"), status
|
||||
if not rel:
|
||||
return None, "self" # start of the same file: no useful link target
|
||||
return url_path(rel), status
|
||||
return None, "missing"
|
||||
|
|
|
|||
|
|
@ -136,6 +136,9 @@ def readme(md: BookMetadata, chapters: list[Chapter], endnotes: Chapter | None,
|
|||
"# danach in diesem Ordner:",
|
||||
f"pandoc {shlex.quote(stem + '.md')} --toc -s{math} -o {shlex.quote(stem + '.html')}",
|
||||
f"pandoc {shlex.quote(stem + '.md')} --toc -o {shlex.quote(stem + '.epub')}",
|
||||
"# PDF über HTML: Pandoc übernimmt dann auch HTML-Tabellen und interne Sprungziele",
|
||||
f"pandoc {shlex.quote(stem + '.md')} -s{math} -o {shlex.quote(stem + '.html')} && "
|
||||
f"pandoc {shlex.quote(stem + '.html')} --toc --pdf-engine=xelatex -o {shlex.quote(stem + '.pdf')}",
|
||||
]
|
||||
out.append("## Weiterverarbeitung")
|
||||
out.append("```bash\n" + "\n".join(commands) + "\n```")
|
||||
|
|
@ -163,7 +166,8 @@ def warnings_md(diag: Diagnostics) -> str:
|
|||
out += ["", f"## {key} ({len(items)})", ""]
|
||||
for d in items:
|
||||
where = " · ".join(x for x in (d.source, d.chapter) if x)
|
||||
out.append(f"- {md_escape(d.message)}" + (f" (`{where}`)" if where else ""))
|
||||
times = f" (×{d.count})" if d.count > 1 else ""
|
||||
out.append(f"- {md_escape(d.message)}{times}" + (f" (`{where}`)" if where else ""))
|
||||
return "\n".join(out) + "\n"
|
||||
|
||||
|
||||
|
|
|
|||
121
tests/test_qa2_open.py
Normal file
121
tests/test_qa2_open.py
Normal file
|
|
@ -0,0 +1,121 @@
|
|||
"""Regression tests for the remaining findings of the independent QA (QA2-09 … QA2-19)."""
|
||||
|
||||
import re
|
||||
|
||||
from conftest import PNG, SVG, Book, Doc
|
||||
|
||||
M = 'xmlns="http://www.w3.org/1998/Math/MathML"'
|
||||
|
||||
|
||||
def html_table(text: str) -> str:
|
||||
return text[text.index("<table"): text.index("</table>")]
|
||||
|
||||
|
||||
def test_qa2_09_math_and_notes_in_html_tables_are_html(convert):
|
||||
body = ('<h1>T</h1><table><tr><td colspan="2">Kopf</td></tr>'
|
||||
f'<tr><td><math {M}><mfrac><mi>a</mi><mi>b</mi></mfrac></math></td>'
|
||||
'<td><span class="InlineEquation"><img src="f.png" alt="$$x^2$$"/></span>'
|
||||
'<a epub:type="noteref" href="#n1">1</a></td></tr></table>'
|
||||
'<aside epub:type="footnote" id="n1"><p>Notiz <em>wichtig</em>.</p></aside>')
|
||||
for mode in ("auto", "tex"):
|
||||
run = convert(Book(docs=[Doc("c.xhtml", body)], media={"f.png": PNG}), "--math", mode, "--footnotes", "inline")
|
||||
table = html_table(run.chapter("001"))
|
||||
visible = re.sub(r'"[^"]*"', '""', table) # ignore attribute values (alt texts)
|
||||
assert "<math" in table and "$" not in visible and "\\(" not in visible # renderable MathML
|
||||
assert '<img src="../media/images/' in table and 'alt="$$x^2$$"' in table # formula image kept
|
||||
assert "(Anmerkung:" not in table and '<sup><a href="#fn-001">1</a></sup>' in table
|
||||
assert '<a id="fn-001"></a>' in run.chapter("001")
|
||||
assert run.report["output_link_check"]["broken_count"] == 0
|
||||
|
||||
|
||||
def test_qa2_10_book_readme_recommends_pdf_via_html(convert):
|
||||
"""Pandoc drops raw HTML (tables, <a id>) when going Markdown -> LaTeX; the HTML route keeps both."""
|
||||
run = convert(Book(docs=[Doc("a.xhtml", "<h1>A</h1><p>x</p>")]))
|
||||
readme = (run.out / "README.md").read_text(encoding="utf-8")
|
||||
assert re.search(r"pandoc \S+\.md -s -o \S+\.html && pandoc \S+\.html --toc --pdf-engine=xelatex -o \S+\.pdf", readme)
|
||||
|
||||
|
||||
def test_qa2_11_code_language_hint(convert):
|
||||
py = "def f(x):\n return x\n\nclass A:\n pass"
|
||||
body = "<h1>C</h1>" + "".join(f"<pre>{py}\n# {i}</pre><pre>print({i})</pre>" for i in range(10))
|
||||
run = convert(Book(docs=[Doc("c.xhtml", body)]))
|
||||
hints = [i for i in run.report["infos"] if i["code"] == "code-language-hint"]
|
||||
assert hints and "--code-language python" in hints[0]["message"]
|
||||
|
||||
|
||||
def test_qa2_12_strict_aborts_on_undecodable_xhtml(convert, tmp_path):
|
||||
import zipfile
|
||||
from conftest import build_epub
|
||||
path = build_epub(tmp_path / "g.epub", Book(docs=[Doc("a.xhtml", "<h1>A</h1><p>x</p>")]))
|
||||
with zipfile.ZipFile(path) as z:
|
||||
entries = {n: z.read(n) for n in z.namelist()}
|
||||
entries["OEBPS/a.xhtml"] = b'<?xml version="1.0" encoding="utf-8"?><html><body><p>\xff\xfe\x00\x81 kaputt</p></body></html>'
|
||||
with zipfile.ZipFile(path, "w") as z:
|
||||
for n, d in entries.items():
|
||||
z.writestr(n, d)
|
||||
assert convert(path).rc == 0
|
||||
assert convert(path, "--strict").rc == 3
|
||||
|
||||
|
||||
def test_qa2_13_suspicious_media_is_skipped_not_fatal(convert, tmp_path):
|
||||
import zipfile
|
||||
from conftest import build_epub
|
||||
path = build_epub(tmp_path / "b.epub", Book(docs=[Doc("a.xhtml", '<h1>A</h1><p><img src="big.png" alt="Riesig"/> Text</p>')],
|
||||
media={"big.png": PNG}))
|
||||
with zipfile.ZipFile(path) as z:
|
||||
entries = [(i, z.read(i)) for i in z.infolist()]
|
||||
with zipfile.ZipFile(path, "w", compression=zipfile.ZIP_DEFLATED) as z:
|
||||
for info, data in entries:
|
||||
z.writestr(info.filename, b"\0" * 60_000_000 if info.filename == "OEBPS/big.png" else data)
|
||||
run = convert(path)
|
||||
assert run.rc == 0
|
||||
assert "Riesig" in run.chapter("001") and "big.png" not in run.chapter("001")
|
||||
assert any(e["code"] == "media-skipped" for e in run.report["errors"])
|
||||
assert run.report["output_link_check"]["broken_count"] == 0
|
||||
|
||||
|
||||
def test_qa2_14_formula_class_on_img_itself(convert):
|
||||
body = '<h1>F</h1><p>Wert <img class="inlineequation" src="f.png" alt="Summe a plus b"/> ist.</p>'
|
||||
run = convert(Book(docs=[Doc("c.xhtml", body)], media={"f.png": PNG}))
|
||||
assert [f["origin"] for f in run.report["formulas"]["items"]] == ["image"]
|
||||
assert any(w["code"] == "formula-image" for w in run.report["warnings"])
|
||||
|
||||
|
||||
def test_qa2_15_diagnostics_count_repeats_and_html_table_formulas(convert):
|
||||
img = '<span class="equation"><img src="f.png" alt="Summe"/></span>'
|
||||
body = (f"<h1>F</h1><p>{img} und {img}</p>"
|
||||
f'<table><tr><td colspan="2"><math {M}><mi>x</mi></math></td></tr><tr><td>a</td><td>b</td></tr></table>')
|
||||
run = convert(Book(docs=[Doc("c.xhtml", body)], media={"f.png": PNG}))
|
||||
w = [w for w in run.report["warnings"] if w["code"] == "formula-image"]
|
||||
assert len(w) == 1 and w[0]["count"] == 2
|
||||
assert run.report["formulas"]["count"] == 3
|
||||
|
||||
|
||||
def test_qa2_16_non_ascii_ids_are_kept(convert):
|
||||
body = '<h1 id="über-uns">Über</h1><p id="été">Sommer</p><p><a href="#été">S</a> <a href="#über-uns">Ü</a></p>'
|
||||
text = convert(Book(docs=[Doc("c.xhtml", body)])).chapter("001")
|
||||
assert '<a id="été"></a>' in text and '<a id="über-uns"></a>' in text
|
||||
assert "[S](#été)" in text
|
||||
|
||||
|
||||
def test_qa2_17_no_empty_fragment_links_and_no_epub_type_attr(convert):
|
||||
body = ('<h1>K</h1><p>Text<a epub:type="noteref" href="#n">1</a></p>'
|
||||
'<table><tr><td colspan="2"><a epub:type="noteref" href="#n">1</a></td></tr><tr><td>a</td><td>b</td></tr></table>'
|
||||
'<aside epub:type="footnote" id="n"><p><a href="c.xhtml">↩</a> Note.</p></aside>')
|
||||
text = convert(Book(docs=[Doc("c.xhtml", body)]), "--footnotes", "keep-links").chapter("001")
|
||||
assert "](#)" not in text
|
||||
assert ' type="noteref"' not in text
|
||||
|
||||
|
||||
def test_qa2_18_undefined_entities_are_reported(convert):
|
||||
body = "<h1>E</h1><p>A &unbekannt; B &auch;</p>"
|
||||
run = convert(Book(docs=[Doc("c.xhtml", body)]))
|
||||
w = [w for w in run.report["warnings"] if w["code"] == "undefined-entity"]
|
||||
assert w and "unbekannt" in w[0]["message"] and "auch" in w[0]["message"]
|
||||
|
||||
|
||||
def test_qa2_19_adjacent_placeholders_without_media(convert):
|
||||
body = '<h1>B</h1><p><img src="a.png" alt="Eins"/><img src="b.svg" alt="Zwei"/></p>'
|
||||
text = convert(Book(docs=[Doc("c.xhtml", body)], media={"a.png": PNG, "b.svg": SVG}), "--no-extract-media").chapter("001")
|
||||
assert "**" not in text
|
||||
assert re.search(r"Bild: Eins.*Bild: Zwei", text)
|
||||
Loading…
Reference in a new issue