pdf2md.py als fester Name, v1–v4 nach History/ verschoben
- pdf2md_v5.py → pdf2md.py; Verweise in Doku und Tests angepasst - pdf2md_v1.py … pdf2md_v4.py nach History/ verschoben - ljutsch_meta.xml entfernt (nicht mehr verwendet) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Dieter Schlüter
parent
37ea397a29
commit
b4b911d37f
13 changed files with 22 additions and 49 deletions
|
|
@ -1,6 +1,6 @@
|
|||
# Bedienungsanleitung: pdf2md_v5.py – PDF → Markdown → EPUB
|
||||
# Bedienungsanleitung: pdf2md.py – PDF → Markdown → EPUB
|
||||
|
||||
`pdf2md_v5.py` wandelt PDF-Bücher in sauberes Markdown um: echte Absätze, Überschriften mit
|
||||
`pdf2md.py` wandelt PDF-Bücher in sauberes Markdown um: echte Absätze, Überschriften mit
|
||||
richtiger Ebene, Bilder und ohne Seitenzahlen. Daraus entsteht auf Wunsch direkt ein EPUB für
|
||||
den PocketBook-Reader. Das funktioniert mit digitalen PDFs (Text lässt sich markieren) ebenso wie
|
||||
mit gescannten PDFs (nur Seitenbilder, der Text wird per OCR erkannt).
|
||||
|
|
@ -31,7 +31,7 @@ Prüfen, ob alles da ist:
|
|||
```bash
|
||||
tesseract --list-langs # muss "deu" enthalten
|
||||
pandoc --version # Version 3.x
|
||||
python3 pdf2md_v5.py --help # zeigt alle Optionen
|
||||
python3 pdf2md.py --help # zeigt alle Optionen
|
||||
```
|
||||
|
||||
---
|
||||
|
|
@ -40,7 +40,7 @@ python3 pdf2md_v5.py --help # zeigt alle Optionen
|
|||
|
||||
```bash
|
||||
cd ~/Downloads/ljutsch
|
||||
python3 pdf2md_v5.py --epub Operation_Ljutsch_Band_1_-_Reinhard_Otto_Kranz.pdf Output/band1.md
|
||||
python3 pdf2md.py --epub Operation_Ljutsch_Band_1_-_Reinhard_Otto_Kranz.pdf Output/band1.md
|
||||
```
|
||||
|
||||
Das Programm meldet zuerst, was es in der PDF gefunden hat:
|
||||
|
|
@ -80,7 +80,7 @@ EPUB zum Vergleichen:
|
|||
|
||||
```bash
|
||||
cd ~/Downloads/ljutsch
|
||||
python3 pdf2md_v5.py --epub Operation_Ljutsch_Band_1_-_Reinhard_Otto_Kranz.pdf Output/band1.md
|
||||
python3 pdf2md.py --epub Operation_Ljutsch_Band_1_-_Reinhard_Otto_Kranz.pdf Output/band1.md
|
||||
```
|
||||
|
||||
Wenn du kein Cover und kein Probe-EPUB brauchst, reicht `--images` statt `--epub`.
|
||||
|
|
@ -144,7 +144,7 @@ pandoc band1.md \
|
|||
--epub-cover-image band1_images/cover.jpg
|
||||
```
|
||||
|
||||
Das sind dieselben Einstellungen, die `pdf2md_v5.py --epub` verwendet. Titel und Autor trägst du
|
||||
Das sind dieselben Einstellungen, die `pdf2md.py --epub` verwendet. Titel und Autor trägst du
|
||||
selbst ein. Ohne Cover lässt du die letzte Zeile weg und entfernst den `\` am Ende der Zeile davor.
|
||||
|
||||
Schritt 2 und 3 kannst du beliebig oft wiederholen. **Achtung:** Schritt 1 erneut auszuführen
|
||||
|
|
@ -158,7 +158,7 @@ Dateinamen.
|
|||
Bei Scans liest das Programm den Text per OCR (Tesseract). Der Aufruf ist derselbe:
|
||||
|
||||
```bash
|
||||
python3 pdf2md_v5.py --epub scan.pdf Output/scan.md
|
||||
python3 pdf2md.py --epub scan.pdf Output/scan.md
|
||||
```
|
||||
|
||||
| Situation | Option |
|
||||
|
|
|
|||
|
|
@ -1,19 +1,19 @@
|
|||
# CLAUDE.md
|
||||
|
||||
PDF → Markdown → EPUB für den PocketBook-Reader (digitale und gescannte PDFs).
|
||||
Aktuelles Programm: `pdf2md_v5.py` (Einzeldatei). v1–v4 sind Historie, nicht ändern.
|
||||
Aktuelles Programm: `pdf2md.py` (Einzeldatei; Name bleibt fest, keine Versionsnummer im Dateinamen). v1–v4 liegen in `History/` (Historie, nicht ändern).
|
||||
|
||||
## Befehle
|
||||
|
||||
- Tests: `python3 -m pytest` (Fixtures werden synthetisch erzeugt; OCR braucht `tesseract-ocr-deu`)
|
||||
- Lint: `python3 -m ruff check pdf2md_v5.py tests/`
|
||||
- Echttest: `python3 pdf2md_v5.py --epub Operation_Ljutsch_Band_1_-_Reinhard_Otto_Kranz.pdf <scratch>/band1.md`
|
||||
- Lint: `python3 -m ruff check pdf2md.py tests/`
|
||||
- Echttest: `python3 pdf2md.py --epub Operation_Ljutsch_Band_1_-_Reinhard_Otto_Kranz.pdf <scratch>/band1.md`
|
||||
|
||||
## Konventionen
|
||||
|
||||
- Hilfetexte, Meldungen, Kommentare und Docstrings auf Deutsch (wie v4); Bezeichner Englisch.
|
||||
- Echttests nie nach `Fertig/` schreiben (dort liegen die Ergebnisse des Nutzers).
|
||||
- PDFs/EPUBs/Bilder sind per `.gitignore` ausgeschlossen – keine Binärdateien committen.
|
||||
- `pdf2md_v5.py` patcht zwei Funktionen in `pymupdf4llm.helpers.document_layout`
|
||||
- `pdf2md.py` patcht zwei Funktionen in `pymupdf4llm.helpers.document_layout`
|
||||
(`get_styled_text`, `text_to_md`). Nach einem pymupdf4llm-Update prüfen, ob
|
||||
`tests/test_normalize.py::test_pymupdf4llm_patch_keeps_compound_hyphen` noch grün ist.
|
||||
|
|
|
|||
146
History/pdf2md_v1.py
Normal file
146
History/pdf2md_v1.py
Normal file
|
|
@ -0,0 +1,146 @@
|
|||
#!/usr/bin/env python3
|
||||
"""PDF nach Markdown konvertieren und weiche Zeilenumbrüche entfernen."""
|
||||
|
||||
import argparse
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pymupdf4llm
|
||||
|
||||
|
||||
HEADING = re.compile(r"^\s{0,3}#{1,6}\s+")
|
||||
LIST_ITEM = re.compile(r"^\s{0,3}(?:[-+*]|\d+[.)])\s+")
|
||||
BLOCKQUOTE = re.compile(r"^\s{0,3}>\s?")
|
||||
FENCE = re.compile(r"^\s{0,3}(`{3,}|~{3,})")
|
||||
RULE = re.compile(r"^\s{0,3}(?:-\s*){3,}$|^\s{0,3}(?:\*\s*){3,}$")
|
||||
IMAGE = re.compile(r"^\s*!\[.*?\]\(.*\)\s*$")
|
||||
HTML_BLOCK = re.compile(r"^\s*</?(?:table|tr|td|th|div|pre|p|h[1-6])\b", re.I)
|
||||
|
||||
|
||||
def is_structural(line: str) -> bool:
|
||||
"""Zeilen, die nicht mit dem vorherigen Fließtext verbunden werden."""
|
||||
return bool(
|
||||
HEADING.match(line)
|
||||
or LIST_ITEM.match(line)
|
||||
or BLOCKQUOTE.match(line)
|
||||
or RULE.match(line)
|
||||
or IMAGE.match(line)
|
||||
or HTML_BLOCK.match(line)
|
||||
or line.lstrip().startswith("|")
|
||||
or line.startswith((" ", "\t"))
|
||||
)
|
||||
|
||||
|
||||
def normalize_markdown(markdown: str) -> str:
|
||||
result = []
|
||||
paragraph = []
|
||||
in_fence = False
|
||||
fence_char = ""
|
||||
fence_length = 0
|
||||
|
||||
def flush_paragraph():
|
||||
if paragraph:
|
||||
result.append(" ".join(paragraph))
|
||||
paragraph.clear()
|
||||
|
||||
for original in markdown.replace("\r\n", "\n").replace("\r", "\n").split("\n"):
|
||||
line = original.rstrip()
|
||||
marker = FENCE.match(line)
|
||||
|
||||
if in_fence:
|
||||
result.append(line)
|
||||
if marker:
|
||||
token = marker.group(1)
|
||||
if token[0] == fence_char and len(token) >= fence_length:
|
||||
in_fence = False
|
||||
continue
|
||||
|
||||
if marker:
|
||||
flush_paragraph()
|
||||
result.append(line)
|
||||
in_fence = True
|
||||
fence_char = marker.group(1)[0]
|
||||
fence_length = len(marker.group(1))
|
||||
continue
|
||||
|
||||
if not line.strip():
|
||||
flush_paragraph()
|
||||
if result and result[-1] != "":
|
||||
result.append("")
|
||||
continue
|
||||
|
||||
if is_structural(line):
|
||||
flush_paragraph()
|
||||
result.append(line)
|
||||
continue
|
||||
|
||||
paragraph.append(line.strip())
|
||||
|
||||
flush_paragraph()
|
||||
|
||||
return "\n".join(result).strip() + "\n"
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="PDF in Markdown umwandeln; Zeilen innerhalb von Absätzen verbinden."
|
||||
)
|
||||
parser.add_argument("pdf", type=Path, help="Eingabe-PDF")
|
||||
parser.add_argument(
|
||||
"markdown",
|
||||
type=Path,
|
||||
nargs="?",
|
||||
help="Ausgabe-Datei; Standard: gleicher Name mit .md",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--ocr-language",
|
||||
default="deu",
|
||||
help="Tesseract-Sprachcode bei erforderlicher OCR (Standard: deu)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-ocr",
|
||||
action="store_true",
|
||||
help="OCR abschalten, wenn die PDF sicher durchsuchbaren Text enthält",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--keep-headers",
|
||||
action="store_true",
|
||||
help="Wiederkehrende Seitenköpfe in die Ausgabe übernehmen",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--keep-footers",
|
||||
action="store_true",
|
||||
help="Wiederkehrende Seitenfüße in die Ausgabe übernehmen",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
source = args.pdf.expanduser()
|
||||
target = (args.markdown or source.with_suffix(".md")).expanduser()
|
||||
|
||||
if not source.is_file():
|
||||
parser.error(f"Eingabedatei nicht gefunden: {source}")
|
||||
if source.resolve() == target.resolve():
|
||||
parser.error("Eingabe- und Ausgabedatei müssen verschieden sein.")
|
||||
|
||||
try:
|
||||
markdown = pymupdf4llm.to_markdown(
|
||||
str(source),
|
||||
header=args.keep_headers,
|
||||
footer=args.keep_footers,
|
||||
use_ocr=not args.no_ocr,
|
||||
ocr_language=args.ocr_language,
|
||||
)
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(normalize_markdown(markdown), encoding="utf-8")
|
||||
except Exception as exc:
|
||||
print(f"Fehler bei der Konvertierung: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
print(f"Markdown geschrieben: {target}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
|
||||
181
History/pdf2md_v2.py
Normal file
181
History/pdf2md_v2.py
Normal file
|
|
@ -0,0 +1,181 @@
|
|||
#!/usr/bin/env python3
|
||||
"""PDF nach Markdown konvertieren und Absätze anhand von Satzenden verbinden."""
|
||||
|
||||
import argparse
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pymupdf4llm
|
||||
from pymupdf4llm.ocr import tesseract_api
|
||||
|
||||
|
||||
HEADING = re.compile(r"^\s{0,3}#{1,6}\s+")
|
||||
LIST_ITEM = re.compile(r"^\s{0,3}(?:[-+*]|\d+[.)])\s+")
|
||||
BLOCKQUOTE = re.compile(r"^\s{0,3}>\s?")
|
||||
FENCE = re.compile(r"^\s{0,3}(`{3,}|~{3,})")
|
||||
RULE = re.compile(r"^\s{0,3}(?:-\s*){3,}$|^\s{0,3}(?:\*\s*){3,}$")
|
||||
IMAGE = re.compile(r"^\s*!\[.*?\]\(.*\)\s*$")
|
||||
HTML_BLOCK = re.compile(
|
||||
r"^\s*</?(?:table|tr|td|th|div|pre|p|h[1-6])\b",
|
||||
re.I,
|
||||
)
|
||||
|
||||
# Satzzeichen, optional gefolgt von schließenden Anführungszeichen/Klammern.
|
||||
SENTENCE_END = re.compile(r"""[.!?…]+[”’"'»›)\]]*$""")
|
||||
|
||||
|
||||
def ends_sentence(text: str) -> bool:
|
||||
return bool(SENTENCE_END.search(text.rstrip()))
|
||||
|
||||
|
||||
def is_structural(line: str) -> bool:
|
||||
"""Markdown-Zeilen, die nicht in Fließtext aufgehen sollen."""
|
||||
return bool(
|
||||
LIST_ITEM.match(line)
|
||||
or BLOCKQUOTE.match(line)
|
||||
or RULE.match(line)
|
||||
or IMAGE.match(line)
|
||||
or HTML_BLOCK.match(line)
|
||||
or line.lstrip().startswith("|")
|
||||
or line.startswith((" ", "\t"))
|
||||
)
|
||||
|
||||
|
||||
def normalize_markdown(markdown: str) -> str:
|
||||
result: list[str] = []
|
||||
paragraph: list[str] = []
|
||||
|
||||
in_fence = False
|
||||
fence_char = ""
|
||||
fence_length = 0
|
||||
|
||||
def add_blank_line() -> None:
|
||||
if result and result[-1] != "":
|
||||
result.append("")
|
||||
|
||||
def flush_paragraph() -> None:
|
||||
if paragraph:
|
||||
result.append(" ".join(paragraph))
|
||||
paragraph.clear()
|
||||
|
||||
for original in markdown.replace("\r\n", "\n").replace("\r", "\n").split("\n"):
|
||||
line = original.rstrip()
|
||||
marker = FENCE.match(line)
|
||||
|
||||
if in_fence:
|
||||
result.append(line)
|
||||
if marker:
|
||||
token = marker.group(1)
|
||||
if token[0] == fence_char and len(token) >= fence_length:
|
||||
in_fence = False
|
||||
continue
|
||||
|
||||
if marker:
|
||||
flush_paragraph()
|
||||
add_blank_line()
|
||||
result.append(line)
|
||||
in_fence = True
|
||||
fence_char = marker.group(1)[0]
|
||||
fence_length = len(marker.group(1))
|
||||
continue
|
||||
|
||||
if HEADING.match(line):
|
||||
flush_paragraph()
|
||||
add_blank_line()
|
||||
result.append(line)
|
||||
result.append("")
|
||||
continue
|
||||
|
||||
if is_structural(line):
|
||||
flush_paragraph()
|
||||
result.append(line)
|
||||
continue
|
||||
|
||||
if not line.strip():
|
||||
# Eine Leerzeile ist nur nach einem Satzende eine Absatzgrenze.
|
||||
# Andernfalls bleibt paragraph bestehen und die nächste
|
||||
# Textzeile wird mit einem Leerzeichen angehängt.
|
||||
if paragraph and ends_sentence(paragraph[-1]):
|
||||
flush_paragraph()
|
||||
add_blank_line()
|
||||
elif not paragraph:
|
||||
add_blank_line()
|
||||
continue
|
||||
|
||||
paragraph.append(line.strip())
|
||||
|
||||
flush_paragraph()
|
||||
return "\n".join(result).strip() + "\n"
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description=(
|
||||
"PDF in Markdown umwandeln; Absätze nur nach Satzende trennen."
|
||||
)
|
||||
)
|
||||
parser.add_argument("pdf", type=Path, help="Eingabe-PDF")
|
||||
parser.add_argument(
|
||||
"markdown",
|
||||
type=Path,
|
||||
nargs="?",
|
||||
help="Ausgabe-Datei; Standard: gleicher Name mit .md",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--ocr-language",
|
||||
default="deu",
|
||||
help="Tesseract-Sprachcode bei OCR (Standard: deu)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-ocr",
|
||||
action="store_true",
|
||||
help="OCR deaktivieren",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--keep-headers",
|
||||
action="store_true",
|
||||
help="Wiederkehrende Seitenköpfe übernehmen",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--keep-footers",
|
||||
action="store_true",
|
||||
help="Wiederkehrende Seitenfüße übernehmen",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
source = args.pdf.expanduser()
|
||||
target = (args.markdown or source.with_suffix(".md")).expanduser()
|
||||
|
||||
if not source.is_file():
|
||||
parser.error(f"Eingabedatei nicht gefunden: {source}")
|
||||
|
||||
if source.resolve() == target.resolve():
|
||||
parser.error("Eingabe- und Ausgabedatei müssen verschieden sein.")
|
||||
|
||||
try:
|
||||
markdown = pymupdf4llm.to_markdown(
|
||||
str(source),
|
||||
header=args.keep_headers,
|
||||
footer=args.keep_footers,
|
||||
use_ocr=not args.no_ocr,
|
||||
ocr_language=args.ocr_language,
|
||||
ocr_function=(
|
||||
tesseract_api.exec_ocr if not args.no_ocr else None
|
||||
),
|
||||
)
|
||||
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(normalize_markdown(markdown), encoding="utf-8")
|
||||
|
||||
except Exception as exc:
|
||||
print(f"Fehler bei der Konvertierung: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
print(f"Markdown geschrieben: {target}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
|
||||
249
History/pdf2md_v3.py
Normal file
249
History/pdf2md_v3.py
Normal file
|
|
@ -0,0 +1,249 @@
|
|||
#!/usr/bin/env python3
|
||||
"""PDF nach Markdown konvertieren und Fließtext nach Satzenden gliedern."""
|
||||
|
||||
import argparse
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pymupdf4llm
|
||||
from pymupdf4llm.ocr import tesseract_api
|
||||
|
||||
|
||||
HEADING = re.compile(r"^\s{0,3}#{1,6}\s+\S")
|
||||
|
||||
# Punkt, Fragezeichen, Ausrufezeichen oder Auslassungspunkte;
|
||||
# danach dürfen schließende Anführungszeichen/Klammern folgen.
|
||||
SENTENCE_END = re.compile(r"""[.!?…]+[”’"'»›)\]]*$""")
|
||||
|
||||
FENCE = re.compile(r"^\s{0,3}(`{3,}|~{3,})")
|
||||
LIST_ITEM = re.compile(r"^\s{0,3}(?:[-+*]|\d+[.)])\s+")
|
||||
BLOCKQUOTE = re.compile(r"^\s{0,3}>\s?")
|
||||
RULE = re.compile(r"^\s{0,3}(?:-\s*){3,}$|^\s{0,3}(?:\*\s*){3,}$")
|
||||
IMAGE = re.compile(r"^\s*!\[.*?\]\(.*\)\s*$")
|
||||
|
||||
# Beispiel:
|
||||
# **==> picture [426 x 480] intentionally omitted <==**
|
||||
OMITTED_IMAGE = re.compile(
|
||||
r"^\s*(?:\*\*|__)?\s*==>\s*"
|
||||
r"(?:picture|image|bild)\b.*?"
|
||||
r"(?:intentionally omitted|omitted|ausgelassen)\s*"
|
||||
r"<==\s*(?:\*\*|__)?\s*$",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
CONTENTS_TITLE = re.compile(
|
||||
r"^\s*(?:#{1,6}\s+)?"
|
||||
r"(?:inhaltsverzeichnis|inhalt|contents|table of contents)"
|
||||
r"\s*$",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
# Inhaltsverzeichnis mit Füllpunkten oder Abstand vor einer Seitenzahl:
|
||||
# Kapitel 1 .......... 12
|
||||
# Kapitel 1 12
|
||||
# Kapitel 1 xiv
|
||||
TOC_WITH_PAGE = re.compile(
|
||||
r"^\s*(?:[-*]\s+)?\S.+?"
|
||||
r"(?:\.{2,}|\s{2,})"
|
||||
r"(?:\d+|[ivxlcdm]+)\s*$",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
# Einträge ohne Seitenzahl, sofern bereits ein Inhaltsverzeichnis erkannt wurde:
|
||||
# 1 Einleitung
|
||||
# 1.2 Vorgeschichte
|
||||
# Kapitel 3 – Die Reise
|
||||
TOC_NUMBERED = re.compile(
|
||||
r"^\s*(?:[-*]\s+)?"
|
||||
r"(?:\d+(?:\.\d+)*\.?\s+|Kapitel\s+\d+\b|Chapter\s+\d+\b)"
|
||||
r".+",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
HTML_BLOCK = re.compile(
|
||||
r"^\s*</?(?:table|tr|td|th|div|pre|p|h[1-6])\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
def ends_sentence(text: str) -> bool:
|
||||
return bool(SENTENCE_END.search(text.rstrip()))
|
||||
|
||||
|
||||
def is_other_structure(line: str) -> bool:
|
||||
"""Markdown-Strukturen, die nicht zu Fließtext verbunden werden."""
|
||||
return bool(
|
||||
LIST_ITEM.match(line)
|
||||
or BLOCKQUOTE.match(line)
|
||||
or RULE.match(line)
|
||||
or IMAGE.match(line)
|
||||
or HTML_BLOCK.match(line)
|
||||
or line.lstrip().startswith("|")
|
||||
or line.startswith((" ", "\t"))
|
||||
)
|
||||
|
||||
|
||||
def normalize_markdown(markdown: str) -> str:
|
||||
result: list[str] = []
|
||||
paragraph: list[str] = []
|
||||
|
||||
in_fence = False
|
||||
fence_char = ""
|
||||
fence_length = 0
|
||||
in_contents = False
|
||||
|
||||
def blank_line() -> None:
|
||||
if result and result[-1] != "":
|
||||
result.append("")
|
||||
|
||||
def flush_paragraph() -> None:
|
||||
if paragraph:
|
||||
result.append(" ".join(paragraph))
|
||||
paragraph.clear()
|
||||
|
||||
def standalone(line: str) -> None:
|
||||
flush_paragraph()
|
||||
blank_line()
|
||||
result.append(line)
|
||||
result.append("")
|
||||
|
||||
lines = markdown.replace("\r\n", "\n").replace("\r", "\n").split("\n")
|
||||
|
||||
for original in lines:
|
||||
line = original.rstrip()
|
||||
marker = FENCE.match(line)
|
||||
|
||||
if in_fence:
|
||||
result.append(line)
|
||||
if marker:
|
||||
token = marker.group(1)
|
||||
if token[0] == fence_char and len(token) >= fence_length:
|
||||
in_fence = False
|
||||
continue
|
||||
|
||||
if marker:
|
||||
flush_paragraph()
|
||||
blank_line()
|
||||
result.append(line)
|
||||
in_fence = True
|
||||
fence_char = marker.group(1)[0]
|
||||
fence_length = len(marker.group(1))
|
||||
continue
|
||||
|
||||
if not line.strip():
|
||||
# Leerzeile nur dann als Absatzgrenze behandeln,
|
||||
# wenn der bisherige Fließtext mit einem Satzende endet.
|
||||
if paragraph and ends_sentence(paragraph[-1]):
|
||||
flush_paragraph()
|
||||
blank_line()
|
||||
elif not paragraph:
|
||||
blank_line()
|
||||
continue
|
||||
|
||||
if CONTENTS_TITLE.match(line):
|
||||
standalone(line)
|
||||
in_contents = True
|
||||
continue
|
||||
|
||||
if HEADING.match(line):
|
||||
standalone(line)
|
||||
in_contents = False
|
||||
continue
|
||||
|
||||
if OMITTED_IMAGE.match(line):
|
||||
standalone(line)
|
||||
continue
|
||||
|
||||
if TOC_WITH_PAGE.match(line):
|
||||
standalone(line)
|
||||
continue
|
||||
|
||||
if in_contents and TOC_NUMBERED.match(line):
|
||||
standalone(line)
|
||||
continue
|
||||
|
||||
if in_contents and ends_sentence(line):
|
||||
# Erster normaler Satz nach einem erkannten Inhaltsverzeichnis.
|
||||
in_contents = False
|
||||
|
||||
if is_other_structure(line):
|
||||
flush_paragraph()
|
||||
result.append(line)
|
||||
continue
|
||||
|
||||
# Einfache Zeilenumbrüche und nicht bestätigte Absatzgrenzen
|
||||
# werden beim späteren Flush durch Leerzeichen ersetzt.
|
||||
paragraph.append(line.strip())
|
||||
|
||||
flush_paragraph()
|
||||
return "\n".join(result).strip() + "\n"
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="PDF in Markdown umwandeln; Fließtext nur nach Satzende trennen."
|
||||
)
|
||||
parser.add_argument("pdf", type=Path, help="Eingabe-PDF")
|
||||
parser.add_argument(
|
||||
"markdown",
|
||||
type=Path,
|
||||
nargs="?",
|
||||
help="Ausgabe-Datei; Standard: gleicher Name mit .md",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--ocr-language",
|
||||
default="deu",
|
||||
help="Tesseract-Sprachcode bei OCR (Standard: deu)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-ocr",
|
||||
action="store_true",
|
||||
help="OCR deaktivieren",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--keep-headers",
|
||||
action="store_true",
|
||||
help="Wiederkehrende Seitenköpfe übernehmen",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--keep-footers",
|
||||
action="store_true",
|
||||
help="Wiederkehrende Seitenfüße übernehmen",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
source = args.pdf.expanduser()
|
||||
target = (args.markdown or source.with_suffix(".md")).expanduser()
|
||||
|
||||
if not source.is_file():
|
||||
parser.error(f"Eingabedatei nicht gefunden: {source}")
|
||||
if source.resolve() == target.resolve():
|
||||
parser.error("Eingabe- und Ausgabedatei müssen verschieden sein.")
|
||||
|
||||
try:
|
||||
markdown = pymupdf4llm.to_markdown(
|
||||
str(source),
|
||||
header=args.keep_headers,
|
||||
footer=args.keep_footers,
|
||||
use_ocr=not args.no_ocr,
|
||||
ocr_language=args.ocr_language,
|
||||
ocr_function=(
|
||||
tesseract_api.exec_ocr if not args.no_ocr else None
|
||||
),
|
||||
)
|
||||
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(normalize_markdown(markdown), encoding="utf-8")
|
||||
|
||||
except Exception as exc:
|
||||
print(f"Fehler bei der Konvertierung: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
print(f"Markdown geschrieben: {target}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
|
||||
269
History/pdf2md_v4.py
Normal file
269
History/pdf2md_v4.py
Normal file
|
|
@ -0,0 +1,269 @@
|
|||
#!/usr/bin/env python3
|
||||
"""PDF nach Markdown konvertieren, Absätze gliedern und optional Bilder speichern."""
|
||||
|
||||
import argparse
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pymupdf4llm
|
||||
from pymupdf4llm.ocr import tesseract_api
|
||||
|
||||
|
||||
HEADING = re.compile(r"^\s{0,3}#{1,6}\s+\S")
|
||||
|
||||
# Satzzeichen, optional gefolgt von schließenden Anführungszeichen/Klammern.
|
||||
SENTENCE_END = re.compile(r"""[.!?…]+[”’"'»›)\]]*$""")
|
||||
|
||||
FENCE = re.compile(r"^\s{0,3}(`{3,}|~{3,})")
|
||||
LIST_ITEM = re.compile(r"^\s{0,3}(?:[-+*]|\d+[.)])\s+")
|
||||
BLOCKQUOTE = re.compile(r"^\s{0,3}>\s?")
|
||||
RULE = re.compile(r"^\s{0,3}(?:-\s*){3,}$|^\s{0,3}(?:\*\s*){3,}$")
|
||||
IMAGE = re.compile(r"^\s*!\[.*?\]\(.*\)\s*$")
|
||||
|
||||
# Beispiel: **==> picture [426 x 480] intentionally omitted <==**
|
||||
OMITTED_IMAGE = re.compile(
|
||||
r"^\s*(?:\*\*|__)?\s*==>\s*"
|
||||
r"(?:picture|image|bild)\b.*?"
|
||||
r"(?:intentionally omitted|omitted|ausgelassen)\s*"
|
||||
r"<==\s*(?:\*\*|__)?\s*$",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
CONTENTS_TITLE = re.compile(
|
||||
r"^\s*(?:#{1,6}\s+)?"
|
||||
r"(?:inhaltsverzeichnis|inhalt|contents|table of contents)"
|
||||
r"\s*$",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
TOC_WITH_PAGE = re.compile(
|
||||
r"^\s*(?:[-*]\s+)?\S.+?"
|
||||
r"(?:\.{2,}|\s{2,})"
|
||||
r"(?:\d+|[ivxlcdm]+)\s*$",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
TOC_NUMBERED = re.compile(
|
||||
r"^\s*(?:[-*]\s+)?"
|
||||
r"(?:\d+(?:\.\d+)*\.?\s+|Kapitel\s+\d+\b|Chapter\s+\d+\b)"
|
||||
r".+",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
HTML_BLOCK = re.compile(
|
||||
r"^\s*</?(?:table|tr|td|th|div|pre|p|h[1-6])\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
def ends_sentence(text: str) -> bool:
|
||||
return bool(SENTENCE_END.search(text.rstrip()))
|
||||
|
||||
|
||||
def is_other_structure(line: str) -> bool:
|
||||
"""Markdown-Strukturen, die nicht zu Fließtext verbunden werden."""
|
||||
return bool(
|
||||
LIST_ITEM.match(line)
|
||||
or BLOCKQUOTE.match(line)
|
||||
or RULE.match(line)
|
||||
or IMAGE.match(line)
|
||||
or HTML_BLOCK.match(line)
|
||||
or line.lstrip().startswith("|")
|
||||
or line.startswith((" ", "\t"))
|
||||
)
|
||||
|
||||
|
||||
def normalize_markdown(markdown: str) -> str:
|
||||
result: list[str] = []
|
||||
paragraph: list[str] = []
|
||||
|
||||
in_fence = False
|
||||
fence_char = ""
|
||||
fence_length = 0
|
||||
in_contents = False
|
||||
|
||||
def blank_line() -> None:
|
||||
if result and result[-1] != "":
|
||||
result.append("")
|
||||
|
||||
def flush_paragraph() -> None:
|
||||
if paragraph:
|
||||
result.append(" ".join(paragraph))
|
||||
paragraph.clear()
|
||||
|
||||
def standalone(line: str) -> None:
|
||||
flush_paragraph()
|
||||
blank_line()
|
||||
result.append(line)
|
||||
result.append("")
|
||||
|
||||
lines = markdown.replace("\r\n", "\n").replace("\r", "\n").split("\n")
|
||||
|
||||
for original in lines:
|
||||
line = original.rstrip()
|
||||
marker = FENCE.match(line)
|
||||
|
||||
if in_fence:
|
||||
result.append(line)
|
||||
if marker:
|
||||
token = marker.group(1)
|
||||
if token[0] == fence_char and len(token) >= fence_length:
|
||||
in_fence = False
|
||||
continue
|
||||
|
||||
if marker:
|
||||
flush_paragraph()
|
||||
blank_line()
|
||||
result.append(line)
|
||||
in_fence = True
|
||||
fence_char = marker.group(1)[0]
|
||||
fence_length = len(marker.group(1))
|
||||
continue
|
||||
|
||||
if not line.strip():
|
||||
if paragraph and ends_sentence(paragraph[-1]):
|
||||
flush_paragraph()
|
||||
blank_line()
|
||||
elif not paragraph:
|
||||
blank_line()
|
||||
continue
|
||||
|
||||
if CONTENTS_TITLE.match(line):
|
||||
standalone(line)
|
||||
in_contents = True
|
||||
continue
|
||||
|
||||
if HEADING.match(line):
|
||||
standalone(line)
|
||||
in_contents = False
|
||||
continue
|
||||
|
||||
if OMITTED_IMAGE.match(line):
|
||||
standalone(line)
|
||||
continue
|
||||
|
||||
if TOC_WITH_PAGE.match(line):
|
||||
standalone(line)
|
||||
continue
|
||||
|
||||
if in_contents and TOC_NUMBERED.match(line):
|
||||
standalone(line)
|
||||
continue
|
||||
|
||||
if in_contents and ends_sentence(line):
|
||||
in_contents = False
|
||||
|
||||
if is_other_structure(line):
|
||||
flush_paragraph()
|
||||
result.append(line)
|
||||
continue
|
||||
|
||||
paragraph.append(line.strip())
|
||||
|
||||
flush_paragraph()
|
||||
return "\n".join(result).strip() + "\n"
|
||||
|
||||
|
||||
def make_image_links_relative(markdown: str, image_dir: Path) -> str:
|
||||
"""Absolute Bildpfade durch Pfade relativ zur Markdown-Datei ersetzen."""
|
||||
absolute_dir = image_dir.resolve().as_posix()
|
||||
relative_dir = image_dir.name
|
||||
|
||||
return markdown.replace(
|
||||
absolute_dir + "/",
|
||||
relative_dir + "/",
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description=(
|
||||
"PDF in Markdown umwandeln; Fließtext nach Satzenden gliedern "
|
||||
"und optional Grafiken speichern."
|
||||
)
|
||||
)
|
||||
parser.add_argument("pdf", type=Path, help="Eingabe-PDF")
|
||||
parser.add_argument(
|
||||
"markdown",
|
||||
type=Path,
|
||||
nargs="?",
|
||||
help="Ausgabe-Datei; Standard: gleicher Name mit .md",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--ocr-language",
|
||||
default="deu",
|
||||
help="Tesseract-Sprachcode bei OCR (Standard: deu)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-ocr",
|
||||
action="store_true",
|
||||
help="OCR deaktivieren",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--images",
|
||||
action="store_true",
|
||||
help="Grafiken speichern und in Markdown verlinken",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--keep-headers",
|
||||
action="store_true",
|
||||
help="Wiederkehrende Seitenköpfe übernehmen",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--keep-footers",
|
||||
action="store_true",
|
||||
help="Wiederkehrende Seitenfüße übernehmen",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
source = args.pdf.expanduser()
|
||||
target = (args.markdown or source.with_suffix(".md")).expanduser()
|
||||
|
||||
if not source.is_file():
|
||||
parser.error(f"Eingabedatei nicht gefunden: {source}")
|
||||
|
||||
if source.resolve() == target.resolve():
|
||||
parser.error("Eingabe- und Ausgabedatei müssen verschieden sein.")
|
||||
|
||||
image_dir = target.parent / f"{target.stem}_images"
|
||||
|
||||
try:
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
if args.images:
|
||||
image_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
markdown = pymupdf4llm.to_markdown(
|
||||
str(source),
|
||||
header=args.keep_headers,
|
||||
footer=args.keep_footers,
|
||||
use_ocr=not args.no_ocr,
|
||||
ocr_language=args.ocr_language,
|
||||
ocr_function=(
|
||||
tesseract_api.exec_ocr if not args.no_ocr else None
|
||||
),
|
||||
write_images=args.images,
|
||||
image_path=str(image_dir.resolve()) if args.images else "",
|
||||
image_format="png",
|
||||
)
|
||||
|
||||
if args.images:
|
||||
markdown = make_image_links_relative(markdown, image_dir)
|
||||
|
||||
target.write_text(normalize_markdown(markdown), encoding="utf-8")
|
||||
|
||||
except Exception as exc:
|
||||
print(f"Fehler bei der Konvertierung: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
print(f"Markdown geschrieben: {target}")
|
||||
if args.images:
|
||||
print(f"Bilder gespeichert in: {image_dir}")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
|
||||
12
README.md
12
README.md
|
|
@ -22,13 +22,13 @@ dann EPUB“: **[BEDIENUNGSANLEITUNG.md](BEDIENUNGSANLEITUNG.md)**.
|
|||
|
||||
```bash
|
||||
# digitales oder gescanntes PDF -> Markdown (OCR wird automatisch entschieden)
|
||||
python3 pdf2md_v5.py buch.pdf buch.md
|
||||
python3 pdf2md.py buch.pdf buch.md
|
||||
|
||||
# zusätzlich Bilder und EPUB (Titel/Autor aus den PDF-Metadaten, Seite 1 als Cover)
|
||||
python3 pdf2md_v5.py --epub buch.pdf buch.md
|
||||
python3 pdf2md.py --epub buch.pdf buch.md
|
||||
|
||||
# Scan in Deutsch und Englisch, 8 parallele OCR-Prozesse
|
||||
python3 pdf2md_v5.py --ocr-language deu+eng --jobs 8 scan.pdf scan.md
|
||||
python3 pdf2md.py --ocr-language deu+eng --jobs 8 scan.pdf scan.md
|
||||
```
|
||||
|
||||
| Option | Bedeutung |
|
||||
|
|
@ -67,7 +67,7 @@ synthetisch erzeugt.
|
|||
## Grenzen
|
||||
|
||||
- Die Ergebnisqualität hängt von pymupdf4llm/pymupdf-layout ab. Zwei Schwächen der
|
||||
Bibliothek gleicht `pdf2md_v5.py` per Patch aus (Bindestriche am Zeilenende, Absätze in
|
||||
Bibliothek gleicht `pdf2md.py` per Patch aus (Bindestriche am Zeilenende, Absätze in
|
||||
Scan-Blöcken). Eine weitere ist bekannt: Bei Schriften mit Ligaturen (ff, fi) mit
|
||||
übergroßer Glyphenbox kann pymupdf4llm einzelne Zeilen verlieren. Bei den Ljutsch-Bänden
|
||||
tritt das nicht auf (geprüft per Wortabgleich PDF ↔ Markdown).
|
||||
|
|
@ -76,7 +76,7 @@ synthetisch erzeugt.
|
|||
|
||||
## Dateien
|
||||
|
||||
- `pdf2md_v5.py` – aktuelle Version
|
||||
- `pdf2md_v1.py` … `pdf2md_v4.py` – frühere Entwicklungsstände (siehe `Info/`)
|
||||
- `pdf2md.py` – aktuelle Version
|
||||
- `History/pdf2md_v1.py` … `History/pdf2md_v4.py` – frühere Entwicklungsstände (siehe `Info/`)
|
||||
- `Info/` – Recherche und Entstehungsgeschichte der Skripte
|
||||
- `tests/` – pytest-Suite
|
||||
|
|
|
|||
2
TODO.md
2
TODO.md
|
|
@ -38,7 +38,7 @@ speichern. Damit lassen sich:
|
|||
- EPUB aus den Einzeldateien: pandoc nimmt mehrere Eingabedateien in der Reihenfolge von
|
||||
`index.md`; Umsortieren = Reihenfolge im Index ändern.
|
||||
|
||||
**Ansatzpunkte im Code (`pdf2md_v5.py`):**
|
||||
**Ansatzpunkte im Code (`pdf2md.py`):**
|
||||
|
||||
- Schnitt **nach** `normalize_markdown()` an den Überschriften `##` (Teil) und `###` (Kapitel);
|
||||
die Ebenen kommen aus `load_outline()`.
|
||||
|
|
|
|||
|
|
@ -1,27 +0,0 @@
|
|||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<metadata>
|
||||
<identifier>ljutsch</identifier>
|
||||
<mediatype>texts</mediatype>
|
||||
<collection>opensource</collection>
|
||||
<collection>community</collection>
|
||||
<description>Reinhard Otto Kranz<br /></description>
|
||||
<scanner>Internet Archive HTML5 Uploader 1.6.4</scanner>
|
||||
<subject>ljutsch</subject>
|
||||
<title>Ljutsch</title>
|
||||
<uploader>schamhaar.friseur@gmail.com</uploader>
|
||||
<publicdate>2022-05-30 15:09:50</publicdate>
|
||||
<addeddate>2022-05-30 15:09:50</addeddate>
|
||||
<identifier-access>http://archive.org/details/ljutsch</identifier-access>
|
||||
<identifier-ark>ark:/13960/s2jf47qc96n</identifier-ark>
|
||||
<ppi>300</ppi>
|
||||
<curation>[curator]validator@archive.org[/curator][date]20220530154119[/date][comment]checked for malware[/comment]</curation>
|
||||
<ocr>tesseract 5.0.0-1-g862e</ocr>
|
||||
<ocr_parameters>-l deu+Latin</ocr_parameters>
|
||||
<ocr_module_version>0.0.15</ocr_module_version>
|
||||
<ocr_autonomous>true</ocr_autonomous>
|
||||
<ocr_detected_script>Latin</ocr_detected_script>
|
||||
<ocr_detected_script_conf>1.0000</ocr_detected_script_conf>
|
||||
<ocr_detected_lang>de</ocr_detected_lang>
|
||||
<ocr_detected_lang_conf>1.0000</ocr_detected_lang_conf>
|
||||
<language>deu</language>
|
||||
</metadata>
|
||||
|
|
@ -3,7 +3,7 @@
|
|||
import pymupdf
|
||||
import pytest
|
||||
|
||||
import pdf2md_v5 as p
|
||||
import pdf2md as p
|
||||
from conftest import needs_tesseract
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ import pytest
|
|||
|
||||
from conftest import ROOT, TITLE, needs_pandoc, needs_tesseract
|
||||
|
||||
SCRIPT = ROOT / "pdf2md_v5.py"
|
||||
SCRIPT = ROOT / "pdf2md.py"
|
||||
|
||||
|
||||
def run(*args, cwd=None) -> subprocess.CompletedProcess:
|
||||
|
|
@ -132,7 +132,7 @@ def test_png_on_request(pdfs, tmp_path):
|
|||
|
||||
|
||||
def test_image_links_relative_to_markdown_with_relative_output(pdfs, tmp_path):
|
||||
# Aufruf wie "python3 pdf2md_v5.py --images buch.pdf Output/buch.md" aus dem Projektordner
|
||||
# Aufruf wie "python3 pdf2md.py --images buch.pdf Output/buch.md" aus dem Projektordner
|
||||
run(pdfs["digital"], "Output/buch.md", "--images", cwd=tmp_path)
|
||||
md = (tmp_path / "Output" / "buch.md").read_text(encoding="utf-8")
|
||||
links = re.findall(r"!\[\]\(([^)]+)\)", md)
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ from pathlib import Path
|
|||
import pymupdf
|
||||
import pytest
|
||||
|
||||
import pdf2md_v5 as p
|
||||
import pdf2md as p
|
||||
|
||||
|
||||
def norm(md: str, outline=None, **kwargs) -> str:
|
||||
|
|
|
|||
Loading…
Reference in a new issue