diff --git a/BEDIENUNGSANLEITUNG.md b/BEDIENUNGSANLEITUNG.md index c2caaca..6324147 100644 --- a/BEDIENUNGSANLEITUNG.md +++ b/BEDIENUNGSANLEITUNG.md @@ -3,8 +3,10 @@ Dieses Programm ersetzt Formelbilder in den Markdown-Dateien von epub2md und pdf2md durch echte LaTeX-Formeln. Dadurch passen die Formeln beim Neubau als EPUB oder PDF in Größe und Schrift zum Text. -Es ersetzt eine Formel nur, wenn **drei unabhängige Bildmodelle dieselbe Formel lesen** (zwei lokale, eines -in der Cloud) und das gerenderte LaTeX dem Bild grob ähnelt. Bei Zweifeln fragt es dich in einer +Es liest jede Formel zuerst mit dem Formel-OCR **pix2text** und ersetzt sie nur, wenn das Bildmodell +**Qwen** dieselbe Formel liest (bei Uneinigkeit darf das zweite Bildmodell Gemma pix2text bestätigen) und +das gerenderte LaTeX dem Bild grob ähnelt. Eingesetzt wird die Lesart von pix2text, ohne ihre +überflüssigen Leerzeichen. Alles läuft lokal. Bei Zweifeln fragt es dich in einer Review-Seite im Browser. Ohne deine Zustimmung bleibt die Formel ein Bild. „Dieselbe Formel“ heißt: jedes Zeichen stimmt und steht an der richtigen Stelle. Abstände, Pfeillängen, @@ -41,12 +43,28 @@ CUDA_VISIBLE_DEVICES=2 ~/.local/bin/llama-server \ GPU 2 ist währenddessen für Chatterbox-TTS belegt. Beenden mit Strg+C in beiden Terminals. -### Dritte Lesart aus der Cloud +### pix2text (Formel-OCR) -Die Bestätigung kommt von **GLM-4.6V** über OpenRouter (andere Modellfamilie als Qwen und Gemma). Dafür -muss `OPENROUTER_API_KEY` gesetzt sein. GLM liest nur die Formeln, bei denen Qwen und Gemma schon einig -sind. Beim Gehrke-Brückenkurs (1505 Formelbilder) waren das 967 Anfragen für zusammen etwa 0,15–0,20 $. -Ohne Cloud (`--no-cloud`) wird nichts automatisch ersetzt, alles Einige landet im Review. +pix2text läuft ohne Server im Programm selbst, auf der ersten sichtbaren Grafikkarte (sonst auf der CPU, +dann ~5× langsamer). Einmalig: + +```bash +pip install "pix2text[multilingual]" +pip install --no-deps pix2tex munch # nur für post_process (Leerzeichen) +pip uninstall -y onnxruntime && pip install onnxruntime-gpu # für die GPU +``` + +Das Modell (~300 MB) wird beim ersten Lauf nach `~/.pix2text` geladen. Gerät wählen mit +`FORMULA_P2T_DEVICE=cpu` oder `cuda:N` (`N` zählt innerhalb von `CUDA_VISIBLE_DEVICES`). + +Gemma (Port 8012) wird nur für Formeln gefragt, bei denen pix2text und Qwen uneinig sind. Läuft Gemma +nicht, bleiben diese Formeln im Review. + +### Frühere Regel mit Cloud (optional) + +Bis Version 0.2.0 kam die Bestätigung von **GLM-4.6V** über OpenRouter (Qwen = Gemma = GLM). Diese Regel +ist weiter wählbar: im Arbeitsordner in `workspace.json` unter `settings` den Eintrag +`"consensus_rule": "vlm3"` setzen; dann muss `OPENROUTER_API_KEY` gesetzt sein. --- @@ -168,9 +186,10 @@ geändert, wird sie übersprungen. `--force` setzt sie trotzdem zurück, dann is ```bash python3 formula_img2latex.py scan AUSGABE # Bilder finden -python3 formula_img2latex.py recognize AUSGABE --engine local # lokal erkennen +python3 formula_img2latex.py recognize AUSGABE --engine p2t # pix2text (zuerst) +python3 formula_img2latex.py recognize AUSGABE --engine local # Qwen (bestätigt) python3 formula_img2latex.py verify AUSGABE # prüfen und entscheiden -python3 formula_img2latex.py recognize AUSGABE --engine cloud --only-unresolved +python3 formula_img2latex.py recognize AUSGABE --engine local2 --only-unresolved # Gemma (Stichwahl) python3 formula_img2latex.py verify AUSGABE python3 formula_img2latex.py status AUSGABE ``` @@ -214,6 +233,6 @@ Entscheidung mit Grund und den Bildabgleich. Der Ordner `vergleich/` wird nicht Ja. Das Bild hatte die Farbe eingebaut, LaTeX-Formeln haben keinen Hintergrund. **Werden Buchinhalte ins Internet geschickt?** -Nur für die dritte Lesart: die einzelnen Formelbilder, bei denen Qwen und Gemma einig sind, samt eines -kurzen Textausschnitts um die Formel (höchstens 240 Zeichen), gehen an OpenRouter/GLM. Mit `--no-cloud` bleibt alles lokal; dann wird nichts automatisch -ersetzt. +Nein, mit der Standardregel bleibt alles lokal. Nur die frühere Regel `vlm3` schickt die Formelbilder, bei +denen Qwen und Gemma einig sind, samt eines kurzen Textausschnitts um die Formel (höchstens 240 Zeichen) +an OpenRouter/GLM. diff --git a/CHANGELOG.md b/CHANGELOG.md index ef87880..e9aa946 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,31 @@ # Changelog +## 0.3.0 – 2026-10-09 + +Neue Standardregel `p2t`: **pix2text liest zuerst, Qwen bestätigt, Gemma nur bei Uneinigkeit**; GLM (Cloud) +wird nicht mehr benutzt. + +- **pix2text** (Formel-OCR MFR 1.5, ONNX) als Engine `p2t`, im Prozess, auf der GPU mit `onnxruntime-gpu` + (`FORMULA_P2T_DEVICE`: `auto`, `cpu`, `cuda:N`). Allein am häufigsten richtig (Göllmann Band 1: 90 % / + 85 %) und am schnellsten (0,1 s/Formel auf der GPU, Qwen 0,7 s, Gemma 1,8 s). +- **Regel:** übernommen wird immer pix2texts Lesart (trifft den Fettdruck am besten), wenn Qwen sie + bestätigt oder – nur bei Uneinigkeit gefragt – Gemma. Qwen und Gemma gemeinsam gegen pix2text gehen + an einen Menschen (so kamen an Band 1 die gemeinsamen Fehler der beiden Bildmodelle durch). Der + Schlusspunkt muss bestätigt sein. Bild-Veto und Pandoc wie bisher, geprüft auf pix2texts Lesart. + Die alte Regel bleibt als `"consensus_rule": "vlm3"` in `workspace.json` wählbar. +- **pix2text-Ausgabe bereinigt:** Leerzeichen mit `post_process` aus pix2tex (installiert mit + `--no-deps`), Fettschreibweisen `\mathrm{\boldmath~ x ~}` / `\mathrm{\bf~ r o t}` → + `\boldsymbol{x}` / `\mathbf{rot}`. An 400 Lesarten geprüft: gleiches gesetztes Bild. +- **Normalform:** `\operatorname{l i m}` = `\lim`, `\ddot{u}` in Text = ü, `^{\prime}` = `'`, + abschließendes `\\` vor `\end`, `\begin{array}[t]`, `\mathop{…}`. +- **calibrate:** beliebig viele Engines (`--engines`), Regel-Tabelle, `--reuse` rechnet aus einem + früheren Bericht ohne Modellabfragen; wahre Formeln aus dem alt-Text werden bereinigt + (einzeiliges `aligned`, `\mathop{}\!\mathrm{d}`, upgreek). +- **Werkzeuge:** `tools/zeittest.py` (Zeit und Trefferquote je Tool), `tools/regeltest.py` (Standardregel + an Formeln mit bekanntem TeX, Seite zum Mitbeurteilen; `--reuse`, `--exclude`). +- **Messung:** Göllmann Band 1 (2 × 150) 86 % / 85 % automatisch, 0 falsch (bisher 78 % / 70 %); + Band 2, zwei frische Stichproben zu 50: 78 % und 86 % automatisch, 0 falsch. + ## 0.2.0 – 2026-10-07 Entscheider ist jetzt der **Konsens dreier Bildmodelle** statt eines Pixelvergleichs. diff --git a/README.md b/README.md index 353afb7..cbdf4d0 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,8 @@ Nachbearbeitungs-Werkzeug für die Markdown-Ausgaben von [epub2md](https://kitux.de/forgejo/dschlueter/epub2md) und [pdf2md](https://kitux.de/forgejo/dschlueter/pdf2md): Formeln, die nur als Bild vorliegen, werden durch -LaTeX ersetzt – **aber nur, wenn drei unabhängige Bildmodelle dieselbe Formel lesen** und ein grober +LaTeX ersetzt – **aber nur, wenn zwei unabhängige Erkenner dieselbe Formel lesen** (zuerst das Formel-OCR +pix2text, bestätigt vom Bildmodell Qwen; bei Uneinigkeit darf Gemma pix2text bestätigen) und ein grober Bildabgleich nicht widerspricht. Alles andere geht in eine Review-Galerie oder bleibt Bild. Schritt-für-Schritt-Anleitung: [BEDIENUNGSANLEITUNG.md](BEDIENUNGSANLEITUNG.md). Änderungen je Version: @@ -14,13 +15,22 @@ Schritt-für-Schritt-Anleitung: [BEDIENUNGSANLEITUNG.md](BEDIENUNGSANLEITUNG.md) > Jedes Zeichen muss stimmen und an der richtigen Stelle relativ zu den anderen stehen. Abstände, > Pfeil- und Bruchstrichlängen, Schriftgröße, kursiv/aufrecht und Fettdruck dürfen abweichen. -Eine Formel wird automatisch ersetzt, wenn +Eine Formel wird automatisch ersetzt (Regel `p2t`, Standard), wenn -1. **Qwen3-VL-30B-A3B** (lokal, llama.cpp) und **Gemma-4-26B-A4B** (lokal) dieselbe Formel lesen, -2. **GLM-4.6V** (Cloud über OpenRouter, andere Modellfamilie) das bestätigt – ein fehlender Schlusspunkt - zählt dabei nicht, -3. Pandoc sie nach MathML umsetzen kann (sonst bliebe sie im EPUB/PDF roher TeX-Text) und -4. das gerenderte LaTeX dem Original grob ähnelt (Bild-Veto, elastischer Vergleich, Ähnlichkeit ≥ 0,65). +1. **pix2text** (Formel-OCR MFR 1.5, lokal, GPU) sie zuerst liest und **Qwen3-VL-30B-A3B** (Bildmodell, + lokal, llama.cpp) dieselbe Formel liest **oder**, wenn die beiden uneinig sind, **Gemma-4-26B-A4B** + (lokal) die Lesart von pix2text bestätigt. Übernommen wird immer **pix2texts Lesart** – sie trifft den + Fettdruck am besten. Ihre überflüssigen Leerzeichen entfernt vorher `post_process` aus pix2tex (LaTeX-OCR): + `\frac{\alpha} {3 6 0}` → `\frac{\alpha}{360}`; am gesetzten Bild ändert das nichts, geprüft an + 400 Lesarten. Auch der Schlusspunkt muss bestätigt sein, +2. Pandoc sie nach MathML umsetzen kann (sonst bliebe sie im EPUB/PDF roher TeX-Text) und +3. das gerenderte LaTeX dem Original grob ähnelt (Bild-Veto, elastischer Vergleich, Ähnlichkeit ≥ 0,65). + +Sind Qwen und Gemma gemeinsam gegen pix2text einig, entscheidet ein Mensch: Genau das sind die Fehler, +die beide Bildmodelle teilen (`\mathcal{E}_r` statt `\varepsilon_r`, erfundenes `\tilde{z}`). + +Die frühere Regel `vlm3` (Qwen = Gemma, bestätigt von **GLM-4.6V** über OpenRouter) bleibt wählbar: +`"consensus_rule": "vlm3"` unter `settings` in `workspace.json` des Arbeitsordners. „Dieselbe Formel“ prüft `consensus.py`: gleiche Normalform (Abstände, `\left`/`\right`, überflüssige Klammern, `\vec`/`\overrightarrow`, `\frac`/`\dfrac`, `\iff`/`\Leftrightarrow`, `pmatrix`/`\left(\begin{matrix}`, @@ -32,6 +42,25 @@ Sind sich die Modelle uneinig oder widerspricht das Veto, entscheidet ein Mensch ## Messung +### Regel `p2t` (Standard) + +Göllmann Band 1, 150 echte Bilder und 150 Nachbildungen im Gehrke-Stil, Lesarten aller Modelle: + +| Regel | echt: automatisch / falsch | Gehrke-Stil: automatisch / falsch | +| --- | --- | --- | +| pix2text = Qwen | 81 % / 0 | 76 % / 0 | +| + Stichwahl: Gemma bestätigt pix2text, Qwens Lesart (erste Fassung) | 85 % / 0 | 80 % / 0 | +| **pix2text zuerst und übernommen, bereinigt, Stichwahl Gemma** (Standard) | **86 % / 0** | **85 % / 0** | +| + Stichwahl auch Qwen = Gemma gegen pix2text | 92 % / 1 | 91 % / 2 | +| Qwen = Gemma = GLM (Regel `vlm3`) | 78 % / 0 | 70 % / 0 | + +Allein richtig: pix2text 90 % / 85 %, Qwen 86 % / 84 %, Gemma 86 % / 80 %, GLM 86 % / 83 %. +Zeit je Formel (seriell, 15 Formeln): pix2text 0,10 s (GPU) bzw. 0,51 s (CPU), Qwen 0,72 s, GLM 1,19 s, +Gemma 1,77 s. Gegenprobe an 50 frischen Formeln aus Göllmann Band 2 (`tools/regeltest.py`): +37 automatisch (74 %), **0 falsch**, 13 Review; Gemma wurde für 16 Formeln gefragt. + +### Regel `vlm3` (Standard bis 0.2.0) + An Formelbildern mit bekanntem TeX (Göllmann, Springer: TeX im alt-Text), je 150 echte Bilder und 150 Nachbildungen im Stil von Gehrke (LaTeX + dvipng mit Bitmap-Schriften, ~102 dpi, JPEG): @@ -63,12 +92,12 @@ Gehrke, Brückenkurs Mathematik (EPUB, 1505 Formelbilder): siehe [Stand](#stand) ```text scan Bildverweise in chapters/*.md (bzw. pdf2md-Markdown) finden, Formel/Abbildung vorklassifizieren -run in Paketen zu 50: Qwen + Gemma lesen (parallel), prüfen, GLM nur für Einige, prüfen; - nach jedem Paket eine Statuszeile +run in Paketen zu 50: pix2text, dann Qwen lesen, prüfen, Gemma nur für Uneinige, prüfen; + nach jedem Paket eine Statuszeile (Regel vlm3: Qwen + Gemma, GLM nur für Einige) review Zweifelsfälle im Browser: Original / Render / Overlay / Diff, LaTeX korrigieren und neu prüfen apply angenommene Formeln ins Markdown schreiben (aus einer Sicherung, idempotent) revert Markdown-Dateien auf den Stand vor apply zurücksetzen -recognize einzelne Engine (local, local2, cloud) laufen lassen +recognize einzelne Engine (local, local2, cloud, p2t) laufen lassen verify prüfen und entscheiden (ohne neue Modellabfragen) status Zählung je Status calibrate Messung an Formelbildern mit bekanntem TeX (--consensus: Konsensregel) @@ -90,14 +119,19 @@ Status je Formel: `auto_accepted`, `needs_review`, `rejected_keep_image`, `rende | --- | --- | --- | | `local` | llama.cpp `http://127.0.0.1:8011/v1` (Qwen3-VL-30B-A3B-Instruct Q4_K_M + mmproj) | `FORMULA_LOCAL_URL`, `FORMULA_LOCAL_MODEL` | | `local2` | llama.cpp `http://127.0.0.1:8012/v1` (Gemma-4-26B-A4B-it Q4_0 + mmproj) | `FORMULA_LOCAL2_URL`, `FORMULA_LOCAL2_MODEL` | -| `cloud` | OpenRouter, `z-ai/glm-4.6v` | `OPENROUTER_API_KEY`, `FORMULA_CLOUD_MODEL` | +| `cloud` | OpenRouter, `z-ai/glm-4.6v` (nur Regel `vlm3`) | `OPENROUTER_API_KEY`, `FORMULA_CLOUD_MODEL` | +| `p2t` | pix2text MFR 1.5 (ONNX) im Prozess, erste sichtbare GPU, sonst CPU | `FORMULA_P2T_DEVICE` (`auto`, `cpu`, `cuda:N`), `FORMULA_P2T_MODEL` | + +pix2text auf der GPU braucht `onnxruntime-gpu` statt `onnxruntime` (cuDNN kommt aus den nvidia-Paketen +von torch). Für `post_process` wird `pix2tex` ohne seine Abhängigkeiten installiert +(`pip install --no-deps pix2tex munch`), weil es sonst `timm==0.5.4` erzwingen würde. `cuda:N` zählt innerhalb von `CUDA_VISIBLE_DEVICES`. `FORMULA__THINKING=1` schaltet das Nachdenken wieder ein. Antworten sind auf 800 Tokens begrenzt (lange mehrzeilige Formeln brauchen ~400; einzelne Antworten von Gemma liefen sonst in Wiederholungsschleifen bis 165 s). Jede Lesart wird mit Modell, Prompt-Version und Nachdenk-Einstellung gespeichert; dieselbe Frage wird nie zweimal gestellt, ein abgebrochener Lauf setzt dort fort. -Kosten der Cloud-Stufe: GLM liest nur Formeln, bei denen Qwen und Gemma einig sind. Gehrke: 967 +Kosten der Cloud-Stufe (nur Regel `vlm3`): GLM liest nur Formeln, bei denen Qwen und Gemma einig sind. Gehrke: 967 Anfragen, Ø 2,6 s, zusammen rund 0,15–0,20 $. ## Bildvergleich (Veto) @@ -105,9 +139,9 @@ Anfragen, Ø 2,6 s, zusammen rund 0,15–0,20 $. `compare.py` rendert die Lesart mit pdflatex (Schrift je Buch automatisch: `cm`, `times`, `txfonts`, `palatino`, `stix`, `euler`), richtet sie auf das Original aus und verzerrt sie **elastisch** (dynamische Zeitverzerrung über Spalten und Zeilen; Dehnen kostet proportional zur Tinte), damit -Abstände und Längen keine Rolle spielen. Die Ähnlichkeit danach ist das Veto. Geprüft wird nur die -Lesart, die übernommen würde (die von Qwen; im Konsens ist sie gleichwertig zu den anderen) – eine -später eintreffende dritte Lesart löst keine neue Prüfung aus. Die Prüfergebnisse tragen eine +Abstände und Längen keine Rolle spielen. Die Ähnlichkeit danach ist das Veto. Geprüft wird nur +die Lesart, die übernommen werden kann: die von pix2text (Regel `vlm3`: die von Qwen). Bestätigende +Lesarten werden nie übernommen und brauchen keine Prüfung. Die Prüfergebnisse tragen eine Versionsnummer des Verfahrens und werden nach Änderungen neu berechnet. Der alte Pixel-Entscheider mit Verwechslungsvarianten und Kontrasttest bleibt als Modus `decide=pixel` erhalten. @@ -145,6 +179,11 @@ sind nicht Teil dieser Version. - `tools/vergleich.py AUSGABE` – feste Stichprobe (50 Formeln), verarbeitet wie `run`, schreibt `vergleich//index.html` mit Original, gerenderten Lesarten, Entscheidung und Veto-Overlay. - `tools/konsens_bericht.py BERICHT.json EPUB` – Übersichtsseite zu `calibrate --consensus --report`. +- `tools/zeittest.py EPUB` – Zeit und Trefferquote je Tool, seriell an 15 Formeln mit bekanntem TeX. +- `tools/regeltest.py EPUB [--n 50] [--reuse ergebnis.json]` – die Standardregel an Formeln mit bekanntem + TeX, wie `run`; Seite mit Original, wahrer Formel, allen Lesarten und Entscheidung. +- `calibrate --consensus --engines local,local2,cloud,p2t --report X.json` misst alle Regeln; + `--reuse X.json` rechnet sie ohne Modellabfragen neu. - `vergleich/` ist nicht versioniert (Ausschnitte aus Büchern). ## Stand @@ -209,7 +248,8 @@ formula_img2latex/ ├── pipeline.py scan, recognize, verify, Entscheidung (Konsens/Pixel), run in Paketen, apply/revert ├── consensus.py Normalform, Render-Gleichheit: sagen zwei Lesarten dasselbe? ├── decide.py Annahmeregeln und Schwellen (Konsens- und Pixel-Modus) -├── engines/ Vision-Modelle über die OpenAI-API (llama.cpp, OpenRouter), Logprobs, Prompt-Version +├── engines/ Vision-Modelle über die OpenAI-API (llama.cpp, OpenRouter), Logprobs, Prompt-Version; +│ p2t.py: pix2text-Formel-OCR (ONNX, CPU/GPU) ├── compare.py Ausrichtung, elastische Verzerrung, Kennzahlen, Kontrasttest, Diff-Bilder ├── confusables.py Verwechslungsvarianten (Pixel-Modus) ├── verify.py Bildprüfung einer Lesart (Veto bzw. Pixel-Entscheider) diff --git a/formula_img2latex/__init__.py b/formula_img2latex/__init__.py index 3c853e7..cae0f3a 100644 --- a/formula_img2latex/__init__.py +++ b/formula_img2latex/__init__.py @@ -1,3 +1,3 @@ """formula_img2latex: Formelbilder in Markdown geprüft durch LaTeX ersetzen.""" -__version__ = "0.2.0" +__version__ = "0.3.0" diff --git a/formula_img2latex/calibrate.py b/formula_img2latex/calibrate.py index 416f717..67a6287 100644 --- a/formula_img2latex/calibrate.py +++ b/formula_img2latex/calibrate.py @@ -43,6 +43,22 @@ from .verify import verify SYNTH_PX_PER_PT = 1.41 +# upright Greek of the upgreek package (Springer alt texts): pdflatex here has no upgreek, and +# upright vs. italic is the same symbol for the project +_UPGREEK = re.compile(r"\\[Uu]p(alpha|beta|gamma|delta|epsilon|varepsilon|zeta|eta|theta|vartheta|iota|kappa|lambda|mu|nu|" + r"xi|pi|varpi|rho|varrho|sigma|varsigma|tau|upsilon|phi|varphi|chi|psi|omega)(?![A-Za-z])", re.I) + + +def clean_truth(tex: str) -> str: + """Alt-text TeX as a plain formula: no one-line aligned wrapper, \\mathop{}\\!\\mathrm{d} -> \\mathrm{d}, + upgreek -> standard Greek. Only the spelling changes, never a symbol.""" + m = re.fullmatch(r"\s*\\begin\{aligned\}\s*(?:\\displaystyle)?(.*?)\\end\{aligned\}\s*", tex, re.S) + if m and "\\\\" not in m.group(1) and "&" not in m.group(1): + tex = m.group(1).strip() + tex = re.sub(r"\\mathop\{\}\\!", "", tex) + return _UPGREEK.sub(lambda g: "\\" + (g.group(1).capitalize() if g.group(0)[1] == "U" else g.group(1).lower()), tex) + + def corpus(epub: Path) -> list[tuple[str, bytes, str, bool]]: """(name, image bytes, true TeX, display) for every image with TeX in its alt text.""" out = [] @@ -59,7 +75,7 @@ def corpus(epub: Path) -> list[tuple[str, bytes, str, bool]]: if not (src and alt and alt.group(1).lstrip().startswith("$")): continue path = posixpath.normpath(posixpath.join(base, src.group(1))) - tex = strip_delimiters(html.unescape(alt.group(1))) + tex = clean_truth(strip_delimiters(html.unescape(alt.group(1)))) if path in names and not unsafe_reason(tex): display = "IEq" not in path and "inline" not in tag.lower() out.append((path, z.read(path), tex, display)) @@ -224,26 +240,63 @@ def _print(s: dict[str, Any], log: Any) -> None: # ------------------------------------------------------------------------- consensus calibration def _consensus_eval(args: tuple) -> dict[str, Any]: - """Worker: compare both readings with each other and with the truth; rough image check.""" + """Worker: compare all readings with each other and with the truth; rough image check.""" from .consensus import same_reading - name, data, truth, display, preset, ra, rb = args + from .pipeline import _strip_final_punct + name, data, truth, display, preset, readings = args os.environ.setdefault("OMP_NUM_THREADS", "1") - ta, tb = ra.get("tex") or "", rb.get("tex") or "" - res: dict[str, Any] = {"name": name, "truth": truth, "display": display, "a": ra, "b": rb, - "a_ok": bool(same_reading(ta, truth, display, preset)), - "b_ok": bool(same_reading(tb, truth, display, preset)), - "agree": same_reading(ta, tb, display, preset)} - res["veto"] = None - if res["agree"]: - v = verify(load_ink(data, name), ta, display, preset, check_variants=False) - res["veto"] = {"ok": v.ok, "reason": v.reason, "metrics": v.metrics} - return res + texes = [r.get("tex") or "" for r in readings] + ok = [bool(same_reading(t, truth, display, preset)) for t in texes] + agree: list[tuple[int, int, str]] = [] + for i in range(len(texes)): + for j in range(i + 1, len(texes)): + a, b = texes[i], texes[j] + # as in pipeline.decide_all_consensus: a confirming cloud reading may lack the final period; + # pix2text's reading is used under rule p2t, so it is compared strictly + if j >= 2 and readings[j].get("engine") != "p2t" and readings[i].get("engine") != "p2t": + a, b = _strip_final_punct(a), _strip_final_punct(b) + how = same_reading(a, b, display, preset) + if how: + agree.append((i, j, how)) + vetos: dict[int, dict[str, Any]] = {} + for k in range(len(texes)): # only readings that may be used: primary pair and pix2text (rule p2t) + if (k < 2 or readings[k].get("engine") == "p2t") and any(k in (i, j) for i, j, _ in agree): + v = verify(load_ink(data, name), texes[k], display, preset, check_variants=False) + vetos[k] = {"ok": v.ok, "reason": v.reason, "metrics": v.metrics} + pair = next((how for i, j, how in agree if (i, j) == (0, 1)), None) + # "a", "b", "agree", "veto": the two primary models (read by tools/konsens_bericht.py) + return {"name": name, "truth": truth, "display": display, "readings": readings, "ok": ok, "agree_all": agree, + "vetos": vetos, "a": readings[0], "b": readings[1], "a_ok": ok[0], "b_ok": ok[1], "agree": pair, + "veto": vetos.get(0) if pair else None} + + +def _reading_cache(report: Path | None) -> dict[tuple[str, str, str], dict[str, Any]]: + """(set label, image name, engine) -> reading from an earlier --report: models are not asked twice. + + pix2text readings are tidied again (reports from before tidy_p2t hold the raw spaced output).""" + from .engines.p2t import tidy_p2t + if not report or not report.exists(): + return {} + old = json.loads(report.read_text(encoding="utf-8")) + cache = {} + for s in old.get("sets", []): + for r in s.get("results", []): + for rd in r.get("readings") or [r.get("a") or {}, r.get("b") or {}]: + if rd.get("engine") and not rd.get("error"): + if rd["engine"] == "p2t" and rd.get("tex"): + rd = dict(rd, tex=tidy_p2t(rd.get("raw") or rd["tex"])) + cache[(s["label"], r["name"], rd["engine"])] = rd + return cache def calibrate_consensus(epubs: list[Path], samples: int = 150, preset: str | None = None, synthetic: bool = True, report: Path | None = None, workers: int = 0, quiet: bool = False, - engines: tuple[str, str] = ("local", "local2")) -> dict[str, Any]: - """Measure the two-model consensus rule on images whose TeX is known.""" + engines: tuple[str, ...] = ("local", "local2"), reuse: Path | None = None) -> dict[str, Any]: + """Measure consensus rules on images whose TeX is known. + + *engines*: the first two are the primary pair (local, local2); further engines (cloud, p2t) + give confirming readings. *reuse*: report of an earlier run whose readings are taken over. + """ from concurrent.futures import ThreadPoolExecutor from .decide import consensus_thresholds, decide_consensus @@ -253,10 +306,10 @@ def calibrate_consensus(epubs: list[Path], samples: int = 150, preset: str | Non if not quiet: print(m, file=sys.stderr, flush=True) + if len(engines) < 2: + raise ValueError("mindestens zwei Engines nötig") + cache = _reading_cache(reuse) eng = [make_engine(e) for e in engines] - for e in eng: - e.check() - log("Modelle: " + ", ".join(f"{e.name}={e.model}" for e in eng)) rng = random.Random(42) sets: list[tuple[str, list[tuple[str, bytes, str, bool]], str]] = [] for epub in epubs: @@ -268,21 +321,28 @@ def calibrate_consensus(epubs: list[Path], samples: int = 150, preset: str | Non if synthetic: syn = [(n + "#synth.jpg", img, t, d) for n, _d, t, d in smp for img in [synth_jpeg(t, d, rng)] if img] sets.append((f"{epub.stem[:40]} [dvipng, Gehrke-Stil]", syn, "cm")) + todo = [(k, label, i) for label, items, _p in sets for k in range(len(eng)) for i in range(len(items)) + if (label, items[i][0], eng[k].name) not in cache] + for k in sorted({k for k, _, _ in todo}): + eng[k].check() + log("Modelle: " + ", ".join(f"{e.name}={e.model}" for e in eng) + + (f"; {len(cache)} Lesarten aus {reuse}" if cache else "")) summary: dict[str, Any] = {"engines": [f"{e.name}={e.model}" for e in eng], "sets": []} for label, items, p in sets: log(f"{label}: {len(items)} Formeln lesen …") - readings: list[list[dict[str, Any]]] = [[{}] * len(items) for _ in eng] + readings: list[list[dict[str, Any]]] = [ + [cache.get((label, n, e.name), {}) for n, _d, _t, _disp in items] for e in eng] def ask(k: int, i: int) -> None: n, d, _t, disp = items[i] readings[k][i] = eng[k].recognize(d, n, "", disp).to_json() with ThreadPoolExecutor(max_workers=8) as pool: - list(pool.map(lambda ki: ask(*ki), [(k, i) for k in range(len(eng)) for i in range(len(items))])) - jobs = [(n, d, t, disp, p, readings[0][i], readings[1][i]) for i, (n, d, t, disp) in enumerate(items)] + list(pool.map(lambda ki: ask(*ki), [(k, i) for k, lb, i in todo if lb == label])) + jobs = [(n, d, t, disp, p, [readings[k][i] for k in range(len(eng))]) for i, (n, d, t, disp) in enumerate(items)] with ProcessPoolExecutor(max_workers=workers or max(1, (os.cpu_count() or 2) - 2)) as ex: results = list(ex.map(_consensus_eval, jobs)) - s = _summarize_consensus(label, results, consensus_thresholds, decide_consensus) + s = _summarize_consensus(label, results, [e.name for e in eng], consensus_thresholds, decide_consensus) summary["sets"].append(s) s["results"] = results _print_consensus(s, log) @@ -292,19 +352,74 @@ def calibrate_consensus(epubs: list[Path], samples: int = 150, preset: str | Non return summary -def _summarize_consensus(label: str, results: list[dict[str, Any]], consensus_thresholds: Any, +# Rules to compare: (label, mode, groups). A group lists engines; the first two are the primary pair +# and every further one must confirm it (decide_consensus with require_third). "any": accepted if one +# group accepts (the first accepting group supplies the reading); "all": every group must accept; +# "tiebreak": decide_pair_tiebreak with the group (Qwen, pix2text, Gemma) – the standard rule "p2t". +CONSENSUS_RULES: list[tuple[str, str, list[tuple[str, ...]]]] = [ + ("Qwen = Gemma", "any", [("local", "local2")]), + ("Qwen = Gemma = GLM (bisher)", "any", [("local", "local2", "cloud")]), + ("B: Qwen = Gemma = p2t", "any", [("local", "local2", "p2t")]), + ("Qwen = Gemma, GLM oder p2t bestätigt", "any", [("local", "local2", "cloud"), ("local", "local2", "p2t")]), + ("A: 3 von 4 (p2t entscheidet bei 2:1)", "any", + [("local", "local2", "cloud"), ("local", "local2", "p2t"), ("local", "cloud", "p2t"), ("local2", "cloud", "p2t")]), + ("C: Qwen = Gemma = GLM = p2t", "all", [("local", "local2", "cloud"), ("local", "local2", "p2t")]), + ("Standard: p2t = Qwen, Stichwahl Gemma = p2t", "tiebreak", [("local", "p2t", "local2")]), +] + + +def _rule_outcome(r: dict[str, Any], idx: list[int], th: dict[str, float], decide_fn: Any, + tiebreak: bool = False) -> int | None: + """Index of the accepted reading if the readings *idx* (primary pair first) accept, else None.""" + sub = [r["readings"][k] for k in idx] + pos = {k: n for n, k in enumerate(idx)} + agree = [(pos[i], pos[j], how) for i, j, how in r["agree_all"] if i in pos and j in pos] + t = th if tiebreak else dict(th, require_third=1 if len(idx) > 2 else 0) + _st, _why, use = decide_fn(sub, agree, None, True, None, t) # which reading would be used + if use is None: + return None + veto = r["vetos"].get(idx[use]) + st, _why, use = decide_fn(sub, agree, veto, True, None, t) + return idx[use] if st == "auto_accepted" else None + + +def _summarize_consensus(label: str, results: list[dict[str, Any]], names: list[str], consensus_thresholds: Any, decide_consensus: Any) -> dict[str, Any]: + from .decide import decide_pair_tiebreak n = len(results) - out: dict[str, Any] = {"label": label, "formulas": n, + out: dict[str, Any] = {"label": label, "formulas": n, "engines": names, + "correct": {e: sum(r["ok"][k] for r in results) for k, e in enumerate(names)}, "a_correct": sum(r["a_ok"] for r in results), "b_correct": sum(r["b_ok"] for r in results), "agree": sum(bool(r["agree"]) for r in results), "agree_wrong": sum(bool(r["agree"]) and not r["a_ok"] for r in results), - "sweep": []} + "rules": [], "sweep": []} out["agree_wrong_examples"] = [{"truth": r["truth"], "a": r["a"].get("tex"), "b": r["b"].get("tex")} for r in results if r["agree"] and not r["a_ok"]][:10] + th0 = consensus_thresholds() + for rule, mode, groups in CONSENSUS_RULES: + if any(e not in names for g in groups for e in g): + continue + acc = wrong = 0 + examples = [] + for r in results: + if mode == "tiebreak": + used = [_rule_outcome(r, [names.index(e) for e in g], th0, decide_pair_tiebreak, True) + for g in groups] + else: + used = [_rule_outcome(r, [names.index(e) for e in g], th0, decide_consensus) for g in groups] + ok_used = [u for u in used if u is not None] + use = (ok_used[0] if ok_used else None) if mode == "any" else (used[0] if None not in used else None) + if use is None: + continue + acc += 1 + if not r["ok"][use]: + wrong += 1 + examples.append({"truth": r["truth"], "used": r["readings"][use].get("tex"), + "readings": {e: rd.get("tex") for e, rd in zip(names, r["readings"])}}) + out["rules"].append({"rule": rule, "accepted": acc, "wrong": wrong, "wrong_examples": examples[:5]}) for pm in (0.0, 0.8, 0.9, 0.95, 0.98): for pn in (0.0, 0.2, 0.3, 0.5): - th = consensus_thresholds({"min_p_mean": pm, "min_p_min": pn}) + th = consensus_thresholds({"min_p_mean": pm, "min_p_min": pn, "require_third": 0}) acc = wrong = 0 for r in results: agree = [(0, 1, r["agree"])] if r["agree"] else [] @@ -319,11 +434,14 @@ def _summarize_consensus(label: str, results: list[dict[str, Any]], consensus_th def _print_consensus(s: dict[str, Any], log: Any) -> None: n = max(1, s["formulas"]) log(f" {s['label']}: {s['formulas']} Formeln") - log(f" Modell A richtig {s['a_correct'] / n:.0%}, Modell B richtig {s['b_correct'] / n:.0%}, " - f"einig {s['agree'] / n:.0%}, davon FALSCH {s['agree_wrong']}") - for ex in s["agree_wrong_examples"][:5]: - log(f" einig, aber falsch: {ex['a']} (richtig: {ex['truth']})") - log(" Schwellen (Mittel/Minimum) → automatisch angenommen / davon falsch:") + log(" allein richtig: " + ", ".join(f"{e} {c / n:.0%}" for e, c in s["correct"].items()) + + f"; Qwen und Gemma einig {s['agree'] / n:.0%}, davon FALSCH {s['agree_wrong']}") + log(" Regel → automatisch angenommen / davon falsch:") + for row in s["rules"]: + log(f" {row['rule']:<42} {row['accepted']:4d} ({row['accepted'] / n:.0%}) / {row['wrong']}") + for ex in row["wrong_examples"][:3]: + log(f" falsch: {ex['used']} (richtig: {ex['truth']})") + log(" Qwen = Gemma mit Schwellen (Mittel/Minimum) → automatisch angenommen / davon falsch:") for row in s["sweep"]: log(f" p_mean ≥ {row['min_p_mean']:.2f}, p_min ≥ {row['min_p_min']:.2f}: " f"{row['accepted']:4d} ({row['accepted'] / n:.0%}) / {row['wrong']}") diff --git a/formula_img2latex/cli.py b/formula_img2latex/cli.py index febc1fa..d22ee95 100644 --- a/formula_img2latex/cli.py +++ b/formula_img2latex/cli.py @@ -123,7 +123,12 @@ def build_parser() -> argparse.ArgumentParser: sp.add_argument("--report", type=Path, default=None, help="Bericht als JSON speichern") sp.add_argument("--workers", type=int, default=0) sp.add_argument("--consensus", action="store_true", - help="Konsensregel messen: beide lokalen Modelle (local, local2) lesen die Formeln") + help="Konsensregeln messen: die Engines aus --engines lesen die Formeln") + sp.add_argument("--engines", default="local,local2", + help="Engines für --consensus, die ersten beiden bilden das Hauptpaar " + "(Standard local,local2; z. B. local,local2,cloud,p2t)") + sp.add_argument("--reuse", type=Path, default=None, + help="Lesarten aus einem früheren --report übernehmen (Modelle nicht erneut fragen)") sp.add_argument("--quiet", "-q", action="store_true") return p @@ -166,7 +171,8 @@ def _dispatch(args: argparse.Namespace) -> int: if args.command == "calibrate" and args.consensus: from .calibrate import calibrate_consensus calibrate_consensus(args.epubs, samples=args.samples, preset=None if args.preset == "auto" else args.preset, - synthetic=args.synthetic, report=args.report, workers=args.workers, quiet=args.quiet) + synthetic=args.synthetic, report=args.report, workers=args.workers, quiet=args.quiet, + engines=tuple(e.strip() for e in args.engines.split(",") if e.strip()), reuse=args.reuse) return EXIT_OK if args.command == "calibrate": from .calibrate import calibrate diff --git a/formula_img2latex/consensus.py b/formula_img2latex/consensus.py index eb0ca3e..b86af31 100644 --- a/formula_img2latex/consensus.py +++ b/formula_img2latex/consensus.py @@ -26,7 +26,7 @@ _DROP = {"\\,", "\\;", "\\:", "\\!", "\\ ", "~", "\\quad", "\\qquad", "\\left", "\\Bigg", "\\bigl", "\\bigr", "\\Bigl", "\\Bigr", "\\biggl", "\\biggr", "\\displaystyle", "\\textstyle", "\\scriptstyle", "\\limits", "\\nolimits", "\\middle", "\\hfill", "\\enspace", "\\thinspace", "\\medspace"} # wrappers whose argument is kept (upright vs. italic is the same symbol) -_UNWRAP = {"\\mathrm", "\\mathit", "\\operatorname", "\\textrm", "\\textit", "\\mathnormal", "\\mbox", "\\text", +_UNWRAP = {"\\mathrm", "\\mathit", "\\operatorname", "\\mathop", "\\textrm", "\\textit", "\\mathnormal", "\\mbox", "\\text", "\\textnormal", "\\textup"} _SYNONYM = { "\\overrightarrow": "\\vec", "\\overline": "\\bar", "\\widehat": "\\hat", "\\widetilde": "\\tilde", @@ -73,7 +73,8 @@ def canonical(tex: str, bold: bool = False) -> str: body = re.sub(r"\\\\\s*(?=\\end\{)", "", body) # a final \\ before \end{...} adds no row body = re.sub(r"\{?\\ss\{?\}?\}?", "ß", body) body = re.sub(r"\{([^{}]*)\}\s*\\atop\s*\{([^{}]*)\}", r"\\substack{\1\\\\\2}", body) - body = re.sub(r"\\begin\{array\}\{[^{}]*\}", r"\\begin{matrix}", body).replace("\\end{array}", "\\end{matrix}") + body = re.sub(r"\\begin\{array\}\s*(?:\[[^\]]*\])?\s*\{[^{}]*\}", r"\\begin{matrix}", body) # [t]: position only + body = body.replace("\\end{array}", "\\end{matrix}") for env, (lft, rgt) in _ENV_DELIMS.items(): body = body.replace(f"\\begin{{{env}}}", f"{lft} \\begin{{MATRIX}}").replace(f"\\end{{{env}}}", f"\\end{{MATRIX}} {rgt}") body = re.sub(r"\\not\s*=", r"\\ne ", body) diff --git a/formula_img2latex/decide.py b/formula_img2latex/decide.py index f488b53..6ff0b51 100644 --- a/formula_img2latex/decide.py +++ b/formula_img2latex/decide.py @@ -137,7 +137,16 @@ def decide_consensus(readings: list[dict[str, Any]], agree: list[tuple[int, int, return "needs_review", f"zwei Modelle einig ({how}), dritte Lesart weicht ab: {texts}", i tie = " (dritte Lesart bestätigt)" use = min(i, j) # the primary model's reading (it is the one checked against the image) - problems = [p for p in (_sure(readings[i], th), _sure(readings[j], th)) if p] + problems = _checks([readings[i], readings[j]], veto, pandoc_ok, book_px_per_pt, th) + if problems: + return "needs_review", f"einig ({how}){tie}; " + "; ".join(problems), use + return "auto_accepted", f"zwei Modelle einig ({how}){tie}, sicher, Bild-Veto bestanden", use + + +def _checks(agreeing: list[dict[str, Any]], veto: dict[str, Any] | None, pandoc_ok: bool, + book_px_per_pt: float | None, th: dict[str, float]) -> list[str]: + """Problems of an agreed reading: unsure readings, Pandoc, rough image check (German reasons).""" + problems = [p for p in (_sure(r, th) for r in agreeing) if p] if not pandoc_ok: problems.append("Pandoc kann die Formel nicht nach MathML umsetzen") if veto is None or not veto.get("ok"): @@ -149,6 +158,53 @@ def decide_consensus(readings: list[dict[str, Any]], agree: list[tuple[int, int, if (th["veto_scale_dev"] > 0 and book_px_per_pt and abs(m.get("px_per_pt", book_px_per_pt) / book_px_per_pt - 1) > th["veto_scale_dev"]): problems.append(f"Bild-Veto: Schriftgröße weicht um {abs(m['px_per_pt'] / book_px_per_pt - 1):.0%} ab") + return problems + + +# Marker in the reason of a formula that waits for the tie-break reading (see pipeline.needs_tiebreak). +TIEBREAK_MISSING = "Stichwahl fehlt" + + +def decide_pair_tiebreak(readings: list[dict[str, Any] | None], agree: list[tuple[int, int, str]], + veto: dict[str, Any] | None, pandoc_ok: bool, book_px_per_pt: float | None, + th: dict[str, float]) -> tuple[str, str, int | None]: + """Rule "p2t": pix2text reads first, Qwen confirms, Gemma only where the two disagree. + + The reading used is always pix2text's (spaces tidied; it reads bold type best). *readings*: + [Qwen, pix2text, Gemma or None]. *agree*: pairs (i, j, how) of readings that say the same. + *veto*: verification of pix2text's reading. Returns (status, reason, index of the reading to use + or None). + + Gemma may only confirm pix2text. Qwen and Gemma together against pix2text go to a human: in the + measurement exactly these were the errors both VLMs share (\\mathcal{E}_r for \\varepsilon_r, an + invented \\tilde{z}). + """ + q, p = readings[0], readings[1] + g = readings[2] if len(readings) > 2 else None + pairs = {frozenset((i, j)): how for i, j, how in agree} + if q is None or p is None: + missing = "Qwen" if q is None else "pix2text" + return "needs_review", f"Lesart von {missing} fehlt", 1 if p is not None and p.get("tex") else None + if q.get("is_formula") is False and (g is None or g.get("is_formula") is False) and (g or not p.get("tex")): + return "not_formula", "Qwen" + (" und Gemma" if g else "") + ": keine Formel", None + first = 1 if p.get("tex") else None # reading shown for review: pix2text's + if frozenset((0, 1)) in pairs: + how, who, confirm = pairs[frozenset((0, 1))], "pix2text und Qwen einig", q + elif g is None: + return "needs_review", f"pix2text und Qwen uneinig, {TIEBREAK_MISSING}", first + elif frozenset((1, 2)) in pairs: + how, who, confirm = pairs[frozenset((1, 2))], "Stichwahl: Gemma bestätigt pix2text", g + elif frozenset((0, 2)) in pairs: + return ("needs_review", "Qwen und Gemma einig gegen pix2text – Mensch entscheidet: " + f"Qwen: {q.get('tex') or '–'}", first) + else: + texts = " | ".join(f"{n}: {r.get('tex') or '–'}" for n, r in (("Qwen", q), ("pix2text", p), ("Gemma", g))) + return "needs_review", f"alle uneinig: {texts}", first + use = 1 + if not p.get("tex"): + return "needs_review", f"{who}, aber ohne LaTeX", None + # the confidence check (token probabilities, off by default) covers the confirming VLM reading + problems = _checks([confirm], veto, pandoc_ok, book_px_per_pt, th) if problems: - return "needs_review", f"einig ({how}){tie}; " + "; ".join(problems), use - return "auto_accepted", f"zwei Modelle einig ({how}){tie}, sicher, Bild-Veto bestanden", use + return "needs_review", f"{who} ({how}); " + "; ".join(problems), use + return "auto_accepted", f"{who} ({how}), Bild-Veto bestanden", use diff --git a/formula_img2latex/engines/__init__.py b/formula_img2latex/engines/__init__.py index b10e61b..ab06388 100644 --- a/formula_img2latex/engines/__init__.py +++ b/formula_img2latex/engines/__init__.py @@ -47,5 +47,5 @@ def make_engine(name: str, model: str | None = None, url: str | None = None) -> ) if name == "p2t": return Pix2TextEngine(model=model or os.environ.get("FORMULA_P2T_MODEL", DEFAULT_P2T_MODEL), - device=os.environ.get("FORMULA_P2T_DEVICE", "cpu")) + device=os.environ.get("FORMULA_P2T_DEVICE", "auto")) raise ValueError(f"unbekannte Engine {name!r} (local, local2, cloud, p2t)") diff --git a/formula_img2latex/engines/p2t.py b/formula_img2latex/engines/p2t.py index f543fb6..c553ed7 100644 --- a/formula_img2latex/engines/p2t.py +++ b/formula_img2latex/engines/p2t.py @@ -1,12 +1,14 @@ """Pix2Text formula OCR (MFR model) as a local candidate generator. Not a vision-language model but a dedicated image-to-LaTeX transformer: another kind of model than -the VLM engines, so its errors are likely to be different ones. Runs on the CPU (ONNX), no server. +the VLM engines, so its errors are likely to be different ones. Runs in-process (ONNX), no server: +on the CPU, or on a GPU with onnxruntime-gpu (device "cuda:N"; N counts within CUDA_VISIBLE_DEVICES). """ from __future__ import annotations import io +import re import threading import time @@ -17,9 +19,48 @@ from .vlm import Recognition, prepare_image DEFAULT_MODEL = "mfr-1.5" +# Pix2Text's bold: \mathrm{\boldmath~ x ~} (an error in math mode), \mathrm{{\bf~ r o t}}, \mathrm{\bf~ r o t} +_BOLDMATH = re.compile(r"\\mathrm\s*\{\s*\\boldmath\s*~?\s*([^{}~\\]*?)\s*~?\s*\}") +_BF = re.compile(r"\\mathrm\s*\{\s*(\{)?\s*\\bf\s*~?\s*([^{}~\\]*?)\s*~?\s*(?(1)\})\s*\}") + + +def tidy_p2t(tex: str) -> str: + """Pix2Text output without superfluous spaces: `\\frac{\\alpha} {3 6 0}` -> `\\frac{\\alpha}{360}`. + + The spaces are removed by post_process of pix2tex (LaTeX-OCR), which keeps a space between two + letters (`a x`, `\\cdot x`) and the control space `\\ `; TeX ignores the removed spaces in math + mode, so the typeset formula does not change. Before that, Pix2Text's bold spellings become + \\boldsymbol / \\mathbf. + """ + from pix2tex.utils.utils import post_process # imports torch: only when needed + + tex = _BOLDMATH.sub(lambda m: "\\boldsymbol{" + re.sub(r"\s+", "", m.group(1)) + "}", tex) + tex = _BF.sub(lambda m: "\\mathbf{" + re.sub(r"\s+", "", m.group(2)) + "}", tex) + return post_process(tex).strip() + + +def resolve_device(device: str) -> str: + """Device "auto": the first visible GPU if onnxruntime has CUDA, else the CPU.""" + if device != "auto": + return device + import onnxruntime + return "cuda" if "CUDAExecutionProvider" in onnxruntime.get_available_providers() else "cpu" + + +def ort_provider(device: str) -> tuple[str, dict]: + """ONNX Runtime provider and its options for "cpu", "cuda" or "cuda:N". + + Set explicitly: pix2text otherwise takes the first available provider (TensorRT with onnxruntime-gpu). + """ + if device == "cpu": + return "CPUExecutionProvider", {} + if device == "cuda" or device.startswith("cuda:"): + return "CUDAExecutionProvider", {"device_id": int(device.partition(":")[2] or 0)} + raise ValueError(f"unbekanntes Gerät {device!r} (cpu, cuda, cuda:N)") + class Pix2TextEngine: - def __init__(self, model: str = DEFAULT_MODEL, device: str = "cpu") -> None: + def __init__(self, model: str = DEFAULT_MODEL, device: str = "auto") -> None: self.name = "p2t" self.model_name = model self.model = f"pix2text-{model}" @@ -38,7 +79,21 @@ class Pix2TextEngine: raise FormulaError(f"pix2text ist nicht installiert ({e}): pip install 'pix2text[multilingual]'", EXIT_TOOL) from e try: - self._ocr = LatexOCR(model_name=self.model_name, model_backend="onnx", device=self.device) + import pix2tex.utils.utils # noqa: F401 (post_process for tidy_p2t) + except ImportError as e: + raise FormulaError(f"pix2tex ist nicht installiert ({e}): pip install --no-deps pix2tex munch", + EXIT_TOOL) from e + self.device = resolve_device(self.device) + provider, options = ort_provider(self.device) + if provider == "CUDAExecutionProvider": + import onnxruntime + if provider not in onnxruntime.get_available_providers(): + raise FormulaError(f"Gerät {self.device}: onnxruntime ohne CUDA – pip install onnxruntime-gpu", + EXIT_TOOL) + onnxruntime.preload_dlls() # cuDNN/cuBLAS from the nvidia-* pip packages (installed with torch) + try: + self._ocr = LatexOCR(model_name=self.model_name, model_backend="onnx", device=self.device, + more_model_configs={"provider": provider, "provider_options": options}) except Exception as e: # download or model files broken: any exception type is possible here raise FormulaError(f"Pix2Text-Modell {self.model_name} nicht ladbar ({e})", EXIT_TOOL) from e @@ -51,9 +106,10 @@ class Pix2TextEngine: with self._lock: # _one_batch instead of recognize(): the latter draws a tqdm bar for every image out = self._ocr._one_batch([img], None)[0] - tex = self._ocr.post_process(out["text"]).strip() + raw = self._ocr.post_process(out["text"]).strip() except Exception as e: # model not loadable or one unreadable image: must not stop the run return Recognition(self.name, self.model, None, None, "", str(e)[:300], time.time() - t0) score = float(out.get("score") or 0.0) or None - return Recognition(self.name, self.model, tex or None, None, tex[:2000], None, time.time() - t0, + tex = tidy_p2t(raw) + return Recognition(self.name, self.model, tex or None, None, raw[:2000], None, time.time() - t0, p_mean=score) diff --git a/formula_img2latex/pipeline.py b/formula_img2latex/pipeline.py index 1f4a5a0..db80c66 100644 --- a/formula_img2latex/pipeline.py +++ b/formula_img2latex/pipeline.py @@ -20,8 +20,10 @@ from PIL import Image from .checks import pandoc_mathml_ok from .classify import classify from .compare import diff_image -from .decide import consensus_thresholds, decide, decide_consensus, thresholds +from .decide import (TIEBREAK_MISSING, consensus_thresholds, decide, decide_consensus, decide_pair_tiebreak, + thresholds) from .engines import make_engine +from .errors import FormulaError from .engines.vlm import PROMPT_VERSION from .imageio import load_ink from .render import PRESETS @@ -113,10 +115,15 @@ def needs_third(f: Formula) -> bool: return f.status == "needs_review" and "dritte Lesart fehlt" in (f.reason or "") +def needs_tiebreak(f: Formula) -> bool: + """Rule "p2t": pix2text and Qwen disagree; Gemma's tie-break reading is missing.""" + return f.status == "needs_review" and TIEBREAK_MISSING in (f.reason or "") + + def recognize(source: Source, ws: Workspace, engine_name: str, model: str | None = None, url: str | None = None, only_unresolved: bool = False, include_figures: bool = False, limit: int = 0, workers: int = 4, quiet: bool = False, only_ids: set[str] | None = None, - only_needing_third: bool = False) -> dict[str, int]: + only_needing_third: bool = False, only_needing_tiebreak: bool = False) -> dict[str, int]: engine = make_engine(engine_name, model, url) engine.check() todo = [] @@ -127,6 +134,8 @@ def recognize(source: Source, ws: Workspace, engine_name: str, model: str | None continue if only_needing_third and not needs_third(f): continue + if only_needing_tiebreak and not needs_tiebreak(f): + continue if any(c.get("engine") == engine.name and c.get("model") == engine.model and c.get("prompt") == PROMPT_VERSION and c.get("thinking") == engine.thinking for c in f.candidates): continue # cached: never ask the same model the same question twice @@ -246,6 +255,8 @@ def verify_all(source: Source, ws: Workspace, preset: str | None = None, include formulas = [f for f in ws.all() if _wanted(f, include_figures) and _distinct_candidates(f) and (only_ids is None or f.id in only_ids)] mode = decide_mode(ws) + if mode == "consensus" and consensus_rule(ws) == "vlm3": + mode = "consensus-vlm3" formulas = [f for f in formulas if _verify_texes(f, mode)] if not redo: formulas = [f for f in formulas if not _verified_current(f, mode)] @@ -288,10 +299,13 @@ VERIFY_VERSION = 3 # bump when the comparison changes, so stored image checks def _verify_texes(f: Formula, mode: str) -> list[str]: - """Readings to check against the image. Consensus: only the reading that would be used (the - first primary reading) – the veto needs nothing else, and a later third reading changes nothing.""" - if mode != "consensus": + """Readings to check against the image: only the one that may be used. Mode "consensus" (rule + p2t): pix2text's. "consensus-vlm3": the first primary reading (Qwen's). "pixel": all candidates.""" + if mode == "pixel": return _distinct_candidates(f) + if mode == "consensus": + p = latest_by_engine(f).get("p2t") + return [split_tag(strip_delimiters(p["tex"]))[0]] if p and p.get("tex") else [] for r in readings_of(f): if r.get("tex"): return [split_tag(strip_delimiters(r["tex"]))[0]] @@ -312,6 +326,20 @@ def decide_mode(ws: Workspace) -> str: return ws.settings.get("decide") or "consensus" +def consensus_rule(ws: Workspace) -> str: + """'p2t' (pix2text = Qwen, Gemma as tie-break; default) or 'vlm3' (Qwen = Gemma = GLM).""" + return ws.settings.get("consensus_rule") or "p2t" + + +def latest_by_engine(f: Formula) -> dict[str, dict]: + """Latest error-free reading per engine.""" + latest: dict[str, dict] = {} + for c in f.candidates: + if not c.get("error"): + latest[c.get("engine", "")] = c + return latest + + def readings_of(f: Formula) -> list[dict]: """Latest reading per model; local engines first (primary pair), then the rest (tie-break).""" latest: dict[str, dict] = {} @@ -369,21 +397,32 @@ def _tex_key(tex: str) -> str: def decide_all_consensus(ws: Workspace, quiet: bool, preset: str, only_ids: set[str] | None = None) -> dict[str, int]: from .consensus import same_reading th = consensus_thresholds(ws.settings.get("consensus_thresholds")) + rule = consensus_rule(ws) + decide_fn = decide_pair_tiebreak if rule == "p2t" else decide_consensus formulas = [f for f in ws.all() if f.candidates and f.status not in HUMAN_STATUSES and (only_ids is None or f.id in only_ids)] plan = [] for f in formulas: - rs = readings_of(f) bold = bool(th.get("bold_relevant")) + if rule == "p2t": + latest = latest_by_engine(f) + rs = [latest.get("local"), latest.get("p2t"), latest.get("local2")] + pairs = [(0, 1), (1, 2), (0, 2)] + else: + rs = readings_of(f) + pairs = [(i, j) for i in range(len(rs)) for j in range(i + 1, len(rs))] agree = [] - for i in range(len(rs)): - for j in range(i + 1, len(rs)): - a, b = rs[i].get("tex") or "", rs[j].get("tex") or "" - if j >= 2: # the confirming reading: a missing final period/comma is no contradiction - a, b = _strip_final_punct(a), _strip_final_punct(b) - how = same_reading(a, b, f.display, preset, bold) - if how: - agree.append((i, j, how)) + for i, j in pairs: + if rs[i] is None or rs[j] is None: + continue + a, b = rs[i].get("tex") or "", rs[j].get("tex") or "" + # rule vlm3: a confirming reading (j >= 2) may lack the final period or comma. Rule p2t: strict, + # pix2text's reading is the one used, so its final punctuation must be confirmed too + if rule != "p2t" and j >= 2: + a, b = _strip_final_punct(a), _strip_final_punct(b) + how = same_reading(a, b, f.display, preset, bold) + if how: + agree.append((i, j, how)) plan.append((f, rs, agree)) # Book-wide scale from agreed readings that resemble their original well. by_tex = {f.id: {_tex_key(r["tex"]): r for r in (f.verification or {}).get("all", [])} for f, _, _ in plan} @@ -397,20 +436,20 @@ def decide_all_consensus(ws: Workspace, quiet: bool, preset: str, only_ids: set[ ws.settings["book_px_per_pt"] = book_ppt chosen = [] for f, rs, agree in plan: - status, reason, use = decide_consensus(rs, agree, None, True, book_ppt, th) # preliminary: which reading + status, reason, use = decide_fn(rs, agree, None, True, book_ppt, th) # preliminary: which reading chosen.append(rs[use]["tex"] if use is not None and rs[use].get("tex") else None) pandoc = pandoc_mathml_ok([(t or "x", f.display) for (f, _, _), t in zip(plan, chosen)]) if plan else [] counts: dict[str, int] = {} for (f, rs, agree), tex, pok in zip(plan, chosen, pandoc): veto = by_tex[f.id].get(_tex_key(tex)) if tex else None - status, reason, use = decide_consensus(rs, agree, veto, pok, book_ppt, th) + status, reason, use = decide_fn(rs, agree, veto, pok, book_ppt, th) f.status, f.reason = status, reason f.final_tex = _with_tag(f, tex) if status == "auto_accepted" and tex else None f.decided_at = now_iso() ws.save(f) counts[status] = counts.get(status, 0) + 1 ws.save_meta() - log("Entscheidung (Konsens): " + ", ".join(f"{k} {v}" for k, v in sorted(counts.items())), quiet) + log(f"Entscheidung (Konsens, Regel {rule}): " + ", ".join(f"{k} {v}" for k, v in sorted(counts.items())), quiet) return counts @@ -526,25 +565,39 @@ def run_consensus(source: Source, ws: Workspace, chunk: int = 50, preset: str | include_figures: bool = False, limit: int = 0, cloud_model: str | None = None, use_cloud: bool = True, ids: list[str] | None = None, quiet: bool = False, report: Callable[[str], None] | None = None) -> dict[str, int]: - """Process formulas in chunks: local, local2, verify; cloud only where both agree; verify again. + """Process formulas in chunks and report a status line (processed / accepted / review / image) after each. - After every chunk a status line (processed / accepted / review / image) is reported. + Rule "p2t": pix2text, then Qwen, verify; Gemma only where they disagree; decide again. + Rule "vlm3": Qwen and Gemma, verify; cloud only where both agree; verify again. """ say = report or (lambda m: print(m, flush=True)) todo = ids or [f.id for f in ws.all() if _wanted(f, include_figures)] if limit: todo = todo[:limit] + rule = consensus_rule(ws) + tiebreak_ok = True totals: dict[str, int] = {} done: list[str] = [] for start in range(0, len(todo), chunk): part = set(todo[start:start + chunk]) - # both local models run on their own GPU: ask them at the same time - with ThreadPoolExecutor(max_workers=2) as both: - for fut in [both.submit(recognize, source, ws, eng, only_ids=part, include_figures=include_figures, - quiet=True) for eng in ("local", "local2")]: - fut.result() + if rule == "p2t": # pix2text first (fast), then Qwen + for eng in ("p2t", "local"): + recognize(source, ws, eng, only_ids=part, include_figures=include_figures, quiet=True) + else: # both VLMs run on their own GPU: ask them at the same time + with ThreadPoolExecutor(max_workers=2) as both: + for fut in [both.submit(recognize, source, ws, eng, only_ids=part, include_figures=include_figures, + quiet=True) for eng in ("local", "local2")]: + fut.result() verify_all(source, ws, preset, include_figures, quiet=True, only_ids=part) - if use_cloud: + if rule == "p2t" and tiebreak_ok and any(needs_tiebreak(f) for f in map(ws.load, part) if f): + try: + recognize(source, ws, "local2", only_ids=part, only_needing_tiebreak=True, + include_figures=include_figures, quiet=True) + verify_all(source, ws, preset, include_figures, quiet=True, only_ids=part) + except FormulaError as e: # Gemma not running: the disputed formulas stay in review + tiebreak_ok = False + say(f"Stichwahl (Gemma) nicht möglich, strittige Formeln bleiben im Review: {e}") + if rule != "p2t" and use_cloud: recognize(source, ws, "cloud", cloud_model, only_ids=part, only_needing_third=True, include_figures=include_figures, workers=8, quiet=True) verify_all(source, ws, preset, include_figures, quiet=True, only_ids=part) diff --git a/tests/test_pipeline.py b/tests/test_pipeline.py index 4a032c7..e988b24 100644 --- a/tests/test_pipeline.py +++ b/tests/test_pipeline.py @@ -4,6 +4,8 @@ from __future__ import annotations from pathlib import Path +import pytest + from conftest import needs_latex, needs_pandoc from formula_img2latex import pipeline from formula_img2latex.cli import main @@ -120,6 +122,7 @@ def _reading(engine: str, model: str, tex: str | None, p_mean: float = 0.99, p_m @needs_pandoc def test_consensus_mode(epub2md_dir: Path) -> None: source, ws = pipeline.open_all(epub2md_dir, None, create=True) + ws.settings["consensus_rule"] = "vlm3" # the earlier rule Qwen = Gemma = GLM pipeline.scan(source, ws, quiet=True) by = {f.asset.rsplit("/", 1)[1]: f for f in ws.all()} inline, display = by["0001-inline.jpg"], by["0002-display.jpg"] @@ -155,6 +158,63 @@ def test_consensus_mode(epub2md_dir: Path) -> None: assert i.status == "needs_review" and "weicht ab" in i.reason, i.reason +@needs_latex +@needs_pandoc +def test_consensus_rule_p2t(epub2md_dir: Path) -> None: + """Standard rule: pix2text first, Qwen confirms; where they differ Gemma may confirm pix2text. + The reading used is always pix2text's.""" + source, ws = pipeline.open_all(epub2md_dir, None, create=True) + pipeline.scan(source, ws, quiet=True) + by = {f.asset.rsplit("/", 1)[1]: f for f in ws.all()} + inline, display = by["0001-inline.jpg"], by["0002-display.jpg"] + inline.candidates = [_reading("local", "Q", r"L=\{-\sqrt{2};\sqrt{2}\}"), + _reading("p2t", "P", r"L=\{-\sqrt{2};\sqrt{2}\}")] + display.candidates = [_reading("local", "Q", r"x_{3,k}=\arcsin\left(-\frac{3}{5}\right)+2k\pi"), + _reading("p2t", "P", r"x_{2,k}=\operatorname{arcsin}(-\frac{3}{5})+2k\pi")] + ws.save(inline) + ws.save(display) + pipeline.verify_all(source, ws, preset="cm", workers=2, quiet=True) + i, d = ws.load(inline.id), ws.load(display.id) + assert i.status == "auto_accepted" and "pix2text und Qwen einig" in i.reason, i.reason + assert i.final_tex == r"L=\{-\sqrt{2};\sqrt{2}\}" # pix2text's reading + assert pipeline.needs_tiebreak(d), d.reason + + # Gemma confirms pix2text -> accepted with pix2text's reading + d.candidates.append(_reading("local2", "G", r"x_{2,k} = \arcsin(-\frac{3}{5}) + 2k\pi")) + ws.save(d) + pipeline.verify_all(source, ws, preset="cm", workers=2, quiet=True) + d = ws.load(d.id) + assert d.status == "auto_accepted" and "Gemma bestätigt pix2text" in d.reason, d.reason + assert d.final_tex == r"x_{2,k}=\operatorname{arcsin}(-\frac{3}{5})+2k\pi" + + # Qwen and Gemma together against pix2text -> a human decides + d.candidates[-1] = _reading("local2", "G", r"x_{3,k}=\arcsin(-\frac{3}{5})+2k\pi") + d.status = "pending" + ws.save(d) + pipeline.verify_all(source, ws, preset="cm", workers=2, quiet=True) + d = ws.load(d.id) + assert d.status == "needs_review" and "Mensch entscheidet" in d.reason, d.reason + + +def test_pair_tiebreak_rule() -> None: + from formula_img2latex.decide import consensus_thresholds, decide_pair_tiebreak + th = consensus_thresholds() + q, p, g = _reading("local", "Q", "x^2"), _reading("p2t", "P", "x^{2}"), _reading("local2", "G", "x^2") + veto = {"ok": True, "metrics": {"ncc": 0.9, "px_per_pt": 1.4}} + assert decide_pair_tiebreak([q, p, None], [(0, 1, "normalform")], veto, True, 1.4, th)[::2] == ("auto_accepted", 1) + bad = {"ok": True, "metrics": {"ncc": 0.5, "px_per_pt": 1.4}} + assert decide_pair_tiebreak([q, p, None], [(0, 1, "normalform")], bad, True, 1.4, th)[0] == "needs_review" # veto + assert decide_pair_tiebreak([q, p, None], [(0, 1, "normalform")], veto, False, 1.4, th)[0] == "needs_review" # pandoc + assert "Stichwahl fehlt" in decide_pair_tiebreak([q, p, None], [], veto, True, 1.4, th)[1] + assert decide_pair_tiebreak([q, p, g], [(1, 2, "normalform")], veto, True, 1.4, th)[::2] == ("auto_accepted", 1) + assert decide_pair_tiebreak([q, p, g], [(0, 2, "normalform")], veto, True, 1.4, th)[0] == "needs_review" + assert decide_pair_tiebreak([q, p, g], [], veto, True, 1.4, th)[0] == "needs_review" + assert "fehlt" in decide_pair_tiebreak([q, None, None], [], veto, True, 1.4, th)[1] + nq, ng = _reading("local", "Q", None, is_formula=False), _reading("local2", "G", None, is_formula=False) + assert decide_pair_tiebreak([nq, p, ng], [], None, True, 1.4, th)[0] == "not_formula" + assert decide_pair_tiebreak([nq, p, None], [], None, True, 1.4, th)[0] == "needs_review" # ask Gemma first + + def test_consensus_rules() -> None: from formula_img2latex.decide import consensus_thresholds, decide_consensus th = consensus_thresholds({"min_p_mean": 0.9, "min_p_min": 0.3, "require_third": 0}) @@ -181,6 +241,29 @@ def test_consensus_rules() -> None: assert decide_consensus([a, b, c], [(0, 2, "normalform")], veto, True, 1.4, strict)[0] == "needs_review" # no 2:1 +def test_calibration_rules_with_four_engines() -> None: + from formula_img2latex.calibrate import _summarize_consensus + from formula_img2latex.decide import consensus_thresholds, decide_consensus + names = ["local", "local2", "cloud", "p2t"] + veto = {"ok": True, "metrics": {"ncc": 0.9}} + + def res(agree: list[tuple[int, int]], ok: list[bool]) -> dict: + rs = [_reading(e, e, "x") for e in names] + pair = "normalform" if (0, 1) in agree else None + return {"truth": "x", "readings": rs, "ok": ok, "agree_all": [(i, j, "normalform") for i, j in agree], + "vetos": {0: veto, 1: veto, 3: veto}, "a": rs[0], "b": rs[1], "a_ok": ok[0], "b_ok": ok[1], + "agree": pair, "veto": veto if pair else None} + + all4 = [(0, 1), (0, 2), (0, 3), (1, 2), (1, 3), (2, 3)] + results = [res(all4, [True] * 4), # everyone agrees + res([(0, 1), (0, 3), (1, 3)], [False, False, True, False]), # Q = G = p2t, all wrong, GLM right + res([(0, 2), (0, 3), (2, 3)], [True, False, True, True])] # Gemma differs: 2:1 decided by p2t + s = _summarize_consensus("t", results, names, consensus_thresholds, decide_consensus) + got = [(r["accepted"], r["wrong"]) for r in s["rules"]] + # Qwen = Gemma | + GLM | B (+ p2t) | GLM or p2t confirms | A (3 of 4) | C (all four) | standard rule p2t + assert got == [(2, 1), (1, 0), (2, 1), (2, 1), (3, 1), (1, 0), (3, 1)] + + def test_canonical_form() -> None: from formula_img2latex.consensus import canonical same = [(r"x^{2}+y^{2}", "x^2 + y^2"), (r"\vec{b}", r"\overrightarrow{b}"), (r"\left(a\right)", "(a)"), @@ -190,6 +273,8 @@ def test_canonical_form() -> None: (r"\operatorname{l i m}_{x \to0} \frac{\operatorname{s i n} x} {x}", r"\lim_{x\to 0}\frac{\sin x}{x}"), (r"\mathrm{f \ddot{u} r ~ a l l e ~} x", r"\text{für alle } x"), (r"f^{\prime\prime}(x)", "f''(x)"), (r"\left( \begin{matrix} {1} & {2} \\ \end{matrix} \right)", r"\begin{pmatrix}1&2\end{pmatrix}"), + (r"\begin{array}[t]{cc}a&b\end{array}", r"\begin{array}{cc}a&b\end{array}"), + (r"\mathop{\mathbf{rot}}\boldsymbol{v}", r"\mathbf{rot}\,\boldsymbol{v}"), (r"{\int}_{0}^{\infty} \! e^{-x^{2}} \, d x \!=\! \frac{\sqrt\pi} 2", r"\int_0^\infty e^{-x^2}\,dx=\frac{\sqrt{\pi}}{2}")] differ = [(r"x_{2,k}", r"x_{2;k}"), ("x^{2}", "x_{2}"), (r"\mathbb{R}", "R"), ("a_{n+1}", "a_n+1"), (r"\frac{a}{b+c}", r"\frac{a}{b}+c"), (r"x \in t", r"x \int"), (r"\sqrt[3]{x}", r"\sqrt{x}"), @@ -198,10 +283,30 @@ def test_canonical_form() -> None: assert all(canonical(a) != canonical(b) for a, b in differ) +def test_tidy_p2t() -> None: + pytest.importorskip("pix2tex") + from formula_img2latex.engines.p2t import tidy_p2t + cases = { + r"{\frac{\alpha} {3 6 0}}={\frac{x} {2 \pi}}": r"{\frac{\alpha}{360}}={\frac{x}{2\pi}}", + r"\frac{\mathrm{d}} {\mathrm{d} x} | x |=\frac{1} {x} .": r"\frac{\mathrm{d}}{\mathrm{d}x}|x|=\frac{1}{x}.", + r"\left( \begin{array} {c} {-y} \\ {0} \end{array} \right)": r"\left(\begin{array}{c}{-y}\\ {0}\end{array}\right)", + r"a \cdot x \in t": r"a\cdot x\in t", # space between letters stays + r"A \cdot\mathrm{\boldmath~ v ~}=0": r"A\cdot\boldsymbol{v}=0", # Pix2Text's bold + r"{\mathrm{{\bf~ r o t}}} \, v": r"{\mathbf{rot}}\,v", + r"\mathrm{\bf~ r o t} \, E": r"\mathbf{rot}\,E", + } + for raw, tidy in cases.items(): + assert tidy_p2t(raw) == tidy, (raw, tidy_p2t(raw)) + + def test_p2t_engine_is_lazy() -> None: from formula_img2latex.engines import make_engine e = make_engine("p2t") assert (e.name, e.model, e._ocr) == ("p2t", "pix2text-mfr-1.5", None) # model loads only in check() + from formula_img2latex.engines.p2t import ort_provider + assert ort_provider("cpu") == ("CPUExecutionProvider", {}) + assert ort_provider("cuda") == ("CUDAExecutionProvider", {"device_id": 0}) + assert ort_provider("cuda:2") == ("CUDAExecutionProvider", {"device_id": 2}) def test_logprob_confidence() -> None: diff --git a/tools/regeltest.py b/tools/regeltest.py new file mode 100644 index 0000000..7368bbe --- /dev/null +++ b/tools/regeltest.py @@ -0,0 +1,218 @@ +#!/usr/bin/env python3 +"""Run the standard consensus rule on formulas whose TeX is known and show every case for review. + + python3 tools/regeltest.py EPUB [--n 50] [--name regeltest] [--reuse vergleich/regeltest/ergebnis.json] + [--exclude vergleich/regeltest/ergebnis.json ...] + +Like `run`: pix2text reads every formula first, then Qwen (local); Gemma (local2) only where they +disagree; then the rough image check on pix2text's reading and decide_pair_tiebreak. Writes +vergleich//index.html (not versioned): summary, then per formula the original, the true +formula and every reading set with pdflatex, the decision and whether the result is right. +--reuse takes the readings (and their times) of an earlier run instead of asking the models again; +--exclude leaves out the formulas of earlier runs, so a new sample has none of them. +""" + +from __future__ import annotations + +import argparse +import html +import json +import random +import statistics +import sys +import time +from pathlib import Path + +import numpy as np +from PIL import Image + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from konsens_bericht import orig_img, png_uri # noqa: E402 + +from formula_img2latex.calibrate import _best_preset, corpus # noqa: E402 +from formula_img2latex.checks import pandoc_mathml_ok # noqa: E402 +from formula_img2latex.consensus import same_reading # noqa: E402 +from formula_img2latex.decide import consensus_thresholds, decide_pair_tiebreak # noqa: E402 +from formula_img2latex.engines import make_engine # noqa: E402 +from formula_img2latex.engines.p2t import tidy_p2t # noqa: E402 +from formula_img2latex.imageio import load_ink # noqa: E402 +from formula_img2latex.render import render_many # noqa: E402 +from formula_img2latex.tex import strip_delimiters # noqa: E402 +from formula_img2latex.verify import verify # noqa: E402 + +TOOLS = [("local", "Qwen"), ("p2t", "pix2text"), ("local2", "Gemma")] + + +def agreements(texes: list[str | None], display: bool, preset: str) -> list[tuple[int, int, str]]: + """Pairs (Qwen, pix2text), (pix2text, Gemma), (Qwen, Gemma) as in pipeline.decide_all_consensus (strict).""" + out = [] + for i, j in ((0, 1), (1, 2), (0, 2)): + a, b = texes[i], texes[j] + if a is None or b is None: + continue + how = same_reading(a, b, display, preset) + if how: + out.append((i, j, how)) + return out + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("epub", type=Path) + ap.add_argument("--n", type=int, default=50) + ap.add_argument("--name", default="regeltest") + ap.add_argument("--reuse", type=Path, default=None, help="Lesarten aus einem früheren ergebnis.json") + ap.add_argument("--exclude", type=Path, action="append", default=[], + help="Formeln aus diesem früheren ergebnis.json auslassen (mehrfach möglich)") + args = ap.parse_args() + cached = {} + if args.reuse: + cached = {c["name"]: c["readings"] for c in json.loads(args.reuse.read_text(encoding="utf-8"))["cases"]} + for rs in cached.values(): # earlier runs stored pix2text's raw spaced output + if rs[1] and rs[1].get("tex"): + rs[1]["tex"] = tidy_p2t(rs[1].get("raw") or rs[1]["tex"]) + excluded = {c["name"] for x in args.exclude for c in json.loads(x.read_text(encoding="utf-8"))["cases"]} + + items = [it for it in corpus(args.epub) if it[0] not in excluded] + smp = random.Random(42).sample(items, min(args.n + 1, len(items))) + formulas, warm = smp[:args.n], smp[args.n] + preset = _best_preset(smp[:12]) + th = consensus_thresholds() + eng = {} + for key, label in ([] if cached else TOOLS): + e = make_engine(key) + t0 = time.time() + e.check() + e.recognize(warm[1], warm[0], "", warm[3]) # warm-up, not counted + eng[key] = e + print(f"{label}: {e.model}, bereit in {time.time() - t0:.1f} s", file=sys.stderr, flush=True) + + cases = [] + for k, (name, data, truth, disp) in enumerate(formulas, 1): + old = cached.get(name) + if cached and old is None: + raise SystemExit(f"{name} fehlt in {args.reuse} (andere Stichprobe?)") + reads: list[dict | None] = [None, None, None] + for pos in (1, 0): # pix2text first, then Qwen + reads[pos] = old[pos] if old else eng[TOOLS[pos][0]].recognize(data, name, "", disp).to_json() + texes = [r and r.get("tex") for r in reads] + agree = agreements(texes, disp, preset) + if not any((i, j) == (0, 1) for i, j, _ in agree): # disputed: ask Gemma + reads[2] = old[2] if old else eng["local2"].recognize(data, name, "", disp).to_json() + texes = [r and r.get("tex") for r in reads] + agree = agreements(texes, disp, preset) + _st, _why, use = decide_pair_tiebreak(reads, agree, None, True, None, th) + veto = None + pandoc = True + if use is not None and texes[use]: + v = verify(load_ink(data, name), texes[use], disp, preset, check_variants=False) + veto = {"ok": v.ok, "reason": v.reason, "metrics": v.metrics} + pandoc = pandoc_mathml_ok([(texes[use], disp)])[0] + status, reason, use = decide_pair_tiebreak(reads, agree, veto, pandoc, None, th) + ok = [bool(t) and bool(same_reading(t, truth, disp, preset)) for t in texes] + accepted = status == "auto_accepted" + cases.append({"name": name, "truth": truth, "display": disp, "readings": reads, "ok": ok, "agree": agree, + "status": status, "reason": reason, "use": use, "veto": veto, + "right": bool(accepted and ok[use])}) + mark = ("✓" if ok[use] else "✗ FALSCH") if accepted else "Review" + print(f"{k:3d} {status:13s} {mark:8s} {reason[:110]}", file=sys.stderr, flush=True) + + n = len(cases) + acc = [c for c in cases if c["status"] == "auto_accepted"] + wrong = [c for c in acc if not c["right"]] + review = [c for c in cases if c["status"] == "needs_review"] + gemma = sum(c["readings"][2] is not None for c in cases) + secs = {key: [c["readings"][p]["seconds"] for c in cases if c["readings"][p]] for p, (key, _l) in enumerate(TOOLS)} + per_formula = [sum(r["seconds"] for r in c["readings"] if r) for c in cases] + summary = {"formulas": n, "accepted": len(acc), "wrong": len(wrong), "review": len(review), + "other": n - len(acc) - len(review), "gemma_asked": gemma, + "mean_s_per_formula": statistics.mean(per_formula), + "mean_s": {k: statistics.mean(v) if v else None for k, v in secs.items()}, + "correct_alone": {key: sum(c["ok"][p] for c in cases if c["readings"][p]) + for p, (key, _l) in enumerate(TOOLS)}} + + out = Path(__file__).resolve().parents[1] / "vergleich" / args.name + out.mkdir(parents=True, exist_ok=True) + (out / "ergebnis.json").write_text(json.dumps({"epub": args.epub.name, "preset": preset, "summary": summary, + "cases": cases}, ensure_ascii=False, indent=1, default=str), + encoding="utf-8") + + def order(c: dict) -> int: + return 0 if c["status"] == "auto_accepted" and not c["right"] else 1 if c["status"] != "auto_accepted" else 2 + + shown = sorted(cases, key=order) + texes = [] + for c in shown: + texes.append((strip_delimiters(c["truth"]), c["display"])) + texes += [((r or {}).get("tex") or "\\text{–}", c["display"]) for r in c["readings"]] + rendered = render_many(texes, preset) + by_name = {f[0]: f[1] for f in formulas} + lab = dict(TOOLS) + head = (f"

Regeltest: {html.escape(args.epub.name[:70])}

" + "

Regel: pix2text liest zuerst, Qwen bestätigt → annehmen. Sonst fragt Gemma; Gemma = pix2text → annehmen; " + "Qwen = Gemma gegen pix2text → Mensch. Übernommen wird immer pix2texts Lesart (Leerzeichen bereinigt). " + "Annahme nur mit Pandoc und Bild-Veto. " + "✓/✗ = gleiche Zeichen an gleicher Stelle wie die wahre Formel (Abstände, kursiv/aufrecht, Fettdruck egal).

" + "" + f"" + f"" + f"" + f"" + f"" + f"" + + "".join(f"" for k, v in summary["mean_s"].items() if v) + + "".join(f"" + for k, v in summary["correct_alone"].items()) + + "
Formeln{n}
automatisch angenommen{len(acc)} ({len(acc) / n:.0%})
davon FALSCH{len(wrong)}
Review (Mensch){len(review)} ({len(review) / n:.0%})
Gemma gefragt{gemma} ({gemma / n:.0%})
Ø Zeit pro Formel (alle Tools nacheinander){summary['mean_s_per_formula']:.2f} s
Ø {lab[k]}{v:.2f} s
{lab[k]} allein richtig{v}/{len(secs[k])}
") + parts = [head] + per = 4 + for i, c in enumerate(shown): + imgs = [] + for rr in rendered[per * i: per * (i + 1)]: + if rr.ok: + g = (rr.rasterize(2.0) * 255).astype(np.uint8) + imgs.append(f"") + else: + imgs.append("nicht setzbar") + if c["status"] == "auto_accepted": + cls, verdict = ("good", "angenommen, richtig") if c["right"] else ("bad", "angenommen, FALSCH") + elif c["status"] == "needs_review": + cls, verdict = "rev", "Review" + else: + cls, verdict = "rev", c["status"] + ncc = (c["veto"] or {}).get("metrics", {}).get("ncc") + rows = [f"Original", + f"richtig{imgs[0]}{html.escape(c['truth'])}"] + for p, (key, label) in enumerate(TOOLS): + r = c["readings"][p] + if r is None: + rows.append(f"{label}nicht gefragt") + continue + mark = "✓" if c["ok"][p] else "✗" + used = " ← verwendet" if c["use"] == p and c["status"] == "auto_accepted" else "" + rows.append(f"{mark} {label}{used}
{r['seconds']:.2f} s{imgs[p + 1]}" + f"{html.escape(r.get('tex') or r.get('error') or '–')}" + + (f"
roh: {html.escape(r.get('raw') or '')}" + if key == "p2t" and r.get("raw") and r.get("raw") != r.get("tex") else "") + + "") + parts.append(f"
{verdict} · {html.escape(c['reason'])}" + f"{f' · Bild-Ähnlichkeit {ncc:.2f}' if ncc is not None else ''}
" + f"{html.escape(c['name'])}{''.join(rows)}
") + (out / "index.html").write_text( + "Regeltest" + "".join(parts), + encoding="utf-8") + print(json.dumps(summary, ensure_ascii=False, indent=1)) + print(out / "index.html") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/zeittest.py b/tools/zeittest.py new file mode 100644 index 0000000..8d2ad01 --- /dev/null +++ b/tools/zeittest.py @@ -0,0 +1,145 @@ +#!/usr/bin/env python3 +"""Time and accuracy per recognition tool on formulas whose TeX is known (serial, one at a time). + + python3 tools/zeittest.py EPUB [--n 15] [--tools local,local2,cloud,p2t-cpu,p2t-gpu] [--gpu cuda:0] + +Same sample as `calibrate --consensus` (seed 42, real images). Every tool reads every formula alone, +one request after the other, after one uncounted warm-up on another formula. Writes +vergleich/zeittest/index.html (not versioned): table per tool, then per formula the original, the +true formula and every reading set with pdflatex, with time and verdict; plus ergebnis.json. +""" + +from __future__ import annotations + +import argparse +import html +import json +import random +import statistics +import sys +import time +from pathlib import Path + +import numpy as np +from PIL import Image + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from konsens_bericht import orig_img, png_uri # noqa: E402 + +from formula_img2latex.calibrate import _best_preset, corpus # noqa: E402 +from formula_img2latex.consensus import same_reading # noqa: E402 +from formula_img2latex.engines import make_engine # noqa: E402 +from formula_img2latex.engines.p2t import Pix2TextEngine # noqa: E402 +from formula_img2latex.render import render_many # noqa: E402 +from formula_img2latex.tex import strip_delimiters # noqa: E402 + +LABELS = {"local": "Qwen3-VL-30B (lokal, GPU)", "local2": "Gemma-4-26B (lokal, GPU)", "cloud": "GLM-4.6V (Cloud)", + "p2t-cpu": "pix2text MFR 1.5 (CPU)", "p2t-gpu": "pix2text MFR 1.5 (GPU)"} + + +def engine_for(tool: str, gpu: str): + if tool == "p2t-cpu": + return Pix2TextEngine(device="cpu") + if tool == "p2t-gpu": + return Pix2TextEngine(device=gpu) + return make_engine(tool) + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("epub", type=Path) + ap.add_argument("--n", type=int, default=15) + ap.add_argument("--tools", default="local,local2,cloud,p2t-cpu,p2t-gpu") + ap.add_argument("--gpu", default="cuda", help="Gerät für p2t-gpu (Index relativ zu CUDA_VISIBLE_DEVICES)") + ap.add_argument("--name", default="zeittest") + args = ap.parse_args() + tools = [t.strip() for t in args.tools.split(",") if t.strip()] + + items = corpus(args.epub) + smp = random.Random(42).sample(items, min(150, len(items))) # as calibrate_consensus + formulas, warm = smp[:args.n], smp[args.n] + preset = _best_preset(smp[:12]) + + runs: dict[str, dict] = {} + for tool in tools: + eng = engine_for(tool, args.gpu) + t0 = time.time() + eng.check() + load = time.time() - t0 + eng.recognize(warm[1], warm[0], "", warm[3]) # warm-up, not counted + reads = [] + for name, data, truth, disp in formulas: + r = eng.recognize(data, name, "", disp) + ok = bool(r.tex) and bool(same_reading(r.tex, truth, disp, preset)) + reads.append({"tex": r.tex, "seconds": r.seconds, "ok": ok, "error": r.error, + "tokens_in": r.tokens_in, "tokens_out": r.tokens_out}) + print(f"{tool:8s} {r.seconds:6.2f}s {'✓' if ok else '✗'} {r.tex or r.error}", file=sys.stderr, flush=True) + secs = [x["seconds"] for x in reads] + runs[tool] = {"label": LABELS.get(tool, tool), "model": eng.model, "load_s": load, "reads": reads, + "mean_s": statistics.mean(secs), "median_s": statistics.median(secs), + "correct": sum(x["ok"] for x in reads), "errors": sum(bool(x["error"]) for x in reads), + "tokens_in": sum(x["tokens_in"] for x in reads), "tokens_out": sum(x["tokens_out"] for x in reads)} + + out = Path(__file__).resolve().parents[1] / "vergleich" / args.name + out.mkdir(parents=True, exist_ok=True) + (out / "ergebnis.json").write_text(json.dumps( + {"epub": args.epub.name, "preset": preset, "truth": [f[2] for f in formulas], "runs": runs}, + ensure_ascii=False, indent=1), encoding="utf-8") + + n = len(formulas) + rows = "".join( + f"{html.escape(r['label'])}{r['mean_s']:.2f}{r['median_s']:.2f}" + f"{r['correct']}/{n} ({r['correct'] / n:.0%}){r['load_s']:.1f}" + f"{r['tokens_in'] + r['tokens_out'] or '–'}" for r in runs.values()) + parts = [f"

Zeittest: {html.escape(args.epub.name[:70])}

", + f"

{n} Formeln mit bekanntem TeX (alt-Text), jedes Tool seriell, eine Formel nach der anderen, " + "nach einem nicht gezählten Aufwärmlauf. Alle Formeln mit pdflatex gesetzt. ✓ = gleiche Zeichen an " + "gleicher Stelle wie die wahre Formel (Abstände, kursiv/aufrecht, Fettdruck egal).

", + "" + f"{rows}
ToolØ s/FormelMedian srichtigLaden sTokens
"] + + texes = [] + for i, (_n, _d, truth, disp) in enumerate(formulas): + texes.append((strip_delimiters(truth), disp)) + texes += [(runs[t]["reads"][i]["tex"] or "\\text{–}", disp) for t in tools] + rendered = render_many(texes, preset) + per = 1 + len(tools) + for i, (name, data, truth, _disp) in enumerate(formulas): + imgs = [] + for rr in rendered[per * i: per * (i + 1)]: + if rr.ok: + g = (rr.rasterize(2.0) * 255).astype(np.uint8) + imgs.append(f"") + else: + imgs.append("nicht setzbar") + body = [f"Original", + f"richtig{imgs[0]}{html.escape(truth)}"] + for k, t in enumerate(tools): + r = runs[t]["reads"][i] + mark = "✓" if r["ok"] else "✗" + body.append(f"{mark} {html.escape(runs[t]['label'])}
{r['seconds']:.2f} s" + f"{imgs[k + 1]}{html.escape(r['tex'] or r['error'] or '–')}") + parts.append(f"
Formel {i + 1} · {html.escape(name)}
{''.join(body)}
") + (out / "index.html").write_text( + "Zeittest" + "".join(parts), + encoding="utf-8") + + print(f"\n| Tool | Ø s/Formel | Median s | richtig (n={n}) | Laden s |") + print("|---|---|---|---|---|") + for r in runs.values(): + print(f"| {r['label']} | {r['mean_s']:.2f} | {r['median_s']:.2f} | {r['correct']}/{n} | {r['load_s']:.1f} |") + if "p2t-cpu" in runs and "p2t-gpu" in runs: + same = sum(a["tex"] == b["tex"] for a, b in zip(runs["p2t-cpu"]["reads"], runs["p2t-gpu"]["reads"])) + print(f"\npix2text CPU und GPU gleicher Text: {same}/{n}") + print(out / "index.html") + return 0 + + +if __name__ == "__main__": + sys.exit(main())