dschlueter/formula_img2latex
Watch
1
0
Fork
You've already forked formula_img2latex
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

feat: 0.3.0 – pix2text zuerst, Qwen bestätigt, Gemma nur bei Uneinigkeit

Browse source 
Neue Standardregel p2t (GLM/Cloud wird nicht mehr benutzt):
- pix2text liest zuerst; übernommen wird immer seine Lesart, wenn Qwen sie
  bestätigt oder – nur bei Uneinigkeit gefragt – Gemma. Qwen = Gemma gegen
  pix2text geht an einen Menschen. Schlusspunkt muss bestätigt sein; Bild-Veto
  und Pandoc auf pix2texts Lesart. Alte Regel als consensus_rule=vlm3 wählbar.
- pix2text auf der GPU (onnxruntime-gpu, FORMULA_P2T_DEVICE=auto/cpu/cuda:N)
- Ausgabe bereinigt: post_process aus pix2tex, Fettschreibweisen \boldmath/\bf
  -> \boldsymbol/\mathbf; an 400 Lesarten gleiches gesetztes Bild
- Normalform: \mathop, \begin{array}[t]
- calibrate: --engines, Regel-Tabelle, --reuse; wahre Formeln aus dem alt-Text
  bereinigt (aligned, \mathop{}\!, upgreek)
- tools/zeittest.py, tools/regeltest.py (--reuse, --exclude)

Messung: Göllmann Band 1 (2 x 150) 86 % / 85 % automatisch, 0 falsch
(bisher 78 % / 70 %); Band 2, zwei Stichproben zu 50: 78 % und 86 %, 0 falsch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Dieter Schlüter 2026-10-09 03:45:18 +02:00
commit 4638a12571
14 changed files with 941 additions and 98 deletions
Download patch file Download diff file Expand all files Collapse all files

43
BEDIENUNGSANLEITUNG.md
View file

@ -3,8 +3,10 @@
Dieses Programm ersetzt Formelbilder in den Markdown-Dateien von epub2md und pdf2md durch echte Dieses Programm ersetzt Formelbilder in den Markdown-Dateien von epub2md und pdf2md durch echte
LaTeX-Formeln. Dadurch passen die Formeln beim Neubau als EPUB oder PDF in Größe und Schrift zum Text. LaTeX-Formeln. Dadurch passen die Formeln beim Neubau als EPUB oder PDF in Größe und Schrift zum Text.
Es ersetzt eine Formel nur, wenn **drei unabhängige Bildmodelle dieselbe Formel lesen** (zwei lokale, eines Es liest jede Formel zuerst mit dem Formel-OCR **pix2text** und ersetzt sie nur, wenn das Bildmodell
in der Cloud) und das gerenderte LaTeX dem Bild grob ähnelt. Bei Zweifeln fragt es dich in einer **Qwen** dieselbe Formel liest (bei Uneinigkeit darf das zweite Bildmodell Gemma pix2text bestätigen) und
das gerenderte LaTeX dem Bild grob ähnelt. Eingesetzt wird die Lesart von pix2text, ohne ihre
überflüssigen Leerzeichen. Alles läuft lokal. Bei Zweifeln fragt es dich in einer
Review-Seite im Browser. Ohne deine Zustimmung bleibt die Formel ein Bild. Review-Seite im Browser. Ohne deine Zustimmung bleibt die Formel ein Bild.
„Dieselbe Formel“ heißt: jedes Zeichen stimmt und steht an der richtigen Stelle. Abstände, Pfeillängen, „Dieselbe Formel“ heißt: jedes Zeichen stimmt und steht an der richtigen Stelle. Abstände, Pfeillängen,
@ -41,12 +43,28 @@ CUDA_VISIBLE_DEVICES=2 ~/.local/bin/llama-server \
GPU 2 ist währenddessen für Chatterbox-TTS belegt. Beenden mit Strg+C in beiden Terminals. GPU 2 ist währenddessen für Chatterbox-TTS belegt. Beenden mit Strg+C in beiden Terminals.
### Dritte Lesart aus der Cloud ### pix2text (Formel-OCR)
Die Bestätigung kommt von **GLM-4.6V** über OpenRouter (andere Modellfamilie als Qwen und Gemma). Dafür pix2text läuft ohne Server im Programm selbst, auf der ersten sichtbaren Grafikkarte (sonst auf der CPU,
muss `OPENROUTER_API_KEY` gesetzt sein. GLM liest nur die Formeln, bei denen Qwen und Gemma schon einig dann ~5× langsamer). Einmalig:
sind. Beim Gehrke-Brückenkurs (1505 Formelbilder) waren das 967 Anfragen für zusammen etwa 0,15–0,20 $.
Ohne Cloud (`--no-cloud`) wird nichts automatisch ersetzt, alles Einige landet im Review. ```bash
pip install "pix2text[multilingual]"
pip install --no-deps pix2tex munch # nur für post_process (Leerzeichen)
pip uninstall -y onnxruntime && pip install onnxruntime-gpu # für die GPU
```
Das Modell (~300 MB) wird beim ersten Lauf nach `~/.pix2text` geladen. Gerät wählen mit
`FORMULA_P2T_DEVICE=cpu` oder `cuda:N` (`N` zählt innerhalb von `CUDA_VISIBLE_DEVICES`).
Gemma (Port 8012) wird nur für Formeln gefragt, bei denen pix2text und Qwen uneinig sind. Läuft Gemma
nicht, bleiben diese Formeln im Review.
### Frühere Regel mit Cloud (optional)
Bis Version 0.2.0 kam die Bestätigung von **GLM-4.6V** über OpenRouter (Qwen = Gemma = GLM). Diese Regel
ist weiter wählbar: im Arbeitsordner in `workspace.json` unter `settings` den Eintrag
`"consensus_rule": "vlm3"` setzen; dann muss `OPENROUTER_API_KEY` gesetzt sein.
--- ---
@ -168,9 +186,10 @@ geändert, wird sie übersprungen. `--force` setzt sie trotzdem zurück, dann is
```bash ```bash
python3 formula_img2latex.py scan AUSGABE # Bilder finden python3 formula_img2latex.py scan AUSGABE # Bilder finden
python3 formula_img2latex.py recognize AUSGABE --engine local # lokal erkennen python3 formula_img2latex.py recognize AUSGABE --engine p2t # pix2text (zuerst)
python3 formula_img2latex.py recognize AUSGABE --engine local # Qwen (bestätigt)
python3 formula_img2latex.py verify AUSGABE # prüfen und entscheiden python3 formula_img2latex.py verify AUSGABE # prüfen und entscheiden
python3 formula_img2latex.py recognize AUSGABE --engine cloud --only-unresolved python3 formula_img2latex.py recognize AUSGABE --engine local2 --only-unresolved # Gemma (Stichwahl)
python3 formula_img2latex.py verify AUSGABE python3 formula_img2latex.py verify AUSGABE
python3 formula_img2latex.py status AUSGABE python3 formula_img2latex.py status AUSGABE
``` ```
@ -214,6 +233,6 @@ Entscheidung mit Grund und den Bildabgleich. Der Ordner `vergleich/` wird nicht
Ja. Das Bild hatte die Farbe eingebaut, LaTeX-Formeln haben keinen Hintergrund. Ja. Das Bild hatte die Farbe eingebaut, LaTeX-Formeln haben keinen Hintergrund.
**Werden Buchinhalte ins Internet geschickt?** **Werden Buchinhalte ins Internet geschickt?**
Nur für die dritte Lesart: die einzelnen Formelbilder, bei denen Qwen und Gemma einig sind, samt eines Nein, mit der Standardregel bleibt alles lokal. Nur die frühere Regel `vlm3` schickt die Formelbilder, bei
kurzen Textausschnitts um die Formel (höchstens 240 Zeichen), gehen an OpenRouter/GLM. Mit `--no-cloud` bleibt alles lokal; dann wird nichts automatisch denen Qwen und Gemma einig sind, samt eines kurzen Textausschnitts um die Formel (höchstens 240 Zeichen)
ersetzt. an OpenRouter/GLM.

26
CHANGELOG.md
View file

@ -1,5 +1,31 @@
# Changelog # Changelog
## 0.3.0 – 2026-10-09
Neue Standardregel `p2t`: **pix2text liest zuerst, Qwen bestätigt, Gemma nur bei Uneinigkeit**; GLM (Cloud)
wird nicht mehr benutzt.
- **pix2text** (Formel-OCR MFR 1.5, ONNX) als Engine `p2t`, im Prozess, auf der GPU mit `onnxruntime-gpu`
(`FORMULA_P2T_DEVICE`: `auto`, `cpu`, `cuda:N`). Allein am häufigsten richtig (Göllmann Band 1: 90 % /
85 %) und am schnellsten (0,1 s/Formel auf der GPU, Qwen 0,7 s, Gemma 1,8 s).
- **Regel:** übernommen wird immer pix2texts Lesart (trifft den Fettdruck am besten), wenn Qwen sie
bestätigt oder – nur bei Uneinigkeit gefragt – Gemma. Qwen und Gemma gemeinsam gegen pix2text gehen
an einen Menschen (so kamen an Band 1 die gemeinsamen Fehler der beiden Bildmodelle durch). Der
Schlusspunkt muss bestätigt sein. Bild-Veto und Pandoc wie bisher, geprüft auf pix2texts Lesart.
Die alte Regel bleibt als `"consensus_rule": "vlm3"` in `workspace.json` wählbar.
- **pix2text-Ausgabe bereinigt:** Leerzeichen mit `post_process` aus pix2tex (installiert mit
`--no-deps`), Fettschreibweisen `\mathrm{\boldmath~ x ~}` / `\mathrm{\bf~ r o t}` →
`\boldsymbol{x}` / `\mathbf{rot}`. An 400 Lesarten geprüft: gleiches gesetztes Bild.
- **Normalform:** `\operatorname{l i m}` = `\lim`, `\ddot{u}` in Text = ü, `^{\prime}` = `'`,
abschließendes `\\` vor `\end`, `\begin{array}[t]`, `\mathop{…}`.
- **calibrate:** beliebig viele Engines (`--engines`), Regel-Tabelle, `--reuse` rechnet aus einem
früheren Bericht ohne Modellabfragen; wahre Formeln aus dem alt-Text werden bereinigt
(einzeiliges `aligned`, `\mathop{}\!\mathrm{d}`, upgreek).
- **Werkzeuge:** `tools/zeittest.py` (Zeit und Trefferquote je Tool), `tools/regeltest.py` (Standardregel
an Formeln mit bekanntem TeX, Seite zum Mitbeurteilen; `--reuse`, `--exclude`).
- **Messung:** Göllmann Band 1 (2 × 150) 86 % / 85 % automatisch, 0 falsch (bisher 78 % / 70 %);
Band 2, zwei frische Stichproben zu 50: 78 % und 86 % automatisch, 0 falsch.
## 0.2.0 – 2026-10-07 ## 0.2.0 – 2026-10-07
Entscheider ist jetzt der **Konsens dreier Bildmodelle** statt eines Pixelvergleichs. Entscheider ist jetzt der **Konsens dreier Bildmodelle** statt eines Pixelvergleichs.

72
README.md
View file

@ -3,7 +3,8 @@
Nachbearbeitungs-Werkzeug für die Markdown-Ausgaben von Nachbearbeitungs-Werkzeug für die Markdown-Ausgaben von
[epub2md](https://kitux.de/forgejo/dschlueter/epub2md) und [epub2md](https://kitux.de/forgejo/dschlueter/epub2md) und
[pdf2md](https://kitux.de/forgejo/dschlueter/pdf2md): Formeln, die nur als Bild vorliegen, werden durch [pdf2md](https://kitux.de/forgejo/dschlueter/pdf2md): Formeln, die nur als Bild vorliegen, werden durch
LaTeX ersetzt – **aber nur, wenn drei unabhängige Bildmodelle dieselbe Formel lesen** und ein grober LaTeX ersetzt – **aber nur, wenn zwei unabhängige Erkenner dieselbe Formel lesen** (zuerst das Formel-OCR
pix2text, bestätigt vom Bildmodell Qwen; bei Uneinigkeit darf Gemma pix2text bestätigen) und ein grober
Bildabgleich nicht widerspricht. Alles andere geht in eine Review-Galerie oder bleibt Bild. Bildabgleich nicht widerspricht. Alles andere geht in eine Review-Galerie oder bleibt Bild.
Schritt-für-Schritt-Anleitung: [BEDIENUNGSANLEITUNG.md](BEDIENUNGSANLEITUNG.md). Änderungen je Version: Schritt-für-Schritt-Anleitung: [BEDIENUNGSANLEITUNG.md](BEDIENUNGSANLEITUNG.md). Änderungen je Version:
@ -14,13 +15,22 @@ Schritt-für-Schritt-Anleitung: [BEDIENUNGSANLEITUNG.md](BEDIENUNGSANLEITUNG.md)
> Jedes Zeichen muss stimmen und an der richtigen Stelle relativ zu den anderen stehen. Abstände, > Jedes Zeichen muss stimmen und an der richtigen Stelle relativ zu den anderen stehen. Abstände,
> Pfeil- und Bruchstrichlängen, Schriftgröße, kursiv/aufrecht und Fettdruck dürfen abweichen. > Pfeil- und Bruchstrichlängen, Schriftgröße, kursiv/aufrecht und Fettdruck dürfen abweichen.
Eine Formel wird automatisch ersetzt, wenn Eine Formel wird automatisch ersetzt (Regel `p2t`, Standard), wenn
1. **Qwen3-VL-30B-A3B** (lokal, llama.cpp) und **Gemma-4-26B-A4B** (lokal) dieselbe Formel lesen, 1. **pix2text** (Formel-OCR MFR 1.5, lokal, GPU) sie zuerst liest und **Qwen3-VL-30B-A3B** (Bildmodell,
2. **GLM-4.6V** (Cloud über OpenRouter, andere Modellfamilie) das bestätigt – ein fehlender Schlusspunkt lokal, llama.cpp) dieselbe Formel liest **oder**, wenn die beiden uneinig sind, **Gemma-4-26B-A4B**
zählt dabei nicht, (lokal) die Lesart von pix2text bestätigt. Übernommen wird immer **pix2texts Lesart** – sie trifft den
3. Pandoc sie nach MathML umsetzen kann (sonst bliebe sie im EPUB/PDF roher TeX-Text) und Fettdruck am besten. Ihre überflüssigen Leerzeichen entfernt vorher `post_process` aus pix2tex (LaTeX-OCR):
4. das gerenderte LaTeX dem Original grob ähnelt (Bild-Veto, elastischer Vergleich, Ähnlichkeit ≥ 0,65). `\frac{\alpha} {3 6 0}` → `\frac{\alpha}{360}`; am gesetzten Bild ändert das nichts, geprüft an
400 Lesarten. Auch der Schlusspunkt muss bestätigt sein,
2. Pandoc sie nach MathML umsetzen kann (sonst bliebe sie im EPUB/PDF roher TeX-Text) und
3. das gerenderte LaTeX dem Original grob ähnelt (Bild-Veto, elastischer Vergleich, Ähnlichkeit ≥ 0,65).
Sind Qwen und Gemma gemeinsam gegen pix2text einig, entscheidet ein Mensch: Genau das sind die Fehler,
die beide Bildmodelle teilen (`\mathcal{E}_r` statt `\varepsilon_r`, erfundenes `\tilde{z}`).
Die frühere Regel `vlm3` (Qwen = Gemma, bestätigt von **GLM-4.6V** über OpenRouter) bleibt wählbar:
`"consensus_rule": "vlm3"` unter `settings` in `workspace.json` des Arbeitsordners.
„Dieselbe Formel“ prüft `consensus.py`: gleiche Normalform (Abstände, `\left`/`\right`, überflüssige „Dieselbe Formel“ prüft `consensus.py`: gleiche Normalform (Abstände, `\left`/`\right`, überflüssige
Klammern, `\vec`/`\overrightarrow`, `\frac`/`\dfrac`, `\iff`/`\Leftrightarrow`, `pmatrix`/`\left(\begin{matrix}`, Klammern, `\vec`/`\overrightarrow`, `\frac`/`\dfrac`, `\iff`/`\Leftrightarrow`, `pmatrix`/`\left(\begin{matrix}`,
@ -32,6 +42,25 @@ Sind sich die Modelle uneinig oder widerspricht das Veto, entscheidet ein Mensch
## Messung ## Messung
### Regel `p2t` (Standard)
Göllmann Band 1, 150 echte Bilder und 150 Nachbildungen im Gehrke-Stil, Lesarten aller Modelle:
| Regel | echt: automatisch / falsch | Gehrke-Stil: automatisch / falsch |
| --- | --- | --- |
| pix2text = Qwen | 81 % / 0 | 76 % / 0 |
| + Stichwahl: Gemma bestätigt pix2text, Qwens Lesart (erste Fassung) | 85 % / 0 | 80 % / 0 |
| **pix2text zuerst und übernommen, bereinigt, Stichwahl Gemma** (Standard) | **86 % / 0** | **85 % / 0** |
| + Stichwahl auch Qwen = Gemma gegen pix2text | 92 % / 1 | 91 % / 2 |
| Qwen = Gemma = GLM (Regel `vlm3`) | 78 % / 0 | 70 % / 0 |
Allein richtig: pix2text 90 % / 85 %, Qwen 86 % / 84 %, Gemma 86 % / 80 %, GLM 86 % / 83 %.
Zeit je Formel (seriell, 15 Formeln): pix2text 0,10 s (GPU) bzw. 0,51 s (CPU), Qwen 0,72 s, GLM 1,19 s,
Gemma 1,77 s. Gegenprobe an 50 frischen Formeln aus Göllmann Band 2 (`tools/regeltest.py`):
37 automatisch (74 %), **0 falsch**, 13 Review; Gemma wurde für 16 Formeln gefragt.
### Regel `vlm3` (Standard bis 0.2.0)
An Formelbildern mit bekanntem TeX (Göllmann, Springer: TeX im alt-Text), je 150 echte Bilder und 150 An Formelbildern mit bekanntem TeX (Göllmann, Springer: TeX im alt-Text), je 150 echte Bilder und 150
Nachbildungen im Stil von Gehrke (LaTeX + dvipng mit Bitmap-Schriften, ~102 dpi, JPEG): Nachbildungen im Stil von Gehrke (LaTeX + dvipng mit Bitmap-Schriften, ~102 dpi, JPEG):
@ -63,12 +92,12 @@ Gehrke, Brückenkurs Mathematik (EPUB, 1505 Formelbilder): siehe [Stand](#stand)
```text ```text
scan Bildverweise in chapters/*.md (bzw. pdf2md-Markdown) finden, Formel/Abbildung vorklassifizieren scan Bildverweise in chapters/*.md (bzw. pdf2md-Markdown) finden, Formel/Abbildung vorklassifizieren
run in Paketen zu 50: Qwen + Gemma lesen (parallel), prüfen, GLM nur für Einige, prüfen; run in Paketen zu 50: pix2text, dann Qwen lesen, prüfen, Gemma nur für Uneinige, prüfen;
nach jedem Paket eine Statuszeile nach jedem Paket eine Statuszeile (Regel vlm3: Qwen + Gemma, GLM nur für Einige)
review Zweifelsfälle im Browser: Original / Render / Overlay / Diff, LaTeX korrigieren und neu prüfen review Zweifelsfälle im Browser: Original / Render / Overlay / Diff, LaTeX korrigieren und neu prüfen
apply angenommene Formeln ins Markdown schreiben (aus einer Sicherung, idempotent) apply angenommene Formeln ins Markdown schreiben (aus einer Sicherung, idempotent)
revert Markdown-Dateien auf den Stand vor apply zurücksetzen revert Markdown-Dateien auf den Stand vor apply zurücksetzen
recognize einzelne Engine (local, local2, cloud) laufen lassen recognize einzelne Engine (local, local2, cloud, p2t) laufen lassen
verify prüfen und entscheiden (ohne neue Modellabfragen) verify prüfen und entscheiden (ohne neue Modellabfragen)
status Zählung je Status status Zählung je Status
calibrate Messung an Formelbildern mit bekanntem TeX (--consensus: Konsensregel) calibrate Messung an Formelbildern mit bekanntem TeX (--consensus: Konsensregel)
@ -90,14 +119,19 @@ Status je Formel: `auto_accepted`, `needs_review`, `rejected_keep_image`, `rende
| --- | --- | --- | | --- | --- | --- |
| `local` | llama.cpp `http://127.0.0.1:8011/v1` (Qwen3-VL-30B-A3B-Instruct Q4_K_M + mmproj) | `FORMULA_LOCAL_URL`, `FORMULA_LOCAL_MODEL` | | `local` | llama.cpp `http://127.0.0.1:8011/v1` (Qwen3-VL-30B-A3B-Instruct Q4_K_M + mmproj) | `FORMULA_LOCAL_URL`, `FORMULA_LOCAL_MODEL` |
| `local2` | llama.cpp `http://127.0.0.1:8012/v1` (Gemma-4-26B-A4B-it Q4_0 + mmproj) | `FORMULA_LOCAL2_URL`, `FORMULA_LOCAL2_MODEL` | | `local2` | llama.cpp `http://127.0.0.1:8012/v1` (Gemma-4-26B-A4B-it Q4_0 + mmproj) | `FORMULA_LOCAL2_URL`, `FORMULA_LOCAL2_MODEL` |
| `cloud` | OpenRouter, `z-ai/glm-4.6v` | `OPENROUTER_API_KEY`, `FORMULA_CLOUD_MODEL` | | `cloud` | OpenRouter, `z-ai/glm-4.6v` (nur Regel `vlm3`) | `OPENROUTER_API_KEY`, `FORMULA_CLOUD_MODEL` |
| `p2t` | pix2text MFR 1.5 (ONNX) im Prozess, erste sichtbare GPU, sonst CPU | `FORMULA_P2T_DEVICE` (`auto`, `cpu`, `cuda:N`), `FORMULA_P2T_MODEL` |
pix2text auf der GPU braucht `onnxruntime-gpu` statt `onnxruntime` (cuDNN kommt aus den nvidia-Paketen
von torch). Für `post_process` wird `pix2tex` ohne seine Abhängigkeiten installiert
(`pip install --no-deps pix2tex munch`), weil es sonst `timm==0.5.4` erzwingen würde. `cuda:N` zählt innerhalb von `CUDA_VISIBLE_DEVICES`.
`FORMULA_<ENGINE>_THINKING=1` schaltet das Nachdenken wieder ein. Antworten sind auf 800 Tokens begrenzt `FORMULA_<ENGINE>_THINKING=1` schaltet das Nachdenken wieder ein. Antworten sind auf 800 Tokens begrenzt
(lange mehrzeilige Formeln brauchen ~400; einzelne Antworten von Gemma liefen sonst in (lange mehrzeilige Formeln brauchen ~400; einzelne Antworten von Gemma liefen sonst in
Wiederholungsschleifen bis 165 s). Jede Lesart wird mit Modell, Prompt-Version und Nachdenk-Einstellung Wiederholungsschleifen bis 165 s). Jede Lesart wird mit Modell, Prompt-Version und Nachdenk-Einstellung
gespeichert; dieselbe Frage wird nie zweimal gestellt, ein abgebrochener Lauf setzt dort fort. gespeichert; dieselbe Frage wird nie zweimal gestellt, ein abgebrochener Lauf setzt dort fort.
Kosten der Cloud-Stufe: GLM liest nur Formeln, bei denen Qwen und Gemma einig sind. Gehrke: 967 Kosten der Cloud-Stufe (nur Regel `vlm3`): GLM liest nur Formeln, bei denen Qwen und Gemma einig sind. Gehrke: 967
Anfragen, Ø 2,6 s, zusammen rund 0,15–0,20 $. Anfragen, Ø 2,6 s, zusammen rund 0,15–0,20 $.
## Bildvergleich (Veto) ## Bildvergleich (Veto)
@ -105,9 +139,9 @@ Anfragen, Ø 2,6 s, zusammen rund 0,15–0,20 $.
`compare.py` rendert die Lesart mit pdflatex (Schrift je Buch automatisch: `cm`, `times`, `txfonts`, `compare.py` rendert die Lesart mit pdflatex (Schrift je Buch automatisch: `cm`, `times`, `txfonts`,
`palatino`, `stix`, `euler`), richtet sie auf das Original aus und verzerrt sie **elastisch** `palatino`, `stix`, `euler`), richtet sie auf das Original aus und verzerrt sie **elastisch**
(dynamische Zeitverzerrung über Spalten und Zeilen; Dehnen kostet proportional zur Tinte), damit (dynamische Zeitverzerrung über Spalten und Zeilen; Dehnen kostet proportional zur Tinte), damit
Abstände und Längen keine Rolle spielen. Die Ähnlichkeit danach ist das Veto. Geprüft wird nur die Abstände und Längen keine Rolle spielen. Die Ähnlichkeit danach ist das Veto. Geprüft wird nur
Lesart, die übernommen würde (die von Qwen; im Konsens ist sie gleichwertig zu den anderen) – eine die Lesart, die übernommen werden kann: die von pix2text (Regel `vlm3`: die von Qwen). Bestätigende
später eintreffende dritte Lesart löst keine neue Prüfung aus. Die Prüfergebnisse tragen eine Lesarten werden nie übernommen und brauchen keine Prüfung. Die Prüfergebnisse tragen eine
Versionsnummer des Verfahrens und werden nach Änderungen neu berechnet. Der alte Pixel-Entscheider mit Versionsnummer des Verfahrens und werden nach Änderungen neu berechnet. Der alte Pixel-Entscheider mit
Verwechslungsvarianten und Kontrasttest bleibt als Modus `decide=pixel` erhalten. Verwechslungsvarianten und Kontrasttest bleibt als Modus `decide=pixel` erhalten.
@ -145,6 +179,11 @@ sind nicht Teil dieser Version.
- `tools/vergleich.py AUSGABE` – feste Stichprobe (50 Formeln), verarbeitet wie `run`, schreibt - `tools/vergleich.py AUSGABE` – feste Stichprobe (50 Formeln), verarbeitet wie `run`, schreibt
`vergleich/<name>/index.html` mit Original, gerenderten Lesarten, Entscheidung und Veto-Overlay. `vergleich/<name>/index.html` mit Original, gerenderten Lesarten, Entscheidung und Veto-Overlay.
- `tools/konsens_bericht.py BERICHT.json EPUB` – Übersichtsseite zu `calibrate --consensus --report`. - `tools/konsens_bericht.py BERICHT.json EPUB` – Übersichtsseite zu `calibrate --consensus --report`.
- `tools/zeittest.py EPUB` – Zeit und Trefferquote je Tool, seriell an 15 Formeln mit bekanntem TeX.
- `tools/regeltest.py EPUB [--n 50] [--reuse ergebnis.json]` – die Standardregel an Formeln mit bekanntem
TeX, wie `run`; Seite mit Original, wahrer Formel, allen Lesarten und Entscheidung.
- `calibrate --consensus --engines local,local2,cloud,p2t --report X.json` misst alle Regeln;
`--reuse X.json` rechnet sie ohne Modellabfragen neu.
- `vergleich/` ist nicht versioniert (Ausschnitte aus Büchern). - `vergleich/` ist nicht versioniert (Ausschnitte aus Büchern).
## Stand ## Stand
@ -209,7 +248,8 @@ formula_img2latex/
├── pipeline.py scan, recognize, verify, Entscheidung (Konsens/Pixel), run in Paketen, apply/revert ├── pipeline.py scan, recognize, verify, Entscheidung (Konsens/Pixel), run in Paketen, apply/revert
├── consensus.py Normalform, Render-Gleichheit: sagen zwei Lesarten dasselbe? ├── consensus.py Normalform, Render-Gleichheit: sagen zwei Lesarten dasselbe?
├── decide.py Annahmeregeln und Schwellen (Konsens- und Pixel-Modus) ├── decide.py Annahmeregeln und Schwellen (Konsens- und Pixel-Modus)
├── engines/ Vision-Modelle über die OpenAI-API (llama.cpp, OpenRouter), Logprobs, Prompt-Version ├── engines/ Vision-Modelle über die OpenAI-API (llama.cpp, OpenRouter), Logprobs, Prompt-Version;
│ p2t.py: pix2text-Formel-OCR (ONNX, CPU/GPU)
├── compare.py Ausrichtung, elastische Verzerrung, Kennzahlen, Kontrasttest, Diff-Bilder ├── compare.py Ausrichtung, elastische Verzerrung, Kennzahlen, Kontrasttest, Diff-Bilder
├── confusables.py Verwechslungsvarianten (Pixel-Modus) ├── confusables.py Verwechslungsvarianten (Pixel-Modus)
├── verify.py Bildprüfung einer Lesart (Veto bzw. Pixel-Entscheider) ├── verify.py Bildprüfung einer Lesart (Veto bzw. Pixel-Entscheider)

2
formula_img2latex/__init__.py
View file

@ -1,3 +1,3 @@
"""formula_img2latex: Formelbilder in Markdown geprüft durch LaTeX ersetzen.""" """formula_img2latex: Formelbilder in Markdown geprüft durch LaTeX ersetzen."""
__version__ = "0.2.0" __version__ = "0.3.0"

180
formula_img2latex/calibrate.py
View file

@ -43,6 +43,22 @@ from .verify import verify
SYNTH_PX_PER_PT = 1.41 SYNTH_PX_PER_PT = 1.41
# upright Greek of the upgreek package (Springer alt texts): pdflatex here has no upgreek, and
# upright vs. italic is the same symbol for the project
_UPGREEK = re.compile(r"\\[Uu]p(alpha|beta|gamma|delta|epsilon|varepsilon|zeta|eta|theta|vartheta|iota|kappa|lambda|mu|nu|"
r"xi|pi|varpi|rho|varrho|sigma|varsigma|tau|upsilon|phi|varphi|chi|psi|omega)(?![A-Za-z])", re.I)
def clean_truth(tex: str) -> str:
"""Alt-text TeX as a plain formula: no one-line aligned wrapper, \\mathop{}\\!\\mathrm{d} -> \\mathrm{d},
upgreek -> standard Greek. Only the spelling changes, never a symbol."""
m = re.fullmatch(r"\s*\\begin\{aligned\}\s*(?:\\displaystyle)?(.*?)\\end\{aligned\}\s*", tex, re.S)
if m and "\\\\" not in m.group(1) and "&" not in m.group(1):
tex = m.group(1).strip()
tex = re.sub(r"\\mathop\{\}\\!", "", tex)
return _UPGREEK.sub(lambda g: "\\" + (g.group(1).capitalize() if g.group(0)[1] == "U" else g.group(1).lower()), tex)
def corpus(epub: Path) -> list[tuple[str, bytes, str, bool]]: def corpus(epub: Path) -> list[tuple[str, bytes, str, bool]]:
"""(name, image bytes, true TeX, display) for every image with TeX in its alt text.""" """(name, image bytes, true TeX, display) for every image with TeX in its alt text."""
out = [] out = []
@ -59,7 +75,7 @@ def corpus(epub: Path) -> list[tuple[str, bytes, str, bool]]:
if not (src and alt and alt.group(1).lstrip().startswith("$")): if not (src and alt and alt.group(1).lstrip().startswith("$")):
continue continue
path = posixpath.normpath(posixpath.join(base, src.group(1))) path = posixpath.normpath(posixpath.join(base, src.group(1)))
tex = strip_delimiters(html.unescape(alt.group(1))) tex = clean_truth(strip_delimiters(html.unescape(alt.group(1))))
if path in names and not unsafe_reason(tex): if path in names and not unsafe_reason(tex):
display = "IEq" not in path and "inline" not in tag.lower() display = "IEq" not in path and "inline" not in tag.lower()
out.append((path, z.read(path), tex, display)) out.append((path, z.read(path), tex, display))
@ -224,26 +240,63 @@ def _print(s: dict[str, Any], log: Any) -> None:
# ------------------------------------------------------------------------- consensus calibration # ------------------------------------------------------------------------- consensus calibration
def _consensus_eval(args: tuple) -> dict[str, Any]: def _consensus_eval(args: tuple) -> dict[str, Any]:
"""Worker: compare both readings with each other and with the truth; rough image check.""" """Worker: compare all readings with each other and with the truth; rough image check."""
from .consensus import same_reading from .consensus import same_reading
name, data, truth, display, preset, ra, rb = args from .pipeline import _strip_final_punct
name, data, truth, display, preset, readings = args
os.environ.setdefault("OMP_NUM_THREADS", "1") os.environ.setdefault("OMP_NUM_THREADS", "1")
ta, tb = ra.get("tex") or "", rb.get("tex") or "" texes = [r.get("tex") or "" for r in readings]
res: dict[str, Any] = {"name": name, "truth": truth, "display": display, "a": ra, "b": rb, ok = [bool(same_reading(t, truth, display, preset)) for t in texes]
"a_ok": bool(same_reading(ta, truth, display, preset)), agree: list[tuple[int, int, str]] = []
"b_ok": bool(same_reading(tb, truth, display, preset)), for i in range(len(texes)):
"agree": same_reading(ta, tb, display, preset)} for j in range(i + 1, len(texes)):
res["veto"] = None a, b = texes[i], texes[j]
if res["agree"]: # as in pipeline.decide_all_consensus: a confirming cloud reading may lack the final period;
v = verify(load_ink(data, name), ta, display, preset, check_variants=False) # pix2text's reading is used under rule p2t, so it is compared strictly
res["veto"] = {"ok": v.ok, "reason": v.reason, "metrics": v.metrics} if j >= 2 and readings[j].get("engine") != "p2t" and readings[i].get("engine") != "p2t":
return res a, b = _strip_final_punct(a), _strip_final_punct(b)
how = same_reading(a, b, display, preset)
if how:
agree.append((i, j, how))
vetos: dict[int, dict[str, Any]] = {}
for k in range(len(texes)): # only readings that may be used: primary pair and pix2text (rule p2t)
if (k < 2 or readings[k].get("engine") == "p2t") and any(k in (i, j) for i, j, _ in agree):
v = verify(load_ink(data, name), texes[k], display, preset, check_variants=False)
vetos[k] = {"ok": v.ok, "reason": v.reason, "metrics": v.metrics}
pair = next((how for i, j, how in agree if (i, j) == (0, 1)), None)
# "a", "b", "agree", "veto": the two primary models (read by tools/konsens_bericht.py)
return {"name": name, "truth": truth, "display": display, "readings": readings, "ok": ok, "agree_all": agree,
"vetos": vetos, "a": readings[0], "b": readings[1], "a_ok": ok[0], "b_ok": ok[1], "agree": pair,
"veto": vetos.get(0) if pair else None}
def _reading_cache(report: Path | None) -> dict[tuple[str, str, str], dict[str, Any]]:
"""(set label, image name, engine) -> reading from an earlier --report: models are not asked twice.
pix2text readings are tidied again (reports from before tidy_p2t hold the raw spaced output)."""
from .engines.p2t import tidy_p2t
if not report or not report.exists():
return {}
old = json.loads(report.read_text(encoding="utf-8"))
cache = {}
for s in old.get("sets", []):
for r in s.get("results", []):
for rd in r.get("readings") or [r.get("a") or {}, r.get("b") or {}]:
if rd.get("engine") and not rd.get("error"):
if rd["engine"] == "p2t" and rd.get("tex"):
rd = dict(rd, tex=tidy_p2t(rd.get("raw") or rd["tex"]))
cache[(s["label"], r["name"], rd["engine"])] = rd
return cache
def calibrate_consensus(epubs: list[Path], samples: int = 150, preset: str | None = None, synthetic: bool = True, def calibrate_consensus(epubs: list[Path], samples: int = 150, preset: str | None = None, synthetic: bool = True,
report: Path | None = None, workers: int = 0, quiet: bool = False, report: Path | None = None, workers: int = 0, quiet: bool = False,
engines: tuple[str, str] = ("local", "local2")) -> dict[str, Any]: engines: tuple[str, ...] = ("local", "local2"), reuse: Path | None = None) -> dict[str, Any]:
"""Measure the two-model consensus rule on images whose TeX is known.""" """Measure consensus rules on images whose TeX is known.
*engines*: the first two are the primary pair (local, local2); further engines (cloud, p2t)
give confirming readings. *reuse*: report of an earlier run whose readings are taken over.
"""
from concurrent.futures import ThreadPoolExecutor from concurrent.futures import ThreadPoolExecutor
from .decide import consensus_thresholds, decide_consensus from .decide import consensus_thresholds, decide_consensus
@ -253,10 +306,10 @@ def calibrate_consensus(epubs: list[Path], samples: int = 150, preset: str | Non
if not quiet: if not quiet:
print(m, file=sys.stderr, flush=True) print(m, file=sys.stderr, flush=True)
if len(engines) < 2:
raise ValueError("mindestens zwei Engines nötig")
cache = _reading_cache(reuse)
eng = [make_engine(e) for e in engines] eng = [make_engine(e) for e in engines]
for e in eng:
e.check()
log("Modelle: " + ", ".join(f"{e.name}={e.model}" for e in eng))
rng = random.Random(42) rng = random.Random(42)
sets: list[tuple[str, list[tuple[str, bytes, str, bool]], str]] = [] sets: list[tuple[str, list[tuple[str, bytes, str, bool]], str]] = []
for epub in epubs: for epub in epubs:
@ -268,21 +321,28 @@ def calibrate_consensus(epubs: list[Path], samples: int = 150, preset: str | Non
if synthetic: if synthetic:
syn = [(n + "#synth.jpg", img, t, d) for n, _d, t, d in smp for img in [synth_jpeg(t, d, rng)] if img] syn = [(n + "#synth.jpg", img, t, d) for n, _d, t, d in smp for img in [synth_jpeg(t, d, rng)] if img]
sets.append((f"{epub.stem[:40]} [dvipng, Gehrke-Stil]", syn, "cm")) sets.append((f"{epub.stem[:40]} [dvipng, Gehrke-Stil]", syn, "cm"))
todo = [(k, label, i) for label, items, _p in sets for k in range(len(eng)) for i in range(len(items))
if (label, items[i][0], eng[k].name) not in cache]
for k in sorted({k for k, _, _ in todo}):
eng[k].check()
log("Modelle: " + ", ".join(f"{e.name}={e.model}" for e in eng)
+ (f"; {len(cache)} Lesarten aus {reuse}" if cache else ""))
summary: dict[str, Any] = {"engines": [f"{e.name}={e.model}" for e in eng], "sets": []} summary: dict[str, Any] = {"engines": [f"{e.name}={e.model}" for e in eng], "sets": []}
for label, items, p in sets: for label, items, p in sets:
log(f"{label}: {len(items)} Formeln lesen …") log(f"{label}: {len(items)} Formeln lesen …")
readings: list[list[dict[str, Any]]] = [[{}] * len(items) for _ in eng] readings: list[list[dict[str, Any]]] = [
[cache.get((label, n, e.name), {}) for n, _d, _t, _disp in items] for e in eng]
def ask(k: int, i: int) -> None: def ask(k: int, i: int) -> None:
n, d, _t, disp = items[i] n, d, _t, disp = items[i]
readings[k][i] = eng[k].recognize(d, n, "", disp).to_json() readings[k][i] = eng[k].recognize(d, n, "", disp).to_json()
with ThreadPoolExecutor(max_workers=8) as pool: with ThreadPoolExecutor(max_workers=8) as pool:
list(pool.map(lambda ki: ask(*ki), [(k, i) for k in range(len(eng)) for i in range(len(items))])) list(pool.map(lambda ki: ask(*ki), [(k, i) for k, lb, i in todo if lb == label]))
jobs = [(n, d, t, disp, p, readings[0][i], readings[1][i]) for i, (n, d, t, disp) in enumerate(items)] jobs = [(n, d, t, disp, p, [readings[k][i] for k in range(len(eng))]) for i, (n, d, t, disp) in enumerate(items)]
with ProcessPoolExecutor(max_workers=workers or max(1, (os.cpu_count() or 2) - 2)) as ex: with ProcessPoolExecutor(max_workers=workers or max(1, (os.cpu_count() or 2) - 2)) as ex:
results = list(ex.map(_consensus_eval, jobs)) results = list(ex.map(_consensus_eval, jobs))
s = _summarize_consensus(label, results, consensus_thresholds, decide_consensus) s = _summarize_consensus(label, results, [e.name for e in eng], consensus_thresholds, decide_consensus)
summary["sets"].append(s) summary["sets"].append(s)
s["results"] = results s["results"] = results
_print_consensus(s, log) _print_consensus(s, log)
@ -292,19 +352,74 @@ def calibrate_consensus(epubs: list[Path], samples: int = 150, preset: str | Non
return summary return summary
def _summarize_consensus(label: str, results: list[dict[str, Any]], consensus_thresholds: Any, # Rules to compare: (label, mode, groups). A group lists engines; the first two are the primary pair
# and every further one must confirm it (decide_consensus with require_third). "any": accepted if one
# group accepts (the first accepting group supplies the reading); "all": every group must accept;
# "tiebreak": decide_pair_tiebreak with the group (Qwen, pix2text, Gemma) – the standard rule "p2t".
CONSENSUS_RULES: list[tuple[str, str, list[tuple[str, ...]]]] = [
("Qwen = Gemma", "any", [("local", "local2")]),
("Qwen = Gemma = GLM (bisher)", "any", [("local", "local2", "cloud")]),
("B: Qwen = Gemma = p2t", "any", [("local", "local2", "p2t")]),
("Qwen = Gemma, GLM oder p2t bestätigt", "any", [("local", "local2", "cloud"), ("local", "local2", "p2t")]),
("A: 3 von 4 (p2t entscheidet bei 2:1)", "any",
[("local", "local2", "cloud"), ("local", "local2", "p2t"), ("local", "cloud", "p2t"), ("local2", "cloud", "p2t")]),
("C: Qwen = Gemma = GLM = p2t", "all", [("local", "local2", "cloud"), ("local", "local2", "p2t")]),
("Standard: p2t = Qwen, Stichwahl Gemma = p2t", "tiebreak", [("local", "p2t", "local2")]),
]
def _rule_outcome(r: dict[str, Any], idx: list[int], th: dict[str, float], decide_fn: Any,
tiebreak: bool = False) -> int | None:
"""Index of the accepted reading if the readings *idx* (primary pair first) accept, else None."""
sub = [r["readings"][k] for k in idx]
pos = {k: n for n, k in enumerate(idx)}
agree = [(pos[i], pos[j], how) for i, j, how in r["agree_all"] if i in pos and j in pos]
t = th if tiebreak else dict(th, require_third=1 if len(idx) > 2 else 0)
_st, _why, use = decide_fn(sub, agree, None, True, None, t) # which reading would be used
if use is None:
return None
veto = r["vetos"].get(idx[use])
st, _why, use = decide_fn(sub, agree, veto, True, None, t)
return idx[use] if st == "auto_accepted" else None
def _summarize_consensus(label: str, results: list[dict[str, Any]], names: list[str], consensus_thresholds: Any,
decide_consensus: Any) -> dict[str, Any]: decide_consensus: Any) -> dict[str, Any]:
from .decide import decide_pair_tiebreak
n = len(results) n = len(results)
out: dict[str, Any] = {"label": label, "formulas": n, out: dict[str, Any] = {"label": label, "formulas": n, "engines": names,
"correct": {e: sum(r["ok"][k] for r in results) for k, e in enumerate(names)},
"a_correct": sum(r["a_ok"] for r in results), "b_correct": sum(r["b_ok"] for r in results), "a_correct": sum(r["a_ok"] for r in results), "b_correct": sum(r["b_ok"] for r in results),
"agree": sum(bool(r["agree"]) for r in results), "agree": sum(bool(r["agree"]) for r in results),
"agree_wrong": sum(bool(r["agree"]) and not r["a_ok"] for r in results), "agree_wrong": sum(bool(r["agree"]) and not r["a_ok"] for r in results),
"sweep": []} "rules": [], "sweep": []}
out["agree_wrong_examples"] = [{"truth": r["truth"], "a": r["a"].get("tex"), "b": r["b"].get("tex")} out["agree_wrong_examples"] = [{"truth": r["truth"], "a": r["a"].get("tex"), "b": r["b"].get("tex")}
for r in results if r["agree"] and not r["a_ok"]][:10] for r in results if r["agree"] and not r["a_ok"]][:10]
th0 = consensus_thresholds()
for rule, mode, groups in CONSENSUS_RULES:
if any(e not in names for g in groups for e in g):
continue
acc = wrong = 0
examples = []
for r in results:
if mode == "tiebreak":
used = [_rule_outcome(r, [names.index(e) for e in g], th0, decide_pair_tiebreak, True)
for g in groups]
else:
used = [_rule_outcome(r, [names.index(e) for e in g], th0, decide_consensus) for g in groups]
ok_used = [u for u in used if u is not None]
use = (ok_used[0] if ok_used else None) if mode == "any" else (used[0] if None not in used else None)
if use is None:
continue
acc += 1
if not r["ok"][use]:
wrong += 1
examples.append({"truth": r["truth"], "used": r["readings"][use].get("tex"),
"readings": {e: rd.get("tex") for e, rd in zip(names, r["readings"])}})
out["rules"].append({"rule": rule, "accepted": acc, "wrong": wrong, "wrong_examples": examples[:5]})
for pm in (0.0, 0.8, 0.9, 0.95, 0.98): for pm in (0.0, 0.8, 0.9, 0.95, 0.98):
for pn in (0.0, 0.2, 0.3, 0.5): for pn in (0.0, 0.2, 0.3, 0.5):
th = consensus_thresholds({"min_p_mean": pm, "min_p_min": pn}) th = consensus_thresholds({"min_p_mean": pm, "min_p_min": pn, "require_third": 0})
acc = wrong = 0 acc = wrong = 0
for r in results: for r in results:
agree = [(0, 1, r["agree"])] if r["agree"] else [] agree = [(0, 1, r["agree"])] if r["agree"] else []
@ -319,11 +434,14 @@ def _summarize_consensus(label: str, results: list[dict[str, Any]], consensus_th
def _print_consensus(s: dict[str, Any], log: Any) -> None: def _print_consensus(s: dict[str, Any], log: Any) -> None:
n = max(1, s["formulas"]) n = max(1, s["formulas"])
log(f" {s['label']}: {s['formulas']} Formeln") log(f" {s['label']}: {s['formulas']} Formeln")
log(f" Modell A richtig {s['a_correct'] / n:.0%}, Modell B richtig {s['b_correct'] / n:.0%}, " log(" allein richtig: " + ", ".join(f"{e} {c / n:.0%}" for e, c in s["correct"].items())
f"einig {s['agree'] / n:.0%}, davon FALSCH {s['agree_wrong']}") + f"; Qwen und Gemma einig {s['agree'] / n:.0%}, davon FALSCH {s['agree_wrong']}")
for ex in s["agree_wrong_examples"][:5]: log(" Regel → automatisch angenommen / davon falsch:")
log(f" einig, aber falsch: {ex['a']} (richtig: {ex['truth']})") for row in s["rules"]:
log(" Schwellen (Mittel/Minimum) → automatisch angenommen / davon falsch:") log(f" {row['rule']:<42} {row['accepted']:4d} ({row['accepted'] / n:.0%}) / {row['wrong']}")
for ex in row["wrong_examples"][:3]:
log(f" falsch: {ex['used']} (richtig: {ex['truth']})")
log(" Qwen = Gemma mit Schwellen (Mittel/Minimum) → automatisch angenommen / davon falsch:")
for row in s["sweep"]: for row in s["sweep"]:
log(f" p_mean ≥ {row['min_p_mean']:.2f}, p_min ≥ {row['min_p_min']:.2f}: " log(f" p_mean ≥ {row['min_p_mean']:.2f}, p_min ≥ {row['min_p_min']:.2f}: "
f"{row['accepted']:4d} ({row['accepted'] / n:.0%}) / {row['wrong']}") f"{row['accepted']:4d} ({row['accepted'] / n:.0%}) / {row['wrong']}")

10
formula_img2latex/cli.py
View file

@ -123,7 +123,12 @@ def build_parser() -> argparse.ArgumentParser:
sp.add_argument("--report", type=Path, default=None, help="Bericht als JSON speichern") sp.add_argument("--report", type=Path, default=None, help="Bericht als JSON speichern")
sp.add_argument("--workers", type=int, default=0) sp.add_argument("--workers", type=int, default=0)
sp.add_argument("--consensus", action="store_true", sp.add_argument("--consensus", action="store_true",
help="Konsensregel messen: beide lokalen Modelle (local, local2) lesen die Formeln") help="Konsensregeln messen: die Engines aus --engines lesen die Formeln")
sp.add_argument("--engines", default="local,local2",
help="Engines für --consensus, die ersten beiden bilden das Hauptpaar "
"(Standard local,local2; z. B. local,local2,cloud,p2t)")
sp.add_argument("--reuse", type=Path, default=None,
help="Lesarten aus einem früheren --report übernehmen (Modelle nicht erneut fragen)")
sp.add_argument("--quiet", "-q", action="store_true") sp.add_argument("--quiet", "-q", action="store_true")
return p return p
@ -166,7 +171,8 @@ def _dispatch(args: argparse.Namespace) -> int:
if args.command == "calibrate" and args.consensus: if args.command == "calibrate" and args.consensus:
from .calibrate import calibrate_consensus from .calibrate import calibrate_consensus
calibrate_consensus(args.epubs, samples=args.samples, preset=None if args.preset == "auto" else args.preset, calibrate_consensus(args.epubs, samples=args.samples, preset=None if args.preset == "auto" else args.preset,
synthetic=args.synthetic, report=args.report, workers=args.workers, quiet=args.quiet) synthetic=args.synthetic, report=args.report, workers=args.workers, quiet=args.quiet,
engines=tuple(e.strip() for e in args.engines.split(",") if e.strip()), reuse=args.reuse)
return EXIT_OK return EXIT_OK
if args.command == "calibrate": if args.command == "calibrate":
from .calibrate import calibrate from .calibrate import calibrate

5
formula_img2latex/consensus.py
View file

@ -26,7 +26,7 @@ _DROP = {"\\,", "\\;", "\\:", "\\!", "\\ ", "~", "\\quad", "\\qquad", "\\left",
"\\Bigg", "\\bigl", "\\bigr", "\\Bigl", "\\Bigr", "\\biggl", "\\biggr", "\\displaystyle", "\\textstyle", "\\Bigg", "\\bigl", "\\bigr", "\\Bigl", "\\Bigr", "\\biggl", "\\biggr", "\\displaystyle", "\\textstyle",
"\\scriptstyle", "\\limits", "\\nolimits", "\\middle", "\\hfill", "\\enspace", "\\thinspace", "\\medspace"} "\\scriptstyle", "\\limits", "\\nolimits", "\\middle", "\\hfill", "\\enspace", "\\thinspace", "\\medspace"}
# wrappers whose argument is kept (upright vs. italic is the same symbol) # wrappers whose argument is kept (upright vs. italic is the same symbol)
_UNWRAP = {"\\mathrm", "\\mathit", "\\operatorname", "\\textrm", "\\textit", "\\mathnormal", "\\mbox", "\\text", _UNWRAP = {"\\mathrm", "\\mathit", "\\operatorname", "\\mathop", "\\textrm", "\\textit", "\\mathnormal", "\\mbox", "\\text",
"\\textnormal", "\\textup"} "\\textnormal", "\\textup"}
_SYNONYM = { _SYNONYM = {
"\\overrightarrow": "\\vec", "\\overline": "\\bar", "\\widehat": "\\hat", "\\widetilde": "\\tilde", "\\overrightarrow": "\\vec", "\\overline": "\\bar", "\\widehat": "\\hat", "\\widetilde": "\\tilde",
@ -73,7 +73,8 @@ def canonical(tex: str, bold: bool = False) -> str:
body = re.sub(r"\\\\\s*(?=\\end\{)", "", body) # a final \\ before \end{...} adds no row body = re.sub(r"\\\\\s*(?=\\end\{)", "", body) # a final \\ before \end{...} adds no row
body = re.sub(r"\{?\\ss\{?\}?\}?", "ß", body) body = re.sub(r"\{?\\ss\{?\}?\}?", "ß", body)
body = re.sub(r"\{([^{}]*)\}\s*\\atop\s*\{([^{}]*)\}", r"\\substack{\1\\\\\2}", body) body = re.sub(r"\{([^{}]*)\}\s*\\atop\s*\{([^{}]*)\}", r"\\substack{\1\\\\\2}", body)
body = re.sub(r"\\begin\{array\}\{[^{}]*\}", r"\\begin{matrix}", body).replace("\\end{array}", "\\end{matrix}") body = re.sub(r"\\begin\{array\}\s*(?:\[[^\]]*\])?\s*\{[^{}]*\}", r"\\begin{matrix}", body) # [t]: position only
body = body.replace("\\end{array}", "\\end{matrix}")
for env, (lft, rgt) in _ENV_DELIMS.items(): for env, (lft, rgt) in _ENV_DELIMS.items():
body = body.replace(f"\\begin{{{env}}}", f"{lft} \\begin{{MATRIX}}").replace(f"\\end{{{env}}}", f"\\end{{MATRIX}} {rgt}") body = body.replace(f"\\begin{{{env}}}", f"{lft} \\begin{{MATRIX}}").replace(f"\\end{{{env}}}", f"\\end{{MATRIX}} {rgt}")
body = re.sub(r"\\not\s*=", r"\\ne ", body) body = re.sub(r"\\not\s*=", r"\\ne ", body)

62
formula_img2latex/decide.py
View file

@ -137,7 +137,16 @@ def decide_consensus(readings: list[dict[str, Any]], agree: list[tuple[int, int,
return "needs_review", f"zwei Modelle einig ({how}), dritte Lesart weicht ab: {texts}", i return "needs_review", f"zwei Modelle einig ({how}), dritte Lesart weicht ab: {texts}", i
tie = " (dritte Lesart bestätigt)" tie = " (dritte Lesart bestätigt)"
use = min(i, j) # the primary model's reading (it is the one checked against the image) use = min(i, j) # the primary model's reading (it is the one checked against the image)
problems = [p for p in (_sure(readings[i], th), _sure(readings[j], th)) if p] problems = _checks([readings[i], readings[j]], veto, pandoc_ok, book_px_per_pt, th)
if problems:
return "needs_review", f"einig ({how}){tie}; " + "; ".join(problems), use
return "auto_accepted", f"zwei Modelle einig ({how}){tie}, sicher, Bild-Veto bestanden", use
def _checks(agreeing: list[dict[str, Any]], veto: dict[str, Any] | None, pandoc_ok: bool,
book_px_per_pt: float | None, th: dict[str, float]) -> list[str]:
"""Problems of an agreed reading: unsure readings, Pandoc, rough image check (German reasons)."""
problems = [p for p in (_sure(r, th) for r in agreeing) if p]
if not pandoc_ok: if not pandoc_ok:
problems.append("Pandoc kann die Formel nicht nach MathML umsetzen") problems.append("Pandoc kann die Formel nicht nach MathML umsetzen")
if veto is None or not veto.get("ok"): if veto is None or not veto.get("ok"):
@ -149,6 +158,53 @@ def decide_consensus(readings: list[dict[str, Any]], agree: list[tuple[int, int,
if (th["veto_scale_dev"] > 0 and book_px_per_pt if (th["veto_scale_dev"] > 0 and book_px_per_pt
and abs(m.get("px_per_pt", book_px_per_pt) / book_px_per_pt - 1) > th["veto_scale_dev"]): and abs(m.get("px_per_pt", book_px_per_pt) / book_px_per_pt - 1) > th["veto_scale_dev"]):
problems.append(f"Bild-Veto: Schriftgröße weicht um {abs(m['px_per_pt'] / book_px_per_pt - 1):.0%} ab") problems.append(f"Bild-Veto: Schriftgröße weicht um {abs(m['px_per_pt'] / book_px_per_pt - 1):.0%} ab")
return problems
# Marker in the reason of a formula that waits for the tie-break reading (see pipeline.needs_tiebreak).
TIEBREAK_MISSING = "Stichwahl fehlt"
def decide_pair_tiebreak(readings: list[dict[str, Any] | None], agree: list[tuple[int, int, str]],
veto: dict[str, Any] | None, pandoc_ok: bool, book_px_per_pt: float | None,
th: dict[str, float]) -> tuple[str, str, int | None]:
"""Rule "p2t": pix2text reads first, Qwen confirms, Gemma only where the two disagree.
The reading used is always pix2text's (spaces tidied; it reads bold type best). *readings*:
[Qwen, pix2text, Gemma or None]. *agree*: pairs (i, j, how) of readings that say the same.
*veto*: verification of pix2text's reading. Returns (status, reason, index of the reading to use
or None).
Gemma may only confirm pix2text. Qwen and Gemma together against pix2text go to a human: in the
measurement exactly these were the errors both VLMs share (\\mathcal{E}_r for \\varepsilon_r, an
invented \\tilde{z}).
"""
q, p = readings[0], readings[1]
g = readings[2] if len(readings) > 2 else None
pairs = {frozenset((i, j)): how for i, j, how in agree}
if q is None or p is None:
missing = "Qwen" if q is None else "pix2text"
return "needs_review", f"Lesart von {missing} fehlt", 1 if p is not None and p.get("tex") else None
if q.get("is_formula") is False and (g is None or g.get("is_formula") is False) and (g or not p.get("tex")):
return "not_formula", "Qwen" + (" und Gemma" if g else "") + ": keine Formel", None
first = 1 if p.get("tex") else None # reading shown for review: pix2text's
if frozenset((0, 1)) in pairs:
how, who, confirm = pairs[frozenset((0, 1))], "pix2text und Qwen einig", q
elif g is None:
return "needs_review", f"pix2text und Qwen uneinig, {TIEBREAK_MISSING}", first
elif frozenset((1, 2)) in pairs:
how, who, confirm = pairs[frozenset((1, 2))], "Stichwahl: Gemma bestätigt pix2text", g
elif frozenset((0, 2)) in pairs:
return ("needs_review", "Qwen und Gemma einig gegen pix2text – Mensch entscheidet: "
f"Qwen: {q.get('tex') or '–'}", first)
else:
texts = " | ".join(f"{n}: {r.get('tex') or '–'}" for n, r in (("Qwen", q), ("pix2text", p), ("Gemma", g)))
return "needs_review", f"alle uneinig: {texts}", first
use = 1
if not p.get("tex"):
return "needs_review", f"{who}, aber ohne LaTeX", None
# the confidence check (token probabilities, off by default) covers the confirming VLM reading
problems = _checks([confirm], veto, pandoc_ok, book_px_per_pt, th)
if problems: if problems:
return "needs_review", f"einig ({how}){tie}; " + "; ".join(problems), use return "needs_review", f"{who} ({how}); " + "; ".join(problems), use
return "auto_accepted", f"zwei Modelle einig ({how}){tie}, sicher, Bild-Veto bestanden", use return "auto_accepted", f"{who} ({how}), Bild-Veto bestanden", use

2
formula_img2latex/engines/__init__.py
View file

@ -47,5 +47,5 @@ def make_engine(name: str, model: str | None = None, url: str | None = None) ->
) )
if name == "p2t": if name == "p2t":
return Pix2TextEngine(model=model or os.environ.get("FORMULA_P2T_MODEL", DEFAULT_P2T_MODEL), return Pix2TextEngine(model=model or os.environ.get("FORMULA_P2T_MODEL", DEFAULT_P2T_MODEL),
device=os.environ.get("FORMULA_P2T_DEVICE", "cpu")) device=os.environ.get("FORMULA_P2T_DEVICE", "auto"))
raise ValueError(f"unbekannte Engine {name!r} (local, local2, cloud, p2t)") raise ValueError(f"unbekannte Engine {name!r} (local, local2, cloud, p2t)")

66
formula_img2latex/engines/p2t.py
View file

@ -1,12 +1,14 @@
"""Pix2Text formula OCR (MFR model) as a local candidate generator. """Pix2Text formula OCR (MFR model) as a local candidate generator.
Not a vision-language model but a dedicated image-to-LaTeX transformer: another kind of model than Not a vision-language model but a dedicated image-to-LaTeX transformer: another kind of model than
the VLM engines, so its errors are likely to be different ones. Runs on the CPU (ONNX), no server. the VLM engines, so its errors are likely to be different ones. Runs in-process (ONNX), no server:
on the CPU, or on a GPU with onnxruntime-gpu (device "cuda:N"; N counts within CUDA_VISIBLE_DEVICES).
""" """
from __future__ import annotations from __future__ import annotations
import io import io
import re
import threading import threading
import time import time
@ -17,9 +19,48 @@ from .vlm import Recognition, prepare_image
DEFAULT_MODEL = "mfr-1.5" DEFAULT_MODEL = "mfr-1.5"
# Pix2Text's bold: \mathrm{\boldmath~ x ~} (an error in math mode), \mathrm{{\bf~ r o t}}, \mathrm{\bf~ r o t}
_BOLDMATH = re.compile(r"\\mathrm\s*\{\s*\\boldmath\s*~?\s*([^{}~\\]*?)\s*~?\s*\}")
_BF = re.compile(r"\\mathrm\s*\{\s*(\{)?\s*\\bf\s*~?\s*([^{}~\\]*?)\s*~?\s*(?(1)\})\s*\}")
def tidy_p2t(tex: str) -> str:
"""Pix2Text output without superfluous spaces: `\\frac{\\alpha} {3 6 0}` -> `\\frac{\\alpha}{360}`.
The spaces are removed by post_process of pix2tex (LaTeX-OCR), which keeps a space between two
letters (`a x`, `\\cdot x`) and the control space `\\ `; TeX ignores the removed spaces in math
mode, so the typeset formula does not change. Before that, Pix2Text's bold spellings become
\\boldsymbol / \\mathbf.
"""
from pix2tex.utils.utils import post_process # imports torch: only when needed
tex = _BOLDMATH.sub(lambda m: "\\boldsymbol{" + re.sub(r"\s+", "", m.group(1)) + "}", tex)
tex = _BF.sub(lambda m: "\\mathbf{" + re.sub(r"\s+", "", m.group(2)) + "}", tex)
return post_process(tex).strip()
def resolve_device(device: str) -> str:
"""Device "auto": the first visible GPU if onnxruntime has CUDA, else the CPU."""
if device != "auto":
return device
import onnxruntime
return "cuda" if "CUDAExecutionProvider" in onnxruntime.get_available_providers() else "cpu"
def ort_provider(device: str) -> tuple[str, dict]:
"""ONNX Runtime provider and its options for "cpu", "cuda" or "cuda:N".
Set explicitly: pix2text otherwise takes the first available provider (TensorRT with onnxruntime-gpu).
"""
if device == "cpu":
return "CPUExecutionProvider", {}
if device == "cuda" or device.startswith("cuda:"):
return "CUDAExecutionProvider", {"device_id": int(device.partition(":")[2] or 0)}
raise ValueError(f"unbekanntes Gerät {device!r} (cpu, cuda, cuda:N)")
class Pix2TextEngine: class Pix2TextEngine:
def __init__(self, model: str = DEFAULT_MODEL, device: str = "cpu") -> None: def __init__(self, model: str = DEFAULT_MODEL, device: str = "auto") -> None:
self.name = "p2t" self.name = "p2t"
self.model_name = model self.model_name = model
self.model = f"pix2text-{model}" self.model = f"pix2text-{model}"
@ -38,7 +79,21 @@ class Pix2TextEngine:
raise FormulaError(f"pix2text ist nicht installiert ({e}): pip install 'pix2text[multilingual]'", raise FormulaError(f"pix2text ist nicht installiert ({e}): pip install 'pix2text[multilingual]'",
EXIT_TOOL) from e EXIT_TOOL) from e
try: try:
self._ocr = LatexOCR(model_name=self.model_name, model_backend="onnx", device=self.device) import pix2tex.utils.utils # noqa: F401 (post_process for tidy_p2t)
except ImportError as e:
raise FormulaError(f"pix2tex ist nicht installiert ({e}): pip install --no-deps pix2tex munch",
EXIT_TOOL) from e
self.device = resolve_device(self.device)
provider, options = ort_provider(self.device)
if provider == "CUDAExecutionProvider":
import onnxruntime
if provider not in onnxruntime.get_available_providers():
raise FormulaError(f"Gerät {self.device}: onnxruntime ohne CUDA – pip install onnxruntime-gpu",
EXIT_TOOL)
onnxruntime.preload_dlls() # cuDNN/cuBLAS from the nvidia-* pip packages (installed with torch)
try:
self._ocr = LatexOCR(model_name=self.model_name, model_backend="onnx", device=self.device,
more_model_configs={"provider": provider, "provider_options": options})
except Exception as e: # download or model files broken: any exception type is possible here except Exception as e: # download or model files broken: any exception type is possible here
raise FormulaError(f"Pix2Text-Modell {self.model_name} nicht ladbar ({e})", EXIT_TOOL) from e raise FormulaError(f"Pix2Text-Modell {self.model_name} nicht ladbar ({e})", EXIT_TOOL) from e
@ -51,9 +106,10 @@ class Pix2TextEngine:
with self._lock: with self._lock:
# _one_batch instead of recognize(): the latter draws a tqdm bar for every image # _one_batch instead of recognize(): the latter draws a tqdm bar for every image
out = self._ocr._one_batch([img], None)[0] out = self._ocr._one_batch([img], None)[0]
tex = self._ocr.post_process(out["text"]).strip() raw = self._ocr.post_process(out["text"]).strip()
except Exception as e: # model not loadable or one unreadable image: must not stop the run except Exception as e: # model not loadable or one unreadable image: must not stop the run
return Recognition(self.name, self.model, None, None, "", str(e)[:300], time.time() - t0) return Recognition(self.name, self.model, None, None, "", str(e)[:300], time.time() - t0)
score = float(out.get("score") or 0.0) or None score = float(out.get("score") or 0.0) or None
return Recognition(self.name, self.model, tex or None, None, tex[:2000], None, time.time() - t0, tex = tidy_p2t(raw)
return Recognition(self.name, self.model, tex or None, None, raw[:2000], None, time.time() - t0,
p_mean=score) p_mean=score)

85
formula_img2latex/pipeline.py
View file

@ -20,8 +20,10 @@ from PIL import Image
from .checks import pandoc_mathml_ok from .checks import pandoc_mathml_ok
from .classify import classify from .classify import classify
from .compare import diff_image from .compare import diff_image
from .decide import consensus_thresholds, decide, decide_consensus, thresholds from .decide import (TIEBREAK_MISSING, consensus_thresholds, decide, decide_consensus, decide_pair_tiebreak,
thresholds)
from .engines import make_engine from .engines import make_engine
from .errors import FormulaError
from .engines.vlm import PROMPT_VERSION from .engines.vlm import PROMPT_VERSION
from .imageio import load_ink from .imageio import load_ink
from .render import PRESETS from .render import PRESETS
@ -113,10 +115,15 @@ def needs_third(f: Formula) -> bool:
return f.status == "needs_review" and "dritte Lesart fehlt" in (f.reason or "") return f.status == "needs_review" and "dritte Lesart fehlt" in (f.reason or "")
def needs_tiebreak(f: Formula) -> bool:
"""Rule "p2t": pix2text and Qwen disagree; Gemma's tie-break reading is missing."""
return f.status == "needs_review" and TIEBREAK_MISSING in (f.reason or "")
def recognize(source: Source, ws: Workspace, engine_name: str, model: str | None = None, url: str | None = None, def recognize(source: Source, ws: Workspace, engine_name: str, model: str | None = None, url: str | None = None,
only_unresolved: bool = False, include_figures: bool = False, limit: int = 0, only_unresolved: bool = False, include_figures: bool = False, limit: int = 0,
workers: int = 4, quiet: bool = False, only_ids: set[str] | None = None, workers: int = 4, quiet: bool = False, only_ids: set[str] | None = None,
only_needing_third: bool = False) -> dict[str, int]: only_needing_third: bool = False, only_needing_tiebreak: bool = False) -> dict[str, int]:
engine = make_engine(engine_name, model, url) engine = make_engine(engine_name, model, url)
engine.check() engine.check()
todo = [] todo = []
@ -127,6 +134,8 @@ def recognize(source: Source, ws: Workspace, engine_name: str, model: str | None
continue continue
if only_needing_third and not needs_third(f): if only_needing_third and not needs_third(f):
continue continue
if only_needing_tiebreak and not needs_tiebreak(f):
continue
if any(c.get("engine") == engine.name and c.get("model") == engine.model if any(c.get("engine") == engine.name and c.get("model") == engine.model
and c.get("prompt") == PROMPT_VERSION and c.get("thinking") == engine.thinking for c in f.candidates): and c.get("prompt") == PROMPT_VERSION and c.get("thinking") == engine.thinking for c in f.candidates):
continue # cached: never ask the same model the same question twice continue # cached: never ask the same model the same question twice
@ -246,6 +255,8 @@ def verify_all(source: Source, ws: Workspace, preset: str | None = None, include
formulas = [f for f in ws.all() if _wanted(f, include_figures) and _distinct_candidates(f) formulas = [f for f in ws.all() if _wanted(f, include_figures) and _distinct_candidates(f)
and (only_ids is None or f.id in only_ids)] and (only_ids is None or f.id in only_ids)]
mode = decide_mode(ws) mode = decide_mode(ws)
if mode == "consensus" and consensus_rule(ws) == "vlm3":
mode = "consensus-vlm3"
formulas = [f for f in formulas if _verify_texes(f, mode)] formulas = [f for f in formulas if _verify_texes(f, mode)]
if not redo: if not redo:
formulas = [f for f in formulas if not _verified_current(f, mode)] formulas = [f for f in formulas if not _verified_current(f, mode)]
@ -288,10 +299,13 @@ VERIFY_VERSION = 3 # bump when the comparison changes, so stored image checks
def _verify_texes(f: Formula, mode: str) -> list[str]: def _verify_texes(f: Formula, mode: str) -> list[str]:
"""Readings to check against the image. Consensus: only the reading that would be used (the """Readings to check against the image: only the one that may be used. Mode "consensus" (rule
first primary reading) – the veto needs nothing else, and a later third reading changes nothing.""" p2t): pix2text's. "consensus-vlm3": the first primary reading (Qwen's). "pixel": all candidates."""
if mode != "consensus": if mode == "pixel":
return _distinct_candidates(f) return _distinct_candidates(f)
if mode == "consensus":
p = latest_by_engine(f).get("p2t")
return [split_tag(strip_delimiters(p["tex"]))[0]] if p and p.get("tex") else []
for r in readings_of(f): for r in readings_of(f):
if r.get("tex"): if r.get("tex"):
return [split_tag(strip_delimiters(r["tex"]))[0]] return [split_tag(strip_delimiters(r["tex"]))[0]]
@ -312,6 +326,20 @@ def decide_mode(ws: Workspace) -> str:
return ws.settings.get("decide") or "consensus" return ws.settings.get("decide") or "consensus"
def consensus_rule(ws: Workspace) -> str:
"""'p2t' (pix2text = Qwen, Gemma as tie-break; default) or 'vlm3' (Qwen = Gemma = GLM)."""
return ws.settings.get("consensus_rule") or "p2t"
def latest_by_engine(f: Formula) -> dict[str, dict]:
"""Latest error-free reading per engine."""
latest: dict[str, dict] = {}
for c in f.candidates:
if not c.get("error"):
latest[c.get("engine", "")] = c
return latest
def readings_of(f: Formula) -> list[dict]: def readings_of(f: Formula) -> list[dict]:
"""Latest reading per model; local engines first (primary pair), then the rest (tie-break).""" """Latest reading per model; local engines first (primary pair), then the rest (tie-break)."""
latest: dict[str, dict] = {} latest: dict[str, dict] = {}
@ -369,17 +397,28 @@ def _tex_key(tex: str) -> str:
def decide_all_consensus(ws: Workspace, quiet: bool, preset: str, only_ids: set[str] | None = None) -> dict[str, int]: def decide_all_consensus(ws: Workspace, quiet: bool, preset: str, only_ids: set[str] | None = None) -> dict[str, int]:
from .consensus import same_reading from .consensus import same_reading
th = consensus_thresholds(ws.settings.get("consensus_thresholds")) th = consensus_thresholds(ws.settings.get("consensus_thresholds"))
rule = consensus_rule(ws)
decide_fn = decide_pair_tiebreak if rule == "p2t" else decide_consensus
formulas = [f for f in ws.all() if f.candidates and f.status not in HUMAN_STATUSES formulas = [f for f in ws.all() if f.candidates and f.status not in HUMAN_STATUSES
and (only_ids is None or f.id in only_ids)] and (only_ids is None or f.id in only_ids)]
plan = [] plan = []
for f in formulas: for f in formulas:
rs = readings_of(f)
bold = bool(th.get("bold_relevant")) bold = bool(th.get("bold_relevant"))
if rule == "p2t":
latest = latest_by_engine(f)
rs = [latest.get("local"), latest.get("p2t"), latest.get("local2")]
pairs = [(0, 1), (1, 2), (0, 2)]
else:
rs = readings_of(f)
pairs = [(i, j) for i in range(len(rs)) for j in range(i + 1, len(rs))]
agree = [] agree = []
for i in range(len(rs)): for i, j in pairs:
for j in range(i + 1, len(rs)): if rs[i] is None or rs[j] is None:
continue
a, b = rs[i].get("tex") or "", rs[j].get("tex") or "" a, b = rs[i].get("tex") or "", rs[j].get("tex") or ""
if j >= 2: # the confirming reading: a missing final period/comma is no contradiction # rule vlm3: a confirming reading (j >= 2) may lack the final period or comma. Rule p2t: strict,
# pix2text's reading is the one used, so its final punctuation must be confirmed too
if rule != "p2t" and j >= 2:
a, b = _strip_final_punct(a), _strip_final_punct(b) a, b = _strip_final_punct(a), _strip_final_punct(b)
how = same_reading(a, b, f.display, preset, bold) how = same_reading(a, b, f.display, preset, bold)
if how: if how:
@ -397,20 +436,20 @@ def decide_all_consensus(ws: Workspace, quiet: bool, preset: str, only_ids: set[
ws.settings["book_px_per_pt"] = book_ppt ws.settings["book_px_per_pt"] = book_ppt
chosen = [] chosen = []
for f, rs, agree in plan: for f, rs, agree in plan:
status, reason, use = decide_consensus(rs, agree, None, True, book_ppt, th) # preliminary: which reading status, reason, use = decide_fn(rs, agree, None, True, book_ppt, th) # preliminary: which reading
chosen.append(rs[use]["tex"] if use is not None and rs[use].get("tex") else None) chosen.append(rs[use]["tex"] if use is not None and rs[use].get("tex") else None)
pandoc = pandoc_mathml_ok([(t or "x", f.display) for (f, _, _), t in zip(plan, chosen)]) if plan else [] pandoc = pandoc_mathml_ok([(t or "x", f.display) for (f, _, _), t in zip(plan, chosen)]) if plan else []
counts: dict[str, int] = {} counts: dict[str, int] = {}
for (f, rs, agree), tex, pok in zip(plan, chosen, pandoc): for (f, rs, agree), tex, pok in zip(plan, chosen, pandoc):
veto = by_tex[f.id].get(_tex_key(tex)) if tex else None veto = by_tex[f.id].get(_tex_key(tex)) if tex else None
status, reason, use = decide_consensus(rs, agree, veto, pok, book_ppt, th) status, reason, use = decide_fn(rs, agree, veto, pok, book_ppt, th)
f.status, f.reason = status, reason f.status, f.reason = status, reason
f.final_tex = _with_tag(f, tex) if status == "auto_accepted" and tex else None f.final_tex = _with_tag(f, tex) if status == "auto_accepted" and tex else None
f.decided_at = now_iso() f.decided_at = now_iso()
ws.save(f) ws.save(f)
counts[status] = counts.get(status, 0) + 1 counts[status] = counts.get(status, 0) + 1
ws.save_meta() ws.save_meta()
log("Entscheidung (Konsens): " + ", ".join(f"{k} {v}" for k, v in sorted(counts.items())), quiet) log(f"Entscheidung (Konsens, Regel {rule}): " + ", ".join(f"{k} {v}" for k, v in sorted(counts.items())), quiet)
return counts return counts
@ -526,25 +565,39 @@ def run_consensus(source: Source, ws: Workspace, chunk: int = 50, preset: str |
include_figures: bool = False, limit: int = 0, cloud_model: str | None = None, include_figures: bool = False, limit: int = 0, cloud_model: str | None = None,
use_cloud: bool = True, ids: list[str] | None = None, quiet: bool = False, use_cloud: bool = True, ids: list[str] | None = None, quiet: bool = False,
report: Callable[[str], None] | None = None) -> dict[str, int]: report: Callable[[str], None] | None = None) -> dict[str, int]:
"""Process formulas in chunks: local, local2, verify; cloud only where both agree; verify again. """Process formulas in chunks and report a status line (processed / accepted / review / image) after each.
After every chunk a status line (processed / accepted / review / image) is reported. Rule "p2t": pix2text, then Qwen, verify; Gemma only where they disagree; decide again.
Rule "vlm3": Qwen and Gemma, verify; cloud only where both agree; verify again.
""" """
say = report or (lambda m: print(m, flush=True)) say = report or (lambda m: print(m, flush=True))
todo = ids or [f.id for f in ws.all() if _wanted(f, include_figures)] todo = ids or [f.id for f in ws.all() if _wanted(f, include_figures)]
if limit: if limit:
todo = todo[:limit] todo = todo[:limit]
rule = consensus_rule(ws)
tiebreak_ok = True
totals: dict[str, int] = {} totals: dict[str, int] = {}
done: list[str] = [] done: list[str] = []
for start in range(0, len(todo), chunk): for start in range(0, len(todo), chunk):
part = set(todo[start:start + chunk]) part = set(todo[start:start + chunk])
# both local models run on their own GPU: ask them at the same time if rule == "p2t": # pix2text first (fast), then Qwen
for eng in ("p2t", "local"):
recognize(source, ws, eng, only_ids=part, include_figures=include_figures, quiet=True)
else: # both VLMs run on their own GPU: ask them at the same time
with ThreadPoolExecutor(max_workers=2) as both: with ThreadPoolExecutor(max_workers=2) as both:
for fut in [both.submit(recognize, source, ws, eng, only_ids=part, include_figures=include_figures, for fut in [both.submit(recognize, source, ws, eng, only_ids=part, include_figures=include_figures,
quiet=True) for eng in ("local", "local2")]: quiet=True) for eng in ("local", "local2")]:
fut.result() fut.result()
verify_all(source, ws, preset, include_figures, quiet=True, only_ids=part) verify_all(source, ws, preset, include_figures, quiet=True, only_ids=part)
if use_cloud: if rule == "p2t" and tiebreak_ok and any(needs_tiebreak(f) for f in map(ws.load, part) if f):
try:
recognize(source, ws, "local2", only_ids=part, only_needing_tiebreak=True,
include_figures=include_figures, quiet=True)
verify_all(source, ws, preset, include_figures, quiet=True, only_ids=part)
except FormulaError as e: # Gemma not running: the disputed formulas stay in review
tiebreak_ok = False
say(f"Stichwahl (Gemma) nicht möglich, strittige Formeln bleiben im Review: {e}")
if rule != "p2t" and use_cloud:
recognize(source, ws, "cloud", cloud_model, only_ids=part, only_needing_third=True, recognize(source, ws, "cloud", cloud_model, only_ids=part, only_needing_third=True,
include_figures=include_figures, workers=8, quiet=True) include_figures=include_figures, workers=8, quiet=True)
verify_all(source, ws, preset, include_figures, quiet=True, only_ids=part) verify_all(source, ws, preset, include_figures, quiet=True, only_ids=part)

105
tests/test_pipeline.py
View file

@ -4,6 +4,8 @@ from __future__ import annotations
from pathlib import Path from pathlib import Path
import pytest
from conftest import needs_latex, needs_pandoc from conftest import needs_latex, needs_pandoc
from formula_img2latex import pipeline from formula_img2latex import pipeline
from formula_img2latex.cli import main from formula_img2latex.cli import main
@ -120,6 +122,7 @@ def _reading(engine: str, model: str, tex: str | None, p_mean: float = 0.99, p_m
@needs_pandoc @needs_pandoc
def test_consensus_mode(epub2md_dir: Path) -> None: def test_consensus_mode(epub2md_dir: Path) -> None:
source, ws = pipeline.open_all(epub2md_dir, None, create=True) source, ws = pipeline.open_all(epub2md_dir, None, create=True)
ws.settings["consensus_rule"] = "vlm3" # the earlier rule Qwen = Gemma = GLM
pipeline.scan(source, ws, quiet=True) pipeline.scan(source, ws, quiet=True)
by = {f.asset.rsplit("/", 1)[1]: f for f in ws.all()} by = {f.asset.rsplit("/", 1)[1]: f for f in ws.all()}
inline, display = by["0001-inline.jpg"], by["0002-display.jpg"] inline, display = by["0001-inline.jpg"], by["0002-display.jpg"]
@ -155,6 +158,63 @@ def test_consensus_mode(epub2md_dir: Path) -> None:
assert i.status == "needs_review" and "weicht ab" in i.reason, i.reason assert i.status == "needs_review" and "weicht ab" in i.reason, i.reason
@needs_latex
@needs_pandoc
def test_consensus_rule_p2t(epub2md_dir: Path) -> None:
"""Standard rule: pix2text first, Qwen confirms; where they differ Gemma may confirm pix2text.
The reading used is always pix2text's."""
source, ws = pipeline.open_all(epub2md_dir, None, create=True)
pipeline.scan(source, ws, quiet=True)
by = {f.asset.rsplit("/", 1)[1]: f for f in ws.all()}
inline, display = by["0001-inline.jpg"], by["0002-display.jpg"]
inline.candidates = [_reading("local", "Q", r"L=\{-\sqrt{2};\sqrt{2}\}"),
_reading("p2t", "P", r"L=\{-\sqrt{2};\sqrt{2}\}")]
display.candidates = [_reading("local", "Q", r"x_{3,k}=\arcsin\left(-\frac{3}{5}\right)+2k\pi"),
_reading("p2t", "P", r"x_{2,k}=\operatorname{arcsin}(-\frac{3}{5})+2k\pi")]
ws.save(inline)
ws.save(display)
pipeline.verify_all(source, ws, preset="cm", workers=2, quiet=True)
i, d = ws.load(inline.id), ws.load(display.id)
assert i.status == "auto_accepted" and "pix2text und Qwen einig" in i.reason, i.reason
assert i.final_tex == r"L=\{-\sqrt{2};\sqrt{2}\}" # pix2text's reading
assert pipeline.needs_tiebreak(d), d.reason
# Gemma confirms pix2text -> accepted with pix2text's reading
d.candidates.append(_reading("local2", "G", r"x_{2,k} = \arcsin(-\frac{3}{5}) + 2k\pi"))
ws.save(d)
pipeline.verify_all(source, ws, preset="cm", workers=2, quiet=True)
d = ws.load(d.id)
assert d.status == "auto_accepted" and "Gemma bestätigt pix2text" in d.reason, d.reason
assert d.final_tex == r"x_{2,k}=\operatorname{arcsin}(-\frac{3}{5})+2k\pi"
# Qwen and Gemma together against pix2text -> a human decides
d.candidates[-1] = _reading("local2", "G", r"x_{3,k}=\arcsin(-\frac{3}{5})+2k\pi")
d.status = "pending"
ws.save(d)
pipeline.verify_all(source, ws, preset="cm", workers=2, quiet=True)
d = ws.load(d.id)
assert d.status == "needs_review" and "Mensch entscheidet" in d.reason, d.reason
def test_pair_tiebreak_rule() -> None:
from formula_img2latex.decide import consensus_thresholds, decide_pair_tiebreak
th = consensus_thresholds()
q, p, g = _reading("local", "Q", "x^2"), _reading("p2t", "P", "x^{2}"), _reading("local2", "G", "x^2")
veto = {"ok": True, "metrics": {"ncc": 0.9, "px_per_pt": 1.4}}
assert decide_pair_tiebreak([q, p, None], [(0, 1, "normalform")], veto, True, 1.4, th)[::2] == ("auto_accepted", 1)
bad = {"ok": True, "metrics": {"ncc": 0.5, "px_per_pt": 1.4}}
assert decide_pair_tiebreak([q, p, None], [(0, 1, "normalform")], bad, True, 1.4, th)[0] == "needs_review" # veto
assert decide_pair_tiebreak([q, p, None], [(0, 1, "normalform")], veto, False, 1.4, th)[0] == "needs_review" # pandoc
assert "Stichwahl fehlt" in decide_pair_tiebreak([q, p, None], [], veto, True, 1.4, th)[1]
assert decide_pair_tiebreak([q, p, g], [(1, 2, "normalform")], veto, True, 1.4, th)[::2] == ("auto_accepted", 1)
assert decide_pair_tiebreak([q, p, g], [(0, 2, "normalform")], veto, True, 1.4, th)[0] == "needs_review"
assert decide_pair_tiebreak([q, p, g], [], veto, True, 1.4, th)[0] == "needs_review"
assert "fehlt" in decide_pair_tiebreak([q, None, None], [], veto, True, 1.4, th)[1]
nq, ng = _reading("local", "Q", None, is_formula=False), _reading("local2", "G", None, is_formula=False)
assert decide_pair_tiebreak([nq, p, ng], [], None, True, 1.4, th)[0] == "not_formula"
assert decide_pair_tiebreak([nq, p, None], [], None, True, 1.4, th)[0] == "needs_review" # ask Gemma first
def test_consensus_rules() -> None: def test_consensus_rules() -> None:
from formula_img2latex.decide import consensus_thresholds, decide_consensus from formula_img2latex.decide import consensus_thresholds, decide_consensus
th = consensus_thresholds({"min_p_mean": 0.9, "min_p_min": 0.3, "require_third": 0}) th = consensus_thresholds({"min_p_mean": 0.9, "min_p_min": 0.3, "require_third": 0})
@ -181,6 +241,29 @@ def test_consensus_rules() -> None:
assert decide_consensus([a, b, c], [(0, 2, "normalform")], veto, True, 1.4, strict)[0] == "needs_review" # no 2:1 assert decide_consensus([a, b, c], [(0, 2, "normalform")], veto, True, 1.4, strict)[0] == "needs_review" # no 2:1
def test_calibration_rules_with_four_engines() -> None:
from formula_img2latex.calibrate import _summarize_consensus
from formula_img2latex.decide import consensus_thresholds, decide_consensus
names = ["local", "local2", "cloud", "p2t"]
veto = {"ok": True, "metrics": {"ncc": 0.9}}
def res(agree: list[tuple[int, int]], ok: list[bool]) -> dict:
rs = [_reading(e, e, "x") for e in names]
pair = "normalform" if (0, 1) in agree else None
return {"truth": "x", "readings": rs, "ok": ok, "agree_all": [(i, j, "normalform") for i, j in agree],
"vetos": {0: veto, 1: veto, 3: veto}, "a": rs[0], "b": rs[1], "a_ok": ok[0], "b_ok": ok[1],
"agree": pair, "veto": veto if pair else None}
all4 = [(0, 1), (0, 2), (0, 3), (1, 2), (1, 3), (2, 3)]
results = [res(all4, [True] * 4), # everyone agrees
res([(0, 1), (0, 3), (1, 3)], [False, False, True, False]), # Q = G = p2t, all wrong, GLM right
res([(0, 2), (0, 3), (2, 3)], [True, False, True, True])] # Gemma differs: 2:1 decided by p2t
s = _summarize_consensus("t", results, names, consensus_thresholds, decide_consensus)
got = [(r["accepted"], r["wrong"]) for r in s["rules"]]
# Qwen = Gemma | + GLM | B (+ p2t) | GLM or p2t confirms | A (3 of 4) | C (all four) | standard rule p2t
assert got == [(2, 1), (1, 0), (2, 1), (2, 1), (3, 1), (1, 0), (3, 1)]
def test_canonical_form() -> None: def test_canonical_form() -> None:
from formula_img2latex.consensus import canonical from formula_img2latex.consensus import canonical
same = [(r"x^{2}+y^{2}", "x^2 + y^2"), (r"\vec{b}", r"\overrightarrow{b}"), (r"\left(a\right)", "(a)"), same = [(r"x^{2}+y^{2}", "x^2 + y^2"), (r"\vec{b}", r"\overrightarrow{b}"), (r"\left(a\right)", "(a)"),
@ -190,6 +273,8 @@ def test_canonical_form() -> None:
(r"\operatorname{l i m}_{x \to0} \frac{\operatorname{s i n} x} {x}", r"\lim_{x\to 0}\frac{\sin x}{x}"), (r"\operatorname{l i m}_{x \to0} \frac{\operatorname{s i n} x} {x}", r"\lim_{x\to 0}\frac{\sin x}{x}"),
(r"\mathrm{f \ddot{u} r ~ a l l e ~} x", r"\text{für alle } x"), (r"f^{\prime\prime}(x)", "f''(x)"), (r"\mathrm{f \ddot{u} r ~ a l l e ~} x", r"\text{für alle } x"), (r"f^{\prime\prime}(x)", "f''(x)"),
(r"\left( \begin{matrix} {1} & {2} \\ \end{matrix} \right)", r"\begin{pmatrix}1&2\end{pmatrix}"), (r"\left( \begin{matrix} {1} & {2} \\ \end{matrix} \right)", r"\begin{pmatrix}1&2\end{pmatrix}"),
(r"\begin{array}[t]{cc}a&b\end{array}", r"\begin{array}{cc}a&b\end{array}"),
(r"\mathop{\mathbf{rot}}\boldsymbol{v}", r"\mathbf{rot}\,\boldsymbol{v}"),
(r"{\int}_{0}^{\infty} \! e^{-x^{2}} \, d x \!=\! \frac{\sqrt\pi} 2", r"\int_0^\infty e^{-x^2}\,dx=\frac{\sqrt{\pi}}{2}")] (r"{\int}_{0}^{\infty} \! e^{-x^{2}} \, d x \!=\! \frac{\sqrt\pi} 2", r"\int_0^\infty e^{-x^2}\,dx=\frac{\sqrt{\pi}}{2}")]
differ = [(r"x_{2,k}", r"x_{2;k}"), ("x^{2}", "x_{2}"), (r"\mathbb{R}", "R"), ("a_{n+1}", "a_n+1"), differ = [(r"x_{2,k}", r"x_{2;k}"), ("x^{2}", "x_{2}"), (r"\mathbb{R}", "R"), ("a_{n+1}", "a_n+1"),
(r"\frac{a}{b+c}", r"\frac{a}{b}+c"), (r"x \in t", r"x \int"), (r"\sqrt[3]{x}", r"\sqrt{x}"), (r"\frac{a}{b+c}", r"\frac{a}{b}+c"), (r"x \in t", r"x \int"), (r"\sqrt[3]{x}", r"\sqrt{x}"),
@ -198,10 +283,30 @@ def test_canonical_form() -> None:
assert all(canonical(a) != canonical(b) for a, b in differ) assert all(canonical(a) != canonical(b) for a, b in differ)
def test_tidy_p2t() -> None:
pytest.importorskip("pix2tex")
from formula_img2latex.engines.p2t import tidy_p2t
cases = {
r"{\frac{\alpha} {3 6 0}}={\frac{x} {2 \pi}}": r"{\frac{\alpha}{360}}={\frac{x}{2\pi}}",
r"\frac{\mathrm{d}} {\mathrm{d} x} | x |=\frac{1} {x} .": r"\frac{\mathrm{d}}{\mathrm{d}x}|x|=\frac{1}{x}.",
r"\left( \begin{array} {c} {-y} \\ {0} \end{array} \right)": r"\left(\begin{array}{c}{-y}\\ {0}\end{array}\right)",
r"a \cdot x \in t": r"a\cdot x\in t", # space between letters stays
r"A \cdot\mathrm{\boldmath~ v ~}=0": r"A\cdot\boldsymbol{v}=0", # Pix2Text's bold
r"{\mathrm{{\bf~ r o t}}} \, v": r"{\mathbf{rot}}\,v",
r"\mathrm{\bf~ r o t} \, E": r"\mathbf{rot}\,E",
}
for raw, tidy in cases.items():
assert tidy_p2t(raw) == tidy, (raw, tidy_p2t(raw))
def test_p2t_engine_is_lazy() -> None: def test_p2t_engine_is_lazy() -> None:
from formula_img2latex.engines import make_engine from formula_img2latex.engines import make_engine
e = make_engine("p2t") e = make_engine("p2t")
assert (e.name, e.model, e._ocr) == ("p2t", "pix2text-mfr-1.5", None) # model loads only in check() assert (e.name, e.model, e._ocr) == ("p2t", "pix2text-mfr-1.5", None) # model loads only in check()
from formula_img2latex.engines.p2t import ort_provider
assert ort_provider("cpu") == ("CPUExecutionProvider", {})
assert ort_provider("cuda") == ("CUDAExecutionProvider", {"device_id": 0})
assert ort_provider("cuda:2") == ("CUDAExecutionProvider", {"device_id": 2})
def test_logprob_confidence() -> None: def test_logprob_confidence() -> None:

218
tools/regeltest.py Normal file
View file

@ -0,0 +1,218 @@
#!/usr/bin/env python3
"""Run the standard consensus rule on formulas whose TeX is known and show every case for review.
python3 tools/regeltest.py EPUB [--n 50] [--name regeltest] [--reuse vergleich/regeltest/ergebnis.json]
[--exclude vergleich/regeltest/ergebnis.json ...]
Like `run`: pix2text reads every formula first, then Qwen (local); Gemma (local2) only where they
disagree; then the rough image check on pix2text's reading and decide_pair_tiebreak. Writes
vergleich/<name>/index.html (not versioned): summary, then per formula the original, the true
formula and every reading set with pdflatex, the decision and whether the result is right.
--reuse takes the readings (and their times) of an earlier run instead of asking the models again;
--exclude leaves out the formulas of earlier runs, so a new sample has none of them.
"""
from __future__ import annotations
import argparse
import html
import json
import random
import statistics
import sys
import time
from pathlib import Path
import numpy as np
from PIL import Image
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from konsens_bericht import orig_img, png_uri # noqa: E402
from formula_img2latex.calibrate import _best_preset, corpus # noqa: E402
from formula_img2latex.checks import pandoc_mathml_ok # noqa: E402
from formula_img2latex.consensus import same_reading # noqa: E402
from formula_img2latex.decide import consensus_thresholds, decide_pair_tiebreak # noqa: E402
from formula_img2latex.engines import make_engine # noqa: E402
from formula_img2latex.engines.p2t import tidy_p2t # noqa: E402
from formula_img2latex.imageio import load_ink # noqa: E402
from formula_img2latex.render import render_many # noqa: E402
from formula_img2latex.tex import strip_delimiters # noqa: E402
from formula_img2latex.verify import verify # noqa: E402
TOOLS = [("local", "Qwen"), ("p2t", "pix2text"), ("local2", "Gemma")]
def agreements(texes: list[str | None], display: bool, preset: str) -> list[tuple[int, int, str]]:
"""Pairs (Qwen, pix2text), (pix2text, Gemma), (Qwen, Gemma) as in pipeline.decide_all_consensus (strict)."""
out = []
for i, j in ((0, 1), (1, 2), (0, 2)):
a, b = texes[i], texes[j]
if a is None or b is None:
continue
how = same_reading(a, b, display, preset)
if how:
out.append((i, j, how))
return out
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("epub", type=Path)
ap.add_argument("--n", type=int, default=50)
ap.add_argument("--name", default="regeltest")
ap.add_argument("--reuse", type=Path, default=None, help="Lesarten aus einem früheren ergebnis.json")
ap.add_argument("--exclude", type=Path, action="append", default=[],
help="Formeln aus diesem früheren ergebnis.json auslassen (mehrfach möglich)")
args = ap.parse_args()
cached = {}
if args.reuse:
cached = {c["name"]: c["readings"] for c in json.loads(args.reuse.read_text(encoding="utf-8"))["cases"]}
for rs in cached.values(): # earlier runs stored pix2text's raw spaced output
if rs[1] and rs[1].get("tex"):
rs[1]["tex"] = tidy_p2t(rs[1].get("raw") or rs[1]["tex"])
excluded = {c["name"] for x in args.exclude for c in json.loads(x.read_text(encoding="utf-8"))["cases"]}
items = [it for it in corpus(args.epub) if it[0] not in excluded]
smp = random.Random(42).sample(items, min(args.n + 1, len(items)))
formulas, warm = smp[:args.n], smp[args.n]
preset = _best_preset(smp[:12])
th = consensus_thresholds()
eng = {}
for key, label in ([] if cached else TOOLS):
e = make_engine(key)
t0 = time.time()
e.check()
e.recognize(warm[1], warm[0], "", warm[3]) # warm-up, not counted
eng[key] = e
print(f"{label}: {e.model}, bereit in {time.time() - t0:.1f} s", file=sys.stderr, flush=True)
cases = []
for k, (name, data, truth, disp) in enumerate(formulas, 1):
old = cached.get(name)
if cached and old is None:
raise SystemExit(f"{name} fehlt in {args.reuse} (andere Stichprobe?)")
reads: list[dict | None] = [None, None, None]
for pos in (1, 0): # pix2text first, then Qwen
reads[pos] = old[pos] if old else eng[TOOLS[pos][0]].recognize(data, name, "", disp).to_json()
texes = [r and r.get("tex") for r in reads]
agree = agreements(texes, disp, preset)
if not any((i, j) == (0, 1) for i, j, _ in agree): # disputed: ask Gemma
reads[2] = old[2] if old else eng["local2"].recognize(data, name, "", disp).to_json()
texes = [r and r.get("tex") for r in reads]
agree = agreements(texes, disp, preset)
_st, _why, use = decide_pair_tiebreak(reads, agree, None, True, None, th)
veto = None
pandoc = True
if use is not None and texes[use]:
v = verify(load_ink(data, name), texes[use], disp, preset, check_variants=False)
veto = {"ok": v.ok, "reason": v.reason, "metrics": v.metrics}
pandoc = pandoc_mathml_ok([(texes[use], disp)])[0]
status, reason, use = decide_pair_tiebreak(reads, agree, veto, pandoc, None, th)
ok = [bool(t) and bool(same_reading(t, truth, disp, preset)) for t in texes]
accepted = status == "auto_accepted"
cases.append({"name": name, "truth": truth, "display": disp, "readings": reads, "ok": ok, "agree": agree,
"status": status, "reason": reason, "use": use, "veto": veto,
"right": bool(accepted and ok[use])})
mark = ("✓" if ok[use] else "✗ FALSCH") if accepted else "Review"
print(f"{k:3d} {status:13s} {mark:8s} {reason[:110]}", file=sys.stderr, flush=True)
n = len(cases)
acc = [c for c in cases if c["status"] == "auto_accepted"]
wrong = [c for c in acc if not c["right"]]
review = [c for c in cases if c["status"] == "needs_review"]
gemma = sum(c["readings"][2] is not None for c in cases)
secs = {key: [c["readings"][p]["seconds"] for c in cases if c["readings"][p]] for p, (key, _l) in enumerate(TOOLS)}
per_formula = [sum(r["seconds"] for r in c["readings"] if r) for c in cases]
summary = {"formulas": n, "accepted": len(acc), "wrong": len(wrong), "review": len(review),
"other": n - len(acc) - len(review), "gemma_asked": gemma,
"mean_s_per_formula": statistics.mean(per_formula),
"mean_s": {k: statistics.mean(v) if v else None for k, v in secs.items()},
"correct_alone": {key: sum(c["ok"][p] for c in cases if c["readings"][p])
for p, (key, _l) in enumerate(TOOLS)}}
out = Path(__file__).resolve().parents[1] / "vergleich" / args.name
out.mkdir(parents=True, exist_ok=True)
(out / "ergebnis.json").write_text(json.dumps({"epub": args.epub.name, "preset": preset, "summary": summary,
"cases": cases}, ensure_ascii=False, indent=1, default=str),
encoding="utf-8")
def order(c: dict) -> int:
return 0 if c["status"] == "auto_accepted" and not c["right"] else 1 if c["status"] != "auto_accepted" else 2
shown = sorted(cases, key=order)
texes = []
for c in shown:
texes.append((strip_delimiters(c["truth"]), c["display"]))
texes += [((r or {}).get("tex") or "\\text{–}", c["display"]) for r in c["readings"]]
rendered = render_many(texes, preset)
by_name = {f[0]: f[1] for f in formulas}
lab = dict(TOOLS)
head = (f"<h1>Regeltest: {html.escape(args.epub.name[:70])}</h1>"
"<p>Regel: pix2text liest zuerst, Qwen bestätigt → annehmen. Sonst fragt Gemma; Gemma = pix2text → annehmen; "
"Qwen = Gemma gegen pix2text → Mensch. Übernommen wird immer pix2texts Lesart (Leerzeichen bereinigt). "
"Annahme nur mit Pandoc und Bild-Veto. "
"✓/✗ = gleiche Zeichen an gleicher Stelle wie die wahre Formel (Abstände, kursiv/aufrecht, Fettdruck egal).</p>"
"<table class=sum>"
f"<tr><th>Formeln</th><td>{n}</td></tr>"
f"<tr><th>automatisch angenommen</th><td>{len(acc)} ({len(acc) / n:.0%})</td></tr>"
f"<tr><th>davon FALSCH</th><td><b>{len(wrong)}</b></td></tr>"
f"<tr><th>Review (Mensch)</th><td>{len(review)} ({len(review) / n:.0%})</td></tr>"
f"<tr><th>Gemma gefragt</th><td>{gemma} ({gemma / n:.0%})</td></tr>"
f"<tr><th>Ø Zeit pro Formel (alle Tools nacheinander)</th><td>{summary['mean_s_per_formula']:.2f} s</td></tr>"
+ "".join(f"<tr><th>Ø {lab[k]}</th><td>{v:.2f} s</td></tr>" for k, v in summary["mean_s"].items() if v)
+ "".join(f"<tr><th>{lab[k]} allein richtig</th><td>{v}/{len(secs[k])}</td></tr>"
for k, v in summary["correct_alone"].items())
+ "</table>")
parts = [head]
per = 4
for i, c in enumerate(shown):
imgs = []
for rr in rendered[per * i: per * (i + 1)]:
if rr.ok:
g = (rr.rasterize(2.0) * 255).astype(np.uint8)
imgs.append(f"<img src='{png_uri(Image.fromarray(255 - g))}'>")
else:
imgs.append("<i>nicht setzbar</i>")
if c["status"] == "auto_accepted":
cls, verdict = ("good", "angenommen, richtig") if c["right"] else ("bad", "angenommen, FALSCH")
elif c["status"] == "needs_review":
cls, verdict = "rev", "Review"
else:
cls, verdict = "rev", c["status"]
ncc = (c["veto"] or {}).get("metrics", {}).get("ncc")
rows = [f"<tr><th>Original</th><td><img src='{png_uri(orig_img(by_name[c['name']], 2))}'></td><td></td></tr>",
f"<tr><th>richtig</th><td>{imgs[0]}</td><td><code>{html.escape(c['truth'])}</code></td></tr>"]
for p, (key, label) in enumerate(TOOLS):
r = c["readings"][p]
if r is None:
rows.append(f"<tr><th>{label}</th><td><i>nicht gefragt</i></td><td></td></tr>")
continue
mark = "<b class=ok>✓</b>" if c["ok"][p] else "<b class=no>✗</b>"
used = " <b>← verwendet</b>" if c["use"] == p and c["status"] == "auto_accepted" else ""
rows.append(f"<tr><th>{mark} {label}{used}<br><small>{r['seconds']:.2f} s</small></th><td>{imgs[p + 1]}</td>"
f"<td><code>{html.escape(r.get('tex') or r.get('error') or '–')}</code>"
+ (f"<br><small>roh: <code>{html.escape(r.get('raw') or '')}</code></small>"
if key == "p2t" and r.get("raw") and r.get("raw") != r.get("tex") else "")
+ "</td></tr>")
parts.append(f"<section class={cls}><div class=v>{verdict} · {html.escape(c['reason'])}"
f"{f' · Bild-Ähnlichkeit {ncc:.2f}' if ncc is not None else ''}</div>"
f"<small>{html.escape(c['name'])}</small><table>{''.join(rows)}</table></section>")
(out / "index.html").write_text(
"<!doctype html><meta charset=utf-8><title>Regeltest</title><style>"
"body{font:14px system-ui,sans-serif;margin:16px;background:#f6f6f3}section{background:#fff;border:1px solid #ddd;"
"border-left:6px solid #999;border-radius:6px;padding:6px 10px;margin:10px 0}section.good{border-left-color:#1f7a3a}"
"section.bad{border-left-color:#a32020;background:#fff5f5}section.rev{border-left-color:#c88a00}"
".v{font-weight:600;font-size:13px}table.sum{border-collapse:collapse;background:#fff}"
"table.sum td,table.sum th{border:1px solid #ccc;padding:4px 10px}"
"th{text-align:left;font-weight:400;color:#555;width:190px;vertical-align:middle}td{padding:3px 8px;vertical-align:middle}"
".ok{color:#1f7a3a}.no{color:#a32020}code{font-size:12px}img{max-width:640px}</style>" + "".join(parts),
encoding="utf-8")
print(json.dumps(summary, ensure_ascii=False, indent=1))
print(out / "index.html")
return 0
if __name__ == "__main__":
sys.exit(main())

145
tools/zeittest.py Normal file
View file

@ -0,0 +1,145 @@
#!/usr/bin/env python3
"""Time and accuracy per recognition tool on formulas whose TeX is known (serial, one at a time).
python3 tools/zeittest.py EPUB [--n 15] [--tools local,local2,cloud,p2t-cpu,p2t-gpu] [--gpu cuda:0]
Same sample as `calibrate --consensus` (seed 42, real images). Every tool reads every formula alone,
one request after the other, after one uncounted warm-up on another formula. Writes
vergleich/zeittest/index.html (not versioned): table per tool, then per formula the original, the
true formula and every reading set with pdflatex, with time and verdict; plus ergebnis.json.
"""
from __future__ import annotations
import argparse
import html
import json
import random
import statistics
import sys
import time
from pathlib import Path
import numpy as np
from PIL import Image
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from konsens_bericht import orig_img, png_uri # noqa: E402
from formula_img2latex.calibrate import _best_preset, corpus # noqa: E402
from formula_img2latex.consensus import same_reading # noqa: E402
from formula_img2latex.engines import make_engine # noqa: E402
from formula_img2latex.engines.p2t import Pix2TextEngine # noqa: E402
from formula_img2latex.render import render_many # noqa: E402
from formula_img2latex.tex import strip_delimiters # noqa: E402
LABELS = {"local": "Qwen3-VL-30B (lokal, GPU)", "local2": "Gemma-4-26B (lokal, GPU)", "cloud": "GLM-4.6V (Cloud)",
"p2t-cpu": "pix2text MFR 1.5 (CPU)", "p2t-gpu": "pix2text MFR 1.5 (GPU)"}
def engine_for(tool: str, gpu: str):
if tool == "p2t-cpu":
return Pix2TextEngine(device="cpu")
if tool == "p2t-gpu":
return Pix2TextEngine(device=gpu)
return make_engine(tool)
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("epub", type=Path)
ap.add_argument("--n", type=int, default=15)
ap.add_argument("--tools", default="local,local2,cloud,p2t-cpu,p2t-gpu")
ap.add_argument("--gpu", default="cuda", help="Gerät für p2t-gpu (Index relativ zu CUDA_VISIBLE_DEVICES)")
ap.add_argument("--name", default="zeittest")
args = ap.parse_args()
tools = [t.strip() for t in args.tools.split(",") if t.strip()]
items = corpus(args.epub)
smp = random.Random(42).sample(items, min(150, len(items))) # as calibrate_consensus
formulas, warm = smp[:args.n], smp[args.n]
preset = _best_preset(smp[:12])
runs: dict[str, dict] = {}
for tool in tools:
eng = engine_for(tool, args.gpu)
t0 = time.time()
eng.check()
load = time.time() - t0
eng.recognize(warm[1], warm[0], "", warm[3]) # warm-up, not counted
reads = []
for name, data, truth, disp in formulas:
r = eng.recognize(data, name, "", disp)
ok = bool(r.tex) and bool(same_reading(r.tex, truth, disp, preset))
reads.append({"tex": r.tex, "seconds": r.seconds, "ok": ok, "error": r.error,
"tokens_in": r.tokens_in, "tokens_out": r.tokens_out})
print(f"{tool:8s} {r.seconds:6.2f}s {'✓' if ok else '✗'} {r.tex or r.error}", file=sys.stderr, flush=True)
secs = [x["seconds"] for x in reads]
runs[tool] = {"label": LABELS.get(tool, tool), "model": eng.model, "load_s": load, "reads": reads,
"mean_s": statistics.mean(secs), "median_s": statistics.median(secs),
"correct": sum(x["ok"] for x in reads), "errors": sum(bool(x["error"]) for x in reads),
"tokens_in": sum(x["tokens_in"] for x in reads), "tokens_out": sum(x["tokens_out"] for x in reads)}
out = Path(__file__).resolve().parents[1] / "vergleich" / args.name
out.mkdir(parents=True, exist_ok=True)
(out / "ergebnis.json").write_text(json.dumps(
{"epub": args.epub.name, "preset": preset, "truth": [f[2] for f in formulas], "runs": runs},
ensure_ascii=False, indent=1), encoding="utf-8")
n = len(formulas)
rows = "".join(
f"<tr><td>{html.escape(r['label'])}</td><td>{r['mean_s']:.2f}</td><td>{r['median_s']:.2f}</td>"
f"<td>{r['correct']}/{n} ({r['correct'] / n:.0%})</td><td>{r['load_s']:.1f}</td>"
f"<td>{r['tokens_in'] + r['tokens_out'] or '–'}</td></tr>" for r in runs.values())
parts = [f"<h1>Zeittest: {html.escape(args.epub.name[:70])}</h1>",
f"<p>{n} Formeln mit bekanntem TeX (alt-Text), jedes Tool seriell, eine Formel nach der anderen, "
"nach einem nicht gezählten Aufwärmlauf. Alle Formeln mit pdflatex gesetzt. ✓ = gleiche Zeichen an "
"gleicher Stelle wie die wahre Formel (Abstände, kursiv/aufrecht, Fettdruck egal).</p>",
"<table class=sum><tr><th>Tool</th><th>Ø s/Formel</th><th>Median s</th><th>richtig</th>"
f"<th>Laden s</th><th>Tokens</th></tr>{rows}</table>"]
texes = []
for i, (_n, _d, truth, disp) in enumerate(formulas):
texes.append((strip_delimiters(truth), disp))
texes += [(runs[t]["reads"][i]["tex"] or "\\text{–}", disp) for t in tools]
rendered = render_many(texes, preset)
per = 1 + len(tools)
for i, (name, data, truth, _disp) in enumerate(formulas):
imgs = []
for rr in rendered[per * i: per * (i + 1)]:
if rr.ok:
g = (rr.rasterize(2.0) * 255).astype(np.uint8)
imgs.append(f"<img src='{png_uri(Image.fromarray(255 - g))}'>")
else:
imgs.append("<i>nicht setzbar</i>")
body = [f"<tr><th>Original</th><td><img src='{png_uri(orig_img(data, 2))}'></td><td></td></tr>",
f"<tr><th>richtig</th><td>{imgs[0]}</td><td><code>{html.escape(truth)}</code></td></tr>"]
for k, t in enumerate(tools):
r = runs[t]["reads"][i]
mark = "<b class=ok>✓</b>" if r["ok"] else "<b class=no>✗</b>"
body.append(f"<tr><th>{mark} {html.escape(runs[t]['label'])}<br><small>{r['seconds']:.2f} s</small></th>"
f"<td>{imgs[k + 1]}</td><td><code>{html.escape(r['tex'] or r['error'] or '–')}</code></td></tr>")
parts.append(f"<section><div class=v>Formel {i + 1} · {html.escape(name)}</div><table>{''.join(body)}</table></section>")
(out / "index.html").write_text(
"<!doctype html><meta charset=utf-8><title>Zeittest</title><style>"
"body{font:14px system-ui,sans-serif;margin:16px;background:#f6f6f3}section{background:#fff;border:1px solid #ddd;"
"border-radius:6px;padding:6px 10px;margin:10px 0}.v{font-weight:600;font-size:13px;margin-bottom:4px}"
"table.sum{border-collapse:collapse;background:#fff}table.sum td,table.sum th{border:1px solid #ccc;padding:4px 10px}"
"th{text-align:left;font-weight:400;color:#555;width:210px;vertical-align:middle}td{padding:3px 8px;vertical-align:middle}"
".ok{color:#1f7a3a}.no{color:#a32020}code{font-size:12px}img{max-width:640px}</style>" + "".join(parts),
encoding="utf-8")
print(f"\n| Tool | Ø s/Formel | Median s | richtig (n={n}) | Laden s |")
print("|---|---|---|---|---|")
for r in runs.values():
print(f"| {r['label']} | {r['mean_s']:.2f} | {r['median_s']:.2f} | {r['correct']}/{n} | {r['load_s']:.1f} |")
if "p2t-cpu" in runs and "p2t-gpu" in runs:
same = sum(a["tex"] == b["tex"] for a, b in zip(runs["p2t-cpu"]["reads"], runs["p2t-gpu"]["reads"]))
print(f"\npix2text CPU und GPU gleicher Text: {same}/{n}")
print(out / "index.html")
return 0
if __name__ == "__main__":
sys.exit(main())