dschlueter/formula_img2latex
Watch
1
0
Fork
You've already forked formula_img2latex
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

feat: 0.3.0 – pix2text zuerst, Qwen bestätigt, Gemma nur bei Uneinigkeit

Browse source 
Neue Standardregel p2t (GLM/Cloud wird nicht mehr benutzt):
- pix2text liest zuerst; übernommen wird immer seine Lesart, wenn Qwen sie
  bestätigt oder – nur bei Uneinigkeit gefragt – Gemma. Qwen = Gemma gegen
  pix2text geht an einen Menschen. Schlusspunkt muss bestätigt sein; Bild-Veto
  und Pandoc auf pix2texts Lesart. Alte Regel als consensus_rule=vlm3 wählbar.
- pix2text auf der GPU (onnxruntime-gpu, FORMULA_P2T_DEVICE=auto/cpu/cuda:N)
- Ausgabe bereinigt: post_process aus pix2tex, Fettschreibweisen \boldmath/\bf
  -> \boldsymbol/\mathbf; an 400 Lesarten gleiches gesetztes Bild
- Normalform: \mathop, \begin{array}[t]
- calibrate: --engines, Regel-Tabelle, --reuse; wahre Formeln aus dem alt-Text
  bereinigt (aligned, \mathop{}\!, upgreek)
- tools/zeittest.py, tools/regeltest.py (--reuse, --exclude)

Messung: Göllmann Band 1 (2 x 150) 86 % / 85 % automatisch, 0 falsch
(bisher 78 % / 70 %); Band 2, zwei Stichproben zu 50: 78 % und 86 %, 0 falsch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Dieter Schlüter 2026-10-09 03:45:18 +02:00
commit 4638a12571
14 changed files with 941 additions and 98 deletions
Download patch file Download diff file Expand all files Collapse all files

43
BEDIENUNGSANLEITUNG.md
View file

@ -3,8 +3,10 @@
Dieses Programm ersetzt Formelbilder in den Markdown-Dateien von epub2md und pdf2md durch echte
LaTeX-Formeln. Dadurch passen die Formeln beim Neubau als EPUB oder PDF in Größe und Schrift zum Text.
Es ersetzt eine Formel nur, wenn **drei unabhängige Bildmodelle dieselbe Formel lesen** (zwei lokale, eines
in der Cloud) und das gerenderte LaTeX dem Bild grob ähnelt. Bei Zweifeln fragt es dich in einer
Es liest jede Formel zuerst mit dem Formel-OCR **pix2text** und ersetzt sie nur, wenn das Bildmodell
**Qwen** dieselbe Formel liest (bei Uneinigkeit darf das zweite Bildmodell Gemma pix2text bestätigen) und
das gerenderte LaTeX dem Bild grob ähnelt. Eingesetzt wird die Lesart von pix2text, ohne ihre
überflüssigen Leerzeichen. Alles läuft lokal. Bei Zweifeln fragt es dich in einer
Review-Seite im Browser. Ohne deine Zustimmung bleibt die Formel ein Bild.
„Dieselbe Formel“ heißt: jedes Zeichen stimmt und steht an der richtigen Stelle. Abstände, Pfeillängen,
@ -41,12 +43,28 @@ CUDA_VISIBLE_DEVICES=2 ~/.local/bin/llama-server \
GPU 2 ist währenddessen für Chatterbox-TTS belegt. Beenden mit Strg+C in beiden Terminals.
### Dritte Lesart aus der Cloud
### pix2text (Formel-OCR)
Die Bestätigung kommt von **GLM-4.6V** über OpenRouter (andere Modellfamilie als Qwen und Gemma). Dafür
muss `OPENROUTER_API_KEY` gesetzt sein. GLM liest nur die Formeln, bei denen Qwen und Gemma schon einig
sind. Beim Gehrke-Brückenkurs (1505 Formelbilder) waren das 967 Anfragen für zusammen etwa 0,15–0,20 $.
Ohne Cloud (`--no-cloud`) wird nichts automatisch ersetzt, alles Einige landet im Review.
pix2text läuft ohne Server im Programm selbst, auf der ersten sichtbaren Grafikkarte (sonst auf der CPU,
dann ~5× langsamer). Einmalig:
```bash
pip install "pix2text[multilingual]"
pip install --no-deps pix2tex munch # nur für post_process (Leerzeichen)
pip uninstall -y onnxruntime && pip install onnxruntime-gpu # für die GPU
```
Das Modell (~300 MB) wird beim ersten Lauf nach `~/.pix2text` geladen. Gerät wählen mit
`FORMULA_P2T_DEVICE=cpu` oder `cuda:N` (`N` zählt innerhalb von `CUDA_VISIBLE_DEVICES`).
Gemma (Port 8012) wird nur für Formeln gefragt, bei denen pix2text und Qwen uneinig sind. Läuft Gemma
nicht, bleiben diese Formeln im Review.
### Frühere Regel mit Cloud (optional)
Bis Version 0.2.0 kam die Bestätigung von **GLM-4.6V** über OpenRouter (Qwen = Gemma = GLM). Diese Regel
ist weiter wählbar: im Arbeitsordner in `workspace.json` unter `settings` den Eintrag
`"consensus_rule": "vlm3"` setzen; dann muss `OPENROUTER_API_KEY` gesetzt sein.
---
@ -168,9 +186,10 @@ geändert, wird sie übersprungen. `--force` setzt sie trotzdem zurück, dann is
```bash
python3 formula_img2latex.py scan AUSGABE # Bilder finden
python3 formula_img2latex.py recognize AUSGABE --engine local # lokal erkennen
python3 formula_img2latex.py recognize AUSGABE --engine p2t # pix2text (zuerst)
python3 formula_img2latex.py recognize AUSGABE --engine local # Qwen (bestätigt)
python3 formula_img2latex.py verify AUSGABE # prüfen und entscheiden
python3 formula_img2latex.py recognize AUSGABE --engine cloud --only-unresolved
python3 formula_img2latex.py recognize AUSGABE --engine local2 --only-unresolved # Gemma (Stichwahl)
python3 formula_img2latex.py verify AUSGABE
python3 formula_img2latex.py status AUSGABE
```
@ -214,6 +233,6 @@ Entscheidung mit Grund und den Bildabgleich. Der Ordner `vergleich/` wird nicht
Ja. Das Bild hatte die Farbe eingebaut, LaTeX-Formeln haben keinen Hintergrund.
**Werden Buchinhalte ins Internet geschickt?**
Nur für die dritte Lesart: die einzelnen Formelbilder, bei denen Qwen und Gemma einig sind, samt eines
kurzen Textausschnitts um die Formel (höchstens 240 Zeichen), gehen an OpenRouter/GLM. Mit `--no-cloud` bleibt alles lokal; dann wird nichts automatisch
ersetzt.
Nein, mit der Standardregel bleibt alles lokal. Nur die frühere Regel `vlm3` schickt die Formelbilder, bei
denen Qwen und Gemma einig sind, samt eines kurzen Textausschnitts um die Formel (höchstens 240 Zeichen)
an OpenRouter/GLM.

26
CHANGELOG.md
View file

@ -1,5 +1,31 @@
# Changelog
## 0.3.0 – 2026-10-09
Neue Standardregel `p2t`: **pix2text liest zuerst, Qwen bestätigt, Gemma nur bei Uneinigkeit**; GLM (Cloud)
wird nicht mehr benutzt.
- **pix2text** (Formel-OCR MFR 1.5, ONNX) als Engine `p2t`, im Prozess, auf der GPU mit `onnxruntime-gpu`
(`FORMULA_P2T_DEVICE`: `auto`, `cpu`, `cuda:N`). Allein am häufigsten richtig (Göllmann Band 1: 90 % /
85 %) und am schnellsten (0,1 s/Formel auf der GPU, Qwen 0,7 s, Gemma 1,8 s).
- **Regel:** übernommen wird immer pix2texts Lesart (trifft den Fettdruck am besten), wenn Qwen sie
bestätigt oder – nur bei Uneinigkeit gefragt – Gemma. Qwen und Gemma gemeinsam gegen pix2text gehen
an einen Menschen (so kamen an Band 1 die gemeinsamen Fehler der beiden Bildmodelle durch). Der
Schlusspunkt muss bestätigt sein. Bild-Veto und Pandoc wie bisher, geprüft auf pix2texts Lesart.
Die alte Regel bleibt als `"consensus_rule": "vlm3"` in `workspace.json` wählbar.
- **pix2text-Ausgabe bereinigt:** Leerzeichen mit `post_process` aus pix2tex (installiert mit
`--no-deps`), Fettschreibweisen `\mathrm{\boldmath~ x ~}` / `\mathrm{\bf~ r o t}` →
`\boldsymbol{x}` / `\mathbf{rot}`. An 400 Lesarten geprüft: gleiches gesetztes Bild.
- **Normalform:** `\operatorname{l i m}` = `\lim`, `\ddot{u}` in Text = ü, `^{\prime}` = `'`,
abschließendes `\\` vor `\end`, `\begin{array}[t]`, `\mathop{…}`.
- **calibrate:** beliebig viele Engines (`--engines`), Regel-Tabelle, `--reuse` rechnet aus einem
früheren Bericht ohne Modellabfragen; wahre Formeln aus dem alt-Text werden bereinigt
(einzeiliges `aligned`, `\mathop{}\!\mathrm{d}`, upgreek).
- **Werkzeuge:** `tools/zeittest.py` (Zeit und Trefferquote je Tool), `tools/regeltest.py` (Standardregel
an Formeln mit bekanntem TeX, Seite zum Mitbeurteilen; `--reuse`, `--exclude`).
- **Messung:** Göllmann Band 1 (2 × 150) 86 % / 85 % automatisch, 0 falsch (bisher 78 % / 70 %);
Band 2, zwei frische Stichproben zu 50: 78 % und 86 % automatisch, 0 falsch.
## 0.2.0 – 2026-10-07
Entscheider ist jetzt der **Konsens dreier Bildmodelle** statt eines Pixelvergleichs.

72
README.md
View file

@ -3,7 +3,8 @@
Nachbearbeitungs-Werkzeug für die Markdown-Ausgaben von
[epub2md](https://kitux.de/forgejo/dschlueter/epub2md) und
[pdf2md](https://kitux.de/forgejo/dschlueter/pdf2md): Formeln, die nur als Bild vorliegen, werden durch
LaTeX ersetzt – **aber nur, wenn drei unabhängige Bildmodelle dieselbe Formel lesen** und ein grober
LaTeX ersetzt – **aber nur, wenn zwei unabhängige Erkenner dieselbe Formel lesen** (zuerst das Formel-OCR
pix2text, bestätigt vom Bildmodell Qwen; bei Uneinigkeit darf Gemma pix2text bestätigen) und ein grober
Bildabgleich nicht widerspricht. Alles andere geht in eine Review-Galerie oder bleibt Bild.
Schritt-für-Schritt-Anleitung: [BEDIENUNGSANLEITUNG.md](BEDIENUNGSANLEITUNG.md). Änderungen je Version:
@ -14,13 +15,22 @@ Schritt-für-Schritt-Anleitung: [BEDIENUNGSANLEITUNG.md](BEDIENUNGSANLEITUNG.md)
> Jedes Zeichen muss stimmen und an der richtigen Stelle relativ zu den anderen stehen. Abstände,
> Pfeil- und Bruchstrichlängen, Schriftgröße, kursiv/aufrecht und Fettdruck dürfen abweichen.
Eine Formel wird automatisch ersetzt, wenn
Eine Formel wird automatisch ersetzt (Regel `p2t`, Standard), wenn
1. **Qwen3-VL-30B-A3B** (lokal, llama.cpp) und **Gemma-4-26B-A4B** (lokal) dieselbe Formel lesen,
2. **GLM-4.6V** (Cloud über OpenRouter, andere Modellfamilie) das bestätigt – ein fehlender Schlusspunkt
zählt dabei nicht,
3. Pandoc sie nach MathML umsetzen kann (sonst bliebe sie im EPUB/PDF roher TeX-Text) und
4. das gerenderte LaTeX dem Original grob ähnelt (Bild-Veto, elastischer Vergleich, Ähnlichkeit ≥ 0,65).
1. **pix2text** (Formel-OCR MFR 1.5, lokal, GPU) sie zuerst liest und **Qwen3-VL-30B-A3B** (Bildmodell,
lokal, llama.cpp) dieselbe Formel liest **oder**, wenn die beiden uneinig sind, **Gemma-4-26B-A4B**
(lokal) die Lesart von pix2text bestätigt. Übernommen wird immer **pix2texts Lesart** – sie trifft den
Fettdruck am besten. Ihre überflüssigen Leerzeichen entfernt vorher `post_process` aus pix2tex (LaTeX-OCR):
`\frac{\alpha} {3 6 0}` → `\frac{\alpha}{360}`; am gesetzten Bild ändert das nichts, geprüft an
400 Lesarten. Auch der Schlusspunkt muss bestätigt sein,
2. Pandoc sie nach MathML umsetzen kann (sonst bliebe sie im EPUB/PDF roher TeX-Text) und
3. das gerenderte LaTeX dem Original grob ähnelt (Bild-Veto, elastischer Vergleich, Ähnlichkeit ≥ 0,65).
Sind Qwen und Gemma gemeinsam gegen pix2text einig, entscheidet ein Mensch: Genau das sind die Fehler,
die beide Bildmodelle teilen (`\mathcal{E}_r` statt `\varepsilon_r`, erfundenes `\tilde{z}`).
Die frühere Regel `vlm3` (Qwen = Gemma, bestätigt von **GLM-4.6V** über OpenRouter) bleibt wählbar:
`"consensus_rule": "vlm3"` unter `settings` in `workspace.json` des Arbeitsordners.
„Dieselbe Formel“ prüft `consensus.py`: gleiche Normalform (Abstände, `\left`/`\right`, überflüssige
Klammern, `\vec`/`\overrightarrow`, `\frac`/`\dfrac`, `\iff`/`\Leftrightarrow`, `pmatrix`/`\left(\begin{matrix}`,
@ -32,6 +42,25 @@ Sind sich die Modelle uneinig oder widerspricht das Veto, entscheidet ein Mensch
## Messung
### Regel `p2t` (Standard)
Göllmann Band 1, 150 echte Bilder und 150 Nachbildungen im Gehrke-Stil, Lesarten aller Modelle:
| Regel | echt: automatisch / falsch | Gehrke-Stil: automatisch / falsch |
| --- | --- | --- |
| pix2text = Qwen | 81 % / 0 | 76 % / 0 |
| + Stichwahl: Gemma bestätigt pix2text, Qwens Lesart (erste Fassung) | 85 % / 0 | 80 % / 0 |
| **pix2text zuerst und übernommen, bereinigt, Stichwahl Gemma** (Standard) | **86 % / 0** | **85 % / 0** |
| + Stichwahl auch Qwen = Gemma gegen pix2text | 92 % / 1 | 91 % / 2 |
| Qwen = Gemma = GLM (Regel `vlm3`) | 78 % / 0 | 70 % / 0 |
Allein richtig: pix2text 90 % / 85 %, Qwen 86 % / 84 %, Gemma 86 % / 80 %, GLM 86 % / 83 %.
Zeit je Formel (seriell, 15 Formeln): pix2text 0,10 s (GPU) bzw. 0,51 s (CPU), Qwen 0,72 s, GLM 1,19 s,
Gemma 1,77 s. Gegenprobe an 50 frischen Formeln aus Göllmann Band 2 (`tools/regeltest.py`):
37 automatisch (74 %), **0 falsch**, 13 Review; Gemma wurde für 16 Formeln gefragt.
### Regel `vlm3` (Standard bis 0.2.0)
An Formelbildern mit bekanntem TeX (Göllmann, Springer: TeX im alt-Text), je 150 echte Bilder und 150
Nachbildungen im Stil von Gehrke (LaTeX + dvipng mit Bitmap-Schriften, ~102 dpi, JPEG):
@ -63,12 +92,12 @@ Gehrke, Brückenkurs Mathematik (EPUB, 1505 Formelbilder): siehe [Stand](#stand)
```text
scan Bildverweise in chapters/*.md (bzw. pdf2md-Markdown) finden, Formel/Abbildung vorklassifizieren
run in Paketen zu 50: Qwen + Gemma lesen (parallel), prüfen, GLM nur für Einige, prüfen;
nach jedem Paket eine Statuszeile
run in Paketen zu 50: pix2text, dann Qwen lesen, prüfen, Gemma nur für Uneinige, prüfen;
nach jedem Paket eine Statuszeile (Regel vlm3: Qwen + Gemma, GLM nur für Einige)
review Zweifelsfälle im Browser: Original / Render / Overlay / Diff, LaTeX korrigieren und neu prüfen
apply angenommene Formeln ins Markdown schreiben (aus einer Sicherung, idempotent)
revert Markdown-Dateien auf den Stand vor apply zurücksetzen
recognize einzelne Engine (local, local2, cloud) laufen lassen
recognize einzelne Engine (local, local2, cloud, p2t) laufen lassen
verify prüfen und entscheiden (ohne neue Modellabfragen)
status Zählung je Status
calibrate Messung an Formelbildern mit bekanntem TeX (--consensus: Konsensregel)
@ -90,14 +119,19 @@ Status je Formel: `auto_accepted`, `needs_review`, `rejected_keep_image`, `rende
| --- | --- | --- |
| `local` | llama.cpp `http://127.0.0.1:8011/v1` (Qwen3-VL-30B-A3B-Instruct Q4_K_M + mmproj) | `FORMULA_LOCAL_URL`, `FORMULA_LOCAL_MODEL` |
| `local2` | llama.cpp `http://127.0.0.1:8012/v1` (Gemma-4-26B-A4B-it Q4_0 + mmproj) | `FORMULA_LOCAL2_URL`, `FORMULA_LOCAL2_MODEL` |
| `cloud` | OpenRouter, `z-ai/glm-4.6v` | `OPENROUTER_API_KEY`, `FORMULA_CLOUD_MODEL` |
| `cloud` | OpenRouter, `z-ai/glm-4.6v` (nur Regel `vlm3`) | `OPENROUTER_API_KEY`, `FORMULA_CLOUD_MODEL` |
| `p2t` | pix2text MFR 1.5 (ONNX) im Prozess, erste sichtbare GPU, sonst CPU | `FORMULA_P2T_DEVICE` (`auto`, `cpu`, `cuda:N`), `FORMULA_P2T_MODEL` |
pix2text auf der GPU braucht `onnxruntime-gpu` statt `onnxruntime` (cuDNN kommt aus den nvidia-Paketen
von torch). Für `post_process` wird `pix2tex` ohne seine Abhängigkeiten installiert
(`pip install --no-deps pix2tex munch`), weil es sonst `timm==0.5.4` erzwingen würde. `cuda:N` zählt innerhalb von `CUDA_VISIBLE_DEVICES`.
`FORMULA_<ENGINE>_THINKING=1` schaltet das Nachdenken wieder ein. Antworten sind auf 800 Tokens begrenzt
(lange mehrzeilige Formeln brauchen ~400; einzelne Antworten von Gemma liefen sonst in
Wiederholungsschleifen bis 165 s). Jede Lesart wird mit Modell, Prompt-Version und Nachdenk-Einstellung
gespeichert; dieselbe Frage wird nie zweimal gestellt, ein abgebrochener Lauf setzt dort fort.
Kosten der Cloud-Stufe: GLM liest nur Formeln, bei denen Qwen und Gemma einig sind. Gehrke: 967
Kosten der Cloud-Stufe (nur Regel `vlm3`): GLM liest nur Formeln, bei denen Qwen und Gemma einig sind. Gehrke: 967
Anfragen, Ø 2,6 s, zusammen rund 0,15–0,20 $.
## Bildvergleich (Veto)
@ -105,9 +139,9 @@ Anfragen, Ø 2,6 s, zusammen rund 0,15–0,20 $.
`compare.py` rendert die Lesart mit pdflatex (Schrift je Buch automatisch: `cm`, `times`, `txfonts`,
`palatino`, `stix`, `euler`), richtet sie auf das Original aus und verzerrt sie **elastisch**
(dynamische Zeitverzerrung über Spalten und Zeilen; Dehnen kostet proportional zur Tinte), damit
Abstände und Längen keine Rolle spielen. Die Ähnlichkeit danach ist das Veto. Geprüft wird nur die
Lesart, die übernommen würde (die von Qwen; im Konsens ist sie gleichwertig zu den anderen) – eine
später eintreffende dritte Lesart löst keine neue Prüfung aus. Die Prüfergebnisse tragen eine
Abstände und Längen keine Rolle spielen. Die Ähnlichkeit danach ist das Veto. Geprüft wird nur
die Lesart, die übernommen werden kann: die von pix2text (Regel `vlm3`: die von Qwen). Bestätigende
Lesarten werden nie übernommen und brauchen keine Prüfung. Die Prüfergebnisse tragen eine
Versionsnummer des Verfahrens und werden nach Änderungen neu berechnet. Der alte Pixel-Entscheider mit
Verwechslungsvarianten und Kontrasttest bleibt als Modus `decide=pixel` erhalten.
@ -145,6 +179,11 @@ sind nicht Teil dieser Version.
- `tools/vergleich.py AUSGABE` – feste Stichprobe (50 Formeln), verarbeitet wie `run`, schreibt
`vergleich/<name>/index.html` mit Original, gerenderten Lesarten, Entscheidung und Veto-Overlay.
- `tools/konsens_bericht.py BERICHT.json EPUB` – Übersichtsseite zu `calibrate --consensus --report`.
- `tools/zeittest.py EPUB` – Zeit und Trefferquote je Tool, seriell an 15 Formeln mit bekanntem TeX.
- `tools/regeltest.py EPUB [--n 50] [--reuse ergebnis.json]` – die Standardregel an Formeln mit bekanntem
TeX, wie `run`; Seite mit Original, wahrer Formel, allen Lesarten und Entscheidung.
- `calibrate --consensus --engines local,local2,cloud,p2t --report X.json` misst alle Regeln;
`--reuse X.json` rechnet sie ohne Modellabfragen neu.
- `vergleich/` ist nicht versioniert (Ausschnitte aus Büchern).
## Stand
@ -209,7 +248,8 @@ formula_img2latex/
├── pipeline.py scan, recognize, verify, Entscheidung (Konsens/Pixel), run in Paketen, apply/revert
├── consensus.py Normalform, Render-Gleichheit: sagen zwei Lesarten dasselbe?
├── decide.py Annahmeregeln und Schwellen (Konsens- und Pixel-Modus)
├── engines/ Vision-Modelle über die OpenAI-API (llama.cpp, OpenRouter), Logprobs, Prompt-Version
├── engines/ Vision-Modelle über die OpenAI-API (llama.cpp, OpenRouter), Logprobs, Prompt-Version;
│ p2t.py: pix2text-Formel-OCR (ONNX, CPU/GPU)
├── compare.py Ausrichtung, elastische Verzerrung, Kennzahlen, Kontrasttest, Diff-Bilder
├── confusables.py Verwechslungsvarianten (Pixel-Modus)
├── verify.py Bildprüfung einer Lesart (Veto bzw. Pixel-Entscheider)

2
formula_img2latex/__init__.py
View file

@ -1,3 +1,3 @@
"""formula_img2latex: Formelbilder in Markdown geprüft durch LaTeX ersetzen."""
__version__ = "0.2.0"
__version__ = "0.3.0"

180
formula_img2latex/calibrate.py
View file

@ -43,6 +43,22 @@ from .verify import verify
SYNTH_PX_PER_PT = 1.41
# upright Greek of the upgreek package (Springer alt texts): pdflatex here has no upgreek, and
# upright vs. italic is the same symbol for the project
_UPGREEK = re.compile(r"\\[Uu]p(alpha|beta|gamma|delta|epsilon|varepsilon|zeta|eta|theta|vartheta|iota|kappa|lambda|mu|nu|"
r"xi|pi|varpi|rho|varrho|sigma|varsigma|tau|upsilon|phi|varphi|chi|psi|omega)(?![A-Za-z])", re.I)
def clean_truth(tex: str) -> str:
"""Alt-text TeX as a plain formula: no one-line aligned wrapper, \\mathop{}\\!\\mathrm{d} -> \\mathrm{d},
upgreek -> standard Greek. Only the spelling changes, never a symbol."""
m = re.fullmatch(r"\s*\\begin\{aligned\}\s*(?:\\displaystyle)?(.*?)\\end\{aligned\}\s*", tex, re.S)
if m and "\\\\" not in m.group(1) and "&" not in m.group(1):
tex = m.group(1).strip()
tex = re.sub(r"\\mathop\{\}\\!", "", tex)
return _UPGREEK.sub(lambda g: "\\" + (g.group(1).capitalize() if g.group(0)[1] == "U" else g.group(1).lower()), tex)
def corpus(epub: Path) -> list[tuple[str, bytes, str, bool]]:
"""(name, image bytes, true TeX, display) for every image with TeX in its alt text."""
out = []
@ -59,7 +75,7 @@ def corpus(epub: Path) -> list[tuple[str, bytes, str, bool]]:
if not (src and alt and alt.group(1).lstrip().startswith("$")):
continue
path = posixpath.normpath(posixpath.join(base, src.group(1)))
tex = strip_delimiters(html.unescape(alt.group(1)))
tex = clean_truth(strip_delimiters(html.unescape(alt.group(1))))
if path in names and not unsafe_reason(tex):
display = "IEq" not in path and "inline" not in tag.lower()
out.append((path, z.read(path), tex, display))
@ -224,26 +240,63 @@ def _print(s: dict[str, Any], log: Any) -> None:
# ------------------------------------------------------------------------- consensus calibration
def _consensus_eval(args: tuple) -> dict[str, Any]:
"""Worker: compare both readings with each other and with the truth; rough image check."""
"""Worker: compare all readings with each other and with the truth; rough image check."""
from .consensus import same_reading
name, data, truth, display, preset, ra, rb = args
from .pipeline import _strip_final_punct
name, data, truth, display, preset, readings = args
os.environ.setdefault("OMP_NUM_THREADS", "1")
ta, tb = ra.get("tex") or "", rb.get("tex") or ""
res: dict[str, Any] = {"name": name, "truth": truth, "display": display, "a": ra, "b": rb,
"a_ok": bool(same_reading(ta, truth, display, preset)),
"b_ok": bool(same_reading(tb, truth, display, preset)),
"agree": same_reading(ta, tb, display, preset)}
res["veto"] = None
if res["agree"]:
v = verify(load_ink(data, name), ta, display, preset, check_variants=False)
res["veto"] = {"ok": v.ok, "reason": v.reason, "metrics": v.metrics}
return res
texes = [r.get("tex") or "" for r in readings]
ok = [bool(same_reading(t, truth, display, preset)) for t in texes]
agree: list[tuple[int, int, str]] = []
for i in range(len(texes)):
for j in range(i + 1, len(texes)):
a, b = texes[i], texes[j]
# as in pipeline.decide_all_consensus: a confirming cloud reading may lack the final period;
# pix2text's reading is used under rule p2t, so it is compared strictly
if j >= 2 and readings[j].get("engine") != "p2t" and readings[i].get("engine") != "p2t":
a, b = _strip_final_punct(a), _strip_final_punct(b)
how = same_reading(a, b, display, preset)
if how:
agree.append((i, j, how))
vetos: dict[int, dict[str, Any]] = {}
for k in range(len(texes)): # only readings that may be used: primary pair and pix2text (rule p2t)
if (k < 2 or readings[k].get("engine") == "p2t") and any(k in (i, j) for i, j, _ in agree):
v = verify(load_ink(data, name), texes[k], display, preset, check_variants=False)
vetos[k] = {"ok": v.ok, "reason": v.reason, "metrics": v.metrics}
pair = next((how for i, j, how in agree if (i, j) == (0, 1)), None)
# "a", "b", "agree", "veto": the two primary models (read by tools/konsens_bericht.py)
return {"name": name, "truth": truth, "display": display, "readings": readings, "ok": ok, "agree_all": agree,
"vetos": vetos, "a": readings[0], "b": readings[1], "a_ok": ok[0], "b_ok": ok[1], "agree": pair,
"veto": vetos.get(0) if pair else None}
def _reading_cache(report: Path | None) -> dict[tuple[str, str, str], dict[str, Any]]:
"""(set label, image name, engine) -> reading from an earlier --report: models are not asked twice.
pix2text readings are tidied again (reports from before tidy_p2t hold the raw spaced output)."""
from .engines.p2t import tidy_p2t
if not report or not report.exists():
return {}
old = json.loads(report.read_text(encoding="utf-8"))
cache = {}
for s in old.get("sets", []):
for r in s.get("results", []):
for rd in r.get("readings") or [r.get("a") or {}, r.get("b") or {}]:
if rd.get("engine") and not rd.get("error"):
if rd["engine"] == "p2t" and rd.get("tex"):
rd = dict(rd, tex=tidy_p2t(rd.get("raw") or rd["tex"]))
cache[(s["label"], r["name"], rd["engine"])] = rd
return cache
def calibrate_consensus(epubs: list[Path], samples: int = 150, preset: str | None = None, synthetic: bool = True,
report: Path | None = None, workers: int = 0, quiet: bool = False,
engines: tuple[str, str] = ("local", "local2")) -> dict[str, Any]:
"""Measure the two-model consensus rule on images whose TeX is known."""
engines: tuple[str, ...] = ("local", "local2"), reuse: Path | None = None) -> dict[str, Any]:
"""Measure consensus rules on images whose TeX is known.
*engines*: the first two are the primary pair (local, local2); further engines (cloud, p2t)
give confirming readings. *reuse*: report of an earlier run whose readings are taken over.
"""
from concurrent.futures import ThreadPoolExecutor
from .decide import consensus_thresholds, decide_consensus
@ -253,10 +306,10 @@ def calibrate_consensus(epubs: list[Path], samples: int = 150, preset: str | Non
if not quiet:
print(m, file=sys.stderr, flush=True)
if len(engines) < 2:
raise ValueError("mindestens zwei Engines nötig")
cache = _reading_cache(reuse)
eng = [make_engine(e) for e in engines]
for e in eng:
e.check()
log("Modelle: " + ", ".join(f"{e.name}={e.model}" for e in eng))
rng = random.Random(42)
sets: list[tuple[str, list[tuple[str, bytes, str, bool]], str]] = []
for epub in epubs:
@ -268,21 +321,28 @@ def calibrate_consensus(epubs: list[Path], samples: int = 150, preset: str | Non
if synthetic:
syn = [(n + "#synth.jpg", img, t, d) for n, _d, t, d in smp for img in [synth_jpeg(t, d, rng)] if img]
sets.append((f"{epub.stem[:40]} [dvipng, Gehrke-Stil]", syn, "cm"))
todo = [(k, label, i) for label, items, _p in sets for k in range(len(eng)) for i in range(len(items))
if (label, items[i][0], eng[k].name) not in cache]
for k in sorted({k for k, _, _ in todo}):
eng[k].check()
log("Modelle: " + ", ".join(f"{e.name}={e.model}" for e in eng)
+ (f"; {len(cache)} Lesarten aus {reuse}" if cache else ""))
summary: dict[str, Any] = {"engines": [f"{e.name}={e.model}" for e in eng], "sets": []}
for label, items, p in sets:
log(f"{label}: {len(items)} Formeln lesen …")
readings: list[list[dict[str, Any]]] = [[{}] * len(items) for _ in eng]
readings: list[list[dict[str, Any]]] = [
[cache.get((label, n, e.name), {}) for n, _d, _t, _disp in items] for e in eng]
def ask(k: int, i: int) -> None:
n, d, _t, disp = items[i]
readings[k][i] = eng[k].recognize(d, n, "", disp).to_json()
with ThreadPoolExecutor(max_workers=8) as pool:
list(pool.map(lambda ki: ask(*ki), [(k, i) for k in range(len(eng)) for i in range(len(items))]))
jobs = [(n, d, t, disp, p, readings[0][i], readings[1][i]) for i, (n, d, t, disp) in enumerate(items)]
list(pool.map(lambda ki: ask(*ki), [(k, i) for k, lb, i in todo if lb == label]))
jobs = [(n, d, t, disp, p, [readings[k][i] for k in range(len(eng))]) for i, (n, d, t, disp) in enumerate(items)]
with ProcessPoolExecutor(max_workers=workers or max(1, (os.cpu_count() or 2) - 2)) as ex:
results = list(ex.map(_consensus_eval, jobs))
s = _summarize_consensus(label, results, consensus_thresholds, decide_consensus)
s = _summarize_consensus(label, results, [e.name for e in eng], consensus_thresholds, decide_consensus)
summary["sets"].append(s)
s["results"] = results
_print_consensus(s, log)
@ -292,19 +352,74 @@ def calibrate_consensus(epubs: list[Path], samples: int = 150, preset: str | Non
return summary
def _summarize_consensus(label: str, results: list[dict[str, Any]], consensus_thresholds: Any,
# Rules to compare: (label, mode, groups). A group lists engines; the first two are the primary pair
# and every further one must confirm it (decide_consensus with require_third). "any": accepted if one
# group accepts (the first accepting group supplies the reading); "all": every group must accept;
# "tiebreak": decide_pair_tiebreak with the group (Qwen, pix2text, Gemma) – the standard rule "p2t".
CONSENSUS_RULES: list[tuple[str, str, list[tuple[str, ...]]]] = [
("Qwen = Gemma", "any", [("local", "local2")]),
("Qwen = Gemma = GLM (bisher)", "any", [("local", "local2", "cloud")]),
("B: Qwen = Gemma = p2t", "any", [("local", "local2", "p2t")]),
("Qwen = Gemma, GLM oder p2t bestätigt", "any", [("local", "local2", "cloud"), ("local", "local2", "p2t")]),
("A: 3 von 4 (p2t entscheidet bei 2:1)", "any",
[("local", "local2", "cloud"), ("local", "local2", "p2t"), ("local", "cloud", "p2t"), ("local2", "cloud", "p2t")]),
("C: Qwen = Gemma = GLM = p2t", "all", [("local", "local2", "cloud"), ("local", "local2", "p2t")]),
("Standard: p2t = Qwen, Stichwahl Gemma = p2t", "tiebreak", [("local", "p2t", "local2")]),
]
def _rule_outcome(r: dict[str, Any], idx: list[int], th: dict[str, float], decide_fn: Any,
tiebreak: bool = False) -> int | None:
"""Index of the accepted reading if the readings *idx* (primary pair first) accept, else None."""
sub = [r["readings"][k] for k in idx]
pos = {k: n for n, k in enumerate(idx)}
agree = [(pos[i], pos[j], how) for i, j, how in r["agree_all"] if i in pos and j in pos]
t = th if tiebreak else dict(th, require_third=1 if len(idx) > 2 else 0)
_st, _why, use = decide_fn(sub, agree, None, True, None, t) # which reading would be used
if use is None:
return None
veto = r["vetos"].get(idx[use])
st, _why, use = decide_fn(sub, agree, veto, True, None, t)
return idx[use] if st == "auto_accepted" else None
def _summarize_consensus(label: str, results: list[dict[str, Any]], names: list[str], consensus_thresholds: Any,
decide_consensus: Any) -> dict[str, Any]:
from .decide import decide_pair_tiebreak
n = len(results)
out: dict[str, Any] = {"label": label, "formulas": n,
out: dict[str, Any] = {"label": label, "formulas": n, "engines": names,
"correct": {e: sum(r["ok"][k] for r in results) for k, e in enumerate(names)},
"a_correct": sum(r["a_ok"] for r in results), "b_correct": sum(r["b_ok"] for r in results),
"agree": sum(bool(r["agree"]) for r in results),
"agree_wrong": sum(bool(r["agree"]) and not r["a_ok"] for r in results),
"sweep": []}
"rules": [], "sweep": []}
out["agree_wrong_examples"] = [{"truth": r["truth"], "a": r["a"].get("tex"), "b": r["b"].get("tex")}
for r in results if r["agree"] and not r["a_ok"]][:10]
th0 = consensus_thresholds()
for rule, mode, groups in CONSENSUS_RULES:
if any(e not in names for g in groups for e in g):
continue
acc = wrong = 0
examples = []
for r in results:
if mode == "tiebreak":
used = [_rule_outcome(r, [names.index(e) for e in g], th0, decide_pair_tiebreak, True)
for g in groups]
else:
used = [_rule_outcome(r, [names.index(e) for e in g], th0, decide_consensus) for g in groups]
ok_used = [u for u in used if u is not None]
use = (ok_used[0] if ok_used else None) if mode == "any" else (used[0] if None not in used else None)
if use is None:
continue
acc += 1
if not r["ok"][use]:
wrong += 1
examples.append({"truth": r["truth"], "used": r["readings"][use].get("tex"),
"readings": {e: rd.get("tex") for e, rd in zip(names, r["readings"])}})
out["rules"].append({"rule": rule, "accepted": acc, "wrong": wrong, "wrong_examples": examples[:5]})
for pm in (0.0, 0.8, 0.9, 0.95, 0.98):
for pn in (0.0, 0.2, 0.3, 0.5):
th = consensus_thresholds({"min_p_mean": pm, "min_p_min": pn})
th = consensus_thresholds({"min_p_mean": pm, "min_p_min": pn, "require_third": 0})
acc = wrong = 0
for r in results:
agree = [(0, 1, r["agree"])] if r["agree"] else []
@ -319,11 +434,14 @@ def _summarize_consensus(label: str, results: list[dict[str, Any]], consensus_th
def _print_consensus(s: dict[str, Any], log: Any) -> None:
n = max(1, s["formulas"])
log(f" {s['label']}: {s['formulas']} Formeln")
log(f" Modell A richtig {s['a_correct'] / n:.0%}, Modell B richtig {s['b_correct'] / n:.0%}, "
f"einig {s['agree'] / n:.0%}, davon FALSCH {s['agree_wrong']}")
for ex in s["agree_wrong_examples"][:5]:
log(f" einig, aber falsch: {ex['a']} (richtig: {ex['truth']})")
log(" Schwellen (Mittel/Minimum) → automatisch angenommen / davon falsch:")
log(" allein richtig: " + ", ".join(f"{e} {c / n:.0%}" for e, c in s["correct"].items())
+ f"; Qwen und Gemma einig {s['agree'] / n:.0%}, davon FALSCH {s['agree_wrong']}")
log(" Regel → automatisch angenommen / davon falsch:")
for row in s["rules"]:
log(f" {row['rule']:<42} {row['accepted']:4d} ({row['accepted'] / n:.0%}) / {row['wrong']}")
for ex in row["wrong_examples"][:3]:
log(f" falsch: {ex['used']} (richtig: {ex['truth']})")
log(" Qwen = Gemma mit Schwellen (Mittel/Minimum) → automatisch angenommen / davon falsch:")
for row in s["sweep"]:
log(f" p_mean ≥ {row['min_p_mean']:.2f}, p_min ≥ {row['min_p_min']:.2f}: "
f"{row['accepted']:4d} ({row['accepted'] / n:.0%}) / {row['wrong']}")

10
formula_img2latex/cli.py
View file

@ -123,7 +123,12 @@ def build_parser() -> argparse.ArgumentParser:
sp.add_argument("--report", type=Path, default=None, help="Bericht als JSON speichern")
sp.add_argument("--workers", type=int, default=0)
sp.add_argument("--consensus", action="store_true",
help="Konsensregel messen: beide lokalen Modelle (local, local2) lesen die Formeln")
help="Konsensregeln messen: die Engines aus --engines lesen die Formeln")
sp.add_argument("--engines", default="local,local2",
help="Engines für --consensus, die ersten beiden bilden das Hauptpaar "
"(Standard local,local2; z. B. local,local2,cloud,p2t)")
sp.add_argument("--reuse", type=Path, default=None,
help="Lesarten aus einem früheren --report übernehmen (Modelle nicht erneut fragen)")
sp.add_argument("--quiet", "-q", action="store_true")
return p
@ -166,7 +171,8 @@ def _dispatch(args: argparse.Namespace) -> int:
if args.command == "calibrate" and args.consensus:
from .calibrate import calibrate_consensus
calibrate_consensus(args.epubs, samples=args.samples, preset=None if args.preset == "auto" else args.preset,
synthetic=args.synthetic, report=args.report, workers=args.workers, quiet=args.quiet)
synthetic=args.synthetic, report=args.report, workers=args.workers, quiet=args.quiet,
engines=tuple(e.strip() for e in args.engines.split(",") if e.strip()), reuse=args.reuse)
return EXIT_OK
if args.command == "calibrate":
from .calibrate import calibrate

5
formula_img2latex/consensus.py
View file

@ -26,7 +26,7 @@ _DROP = {"\\,", "\\;", "\\:", "\\!", "\\ ", "~", "\\quad", "\\qquad", "\\left",
"\\Bigg", "\\bigl", "\\bigr", "\\Bigl", "\\Bigr", "\\biggl", "\\biggr", "\\displaystyle", "\\textstyle",
"\\scriptstyle", "\\limits", "\\nolimits", "\\middle", "\\hfill", "\\enspace", "\\thinspace", "\\medspace"}
# wrappers whose argument is kept (upright vs. italic is the same symbol)
_UNWRAP = {"\\mathrm", "\\mathit", "\\operatorname", "\\textrm", "\\textit", "\\mathnormal", "\\mbox", "\\text",
_UNWRAP = {"\\mathrm", "\\mathit", "\\operatorname", "\\mathop", "\\textrm", "\\textit", "\\mathnormal", "\\mbox", "\\text",
"\\textnormal", "\\textup"}
_SYNONYM = {
"\\overrightarrow": "\\vec", "\\overline": "\\bar", "\\widehat": "\\hat", "\\widetilde": "\\tilde",
@ -73,7 +73,8 @@ def canonical(tex: str, bold: bool = False) -> str:
body = re.sub(r"\\\\\s*(?=\\end\{)", "", body) # a final \\ before \end{...} adds no row
body = re.sub(r"\{?\\ss\{?\}?\}?", "ß", body)
body = re.sub(r"\{([^{}]*)\}\s*\\atop\s*\{([^{}]*)\}", r"\\substack{\1\\\\\2}", body)
body = re.sub(r"\\begin\{array\}\{[^{}]*\}", r"\\begin{matrix}", body).replace("\\end{array}", "\\end{matrix}")
body = re.sub(r"\\begin\{array\}\s*(?:\[[^\]]*\])?\s*\{[^{}]*\}", r"\\begin{matrix}", body) # [t]: position only
body = body.replace("\\end{array}", "\\end{matrix}")
for env, (lft, rgt) in _ENV_DELIMS.items():
body = body.replace(f"\\begin{{{env}}}", f"{lft} \\begin{{MATRIX}}").replace(f"\\end{{{env}}}", f"\\end{{MATRIX}} {rgt}")
body = re.sub(r"\\not\s*=", r"\\ne ", body)

62
formula_img2latex/decide.py
View file

@ -137,7 +137,16 @@ def decide_consensus(readings: list[dict[str, Any]], agree: list[tuple[int, int,
return "needs_review", f"zwei Modelle einig ({how}), dritte Lesart weicht ab: {texts}", i
tie = " (dritte Lesart bestätigt)"
use = min(i, j) # the primary model's reading (it is the one checked against the image)
problems = [p for p in (_sure(readings[i], th), _sure(readings[j], th)) if p]
problems = _checks([readings[i], readings[j]], veto, pandoc_ok, book_px_per_pt, th)
if problems:
return "needs_review", f"einig ({how}){tie}; " + "; ".join(problems), use
return "auto_accepted", f"zwei Modelle einig ({how}){tie}, sicher, Bild-Veto bestanden", use
def _checks(agreeing: list[dict[str, Any]], veto: dict[str, Any] | None, pandoc_ok: bool,
book_px_per_pt: float | None, th: dict[str, float]) -> list[str]:
"""Problems of an agreed reading: unsure readings, Pandoc, rough image check (German reasons)."""
problems = [p for p in (_sure(r, th) for r in agreeing) if p]
if not pandoc_ok:
problems.append("Pandoc kann die Formel nicht nach MathML umsetzen")
if veto is None or not veto.get("ok"):
@ -149,6 +158,53 @@ def decide_consensus(readings: list[dict[str, Any]], agree: list[tuple[int, int,
if (th["veto_scale_dev"] > 0 and book_px_per_pt
and abs(m.get("px_per_pt", book_px_per_pt) / book_px_per_pt - 1) > th["veto_scale_dev"]):
problems.append(f"Bild-Veto: Schriftgröße weicht um {abs(m['px_per_pt'] / book_px_per_pt - 1):.0%} ab")
return problems
# Marker in the reason of a formula that waits for the tie-break reading (see pipeline.needs_tiebreak).
TIEBREAK_MISSING = "Stichwahl fehlt"
def decide_pair_tiebreak(readings: list[dict[str, Any] | None], agree: list[tuple[int, int, str]],
veto: dict[str, Any] | None, pandoc_ok: bool, book_px_per_pt: float | None,
th: dict[str, float]) -> tuple[str, str, int | None]:
"""Rule "p2t": pix2text reads first, Qwen confirms, Gemma only where the two disagree.
The reading used is always pix2text's (spaces tidied; it reads bold type best). *readings*:
[Qwen, pix2text, Gemma or None]. *agree*: pairs (i, j, how) of readings that say the same.
*veto*: verification of pix2text's reading. Returns (status, reason, index of the reading to use
or None).
Gemma may only confirm pix2text. Qwen and Gemma together against pix2text go to a human: in the
measurement exactly these were the errors both VLMs share (\\mathcal{E}_r for \\varepsilon_r, an
invented \\tilde{z}).
"""
q, p = readings[0], readings[1]
g = readings[2] if len(readings) > 2 else None
pairs = {frozenset((i, j)): how for i, j, how in agree}
if q is None or p is None:
missing = "Qwen" if q is None else "pix2text"
return "needs_review", f"Lesart von {missing} fehlt", 1 if p is not None and p.get("tex") else None
if q.get("is_formula") is False and (g is None or g.get("is_formula") is False) and (g or not p.get("tex")):
return "not_formula", "Qwen" + (" und Gemma" if g else "") + ": keine Formel", None
first = 1 if p.get("tex") else None # reading shown for review: pix2text's
if frozenset((0, 1)) in pairs:
how, who, confirm = pairs[frozenset((0, 1))], "pix2text und Qwen einig", q
elif g is None:
return "needs_review", f"pix2text und Qwen uneinig, {TIEBREAK_MISSING}", first
elif frozenset((1, 2)) in pairs:
how, who, confirm = pairs[frozenset((1, 2))], "Stichwahl: Gemma bestätigt pix2text", g
elif frozenset((0, 2)) in pairs:
return ("needs_review", "Qwen und Gemma einig gegen pix2text – Mensch entscheidet: "
f"Qwen: {q.get('tex') or '–'}", first)
else:
texts = " | ".join(f"{n}: {r.get('tex') or '–'}" for n, r in (("Qwen", q), ("pix2text", p), ("Gemma", g)))
return "needs_review", f"alle uneinig: {texts}", first
use = 1
if not p.get("tex"):
return "needs_review", f"{who}, aber ohne LaTeX", None
# the confidence check (token probabilities, off by default) covers the confirming VLM reading
problems = _checks([confirm], veto, pandoc_ok, book_px_per_pt, th)
if problems:
return "needs_review", f"einig ({how}){tie}; " + "; ".join(problems), use
return "auto_accepted", f"zwei Modelle einig ({how}){tie}, sicher, Bild-Veto bestanden", use
return "needs_review", f"{who} ({how}); " + "; ".join(problems), use
return "auto_accepted", f"{who} ({how}), Bild-Veto bestanden", use

2
formula_img2latex/engines/__init__.py
View file

@ -47,5 +47,5 @@ def make_engine(name: str, model: str | None = None, url: str | None = None) ->
)
if name == "p2t":
return Pix2TextEngine(model=model or os.environ.get("FORMULA_P2T_MODEL", DEFAULT_P2T_MODEL),
device=os.environ.get("FORMULA_P2T_DEVICE", "cpu"))
device=os.environ.get("FORMULA_P2T_DEVICE", "auto"))
raise ValueError(f"unbekannte Engine {name!r} (local, local2, cloud, p2t)")

66
formula_img2latex/engines/p2t.py
View file

@ -1,12 +1,14 @@
"""Pix2Text formula OCR (MFR model) as a local candidate generator.
Not a vision-language model but a dedicated image-to-LaTeX transformer: another kind of model than
the VLM engines, so its errors are likely to be different ones. Runs on the CPU (ONNX), no server.
the VLM engines, so its errors are likely to be different ones. Runs in-process (ONNX), no server:
on the CPU, or on a GPU with onnxruntime-gpu (device "cuda:N"; N counts within CUDA_VISIBLE_DEVICES).
"""
from __future__ import annotations
import io
import re
import threading
import time
@ -17,9 +19,48 @@ from .vlm import Recognition, prepare_image
DEFAULT_MODEL = "mfr-1.5"
# Pix2Text's bold: \mathrm{\boldmath~ x ~} (an error in math mode), \mathrm{{\bf~ r o t}}, \mathrm{\bf~ r o t}
_BOLDMATH = re.compile(r"\\mathrm\s*\{\s*\\boldmath\s*~?\s*([^{}~\\]*?)\s*~?\s*\}")
_BF = re.compile(r"\\mathrm\s*\{\s*(\{)?\s*\\bf\s*~?\s*([^{}~\\]*?)\s*~?\s*(?(1)\})\s*\}")
def tidy_p2t(tex: str) -> str:
"""Pix2Text output without superfluous spaces: `\\frac{\\alpha} {3 6 0}` -> `\\frac{\\alpha}{360}`.
The spaces are removed by post_process of pix2tex (LaTeX-OCR), which keeps a space between two
letters (`a x`, `\\cdot x`) and the control space `\\ `; TeX ignores the removed spaces in math
mode, so the typeset formula does not change. Before that, Pix2Text's bold spellings become
\\boldsymbol / \\mathbf.
"""
from pix2tex.utils.utils import post_process # imports torch: only when needed
tex = _BOLDMATH.sub(lambda m: "\\boldsymbol{" + re.sub(r"\s+", "", m.group(1)) + "}", tex)
tex = _BF.sub(lambda m: "\\mathbf{" + re.sub(r"\s+", "", m.group(2)) + "}", tex)
return post_process(tex).strip()
def resolve_device(device: str) -> str:
"""Device "auto": the first visible GPU if onnxruntime has CUDA, else the CPU."""
if device != "auto":
return device
import onnxruntime
return "cuda" if "CUDAExecutionProvider" in onnxruntime.get_available_providers() else "cpu"
def ort_provider(device: str) -> tuple[str, dict]:
"""ONNX Runtime provider and its options for "cpu", "cuda" or "cuda:N".
Set explicitly: pix2text otherwise takes the first available provider (TensorRT with onnxruntime-gpu).
"""
if device == "cpu":
return "CPUExecutionProvider", {}
if device == "cuda" or device.startswith("cuda:"):
return "CUDAExecutionProvider", {"device_id": int(device.partition(":")[2] or 0)}
raise ValueError(f"unbekanntes Gerät {device!r} (cpu, cuda, cuda:N)")
class Pix2TextEngine:
def __init__(self, model: str = DEFAULT_MODEL, device: str = "cpu") -> None:
def __init__(self, model: str = DEFAULT_MODEL, device: str = "auto") -> None:
self.name = "p2t"
self.model_name = model
self.model = f"pix2text-{model}"
@ -38,7 +79,21 @@ class Pix2TextEngine:
raise FormulaError(f"pix2text ist nicht installiert ({e}): pip install 'pix2text[multilingual]'",
EXIT_TOOL) from e
try:
self._ocr = LatexOCR(model_name=self.model_name, model_backend="onnx", device=self.device)
import pix2tex.utils.utils # noqa: F401 (post_process for tidy_p2t)
except ImportError as e:
raise FormulaError(f"pix2tex ist nicht installiert ({e}): pip install --no-deps pix2tex munch",
EXIT_TOOL) from e
self.device = resolve_device(self.device)
provider, options = ort_provider(self.device)
if provider == "CUDAExecutionProvider":
import onnxruntime
if provider not in onnxruntime.get_available_providers():
raise FormulaError(f"Gerät {self.device}: onnxruntime ohne CUDA – pip install onnxruntime-gpu",
EXIT_TOOL)
onnxruntime.preload_dlls() # cuDNN/cuBLAS from the nvidia-* pip packages (installed with torch)
try:
self._ocr = LatexOCR(model_name=self.model_name, model_backend="onnx", device=self.device,
more_model_configs={"provider": provider, "provider_options": options})
except Exception as e: # download or model files broken: any exception type is possible here
raise FormulaError(f"Pix2Text-Modell {self.model_name} nicht ladbar ({e})", EXIT_TOOL) from e
@ -51,9 +106,10 @@ class Pix2TextEngine:
with self._lock:
# _one_batch instead of recognize(): the latter draws a tqdm bar for every image
out = self._ocr._one_batch([img], None)[0]
tex = self._ocr.post_process(out["text"]).strip()
raw = self._ocr.post_process(out["text"]).strip()
except Exception as e: # model not loadable or one unreadable image: must not stop the run
return Recognition(self.name, self.model, None, None, "", str(e)[:300], time.time() - t0)
score = float(out.get("score") or 0.0) or None
return Recognition(self.name, self.model, tex or None, None, tex[:2000], None, time.time() - t0,
tex = tidy_p2t(raw)
return Recognition(self.name, self.model, tex or None, None, raw[:2000], None, time.time() - t0,
p_mean=score)

103
formula_img2latex/pipeline.py
View file

@ -20,8 +20,10 @@ from PIL import Image
from .checks import pandoc_mathml_ok
from .classify import classify
from .compare import diff_image
from .decide import consensus_thresholds, decide, decide_consensus, thresholds
from .decide import (TIEBREAK_MISSING, consensus_thresholds, decide, decide_consensus, decide_pair_tiebreak,
thresholds)
from .engines import make_engine
from .errors import FormulaError
from .engines.vlm import PROMPT_VERSION
from .imageio import load_ink
from .render import PRESETS
@ -113,10 +115,15 @@ def needs_third(f: Formula) -> bool:
return f.status == "needs_review" and "dritte Lesart fehlt" in (f.reason or "")
def needs_tiebreak(f: Formula) -> bool:
"""Rule "p2t": pix2text and Qwen disagree; Gemma's tie-break reading is missing."""
return f.status == "needs_review" and TIEBREAK_MISSING in (f.reason or "")
def recognize(source: Source, ws: Workspace, engine_name: str, model: str | None = None, url: str | None = None,
only_unresolved: bool = False, include_figures: bool = False, limit: int = 0,
workers: int = 4, quiet: bool = False, only_ids: set[str] | None = None,
only_needing_third: bool = False) -> dict[str, int]:
only_needing_third: bool = False, only_needing_tiebreak: bool = False) -> dict[str, int]:
engine = make_engine(engine_name, model, url)
engine.check()
todo = []
@ -127,6 +134,8 @@ def recognize(source: Source, ws: Workspace, engine_name: str, model: str | None
continue
if only_needing_third and not needs_third(f):
continue
if only_needing_tiebreak and not needs_tiebreak(f):
continue
if any(c.get("engine") == engine.name and c.get("model") == engine.model
and c.get("prompt") == PROMPT_VERSION and c.get("thinking") == engine.thinking for c in f.candidates):
continue # cached: never ask the same model the same question twice
@ -246,6 +255,8 @@ def verify_all(source: Source, ws: Workspace, preset: str | None = None, include
formulas = [f for f in ws.all() if _wanted(f, include_figures) and _distinct_candidates(f)
and (only_ids is None or f.id in only_ids)]
mode = decide_mode(ws)
if mode == "consensus" and consensus_rule(ws) == "vlm3":
mode = "consensus-vlm3"
formulas = [f for f in formulas if _verify_texes(f, mode)]
if not redo:
formulas = [f for f in formulas if not _verified_current(f, mode)]
@ -288,10 +299,13 @@ VERIFY_VERSION = 3 # bump when the comparison changes, so stored image checks
def _verify_texes(f: Formula, mode: str) -> list[str]:
"""Readings to check against the image. Consensus: only the reading that would be used (the
first primary reading) – the veto needs nothing else, and a later third reading changes nothing."""
if mode != "consensus":
"""Readings to check against the image: only the one that may be used. Mode "consensus" (rule
p2t): pix2text's. "consensus-vlm3": the first primary reading (Qwen's). "pixel": all candidates."""
if mode == "pixel":
return _distinct_candidates(f)
if mode == "consensus":
p = latest_by_engine(f).get("p2t")
return [split_tag(strip_delimiters(p["tex"]))[0]] if p and p.get("tex") else []
for r in readings_of(f):
if r.get("tex"):
return [split_tag(strip_delimiters(r["tex"]))[0]]
@ -312,6 +326,20 @@ def decide_mode(ws: Workspace) -> str:
return ws.settings.get("decide") or "consensus"
def consensus_rule(ws: Workspace) -> str:
"""'p2t' (pix2text = Qwen, Gemma as tie-break; default) or 'vlm3' (Qwen = Gemma = GLM)."""
return ws.settings.get("consensus_rule") or "p2t"
def latest_by_engine(f: Formula) -> dict[str, dict]:
"""Latest error-free reading per engine."""
latest: dict[str, dict] = {}
for c in f.candidates:
if not c.get("error"):
latest[c.get("engine", "")] = c
return latest
def readings_of(f: Formula) -> list[dict]:
"""Latest reading per model; local engines first (primary pair), then the rest (tie-break)."""
latest: dict[str, dict] = {}
@ -369,21 +397,32 @@ def _tex_key(tex: str) -> str:
def decide_all_consensus(ws: Workspace, quiet: bool, preset: str, only_ids: set[str] | None = None) -> dict[str, int]:
from .consensus import same_reading
th = consensus_thresholds(ws.settings.get("consensus_thresholds"))
rule = consensus_rule(ws)
decide_fn = decide_pair_tiebreak if rule == "p2t" else decide_consensus
formulas = [f for f in ws.all() if f.candidates and f.status not in HUMAN_STATUSES
and (only_ids is None or f.id in only_ids)]
plan = []
for f in formulas:
rs = readings_of(f)
bold = bool(th.get("bold_relevant"))
if rule == "p2t":
latest = latest_by_engine(f)
rs = [latest.get("local"), latest.get("p2t"), latest.get("local2")]
pairs = [(0, 1), (1, 2), (0, 2)]
else:
rs = readings_of(f)
pairs = [(i, j) for i in range(len(rs)) for j in range(i + 1, len(rs))]
agree = []
for i in range(len(rs)):
for j in range(i + 1, len(rs)):
a, b = rs[i].get("tex") or "", rs[j].get("tex") or ""
if j >= 2: # the confirming reading: a missing final period/comma is no contradiction
a, b = _strip_final_punct(a), _strip_final_punct(b)
how = same_reading(a, b, f.display, preset, bold)
if how:
agree.append((i, j, how))
for i, j in pairs:
if rs[i] is None or rs[j] is None:
continue
a, b = rs[i].get("tex") or "", rs[j].get("tex") or ""
# rule vlm3: a confirming reading (j >= 2) may lack the final period or comma. Rule p2t: strict,
# pix2text's reading is the one used, so its final punctuation must be confirmed too
if rule != "p2t" and j >= 2:
a, b = _strip_final_punct(a), _strip_final_punct(b)
how = same_reading(a, b, f.display, preset, bold)
if how:
agree.append((i, j, how))
plan.append((f, rs, agree))
# Book-wide scale from agreed readings that resemble their original well.
by_tex = {f.id: {_tex_key(r["tex"]): r for r in (f.verification or {}).get("all", [])} for f, _, _ in plan}
@ -397,20 +436,20 @@ def decide_all_consensus(ws: Workspace, quiet: bool, preset: str, only_ids: set[
ws.settings["book_px_per_pt"] = book_ppt
chosen = []
for f, rs, agree in plan:
status, reason, use = decide_consensus(rs, agree, None, True, book_ppt, th) # preliminary: which reading
status, reason, use = decide_fn(rs, agree, None, True, book_ppt, th) # preliminary: which reading
chosen.append(rs[use]["tex"] if use is not None and rs[use].get("tex") else None)
pandoc = pandoc_mathml_ok([(t or "x", f.display) for (f, _, _), t in zip(plan, chosen)]) if plan else []
counts: dict[str, int] = {}
for (f, rs, agree), tex, pok in zip(plan, chosen, pandoc):
veto = by_tex[f.id].get(_tex_key(tex)) if tex else None
status, reason, use = decide_consensus(rs, agree, veto, pok, book_ppt, th)
status, reason, use = decide_fn(rs, agree, veto, pok, book_ppt, th)
f.status, f.reason = status, reason
f.final_tex = _with_tag(f, tex) if status == "auto_accepted" and tex else None
f.decided_at = now_iso()
ws.save(f)
counts[status] = counts.get(status, 0) + 1
ws.save_meta()
log("Entscheidung (Konsens): " + ", ".join(f"{k} {v}" for k, v in sorted(counts.items())), quiet)
log(f"Entscheidung (Konsens, Regel {rule}): " + ", ".join(f"{k} {v}" for k, v in sorted(counts.items())), quiet)
return counts
@ -526,25 +565,39 @@ def run_consensus(source: Source, ws: Workspace, chunk: int = 50, preset: str |
include_figures: bool = False, limit: int = 0, cloud_model: str | None = None,
use_cloud: bool = True, ids: list[str] | None = None, quiet: bool = False,
report: Callable[[str], None] | None = None) -> dict[str, int]:
"""Process formulas in chunks: local, local2, verify; cloud only where both agree; verify again.
"""Process formulas in chunks and report a status line (processed / accepted / review / image) after each.
After every chunk a status line (processed / accepted / review / image) is reported.
Rule "p2t": pix2text, then Qwen, verify; Gemma only where they disagree; decide again.
Rule "vlm3": Qwen and Gemma, verify; cloud only where both agree; verify again.
"""
say = report or (lambda m: print(m, flush=True))
todo = ids or [f.id for f in ws.all() if _wanted(f, include_figures)]
if limit:
todo = todo[:limit]
rule = consensus_rule(ws)
tiebreak_ok = True
totals: dict[str, int] = {}
done: list[str] = []
for start in range(0, len(todo), chunk):
part = set(todo[start:start + chunk])
# both local models run on their own GPU: ask them at the same time
with ThreadPoolExecutor(max_workers=2) as both:
for fut in [both.submit(recognize, source, ws, eng, only_ids=part, include_figures=include_figures,
quiet=True) for eng in ("local", "local2")]:
fut.result()
if rule == "p2t": # pix2text first (fast), then Qwen
for eng in ("p2t", "local"):
recognize(source, ws, eng, only_ids=part, include_figures=include_figures, quiet=True)
else: # both VLMs run on their own GPU: ask them at the same time
with ThreadPoolExecutor(max_workers=2) as both:
for fut in [both.submit(recognize, source, ws, eng, only_ids=part, include_figures=include_figures,
quiet=True) for eng in ("local", "local2")]:
fut.result()
verify_all(source, ws, preset, include_figures, quiet=True, only_ids=part)
if use_cloud:
if rule == "p2t" and tiebreak_ok and any(needs_tiebreak(f) for f in map(ws.load, part) if f):
try:
recognize(source, ws, "local2", only_ids=part, only_needing_tiebreak=True,
include_figures=include_figures, quiet=True)
verify_all(source, ws, preset, include_figures, quiet=True, only_ids=part)
except FormulaError as e: # Gemma not running: the disputed formulas stay in review
tiebreak_ok = False
say(f"Stichwahl (Gemma) nicht möglich, strittige Formeln bleiben im Review: {e}")
if rule != "p2t" and use_cloud:
recognize(source, ws, "cloud", cloud_model, only_ids=part, only_needing_third=True,
include_figures=include_figures, workers=8, quiet=True)
verify_all(source, ws, preset, include_figures, quiet=True, only_ids=part)

105
tests/test_pipeline.py
View file

@ -4,6 +4,8 @@ from __future__ import annotations
from pathlib import Path
import pytest
from conftest import needs_latex, needs_pandoc
from formula_img2latex import pipeline
from formula_img2latex.cli import main
@ -120,6 +122,7 @@ def _reading(engine: str, model: str, tex: str | None, p_mean: float = 0.99, p_m
@needs_pandoc
def test_consensus_mode(epub2md_dir: Path) -> None:
source, ws = pipeline.open_all(epub2md_dir, None, create=True)
ws.settings["consensus_rule"] = "vlm3" # the earlier rule Qwen = Gemma = GLM
pipeline.scan(source, ws, quiet=True)
by = {f.asset.rsplit("/", 1)[1]: f for f in ws.all()}
inline, display = by["0001-inline.jpg"], by["0002-display.jpg"]
@ -155,6 +158,63 @@ def test_consensus_mode(epub2md_dir: Path) -> None:
assert i.status == "needs_review" and "weicht ab" in i.reason, i.reason
@needs_latex
@needs_pandoc
def test_consensus_rule_p2t(epub2md_dir: Path) -> None:
"""Standard rule: pix2text first, Qwen confirms; where they differ Gemma may confirm pix2text.
The reading used is always pix2text's."""
source, ws = pipeline.open_all(epub2md_dir, None, create=True)
pipeline.scan(source, ws, quiet=True)
by = {f.asset.rsplit("/", 1)[1]: f for f in ws.all()}
inline, display = by["0001-inline.jpg"], by["0002-display.jpg"]
inline.candidates = [_reading("local", "Q", r"L=\{-\sqrt{2};\sqrt{2}\}"),
_reading("p2t", "P", r"L=\{-\sqrt{2};\sqrt{2}\}")]
display.candidates = [_reading("local", "Q", r"x_{3,k}=\arcsin\left(-\frac{3}{5}\right)+2k\pi"),
_reading("p2t", "P", r"x_{2,k}=\operatorname{arcsin}(-\frac{3}{5})+2k\pi")]
ws.save(inline)
ws.save(display)
pipeline.verify_all(source, ws, preset="cm", workers=2, quiet=True)
i, d = ws.load(inline.id), ws.load(display.id)
assert i.status == "auto_accepted" and "pix2text und Qwen einig" in i.reason, i.reason
assert i.final_tex == r"L=\{-\sqrt{2};\sqrt{2}\}" # pix2text's reading
assert pipeline.needs_tiebreak(d), d.reason
# Gemma confirms pix2text -> accepted with pix2text's reading
d.candidates.append(_reading("local2", "G", r"x_{2,k} = \arcsin(-\frac{3}{5}) + 2k\pi"))
ws.save(d)
pipeline.verify_all(source, ws, preset="cm", workers=2, quiet=True)
d = ws.load(d.id)
assert d.status == "auto_accepted" and "Gemma bestätigt pix2text" in d.reason, d.reason
assert d.final_tex == r"x_{2,k}=\operatorname{arcsin}(-\frac{3}{5})+2k\pi"
# Qwen and Gemma together against pix2text -> a human decides
d.candidates[-1] = _reading("local2", "G", r"x_{3,k}=\arcsin(-\frac{3}{5})+2k\pi")
d.status = "pending"
ws.save(d)
pipeline.verify_all(source, ws, preset="cm", workers=2, quiet=True)
d = ws.load(d.id)
assert d.status == "needs_review" and "Mensch entscheidet" in d.reason, d.reason
def test_pair_tiebreak_rule() -> None:
from formula_img2latex.decide import consensus_thresholds, decide_pair_tiebreak
th = consensus_thresholds()
q, p, g = _reading("local", "Q", "x^2"), _reading("p2t", "P", "x^{2}"), _reading("local2", "G", "x^2")
veto = {"ok": True, "metrics": {"ncc": 0.9, "px_per_pt": 1.4}}
assert decide_pair_tiebreak([q, p, None], [(0, 1, "normalform")], veto, True, 1.4, th)[::2] == ("auto_accepted", 1)
bad = {"ok": True, "metrics": {"ncc": 0.5, "px_per_pt": 1.4}}
assert decide_pair_tiebreak([q, p, None], [(0, 1, "normalform")], bad, True, 1.4, th)[0] == "needs_review" # veto
assert decide_pair_tiebreak([q, p, None], [(0, 1, "normalform")], veto, False, 1.4, th)[0] == "needs_review" # pandoc
assert "Stichwahl fehlt" in decide_pair_tiebreak([q, p, None], [], veto, True, 1.4, th)[1]
assert decide_pair_tiebreak([q, p, g], [(1, 2, "normalform")], veto, True, 1.4, th)[::2] == ("auto_accepted", 1)
assert decide_pair_tiebreak([q, p, g], [(0, 2, "normalform")], veto, True, 1.4, th)[0] == "needs_review"
assert decide_pair_tiebreak([q, p, g], [], veto, True, 1.4, th)[0] == "needs_review"
assert "fehlt" in decide_pair_tiebreak([q, None, None], [], veto, True, 1.4, th)[1]
nq, ng = _reading("local", "Q", None, is_formula=False), _reading("local2", "G", None, is_formula=False)
assert decide_pair_tiebreak([nq, p, ng], [], None, True, 1.4, th)[0] == "not_formula"
assert decide_pair_tiebreak([nq, p, None], [], None, True, 1.4, th)[0] == "needs_review" # ask Gemma first
def test_consensus_rules() -> None:
from formula_img2latex.decide import consensus_thresholds, decide_consensus
th = consensus_thresholds({"min_p_mean": 0.9, "min_p_min": 0.3, "require_third": 0})
@ -181,6 +241,29 @@ def test_consensus_rules() -> None:
assert decide_consensus([a, b, c], [(0, 2, "normalform")], veto, True, 1.4, strict)[0] == "needs_review" # no 2:1
def test_calibration_rules_with_four_engines() -> None:
from formula_img2latex.calibrate import _summarize_consensus
from formula_img2latex.decide import consensus_thresholds, decide_consensus
names = ["local", "local2", "cloud", "p2t"]
veto = {"ok": True, "metrics": {"ncc": 0.9}}
def res(agree: list[tuple[int, int]], ok: list[bool]) -> dict:
rs = [_reading(e, e, "x") for e in names]
pair = "normalform" if (0, 1) in agree else None
return {"truth": "x", "readings": rs, "ok": ok, "agree_all": [(i, j, "normalform") for i, j in agree],
"vetos": {0: veto, 1: veto, 3: veto}, "a": rs[0], "b": rs[1], "a_ok": ok[0], "b_ok": ok[1],
"agree": pair, "veto": veto if pair else None}
all4 = [(0, 1), (0, 2), (0, 3), (1, 2), (1, 3), (2, 3)]
results = [res(all4, [True] * 4), # everyone agrees
res([(0, 1), (0, 3), (1, 3)], [False, False, True, False]), # Q = G = p2t, all wrong, GLM right
res([(0, 2), (0, 3), (2, 3)], [True, False, True, True])] # Gemma differs: 2:1 decided by p2t
s = _summarize_consensus("t", results, names, consensus_thresholds, decide_consensus)
got = [(r["accepted"], r["wrong"]) for r in s["rules"]]
# Qwen = Gemma | + GLM | B (+ p2t) | GLM or p2t confirms | A (3 of 4) | C (all four) | standard rule p2t
assert got == [(2, 1), (1, 0), (2, 1), (2, 1), (3, 1), (1, 0), (3, 1)]
def test_canonical_form() -> None:
from formula_img2latex.consensus import canonical
same = [(r"x^{2}+y^{2}", "x^2 + y^2"), (r"\vec{b}", r"\overrightarrow{b}"), (r"\left(a\right)", "(a)"),
@ -190,6 +273,8 @@ def test_canonical_form() -> None:
(r"\operatorname{l i m}_{x \to0} \frac{\operatorname{s i n} x} {x}", r"\lim_{x\to 0}\frac{\sin x}{x}"),
(r"\mathrm{f \ddot{u} r ~ a l l e ~} x", r"\text{für alle } x"), (r"f^{\prime\prime}(x)", "f''(x)"),
(r"\left( \begin{matrix} {1} & {2} \\ \end{matrix} \right)", r"\begin{pmatrix}1&2\end{pmatrix}"),
(r"\begin{array}[t]{cc}a&b\end{array}", r"\begin{array}{cc}a&b\end{array}"),
(r"\mathop{\mathbf{rot}}\boldsymbol{v}", r"\mathbf{rot}\,\boldsymbol{v}"),
(r"{\int}_{0}^{\infty} \! e^{-x^{2}} \, d x \!=\! \frac{\sqrt\pi} 2", r"\int_0^\infty e^{-x^2}\,dx=\frac{\sqrt{\pi}}{2}")]
differ = [(r"x_{2,k}", r"x_{2;k}"), ("x^{2}", "x_{2}"), (r"\mathbb{R}", "R"), ("a_{n+1}", "a_n+1"),
(r"\frac{a}{b+c}", r"\frac{a}{b}+c"), (r"x \in t", r"x \int"), (r"\sqrt[3]{x}", r"\sqrt{x}"),
@ -198,10 +283,30 @@ def test_canonical_form() -> None:
assert all(canonical(a) != canonical(b) for a, b in differ)
def test_tidy_p2t() -> None:
pytest.importorskip("pix2tex")
from formula_img2latex.engines.p2t import tidy_p2t
cases = {
r"{\frac{\alpha} {3 6 0}}={\frac{x} {2 \pi}}": r"{\frac{\alpha}{360}}={\frac{x}{2\pi}}",
r"\frac{\mathrm{d}} {\mathrm{d} x} | x |=\frac{1} {x} .": r"\frac{\mathrm{d}}{\mathrm{d}x}|x|=\frac{1}{x}.",
r"\left( \begin{array} {c} {-y} \\ {0} \end{array} \right)": r"\left(\begin{array}{c}{-y}\\ {0}\end{array}\right)",
r"a \cdot x \in t": r"a\cdot x\in t", # space between letters stays
r"A \cdot\mathrm{\boldmath~ v ~}=0": r"A\cdot\boldsymbol{v}=0", # Pix2Text's bold
r"{\mathrm{{\bf~ r o t}}} \, v": r"{\mathbf{rot}}\,v",
r"\mathrm{\bf~ r o t} \, E": r"\mathbf{rot}\,E",
}
for raw, tidy in cases.items():
assert tidy_p2t(raw) == tidy, (raw, tidy_p2t(raw))
def test_p2t_engine_is_lazy() -> None:
from formula_img2latex.engines import make_engine
e = make_engine("p2t")
assert (e.name, e.model, e._ocr) == ("p2t", "pix2text-mfr-1.5", None) # model loads only in check()
from formula_img2latex.engines.p2t import ort_provider
assert ort_provider("cpu") == ("CPUExecutionProvider", {})
assert ort_provider("cuda") == ("CUDAExecutionProvider", {"device_id": 0})
assert ort_provider("cuda:2") == ("CUDAExecutionProvider", {"device_id": 2})
def test_logprob_confidence() -> None:

218
tools/regeltest.py Normal file
View file

@ -0,0 +1,218 @@
#!/usr/bin/env python3
"""Run the standard consensus rule on formulas whose TeX is known and show every case for review.
python3 tools/regeltest.py EPUB [--n 50] [--name regeltest] [--reuse vergleich/regeltest/ergebnis.json]
[--exclude vergleich/regeltest/ergebnis.json ...]
Like `run`: pix2text reads every formula first, then Qwen (local); Gemma (local2) only where they
disagree; then the rough image check on pix2text's reading and decide_pair_tiebreak. Writes
vergleich/<name>/index.html (not versioned): summary, then per formula the original, the true
formula and every reading set with pdflatex, the decision and whether the result is right.
--reuse takes the readings (and their times) of an earlier run instead of asking the models again;
--exclude leaves out the formulas of earlier runs, so a new sample has none of them.
"""
from __future__ import annotations
import argparse
import html
import json
import random
import statistics
import sys
import time
from pathlib import Path
import numpy as np
from PIL import Image
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from konsens_bericht import orig_img, png_uri # noqa: E402
from formula_img2latex.calibrate import _best_preset, corpus # noqa: E402
from formula_img2latex.checks import pandoc_mathml_ok # noqa: E402
from formula_img2latex.consensus import same_reading # noqa: E402
from formula_img2latex.decide import consensus_thresholds, decide_pair_tiebreak # noqa: E402
from formula_img2latex.engines import make_engine # noqa: E402
from formula_img2latex.engines.p2t import tidy_p2t # noqa: E402
from formula_img2latex.imageio import load_ink # noqa: E402
from formula_img2latex.render import render_many # noqa: E402
from formula_img2latex.tex import strip_delimiters # noqa: E402
from formula_img2latex.verify import verify # noqa: E402
TOOLS = [("local", "Qwen"), ("p2t", "pix2text"), ("local2", "Gemma")]
def agreements(texes: list[str | None], display: bool, preset: str) -> list[tuple[int, int, str]]:
"""Pairs (Qwen, pix2text), (pix2text, Gemma), (Qwen, Gemma) as in pipeline.decide_all_consensus (strict)."""
out = []
for i, j in ((0, 1), (1, 2), (0, 2)):
a, b = texes[i], texes[j]
if a is None or b is None:
continue
how = same_reading(a, b, display, preset)
if how:
out.append((i, j, how))
return out
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("epub", type=Path)
ap.add_argument("--n", type=int, default=50)
ap.add_argument("--name", default="regeltest")
ap.add_argument("--reuse", type=Path, default=None, help="Lesarten aus einem früheren ergebnis.json")
ap.add_argument("--exclude", type=Path, action="append", default=[],
help="Formeln aus diesem früheren ergebnis.json auslassen (mehrfach möglich)")
args = ap.parse_args()
cached = {}
if args.reuse:
cached = {c["name"]: c["readings"] for c in json.loads(args.reuse.read_text(encoding="utf-8"))["cases"]}
for rs in cached.values(): # earlier runs stored pix2text's raw spaced output
if rs[1] and rs[1].get("tex"):
rs[1]["tex"] = tidy_p2t(rs[1].get("raw") or rs[1]["tex"])
excluded = {c["name"] for x in args.exclude for c in json.loads(x.read_text(encoding="utf-8"))["cases"]}
items = [it for it in corpus(args.epub) if it[0] not in excluded]
smp = random.Random(42).sample(items, min(args.n + 1, len(items)))
formulas, warm = smp[:args.n], smp[args.n]
preset = _best_preset(smp[:12])
th = consensus_thresholds()
eng = {}
for key, label in ([] if cached else TOOLS):
e = make_engine(key)
t0 = time.time()
e.check()
e.recognize(warm[1], warm[0], "", warm[3]) # warm-up, not counted
eng[key] = e
print(f"{label}: {e.model}, bereit in {time.time() - t0:.1f} s", file=sys.stderr, flush=True)
cases = []
for k, (name, data, truth, disp) in enumerate(formulas, 1):
old = cached.get(name)
if cached and old is None:
raise SystemExit(f"{name} fehlt in {args.reuse} (andere Stichprobe?)")
reads: list[dict | None] = [None, None, None]
for pos in (1, 0): # pix2text first, then Qwen
reads[pos] = old[pos] if old else eng[TOOLS[pos][0]].recognize(data, name, "", disp).to_json()
texes = [r and r.get("tex") for r in reads]
agree = agreements(texes, disp, preset)
if not any((i, j) == (0, 1) for i, j, _ in agree): # disputed: ask Gemma
reads[2] = old[2] if old else eng["local2"].recognize(data, name, "", disp).to_json()
texes = [r and r.get("tex") for r in reads]
agree = agreements(texes, disp, preset)
_st, _why, use = decide_pair_tiebreak(reads, agree, None, True, None, th)
veto = None
pandoc = True
if use is not None and texes[use]:
v = verify(load_ink(data, name), texes[use], disp, preset, check_variants=False)
veto = {"ok": v.ok, "reason": v.reason, "metrics": v.metrics}
pandoc = pandoc_mathml_ok([(texes[use], disp)])[0]
status, reason, use = decide_pair_tiebreak(reads, agree, veto, pandoc, None, th)
ok = [bool(t) and bool(same_reading(t, truth, disp, preset)) for t in texes]
accepted = status == "auto_accepted"
cases.append({"name": name, "truth": truth, "display": disp, "readings": reads, "ok": ok, "agree": agree,
"status": status, "reason": reason, "use": use, "veto": veto,
"right": bool(accepted and ok[use])})
mark = ("✓" if ok[use] else "✗ FALSCH") if accepted else "Review"
print(f"{k:3d} {status:13s} {mark:8s} {reason[:110]}", file=sys.stderr, flush=True)
n = len(cases)
acc = [c for c in cases if c["status"] == "auto_accepted"]
wrong = [c for c in acc if not c["right"]]
review = [c for c in cases if c["status"] == "needs_review"]
gemma = sum(c["readings"][2] is not None for c in cases)
secs = {key: [c["readings"][p]["seconds"] for c in cases if c["readings"][p]] for p, (key, _l) in enumerate(TOOLS)}
per_formula = [sum(r["seconds"] for r in c["readings"] if r) for c in cases]
summary = {"formulas": n, "accepted": len(acc), "wrong": len(wrong), "review": len(review),
"other": n - len(acc) - len(review), "gemma_asked": gemma,
"mean_s_per_formula": statistics.mean(per_formula),
"mean_s": {k: statistics.mean(v) if v else None for k, v in secs.items()},
"correct_alone": {key: sum(c["ok"][p] for c in cases if c["readings"][p])
for p, (key, _l) in enumerate(TOOLS)}}
out = Path(__file__).resolve().parents[1] / "vergleich" / args.name
out.mkdir(parents=True, exist_ok=True)
(out / "ergebnis.json").write_text(json.dumps({"epub": args.epub.name, "preset": preset, "summary": summary,
"cases": cases}, ensure_ascii=False, indent=1, default=str),
encoding="utf-8")
def order(c: dict) -> int:
return 0 if c["status"] == "auto_accepted" and not c["right"] else 1 if c["status"] != "auto_accepted" else 2
shown = sorted(cases, key=order)
texes = []
for c in shown:
texes.append((strip_delimiters(c["truth"]), c["display"]))
texes += [((r or {}).get("tex") or "\\text{–}", c["display"]) for r in c["readings"]]
rendered = render_many(texes, preset)
by_name = {f[0]: f[1] for f in formulas}
lab = dict(TOOLS)
head = (f"<h1>Regeltest: {html.escape(args.epub.name[:70])}</h1>"
"<p>Regel: pix2text liest zuerst, Qwen bestätigt → annehmen. Sonst fragt Gemma; Gemma = pix2text → annehmen; "
"Qwen = Gemma gegen pix2text → Mensch. Übernommen wird immer pix2texts Lesart (Leerzeichen bereinigt). "
"Annahme nur mit Pandoc und Bild-Veto. "
"✓/✗ = gleiche Zeichen an gleicher Stelle wie die wahre Formel (Abstände, kursiv/aufrecht, Fettdruck egal).</p>"
"<table class=sum>"
f"<tr><th>Formeln</th><td>{n}</td></tr>"
f"<tr><th>automatisch angenommen</th><td>{len(acc)} ({len(acc) / n:.0%})</td></tr>"
f"<tr><th>davon FALSCH</th><td><b>{len(wrong)}</b></td></tr>"
f"<tr><th>Review (Mensch)</th><td>{len(review)} ({len(review) / n:.0%})</td></tr>"
f"<tr><th>Gemma gefragt</th><td>{gemma} ({gemma / n:.0%})</td></tr>"
f"<tr><th>Ø Zeit pro Formel (alle Tools nacheinander)</th><td>{summary['mean_s_per_formula']:.2f} s</td></tr>"
+ "".join(f"<tr><th>Ø {lab[k]}</th><td>{v:.2f} s</td></tr>" for k, v in summary["mean_s"].items() if v)
+ "".join(f"<tr><th>{lab[k]} allein richtig</th><td>{v}/{len(secs[k])}</td></tr>"
for k, v in summary["correct_alone"].items())
+ "</table>")
parts = [head]
per = 4
for i, c in enumerate(shown):
imgs = []
for rr in rendered[per * i: per * (i + 1)]:
if rr.ok:
g = (rr.rasterize(2.0) * 255).astype(np.uint8)
imgs.append(f"<img src='{png_uri(Image.fromarray(255 - g))}'>")
else:
imgs.append("<i>nicht setzbar</i>")
if c["status"] == "auto_accepted":
cls, verdict = ("good", "angenommen, richtig") if c["right"] else ("bad", "angenommen, FALSCH")
elif c["status"] == "needs_review":
cls, verdict = "rev", "Review"
else:
cls, verdict = "rev", c["status"]
ncc = (c["veto"] or {}).get("metrics", {}).get("ncc")
rows = [f"<tr><th>Original</th><td><img src='{png_uri(orig_img(by_name[c['name']], 2))}'></td><td></td></tr>",
f"<tr><th>richtig</th><td>{imgs[0]}</td><td><code>{html.escape(c['truth'])}</code></td></tr>"]
for p, (key, label) in enumerate(TOOLS):
r = c["readings"][p]
if r is None:
rows.append(f"<tr><th>{label}</th><td><i>nicht gefragt</i></td><td></td></tr>")
continue
mark = "<b class=ok>✓</b>" if c["ok"][p] else "<b class=no>✗</b>"
used = " <b>← verwendet</b>" if c["use"] == p and c["status"] == "auto_accepted" else ""
rows.append(f"<tr><th>{mark} {label}{used}<br><small>{r['seconds']:.2f} s</small></th><td>{imgs[p + 1]}</td>"
f"<td><code>{html.escape(r.get('tex') or r.get('error') or '–')}</code>"
+ (f"<br><small>roh: <code>{html.escape(r.get('raw') or '')}</code></small>"
if key == "p2t" and r.get("raw") and r.get("raw") != r.get("tex") else "")
+ "</td></tr>")
parts.append(f"<section class={cls}><div class=v>{verdict} · {html.escape(c['reason'])}"
f"{f' · Bild-Ähnlichkeit {ncc:.2f}' if ncc is not None else ''}</div>"
f"<small>{html.escape(c['name'])}</small><table>{''.join(rows)}</table></section>")
(out / "index.html").write_text(
"<!doctype html><meta charset=utf-8><title>Regeltest</title><style>"
"body{font:14px system-ui,sans-serif;margin:16px;background:#f6f6f3}section{background:#fff;border:1px solid #ddd;"
"border-left:6px solid #999;border-radius:6px;padding:6px 10px;margin:10px 0}section.good{border-left-color:#1f7a3a}"
"section.bad{border-left-color:#a32020;background:#fff5f5}section.rev{border-left-color:#c88a00}"
".v{font-weight:600;font-size:13px}table.sum{border-collapse:collapse;background:#fff}"
"table.sum td,table.sum th{border:1px solid #ccc;padding:4px 10px}"
"th{text-align:left;font-weight:400;color:#555;width:190px;vertical-align:middle}td{padding:3px 8px;vertical-align:middle}"
".ok{color:#1f7a3a}.no{color:#a32020}code{font-size:12px}img{max-width:640px}</style>" + "".join(parts),
encoding="utf-8")
print(json.dumps(summary, ensure_ascii=False, indent=1))
print(out / "index.html")
return 0
if __name__ == "__main__":
sys.exit(main())

145
tools/zeittest.py Normal file
View file

@ -0,0 +1,145 @@
#!/usr/bin/env python3
"""Time and accuracy per recognition tool on formulas whose TeX is known (serial, one at a time).
python3 tools/zeittest.py EPUB [--n 15] [--tools local,local2,cloud,p2t-cpu,p2t-gpu] [--gpu cuda:0]
Same sample as `calibrate --consensus` (seed 42, real images). Every tool reads every formula alone,
one request after the other, after one uncounted warm-up on another formula. Writes
vergleich/zeittest/index.html (not versioned): table per tool, then per formula the original, the
true formula and every reading set with pdflatex, with time and verdict; plus ergebnis.json.
"""
from __future__ import annotations
import argparse
import html
import json
import random
import statistics
import sys
import time
from pathlib import Path
import numpy as np
from PIL import Image
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from konsens_bericht import orig_img, png_uri # noqa: E402
from formula_img2latex.calibrate import _best_preset, corpus # noqa: E402
from formula_img2latex.consensus import same_reading # noqa: E402
from formula_img2latex.engines import make_engine # noqa: E402
from formula_img2latex.engines.p2t import Pix2TextEngine # noqa: E402
from formula_img2latex.render import render_many # noqa: E402
from formula_img2latex.tex import strip_delimiters # noqa: E402
LABELS = {"local": "Qwen3-VL-30B (lokal, GPU)", "local2": "Gemma-4-26B (lokal, GPU)", "cloud": "GLM-4.6V (Cloud)",
"p2t-cpu": "pix2text MFR 1.5 (CPU)", "p2t-gpu": "pix2text MFR 1.5 (GPU)"}
def engine_for(tool: str, gpu: str):
if tool == "p2t-cpu":
return Pix2TextEngine(device="cpu")
if tool == "p2t-gpu":
return Pix2TextEngine(device=gpu)
return make_engine(tool)
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("epub", type=Path)
ap.add_argument("--n", type=int, default=15)
ap.add_argument("--tools", default="local,local2,cloud,p2t-cpu,p2t-gpu")
ap.add_argument("--gpu", default="cuda", help="Gerät für p2t-gpu (Index relativ zu CUDA_VISIBLE_DEVICES)")
ap.add_argument("--name", default="zeittest")
args = ap.parse_args()
tools = [t.strip() for t in args.tools.split(",") if t.strip()]
items = corpus(args.epub)
smp = random.Random(42).sample(items, min(150, len(items))) # as calibrate_consensus
formulas, warm = smp[:args.n], smp[args.n]
preset = _best_preset(smp[:12])
runs: dict[str, dict] = {}
for tool in tools:
eng = engine_for(tool, args.gpu)
t0 = time.time()
eng.check()
load = time.time() - t0
eng.recognize(warm[1], warm[0], "", warm[3]) # warm-up, not counted
reads = []
for name, data, truth, disp in formulas:
r = eng.recognize(data, name, "", disp)
ok = bool(r.tex) and bool(same_reading(r.tex, truth, disp, preset))
reads.append({"tex": r.tex, "seconds": r.seconds, "ok": ok, "error": r.error,
"tokens_in": r.tokens_in, "tokens_out": r.tokens_out})
print(f"{tool:8s} {r.seconds:6.2f}s {'✓' if ok else '✗'} {r.tex or r.error}", file=sys.stderr, flush=True)
secs = [x["seconds"] for x in reads]
runs[tool] = {"label": LABELS.get(tool, tool), "model": eng.model, "load_s": load, "reads": reads,
"mean_s": statistics.mean(secs), "median_s": statistics.median(secs),
"correct": sum(x["ok"] for x in reads), "errors": sum(bool(x["error"]) for x in reads),
"tokens_in": sum(x["tokens_in"] for x in reads), "tokens_out": sum(x["tokens_out"] for x in reads)}
out = Path(__file__).resolve().parents[1] / "vergleich" / args.name
out.mkdir(parents=True, exist_ok=True)
(out / "ergebnis.json").write_text(json.dumps(
{"epub": args.epub.name, "preset": preset, "truth": [f[2] for f in formulas], "runs": runs},
ensure_ascii=False, indent=1), encoding="utf-8")
n = len(formulas)
rows = "".join(
f"<tr><td>{html.escape(r['label'])}</td><td>{r['mean_s']:.2f}</td><td>{r['median_s']:.2f}</td>"
f"<td>{r['correct']}/{n} ({r['correct'] / n:.0%})</td><td>{r['load_s']:.1f}</td>"
f"<td>{r['tokens_in'] + r['tokens_out'] or '–'}</td></tr>" for r in runs.values())
parts = [f"<h1>Zeittest: {html.escape(args.epub.name[:70])}</h1>",
f"<p>{n} Formeln mit bekanntem TeX (alt-Text), jedes Tool seriell, eine Formel nach der anderen, "
"nach einem nicht gezählten Aufwärmlauf. Alle Formeln mit pdflatex gesetzt. ✓ = gleiche Zeichen an "
"gleicher Stelle wie die wahre Formel (Abstände, kursiv/aufrecht, Fettdruck egal).</p>",
"<table class=sum><tr><th>Tool</th><th>Ø s/Formel</th><th>Median s</th><th>richtig</th>"
f"<th>Laden s</th><th>Tokens</th></tr>{rows}</table>"]
texes = []
for i, (_n, _d, truth, disp) in enumerate(formulas):
texes.append((strip_delimiters(truth), disp))
texes += [(runs[t]["reads"][i]["tex"] or "\\text{–}", disp) for t in tools]
rendered = render_many(texes, preset)
per = 1 + len(tools)
for i, (name, data, truth, _disp) in enumerate(formulas):
imgs = []
for rr in rendered[per * i: per * (i + 1)]:
if rr.ok:
g = (rr.rasterize(2.0) * 255).astype(np.uint8)
imgs.append(f"<img src='{png_uri(Image.fromarray(255 - g))}'>")
else:
imgs.append("<i>nicht setzbar</i>")
body = [f"<tr><th>Original</th><td><img src='{png_uri(orig_img(data, 2))}'></td><td></td></tr>",
f"<tr><th>richtig</th><td>{imgs[0]}</td><td><code>{html.escape(truth)}</code></td></tr>"]
for k, t in enumerate(tools):
r = runs[t]["reads"][i]
mark = "<b class=ok>✓</b>" if r["ok"] else "<b class=no>✗</b>"
body.append(f"<tr><th>{mark} {html.escape(runs[t]['label'])}<br><small>{r['seconds']:.2f} s</small></th>"
f"<td>{imgs[k + 1]}</td><td><code>{html.escape(r['tex'] or r['error'] or '–')}</code></td></tr>")
parts.append(f"<section><div class=v>Formel {i + 1} · {html.escape(name)}</div><table>{''.join(body)}</table></section>")
(out / "index.html").write_text(
"<!doctype html><meta charset=utf-8><title>Zeittest</title><style>"
"body{font:14px system-ui,sans-serif;margin:16px;background:#f6f6f3}section{background:#fff;border:1px solid #ddd;"
"border-radius:6px;padding:6px 10px;margin:10px 0}.v{font-weight:600;font-size:13px;margin-bottom:4px}"
"table.sum{border-collapse:collapse;background:#fff}table.sum td,table.sum th{border:1px solid #ccc;padding:4px 10px}"
"th{text-align:left;font-weight:400;color:#555;width:210px;vertical-align:middle}td{padding:3px 8px;vertical-align:middle}"
".ok{color:#1f7a3a}.no{color:#a32020}code{font-size:12px}img{max-width:640px}</style>" + "".join(parts),
encoding="utf-8")
print(f"\n| Tool | Ø s/Formel | Median s | richtig (n={n}) | Laden s |")
print("|---|---|---|---|---|")
for r in runs.values():
print(f"| {r['label']} | {r['mean_s']:.2f} | {r['median_s']:.2f} | {r['correct']}/{n} | {r['load_s']:.1f} |")
if "p2t-cpu" in runs and "p2t-gpu" in runs:
same = sum(a["tex"] == b["tex"] for a, b in zip(runs["p2t-cpu"]["reads"], runs["p2t-gpu"]["reads"]))
print(f"\npix2text CPU und GPU gleicher Text: {same}/{n}")
print(out / "index.html")
return 0
if __name__ == "__main__":
sys.exit(main())