feat: mdgrep 0.1.0 – boolean full-text search for Markdown
- query language: substrings, phrases, /regex/, AND/OR/NOT (+ UND/ODER/NICHT), parentheses; case-insensitive; ReDoS protection via regex timeouts; syntax errors with stable code and position - SearchEngine over in-memory blocks: filters, predicate, grouping, sorting, paging, time budget, automatic reload, highlighted snippets - Markdown loader with split modes file/heading/list/paragraph - grep-like CLI (mdgrep), JSON output, grep exit codes - Flask blueprint/app with JSON API and widget assets, dev server - framework-free JS widget: live search, URL state, filters, paging, syntax help, en/de labels, CSS custom properties for theming Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dschlueter
commit
9c7b7e81f9
23 changed files with 2642 additions and 0 deletions
6
.gitignore
vendored
Normal file
6
.gitignore
vendored
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
.venv/
|
||||
__pycache__/
|
||||
*.egg-info/
|
||||
dist/
|
||||
build/
|
||||
.pytest_cache/
|
||||
21
LICENSE
Normal file
21
LICENSE
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
MIT License
|
||||
|
||||
Copyright (c) 2026 Dieter Schlüter
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
220
README.md
Normal file
220
README.md
Normal file
|
|
@ -0,0 +1,220 @@
|
|||
# mdgrep
|
||||
|
||||
Boolean full-text search over Markdown (and any other text) – as a **Python library**, a
|
||||
**grep-like command line tool**, a **Flask JSON API** and a **framework-free JavaScript search widget**.
|
||||
|
||||
- substrings, `"phrases"` and `/regular expressions/`, always case-insensitive (Unicode-aware)
|
||||
- `AND` / `OR` / `NOT` with parentheses – German keywords `UND` / `ODER` / `NICHT` work too
|
||||
- results as "boxes" (blocks) with highlighted context, grouping, sorting, filters and paging
|
||||
- protection against catastrophic regular expressions (time limit per search)
|
||||
- no search index: blocks live in memory, a few MB of text are searched in milliseconds
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install "mdgrep @ git+https://kitux.de/forgejo/dschlueter/mdgrep.git@v0.1.0" # library + CLI
|
||||
pip install "mdgrep[web] @ git+https://kitux.de/forgejo/dschlueter/mdgrep.git@v0.1.0" # + Flask API
|
||||
```
|
||||
|
||||
Requires Python ≥ 3.10. Dependencies: [`regex`](https://pypi.org/project/regex/) (and `flask` for `[web]`).
|
||||
|
||||
## Query syntax
|
||||
|
||||
| Query | Meaning |
|
||||
|---|---|
|
||||
| `rust` | substring, case-insensitive – also matches *Rust*, *trusted*, *Rustacean* |
|
||||
| `"open source"` | phrase; any whitespace (incl. line breaks) between the words |
|
||||
| `/llms?\b/` | regular expression (`\/` for a literal slash), case-insensitive |
|
||||
| `a b` · `a AND b` · `a UND b` · `a & b` | both |
|
||||
| `a OR b` · `a ODER b` · `a \| b` | at least one |
|
||||
| `NOT a` · `NICHT a` · `-a` · `!a` | must not occur |
|
||||
| `( … )` | grouping |
|
||||
|
||||
Precedence: `NOT` > `AND` > `OR`, so `a OR b AND c` means `a OR (b AND c)`.
|
||||
Keywords are only recognised in upper case (`and` is a normal search word).
|
||||
Whole words: use a regex, e.g. `/\bgo\b/`.
|
||||
|
||||
Examples: `(rust OR go) AND NOT crypto` · `"machine learning" -/pytorch|tensorflow/` · `llm (privacy OR datenschutz)`
|
||||
|
||||
Invalid queries raise `QuerySyntaxError` with a stable `code`, a `message` and the 0-based `position`:
|
||||
`empty_query`, `empty_phrase`, `unterminated_phrase`, `unterminated_regex`, `invalid_regex`,
|
||||
`missing_operand`, `unexpected_paren`, `unexpected_token`, `unclosed_paren`, `empty_group`,
|
||||
`too_long`, `too_many_terms`.
|
||||
|
||||
## Command line
|
||||
|
||||
```bash
|
||||
mdgrep '(rust OR go) AND NOT crypto' notes/ # search all *.md below notes/
|
||||
mdgrep '/llms?\b/' docs/*.md --split list # one result per list item
|
||||
mdgrep 'todo' . -l # file names only
|
||||
mdgrep 'release' . -c # number of matching blocks
|
||||
mdgrep '"open source"' . --json # JSON (same format as the Web API)
|
||||
python -m mdgrep --help
|
||||
```
|
||||
|
||||
`--split` chooses the unit that is searched and shown: `paragraph` (default), `list`
|
||||
(top-level list items, other text as paragraphs), `heading` (sections) or `file`.
|
||||
Exit codes as with grep: `0` match, `1` no match, `2` error (syntax error, time limit).
|
||||
|
||||
## Python API
|
||||
|
||||
```python
|
||||
from mdgrep import compile_query, load_markdown, SearchEngine, Block
|
||||
|
||||
# 1. Just the query language – on any string
|
||||
q = compile_query('(rust OR go) AND NOT /crypto(currency)?/')
|
||||
q.matches("Rust 2.0 is out") # True
|
||||
q.spans("Rust and Go") # [(0, 4), (9, 11)] – for highlighting
|
||||
|
||||
# 2. Markdown files → blocks → search
|
||||
blocks = load_markdown("docs/", split="list", relative_to="docs/",
|
||||
meta=lambda path: {"lang": path.stem.rsplit("_", 1)[-1]})
|
||||
engine = SearchEngine(blocks)
|
||||
result = engine.search("rust", filters={"lang": "de"}, limit=20)
|
||||
for group in result.groups:
|
||||
print(group.block.source, group.block.line, group.segments)
|
||||
result.to_dict() # JSON-ready, see "JSON format"
|
||||
|
||||
# 3. Your own data – any text source
|
||||
engine = SearchEngine([
|
||||
Block(id="42", title="Ticket 42", text="Login fails with SSO", meta={"status": "open"}),
|
||||
Block(id="43", title="Ticket 43", text="Dark mode toggle broken", meta={"status": "closed"}),
|
||||
])
|
||||
engine.search("login OR sso", filters={"status": "open"})
|
||||
```
|
||||
|
||||
### `Block`
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| `id` | unique id |
|
||||
| `text` | searched and shown as snippet |
|
||||
| `title` | searched and shown as heading (don't repeat it in `text`) |
|
||||
| `anchor` | optional fragment id, used by the widget as link target if no `href` is given |
|
||||
| `source`, `line` | origin (file, 1-based line) |
|
||||
| `meta` | free dict, passed through unchanged (filters work on it) |
|
||||
|
||||
`load_markdown()` adds `meta["section"]` (nearest heading), `meta["kind"]`
|
||||
(`file`/`section`/`item`/`paragraph`) and `meta["markdown"]` (raw source).
|
||||
|
||||
### `SearchEngine(source, *, reload_check=None, reload_interval=60)`
|
||||
|
||||
`source` is a list of blocks or a callable returning them. With a callable plus `reload_check`
|
||||
(returns any comparable token, e.g. newest file mtime) blocks are reloaded automatically when
|
||||
the token changes – checked at most every `reload_interval` seconds:
|
||||
|
||||
```python
|
||||
engine = SearchEngine(lambda: load_markdown("docs/"),
|
||||
reload_check=lambda: max(p.stat().st_mtime for p in Path("docs").rglob("*.md")))
|
||||
```
|
||||
|
||||
### `engine.search(query, **options)`
|
||||
|
||||
| Option | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `filters` | `None` | `{meta_key: value or [values]}`, all keys must match; empty values are ignored |
|
||||
| `predicate` | `None` | extra `callable(block) -> bool` |
|
||||
| `group_by` | `None` | `callable(block) -> key or None`; one result per key, the others in `also` |
|
||||
| `sort_key`, `reverse` | original order | sort matching blocks (e.g. newest first) |
|
||||
| `offset`, `limit` | `0`, `30` | paging, counted in groups (`limit=None` = all) |
|
||||
| `time_budget` | `1.5` | seconds for the whole search; on overrun `truncated=True` |
|
||||
| `snippet_chars`, `context` | `700`, `120` | longer texts are cut to windows around the hits |
|
||||
|
||||
## Web API (Flask)
|
||||
|
||||
```python
|
||||
from mdgrep import SearchEngine, load_markdown
|
||||
from mdgrep.web import create_app, create_blueprint
|
||||
|
||||
engine = SearchEngine(lambda: load_markdown("docs", split="list", relative_to="docs"))
|
||||
app = create_app(engine, filters=["lang"]) # standalone: GET /search
|
||||
# or inside an existing app:
|
||||
app.register_blueprint(create_blueprint(engine, filters=["lang"]), url_prefix="/api")
|
||||
```
|
||||
|
||||
`GET /search?q=…&offset=0&limit=30&<filter>=value` – for each name in `filters` an equally
|
||||
named parameter (`""`/`all` = no restriction, `a,b` = several values). The blueprint also serves the
|
||||
widget assets under `/mdgrep-static/`. `create_blueprint()` accepts the search options above plus
|
||||
`default_limit`, `max_limit`, `max_query_length` and `hide_meta` (meta keys removed from responses,
|
||||
default `("markdown",)`).
|
||||
|
||||
Development server (serves a static site and the API together):
|
||||
|
||||
```bash
|
||||
python -m mdgrep.web --dir examples/docs --split list --static examples/site --port 8000
|
||||
# → http://127.0.0.1:8000/ (API: /api/search)
|
||||
```
|
||||
|
||||
For production run the app with a WSGI server (e.g. `gunicorn -w 2 module:app`) behind a reverse
|
||||
proxy with rate limiting.
|
||||
|
||||
### JSON format
|
||||
|
||||
```json
|
||||
{
|
||||
"query": "rust", "total_groups": 12, "total_blocks": 15,
|
||||
"offset": 0, "limit": 30, "next_offset": null, "truncated": false, "elapsed_ms": 3.2,
|
||||
"results": [{
|
||||
"block": {
|
||||
"id": "a.md:4", "title": "…", "anchor": "", "source": "a.md", "line": 4, "meta": {},
|
||||
"title_segments": [{"t": "…", "hit": false}],
|
||||
"segments": [{"t": "The ", "hit": false}, {"t": "Rust", "hit": true}, {"t": " team …", "hit": false}],
|
||||
"hits": 1
|
||||
},
|
||||
"also": [{"id": "b.md:9", "title": "…", "anchor": "", "source": "b.md", "line": 9, "meta": {}}]
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
Errors: HTTP 400 `{"error": {"code": "unclosed_paren", "message": "…", "position": 0}}`.
|
||||
|
||||
## JavaScript widget
|
||||
|
||||
Framework-free, live search with debounce, cancels outdated requests, keeps the query in the URL
|
||||
(shareable, back button), filters, "load more", collapsible syntax help, error messages with
|
||||
position, English and German labels. All result text is inserted via `textContent` (no XSS).
|
||||
|
||||
```html
|
||||
<link rel="stylesheet" href="/api/mdgrep-static/mdgrep-widget.css">
|
||||
<div id="search"></div>
|
||||
<script src="/api/mdgrep-static/mdgrep-widget.js"></script>
|
||||
<script>
|
||||
MdGrep.mount(document.getElementById('search'), {
|
||||
endpoint: '/api/search',
|
||||
labels: 'de', // 'en' | 'de' | {…overrides}
|
||||
filters: [{ name: 'lang', label: 'Sprache',
|
||||
options: [{ value: '', label: 'Alle' }, { value: 'de', label: 'Deutsch' }] }],
|
||||
badgeFields: ['lang'], // meta keys shown as badges
|
||||
dateField: 'date', // meta key shown as date
|
||||
hrefField: 'href', // meta key with the link target (else #anchor)
|
||||
});
|
||||
</script>
|
||||
```
|
||||
|
||||
Without JavaScript code: `<div data-mdgrep data-endpoint="/api/search" data-labels="de"></div>`.
|
||||
|
||||
Other options: `live`, `debounce` (300 ms), `minChars` (2), `limit` (30), `syncUrl`, `autofocus`,
|
||||
`help`, `href(block)`, `formatBadge(field, value, block)`, `alsoText(block)`.
|
||||
The widget fires a `mdgrep:results` event with the response on its root element.
|
||||
|
||||
For static sites copy the assets instead of serving them from Flask:
|
||||
|
||||
```python
|
||||
import mdgrep
|
||||
mdgrep.export_static("site/static") # writes mdgrep-widget.js / .css
|
||||
```
|
||||
|
||||
Theming via CSS custom properties (defaults are light):
|
||||
`--mdgrep-text`, `--mdgrep-muted`, `--mdgrep-surface`, `--mdgrep-border`, `--mdgrep-accent`,
|
||||
`--mdgrep-mark-bg`, `--mdgrep-mark-text`, `--mdgrep-error`, `--mdgrep-radius`.
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
python -m venv .venv && .venv/bin/pip install -e ".[dev]"
|
||||
.venv/bin/pytest
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
13
examples/docs/languages.md
Normal file
13
examples/docs/languages.md
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
# Programming languages
|
||||
|
||||
## Systems
|
||||
|
||||
- **Rust** guarantees memory safety without a garbage collector. Its borrow checker
|
||||
rejects data races at compile time.
|
||||
- **Go** favours simplicity and fast compilation; goroutines make concurrency cheap.
|
||||
- **Zig** offers manual memory management with compile-time code execution.
|
||||
|
||||
## Scripting
|
||||
|
||||
- **Python** is popular for data science and machine learning (LLMs, PyTorch).
|
||||
- **Lua** is a tiny embeddable language used in games and Neovim.
|
||||
7
examples/docs/tools.md
Normal file
7
examples/docs/tools.md
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
# Tools
|
||||
|
||||
Regular expressions are supported by grep, ripgrep and most editors.
|
||||
|
||||
Large language models (LLMs) can explain a regex, but testing it is still wise.
|
||||
|
||||
Open source licenses such as MIT and GPL differ in their copyleft requirements.
|
||||
21
examples/site/index.html
Normal file
21
examples/site/index.html
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width,initial-scale=1">
|
||||
<title>mdgrep demo</title>
|
||||
<link rel="stylesheet" href="api/mdgrep-static/mdgrep-widget.css">
|
||||
<style>
|
||||
body { font-family: system-ui, sans-serif; max-width: 48rem; margin: 2rem auto; padding: 0 1rem; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>mdgrep demo</h1>
|
||||
<!-- Run: python -m mdgrep.web --dir examples/docs --split list --static examples/site -->
|
||||
<div id="search"></div>
|
||||
<script src="api/mdgrep-static/mdgrep-widget.js"></script>
|
||||
<script>
|
||||
MdGrep.mount(document.getElementById('search'), { endpoint: 'api/search', labels: 'en' });
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
29
pyproject.toml
Normal file
29
pyproject.toml
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
[build-system]
|
||||
requires = ["setuptools>=68"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "mdgrep"
|
||||
version = "0.1.0"
|
||||
description = "Boolean full-text search (substrings, phrases, regex, AND/OR/NOT, parentheses) over Markdown and other text blocks, with CLI, Flask API and a JavaScript search widget"
|
||||
readme = "README.md"
|
||||
license = "MIT"
|
||||
requires-python = ">=3.10"
|
||||
authors = [{ name = "Dieter Schlüter" }]
|
||||
dependencies = ["regex>=2023.0"]
|
||||
|
||||
[project.optional-dependencies]
|
||||
web = ["flask>=2.3"]
|
||||
dev = ["pytest>=7", "flask>=2.3"]
|
||||
|
||||
[project.scripts]
|
||||
mdgrep = "mdgrep.cli:main"
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["src"]
|
||||
|
||||
[tool.setuptools.package-data]
|
||||
mdgrep = ["static/*"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
47
src/mdgrep/__init__.py
Normal file
47
src/mdgrep/__init__.py
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
"""mdgrep – boolean full-text search over Markdown and other text blocks."""
|
||||
|
||||
from .engine import Block, Group, SearchEngine, SearchResult
|
||||
from .markdown import load_markdown, markdown_to_text, split_markdown
|
||||
from .query import Query, QuerySyntaxError, QueryTimeout, compile_query
|
||||
from .snippets import segments
|
||||
|
||||
__version__ = "0.1.0"
|
||||
|
||||
STATIC_FILES = ("mdgrep-widget.js", "mdgrep-widget.css")
|
||||
|
||||
|
||||
def export_static(target_dir) -> list:
|
||||
"""Copy the widget assets (JS + CSS) into `target_dir`, e.g. a static site's asset folder.
|
||||
|
||||
Returns the written paths. Files are only rewritten when their content changed.
|
||||
"""
|
||||
from importlib.resources import files
|
||||
from pathlib import Path
|
||||
|
||||
target = Path(target_dir)
|
||||
target.mkdir(parents=True, exist_ok=True)
|
||||
written = []
|
||||
for name in STATIC_FILES:
|
||||
data = files("mdgrep").joinpath("static", name).read_bytes()
|
||||
dest = target / name
|
||||
if not dest.exists() or dest.read_bytes() != data:
|
||||
dest.write_bytes(data)
|
||||
written.append(dest)
|
||||
return written
|
||||
|
||||
__all__ = [
|
||||
"Block",
|
||||
"Group",
|
||||
"Query",
|
||||
"QuerySyntaxError",
|
||||
"QueryTimeout",
|
||||
"SearchEngine",
|
||||
"SearchResult",
|
||||
"compile_query",
|
||||
"export_static",
|
||||
"load_markdown",
|
||||
"markdown_to_text",
|
||||
"split_markdown",
|
||||
"segments",
|
||||
"__version__",
|
||||
]
|
||||
5
src/mdgrep/__main__.py
Normal file
5
src/mdgrep/__main__.py
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
import sys
|
||||
|
||||
from .cli import main
|
||||
|
||||
sys.exit(main())
|
||||
123
src/mdgrep/cli.py
Normal file
123
src/mdgrep/cli.py
Normal file
|
|
@ -0,0 +1,123 @@
|
|||
"""grep-like command line: mdgrep QUERY [PATH ...]
|
||||
|
||||
Exit codes as with grep: 0 = matches found, 1 = no match, 2 = error.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
from . import __version__
|
||||
from .engine import SearchEngine
|
||||
from .markdown import SPLIT_MODES, load_markdown
|
||||
from .query import QuerySyntaxError, compile_query
|
||||
|
||||
_SYNTAX = """query syntax (case-insensitive):
|
||||
word substring ("rust" matches "Rustacean")
|
||||
"a phrase" exact phrase
|
||||
/regex/ regular expression (\\/ for a literal slash)
|
||||
a b, a AND b both (also UND, &)
|
||||
a OR b either (also ODER, |)
|
||||
NOT a, -a, !a negation (also NICHT)
|
||||
( ... ) grouping; precedence NOT > AND > OR
|
||||
|
||||
examples:
|
||||
mdgrep '(rust OR go) AND NOT crypto' notes/
|
||||
mdgrep '/llms?\\b/ "open source"' --split list docs/*.md
|
||||
"""
|
||||
|
||||
_COLORS = {"file": "\033[35m", "line": "\033[32m", "hit": "\033[1;31m", "dim": "\033[2m", "reset": "\033[0m"}
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
p = argparse.ArgumentParser(
|
||||
prog="mdgrep",
|
||||
description="Boolean full-text search in Markdown files.",
|
||||
epilog=_SYNTAX,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
)
|
||||
p.add_argument("query", help="search query (see syntax below)")
|
||||
p.add_argument("paths", nargs="*", default=["."], help="files or directories (default: .)")
|
||||
p.add_argument("-s", "--split", choices=SPLIT_MODES, default="paragraph",
|
||||
help="unit that is searched and shown (default: paragraph)")
|
||||
p.add_argument("-g", "--glob", default="*.md", help="file pattern in directories (default: *.md)")
|
||||
p.add_argument("-c", "--count", action="store_true", help="only print the number of matching blocks")
|
||||
p.add_argument("-l", "--files-with-matches", action="store_true", help="only print file names")
|
||||
p.add_argument("-j", "--json", action="store_true", help="JSON output (same format as the Web API)")
|
||||
p.add_argument("-C", "--context", type=int, default=80, metavar="N",
|
||||
help="characters of context around hits in long blocks (default: 80)")
|
||||
p.add_argument("--max-chars", type=int, default=400, metavar="N",
|
||||
help="shorten blocks longer than N characters (0 = never, default: 400)")
|
||||
p.add_argument("-m", "--max-count", type=int, default=0, metavar="N",
|
||||
help="stop after N matching blocks (0 = all)")
|
||||
p.add_argument("--timeout", type=float, default=30.0, help="time limit in seconds (default: 30)")
|
||||
p.add_argument("--color", choices=("auto", "always", "never"), default="auto")
|
||||
p.add_argument("-V", "--version", action="version", version=f"mdgrep {__version__}")
|
||||
return p
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
color = args.color == "always" or (args.color == "auto" and sys.stdout.isatty()
|
||||
and os.environ.get("NO_COLOR") is None)
|
||||
c = _COLORS if color else dict.fromkeys(_COLORS, "")
|
||||
|
||||
try:
|
||||
query = compile_query(args.query)
|
||||
except QuerySyntaxError as e:
|
||||
print(f"mdgrep: {e.message}", file=sys.stderr)
|
||||
print(f" {args.query}\n {' ' * e.position}^", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
try:
|
||||
blocks = load_markdown(args.paths, split=args.split, pattern=args.glob)
|
||||
except OSError as e:
|
||||
print(f"mdgrep: {e}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
result = SearchEngine(blocks).search(
|
||||
query,
|
||||
limit=args.max_count or None,
|
||||
time_budget=args.timeout,
|
||||
snippet_chars=args.max_chars or None,
|
||||
context=args.context,
|
||||
)
|
||||
if result.truncated:
|
||||
print("mdgrep: time limit exceeded, results are incomplete", file=sys.stderr)
|
||||
|
||||
if args.json:
|
||||
data = result.to_dict()
|
||||
for r in data["results"]:
|
||||
r["block"]["meta"].pop("markdown", None)
|
||||
json.dump(data, sys.stdout, ensure_ascii=False, indent=2)
|
||||
print()
|
||||
elif args.count:
|
||||
print(result.total_blocks)
|
||||
elif args.files_with_matches:
|
||||
seen = []
|
||||
for g in result.groups:
|
||||
if g.block.source not in seen:
|
||||
seen.append(g.block.source)
|
||||
print(f"{c['file']}{g.block.source}{c['reset']}")
|
||||
else:
|
||||
for g in result.groups:
|
||||
print(f"{c['file']}{g.block.source}{c['reset']}:{c['line']}{g.block.line}{c['reset']}")
|
||||
if g.block.title:
|
||||
title = "".join(f"{c['hit']}{s['t']}{c['reset']}" if s["hit"] else s["t"]
|
||||
for s in g.title_segments)
|
||||
print(f" {title}")
|
||||
text = "".join(f"{c['hit']}{s['t']}{c['reset']}" if s["hit"] else s["t"] for s in g.segments)
|
||||
for line in text.split("\n"):
|
||||
print(f" {line}" if line else "")
|
||||
print()
|
||||
|
||||
if result.truncated:
|
||||
return 2
|
||||
return 0 if result.total_blocks else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
229
src/mdgrep/engine.py
Normal file
229
src/mdgrep/engine.py
Normal file
|
|
@ -0,0 +1,229 @@
|
|||
"""Search engine over in-memory text blocks: filtering, grouping, sorting, paging, time budget."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Callable, Hashable, Iterable
|
||||
|
||||
from .query import Query, QueryTimeout, compile_query
|
||||
from .snippets import segments
|
||||
|
||||
__all__ = ["Block", "SearchEngine", "SearchResult", "Group"]
|
||||
|
||||
|
||||
@dataclass
|
||||
class Block:
|
||||
"""One searchable unit ("box" in the result list).
|
||||
|
||||
`title` and `text` are both searched and shown (title as heading, text as snippet), so
|
||||
loaders should not repeat the title inside the text. `meta` holds arbitrary application
|
||||
data (e.g. language, date, URL) and is passed through to results unchanged.
|
||||
"""
|
||||
id: str
|
||||
text: str
|
||||
title: str = ""
|
||||
anchor: str = ""
|
||||
source: str = ""
|
||||
line: int = 0
|
||||
meta: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
def summary(self) -> dict:
|
||||
return {"id": self.id, "title": self.title, "anchor": self.anchor,
|
||||
"source": self.source, "line": self.line, "meta": self.meta}
|
||||
|
||||
|
||||
@dataclass
|
||||
class Group:
|
||||
"""A primary block with its highlighted snippet plus further blocks of the same group."""
|
||||
key: Hashable
|
||||
block: Block
|
||||
title_segments: list[dict]
|
||||
segments: list[dict]
|
||||
hits: int
|
||||
also: list[Block]
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
data = self.block.summary()
|
||||
data["title_segments"] = self.title_segments
|
||||
data["segments"] = self.segments
|
||||
data["hits"] = self.hits
|
||||
return {"block": data, "also": [b.summary() for b in self.also]}
|
||||
|
||||
|
||||
@dataclass
|
||||
class SearchResult:
|
||||
query: str
|
||||
groups: list[Group]
|
||||
total_groups: int
|
||||
total_blocks: int
|
||||
offset: int
|
||||
limit: int
|
||||
truncated: bool
|
||||
elapsed_ms: float
|
||||
|
||||
@property
|
||||
def next_offset(self) -> int | None:
|
||||
nxt = self.offset + len(self.groups)
|
||||
return nxt if nxt < self.total_groups else None
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
"""JSON contract shared by the Web API, the CLI (--json) and the widget."""
|
||||
return {
|
||||
"query": self.query,
|
||||
"total_groups": self.total_groups,
|
||||
"total_blocks": self.total_blocks,
|
||||
"offset": self.offset,
|
||||
"limit": self.limit,
|
||||
"next_offset": self.next_offset,
|
||||
"truncated": self.truncated,
|
||||
"elapsed_ms": round(self.elapsed_ms, 1),
|
||||
"results": [g.to_dict() for g in self.groups],
|
||||
}
|
||||
|
||||
|
||||
BlockSource = Iterable[Block] | Callable[[], Iterable[Block]]
|
||||
|
||||
|
||||
class SearchEngine:
|
||||
"""Holds blocks in memory and searches them.
|
||||
|
||||
`source` is either an iterable of blocks or a callable returning one. With a callable and
|
||||
`reload_check` (a callable returning any comparable token, e.g. the newest file mtime),
|
||||
the blocks are reloaded automatically when the token changes; the check runs at most
|
||||
every `reload_interval` seconds.
|
||||
"""
|
||||
|
||||
def __init__(self, source: BlockSource, *, reload_check: Callable[[], Any] | None = None,
|
||||
reload_interval: float = 60.0):
|
||||
self._loader = source if callable(source) else None
|
||||
self._reload_check = reload_check
|
||||
self._reload_interval = reload_interval
|
||||
self._lock = threading.Lock()
|
||||
self._token = reload_check() if reload_check else None
|
||||
self._checked_at = time.monotonic()
|
||||
self._blocks: list[Block] = list(source() if callable(source) else source)
|
||||
|
||||
@property
|
||||
def blocks(self) -> list[Block]:
|
||||
self._maybe_reload()
|
||||
return self._blocks
|
||||
|
||||
def reload(self) -> None:
|
||||
"""Reload blocks from the loader (no-op for a static block list)."""
|
||||
if self._loader is None:
|
||||
return
|
||||
blocks = list(self._loader())
|
||||
with self._lock:
|
||||
self._blocks = blocks
|
||||
self._token = self._reload_check() if self._reload_check else None
|
||||
self._checked_at = time.monotonic()
|
||||
|
||||
def _maybe_reload(self) -> None:
|
||||
if self._loader is None or self._reload_check is None:
|
||||
return
|
||||
if time.monotonic() - self._checked_at < self._reload_interval:
|
||||
return
|
||||
self._checked_at = time.monotonic()
|
||||
token = self._reload_check()
|
||||
if token != self._token:
|
||||
self.reload()
|
||||
|
||||
def search(
|
||||
self,
|
||||
query: str | Query,
|
||||
*,
|
||||
filters: dict[str, Any] | None = None,
|
||||
predicate: Callable[[Block], bool] | None = None,
|
||||
group_by: Callable[[Block], Hashable | None] | None = None,
|
||||
sort_key: Callable[[Block], Any] | None = None,
|
||||
reverse: bool = False,
|
||||
offset: int = 0,
|
||||
limit: int | None = 30,
|
||||
time_budget: float | None = 1.5,
|
||||
snippet_chars: int | None = 700,
|
||||
context: int = 120,
|
||||
) -> SearchResult:
|
||||
"""Search all blocks.
|
||||
|
||||
filters {meta_key: value or list of values}; blocks must match all keys.
|
||||
predicate extra filter function on blocks.
|
||||
group_by returns a group key per block (None = block forms its own group). Each group
|
||||
is shown once; its first block after sorting is the primary one.
|
||||
sort_key sort matching blocks (default: original order); `reverse` for descending.
|
||||
limit groups per page (None = all). `offset` counts groups.
|
||||
time_budget seconds for the whole search; on overrun the result is partial and
|
||||
`truncated` is True.
|
||||
snippet_chars shorten long texts to windows around the hits (None = full text).
|
||||
"""
|
||||
started = time.monotonic()
|
||||
q = query if isinstance(query, Query) else compile_query(query)
|
||||
deadline = started + time_budget if time_budget else None
|
||||
candidates = self._filter(self.blocks, filters, predicate)
|
||||
|
||||
matched: list[Block] = []
|
||||
truncated = False
|
||||
for block in candidates:
|
||||
try:
|
||||
if q.matches(f"{block.title}\n{block.text}" if block.title else block.text,
|
||||
deadline=deadline):
|
||||
matched.append(block)
|
||||
except QueryTimeout:
|
||||
truncated = True
|
||||
break
|
||||
|
||||
if sort_key is not None:
|
||||
matched.sort(key=sort_key, reverse=reverse)
|
||||
elif reverse:
|
||||
matched.reverse()
|
||||
|
||||
grouped: dict[Hashable, list[Block]] = {}
|
||||
for i, block in enumerate(matched):
|
||||
key = group_by(block) if group_by else None
|
||||
grouped.setdefault(key if key is not None else ("__block__", i), []).append(block)
|
||||
group_items = list(grouped.items())
|
||||
|
||||
page = group_items[offset:offset + limit] if limit is not None else group_items[offset:]
|
||||
groups = []
|
||||
for key, members in page:
|
||||
primary = members[0]
|
||||
try:
|
||||
spans = q.spans(primary.text, deadline=deadline)
|
||||
title_spans = q.spans(primary.title, deadline=deadline) if primary.title else []
|
||||
except QueryTimeout:
|
||||
spans, title_spans, truncated = [], [], True
|
||||
groups.append(Group(
|
||||
key=key,
|
||||
block=primary,
|
||||
title_segments=segments(primary.title, title_spans),
|
||||
segments=segments(primary.text, spans, max_chars=snippet_chars, context=context),
|
||||
hits=len(spans),
|
||||
also=members[1:],
|
||||
))
|
||||
|
||||
return SearchResult(
|
||||
query=q.text,
|
||||
groups=groups,
|
||||
total_groups=len(group_items),
|
||||
total_blocks=len(matched),
|
||||
offset=offset,
|
||||
limit=limit if limit is not None else len(group_items),
|
||||
truncated=truncated,
|
||||
elapsed_ms=(time.monotonic() - started) * 1000,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _filter(blocks, filters, predicate):
|
||||
conditions = []
|
||||
for key, value in (filters or {}).items():
|
||||
if value is None or value == "" or value == []:
|
||||
continue
|
||||
allowed = set(value) if isinstance(value, (list, tuple, set, frozenset)) else {value}
|
||||
conditions.append((key, allowed))
|
||||
for block in blocks:
|
||||
if any(block.meta.get(k) not in allowed for k, allowed in conditions):
|
||||
continue
|
||||
if predicate is not None and not predicate(block):
|
||||
continue
|
||||
yield block
|
||||
240
src/mdgrep/markdown.py
Normal file
240
src/mdgrep/markdown.py
Normal file
|
|
@ -0,0 +1,240 @@
|
|||
"""Split Markdown files into searchable blocks.
|
||||
|
||||
Split modes:
|
||||
file one block per file
|
||||
heading one block per section (heading + content up to the next heading)
|
||||
list one block per top-level list item (with its indented continuation lines);
|
||||
text outside lists becomes paragraph blocks
|
||||
paragraph one block per blank-line separated paragraph (list items count as paragraphs)
|
||||
|
||||
Every block gets `meta["section"]` (nearest preceding heading text), `meta["kind"]`
|
||||
("file", "section", "item" or "paragraph") and `meta["markdown"]` (the raw source).
|
||||
The searchable `text` is plain text with Markdown syntax removed. In `file` and `heading`
|
||||
mode the heading becomes the block title (and is removed from the text); list items and
|
||||
paragraphs have no title.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Callable, Iterable
|
||||
|
||||
from .engine import Block
|
||||
|
||||
__all__ = ["load_markdown", "split_markdown", "markdown_to_text", "slugify", "SPLIT_MODES"]
|
||||
|
||||
SPLIT_MODES = ("file", "heading", "list", "paragraph")
|
||||
|
||||
_HEADING = re.compile(r"^(#{1,6})\s+(.*?)\s*#*\s*$")
|
||||
_FENCE = re.compile(r"^\s*(```|~~~)")
|
||||
_LIST_ITEM = re.compile(r"^([-*+]|\d{1,9}[.)])\s+")
|
||||
_HR = re.compile(r"^\s*([-*_])(\s*\1){2,}\s*$")
|
||||
|
||||
|
||||
def load_markdown(paths: str | Path | Iterable[str | Path], *, split: str = "paragraph",
|
||||
meta: Callable[[Path], dict] | None = None, encoding: str = "utf-8",
|
||||
pattern: str = "*.md", relative_to: str | Path | None = None) -> list[Block]:
|
||||
"""Load Markdown files (directories are searched recursively for `pattern`).
|
||||
|
||||
`meta(path)` may return a dict merged into every block's meta of that file.
|
||||
`relative_to` makes `Block.source`/`Block.id` relative to that directory (recommended for
|
||||
web use, so no server paths are exposed).
|
||||
"""
|
||||
base = Path(relative_to).resolve() if relative_to is not None else None
|
||||
blocks: list[Block] = []
|
||||
for path in _expand(paths, pattern):
|
||||
text = path.read_text(encoding=encoding)
|
||||
extra = meta(path) if meta else {}
|
||||
source = path.resolve().relative_to(base).as_posix() if base else str(path)
|
||||
blocks.extend(split_markdown(text, source=source, split=split, meta=extra))
|
||||
return blocks
|
||||
|
||||
|
||||
def _expand(paths, pattern):
|
||||
if isinstance(paths, (str, Path)):
|
||||
paths = [paths]
|
||||
files = []
|
||||
for p in map(Path, paths):
|
||||
if p.is_dir():
|
||||
files.extend(sorted(p.rglob(pattern)))
|
||||
else:
|
||||
files.append(p)
|
||||
return files
|
||||
|
||||
|
||||
def split_markdown(text: str, *, source: str = "", split: str = "paragraph",
|
||||
meta: dict | None = None) -> list[Block]:
|
||||
"""Split one Markdown document into blocks (see module docstring for modes)."""
|
||||
if split not in SPLIT_MODES:
|
||||
raise ValueError(f"split must be one of {SPLIT_MODES}, got {split!r}")
|
||||
meta = meta or {}
|
||||
lines = text.splitlines()
|
||||
raw_blocks = {
|
||||
"file": _split_file,
|
||||
"heading": _split_heading,
|
||||
"list": _split_list,
|
||||
"paragraph": _split_paragraph,
|
||||
}[split](lines)
|
||||
|
||||
blocks = []
|
||||
for start, end, kind, section, title, anchor in raw_blocks:
|
||||
raw = "\n".join(lines[start:end]).strip("\n")
|
||||
plain = markdown_to_text(raw)
|
||||
if title and plain.startswith(title):
|
||||
plain = plain[len(title):].lstrip("\n")
|
||||
if not plain.strip() and not title:
|
||||
continue
|
||||
block_meta = {**meta, "section": section, "kind": kind, "markdown": raw}
|
||||
blocks.append(Block(id=f"{source}:{start + 1}", text=plain, title=title, anchor=anchor,
|
||||
source=source, line=start + 1, meta=block_meta))
|
||||
return blocks
|
||||
|
||||
|
||||
# ── Line classification ──────────────────────────────────────────────────
|
||||
|
||||
def _scan(lines):
|
||||
"""Yield (index, line, in_fence, heading_match) for every line."""
|
||||
in_fence = False
|
||||
for i, line in enumerate(lines):
|
||||
if _FENCE.match(line):
|
||||
yield i, line, True, None
|
||||
in_fence = not in_fence
|
||||
continue
|
||||
heading = None if in_fence else _HEADING.match(line)
|
||||
yield i, line, in_fence, heading
|
||||
|
||||
|
||||
def _split_file(lines):
|
||||
title = ""
|
||||
for _, _, in_fence, heading in _scan(lines):
|
||||
if heading:
|
||||
title = heading.group(2)
|
||||
break
|
||||
return [(0, len(lines), "file", "", title, "")]
|
||||
|
||||
|
||||
def _split_heading(lines):
|
||||
out = []
|
||||
start, section, title, anchor = 0, "", "", ""
|
||||
for i, _, _, heading in _scan(lines):
|
||||
if heading:
|
||||
if i > start:
|
||||
out.append((start, i, "section", section, title, anchor))
|
||||
start, section = i, heading.group(2)
|
||||
title, anchor = section, slugify(section)
|
||||
out.append((start, len(lines), "section", section, title, anchor))
|
||||
return out
|
||||
|
||||
|
||||
def _split_list(lines):
|
||||
out = []
|
||||
section = ""
|
||||
cur = None # [start, kind]
|
||||
last_content = -1 # last non-blank line of the current block
|
||||
|
||||
def close(end):
|
||||
nonlocal cur
|
||||
if cur is not None:
|
||||
out.append((cur[0], end, cur[1], section, "", ""))
|
||||
cur = None
|
||||
|
||||
prev_blank = True
|
||||
for i, line, in_fence, heading in _scan(lines):
|
||||
if heading:
|
||||
close(last_content + 1)
|
||||
section = heading.group(2)
|
||||
prev_blank = True
|
||||
continue
|
||||
if not line.strip() and not in_fence:
|
||||
if cur is not None and cur[1] == "paragraph":
|
||||
close(last_content + 1)
|
||||
prev_blank = True
|
||||
continue
|
||||
if in_fence:
|
||||
if cur is None:
|
||||
cur = [i, "paragraph"]
|
||||
elif _LIST_ITEM.match(line):
|
||||
close(last_content + 1)
|
||||
cur = [i, "item"]
|
||||
elif _HR.match(line):
|
||||
close(last_content + 1)
|
||||
prev_blank = True
|
||||
continue
|
||||
elif cur is not None and cur[1] == "item" and (line[:1] in (" ", "\t") or not prev_blank):
|
||||
pass # indented or lazy continuation of the item
|
||||
elif cur is None or cur[1] == "item":
|
||||
close(last_content + 1)
|
||||
cur = [i, "paragraph"]
|
||||
last_content = i
|
||||
prev_blank = False
|
||||
close(last_content + 1)
|
||||
return out
|
||||
|
||||
|
||||
def _split_paragraph(lines):
|
||||
out = []
|
||||
section = ""
|
||||
start = None
|
||||
for i, line, in_fence, heading in _scan(lines):
|
||||
if heading:
|
||||
if start is not None:
|
||||
out.append((start, i, "paragraph", section, "", ""))
|
||||
start = None
|
||||
section = heading.group(2)
|
||||
continue
|
||||
if not line.strip() and not in_fence:
|
||||
if start is not None:
|
||||
out.append((start, i, "paragraph", section, "", ""))
|
||||
start = None
|
||||
continue
|
||||
if _HR.match(line) and not in_fence:
|
||||
continue
|
||||
if start is None:
|
||||
start = i
|
||||
if start is not None:
|
||||
out.append((start, len(lines), "paragraph", section, "", ""))
|
||||
return out
|
||||
|
||||
|
||||
# ── Markdown → plain text ────────────────────────────────────────────────
|
||||
|
||||
_IMAGE = re.compile(r"!\[([^\]]*)\]\([^)]*\)")
|
||||
_LINK = re.compile(r"\[([^\]]+)\]\((?:[^()\s]|\([^()\s]*\))+(?:\s+\"[^\"]*\")?\)")
|
||||
_REF_LINK = re.compile(r"\[([^\]]+)\]\[[^\]]*\]")
|
||||
_AUTOLINK = re.compile(r"<(https?://[^>]+)>")
|
||||
_BOLD = re.compile(r"(\*\*|__)(?=\S)(.+?)(?<=\S)\1")
|
||||
_ITALIC_STAR = re.compile(r"(?<!\*)\*(?=\S)([^*]+?)(?<=\S)\*(?!\*)")
|
||||
_ITALIC_UNDER = re.compile(r"(?<![\w_])_(?=\S)([^_]+?)(?<=\S)_(?![\w_])")
|
||||
_STRIKE = re.compile(r"~~(.+?)~~")
|
||||
_CODE = re.compile(r"`([^`]*)`")
|
||||
_LINE_MARKER = re.compile(r"^\s*(?:#{1,6}\s+|>\s?|(?:[-*+]|\d{1,9}[.)])\s+)")
|
||||
|
||||
|
||||
def markdown_to_text(md: str) -> str:
|
||||
"""Remove Markdown syntax, keep line structure (blank lines collapsed)."""
|
||||
out_lines = []
|
||||
for line in md.splitlines():
|
||||
if _FENCE.match(line) or _HR.match(line):
|
||||
continue
|
||||
line = _LINE_MARKER.sub("", line).strip()
|
||||
line = _IMAGE.sub(r"\1", line)
|
||||
line = _LINK.sub(r"\1", line)
|
||||
line = _REF_LINK.sub(r"\1", line)
|
||||
line = _AUTOLINK.sub(r"\1", line)
|
||||
line = _CODE.sub(r"\1", line)
|
||||
line = _BOLD.sub(r"\2", line)
|
||||
line = _ITALIC_STAR.sub(r"\1", line)
|
||||
line = _ITALIC_UNDER.sub(r"\1", line)
|
||||
line = _STRIKE.sub(r"\1", line)
|
||||
if line or (out_lines and out_lines[-1]):
|
||||
out_lines.append(line)
|
||||
return "\n".join(out_lines).strip()
|
||||
|
||||
|
||||
def slugify(text: str) -> str:
|
||||
"""GitHub-style heading anchor."""
|
||||
text = markdown_to_text(text).lower()
|
||||
text = re.sub(r"[^\w\- ]", "", text)
|
||||
return re.sub(r" ", "-", text.strip())
|
||||
|
||||
328
src/mdgrep/query.py
Normal file
328
src/mdgrep/query.py
Normal file
|
|
@ -0,0 +1,328 @@
|
|||
"""Query language: substrings, phrases, regular expressions, AND/OR/NOT and parentheses.
|
||||
|
||||
Syntax (all matching is case-insensitive):
|
||||
|
||||
word substring match ("rust" also matches "Rustacean")
|
||||
"a phrase" exact phrase, any whitespace between the words
|
||||
/regex/ regular expression (write \\/ for a literal slash)
|
||||
a b, a AND b both (also: UND, &, &&)
|
||||
a OR b either (also: ODER, |, ||)
|
||||
NOT a, -a, !a negation (also: NICHT)
|
||||
( ... ) grouping
|
||||
|
||||
Precedence: NOT binds tighter than AND, AND tighter than OR.
|
||||
Operator keywords are only recognized in upper case; "and" is a normal search word.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
import regex
|
||||
|
||||
__all__ = [
|
||||
"Query",
|
||||
"QuerySyntaxError",
|
||||
"QueryTimeout",
|
||||
"compile_query",
|
||||
]
|
||||
|
||||
_FLAGS = regex.IGNORECASE | regex.UNICODE
|
||||
|
||||
_KEYWORDS = {
|
||||
"AND": "AND", "UND": "AND",
|
||||
"OR": "OR", "ODER": "OR",
|
||||
"NOT": "NOT", "NICHT": "NOT",
|
||||
}
|
||||
|
||||
|
||||
class QuerySyntaxError(ValueError):
|
||||
"""Invalid query. `code` is a stable identifier, `position` a 0-based offset into the query."""
|
||||
|
||||
def __init__(self, message: str, position: int, code: str):
|
||||
super().__init__(f"{message} (at position {position})")
|
||||
self.message = message
|
||||
self.position = position
|
||||
self.code = code
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {"code": self.code, "message": self.message, "position": self.position}
|
||||
|
||||
|
||||
class QueryTimeout(TimeoutError):
|
||||
"""Matching exceeded the time limit (e.g. a catastrophic regular expression)."""
|
||||
|
||||
|
||||
# ── Tokens ────────────────────────────────────────────────────────────────
|
||||
|
||||
@dataclass
|
||||
class _Token:
|
||||
kind: str # LPAREN RPAREN AND OR NOT TERM
|
||||
pos: int
|
||||
value: str = ""
|
||||
term_kind: str = "" # word | phrase | regex
|
||||
|
||||
|
||||
def _tokenize(text: str) -> list[_Token]:
|
||||
tokens: list[_Token] = []
|
||||
i, n = 0, len(text)
|
||||
while i < n:
|
||||
c = text[i]
|
||||
if c.isspace():
|
||||
i += 1
|
||||
elif c in "()":
|
||||
tokens.append(_Token("LPAREN" if c == "(" else "RPAREN", i))
|
||||
i += 1
|
||||
elif c in "&|":
|
||||
kind = "AND" if c == "&" else "OR"
|
||||
tokens.append(_Token(kind, i))
|
||||
i += 2 if text[i:i + 2] == c * 2 else 1
|
||||
elif c == "!" or (c == "-" and i + 1 < n and not text[i + 1].isspace()):
|
||||
tokens.append(_Token("NOT", i))
|
||||
i += 1
|
||||
elif c == '"':
|
||||
j = i + 1
|
||||
buf = []
|
||||
while j < n and text[j] != '"':
|
||||
if text[j] == "\\" and j + 1 < n:
|
||||
j += 1
|
||||
buf.append(text[j])
|
||||
j += 1
|
||||
if j >= n:
|
||||
raise QuerySyntaxError("Unterminated phrase", i, "unterminated_phrase")
|
||||
tokens.append(_Token("TERM", i, "".join(buf), "phrase"))
|
||||
i = j + 1
|
||||
elif c == "/":
|
||||
j = i + 1
|
||||
buf = []
|
||||
while j < n and text[j] != "/":
|
||||
if text[j] == "\\" and j + 1 < n:
|
||||
if text[j + 1] == "/":
|
||||
buf.append("/")
|
||||
else:
|
||||
buf.append(text[j:j + 2])
|
||||
j += 2
|
||||
continue
|
||||
buf.append(text[j])
|
||||
j += 1
|
||||
if j >= n:
|
||||
raise QuerySyntaxError("Unterminated regular expression", i, "unterminated_regex")
|
||||
tokens.append(_Token("TERM", i, "".join(buf), "regex"))
|
||||
i = j + 1
|
||||
else:
|
||||
j = i
|
||||
while j < n and not text[j].isspace() and text[j] not in '()"':
|
||||
j += 1
|
||||
word = text[i:j]
|
||||
if word in _KEYWORDS:
|
||||
tokens.append(_Token(_KEYWORDS[word], i))
|
||||
else:
|
||||
tokens.append(_Token("TERM", i, word, "word"))
|
||||
i = j
|
||||
return tokens
|
||||
|
||||
|
||||
# ── AST ───────────────────────────────────────────────────────────────────
|
||||
|
||||
@dataclass
|
||||
class Term:
|
||||
kind: str # word | phrase | regex
|
||||
source: str
|
||||
pos: int
|
||||
pattern: regex.Pattern = field(repr=False)
|
||||
|
||||
def evaluate(self, text: str, deadline: float | None) -> bool:
|
||||
return self.pattern.search(text, timeout=_remaining(deadline)) is not None
|
||||
|
||||
|
||||
@dataclass
|
||||
class Not:
|
||||
child: object
|
||||
|
||||
def evaluate(self, text, deadline):
|
||||
return not self.child.evaluate(text, deadline)
|
||||
|
||||
|
||||
@dataclass
|
||||
class And:
|
||||
children: list
|
||||
|
||||
def evaluate(self, text, deadline):
|
||||
return all(c.evaluate(text, deadline) for c in self.children)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Or:
|
||||
children: list
|
||||
|
||||
def evaluate(self, text, deadline):
|
||||
return any(c.evaluate(text, deadline) for c in self.children)
|
||||
|
||||
|
||||
def _remaining(deadline: float | None) -> float | None:
|
||||
if deadline is None:
|
||||
return None
|
||||
left = deadline - time.monotonic()
|
||||
if left <= 0:
|
||||
raise QueryTimeout("Search time limit exceeded")
|
||||
return left
|
||||
|
||||
|
||||
def _compile_term(token: _Token) -> Term:
|
||||
if token.term_kind == "regex":
|
||||
if not token.value:
|
||||
raise QuerySyntaxError("Empty regular expression", token.pos, "invalid_regex")
|
||||
source = token.value
|
||||
elif token.term_kind == "phrase":
|
||||
words = token.value.split()
|
||||
if not words:
|
||||
raise QuerySyntaxError("Empty phrase", token.pos, "empty_phrase")
|
||||
source = r"\s+".join(regex.escape(w) for w in words)
|
||||
else:
|
||||
source = regex.escape(token.value)
|
||||
try:
|
||||
pattern = regex.compile(source, _FLAGS)
|
||||
except regex.error as e:
|
||||
raise QuerySyntaxError(f"Invalid regular expression: {e.msg}", token.pos, "invalid_regex") from None
|
||||
return Term(token.term_kind, token.value, token.pos, pattern)
|
||||
|
||||
|
||||
# ── Parser (recursive descent) ────────────────────────────────────────────
|
||||
|
||||
class _Parser:
|
||||
def __init__(self, tokens: list[_Token], length: int):
|
||||
self.tokens = tokens
|
||||
self.i = 0
|
||||
self.length = length
|
||||
self.terms: list[tuple[Term, bool]] = [] # (term, negated)
|
||||
|
||||
def peek(self) -> _Token | None:
|
||||
return self.tokens[self.i] if self.i < len(self.tokens) else None
|
||||
|
||||
def take(self) -> _Token:
|
||||
tok = self.tokens[self.i]
|
||||
self.i += 1
|
||||
return tok
|
||||
|
||||
def end_pos(self) -> int:
|
||||
return self.length
|
||||
|
||||
def parse(self):
|
||||
if not self.tokens:
|
||||
raise QuerySyntaxError("Empty query", 0, "empty_query")
|
||||
node = self.parse_or(negated=False)
|
||||
tok = self.peek()
|
||||
if tok is not None:
|
||||
if tok.kind == "RPAREN":
|
||||
raise QuerySyntaxError("Unexpected closing parenthesis", tok.pos, "unexpected_paren")
|
||||
raise QuerySyntaxError("Unexpected operator", tok.pos, "unexpected_token")
|
||||
return node
|
||||
|
||||
def parse_or(self, negated):
|
||||
children = [self.parse_and(negated)]
|
||||
while (tok := self.peek()) is not None and tok.kind == "OR":
|
||||
self.take()
|
||||
children.append(self.parse_and(negated))
|
||||
return children[0] if len(children) == 1 else Or(children)
|
||||
|
||||
def parse_and(self, negated):
|
||||
children = [self.parse_not(negated)]
|
||||
while (tok := self.peek()) is not None and tok.kind in ("AND", "NOT", "TERM", "LPAREN"):
|
||||
if tok.kind == "AND":
|
||||
self.take()
|
||||
children.append(self.parse_not(negated))
|
||||
return children[0] if len(children) == 1 else And(children)
|
||||
|
||||
def parse_not(self, negated):
|
||||
tok = self.peek()
|
||||
if tok is not None and tok.kind == "NOT":
|
||||
self.take()
|
||||
return Not(self.parse_not(not negated))
|
||||
return self.parse_primary(negated)
|
||||
|
||||
def parse_primary(self, negated):
|
||||
tok = self.peek()
|
||||
if tok is None:
|
||||
raise QuerySyntaxError("Search term expected", self.end_pos(), "missing_operand")
|
||||
if tok.kind == "TERM":
|
||||
self.take()
|
||||
term = _compile_term(tok)
|
||||
self.terms.append((term, negated))
|
||||
return term
|
||||
if tok.kind == "LPAREN":
|
||||
self.take()
|
||||
if (nxt := self.peek()) is not None and nxt.kind == "RPAREN":
|
||||
raise QuerySyntaxError("Empty parentheses", tok.pos, "empty_group")
|
||||
node = self.parse_or(negated)
|
||||
close = self.peek()
|
||||
if close is None or close.kind != "RPAREN":
|
||||
raise QuerySyntaxError("Missing closing parenthesis", tok.pos, "unclosed_paren")
|
||||
self.take()
|
||||
return node
|
||||
if tok.kind == "RPAREN":
|
||||
raise QuerySyntaxError("Unexpected closing parenthesis", tok.pos, "unexpected_paren")
|
||||
raise QuerySyntaxError("Search term expected", tok.pos, "missing_operand")
|
||||
|
||||
|
||||
# ── Public API ────────────────────────────────────────────────────────────
|
||||
|
||||
class Query:
|
||||
"""A compiled query. Use `matches()` to test a text and `spans()` for highlighting."""
|
||||
|
||||
def __init__(self, text: str, root, terms: list[tuple[Term, bool]]):
|
||||
self.text = text
|
||||
self._root = root
|
||||
self._terms = terms
|
||||
|
||||
@property
|
||||
def positive_terms(self) -> list[Term]:
|
||||
"""Terms that contribute to a match (not under an odd number of NOTs)."""
|
||||
return [t for t, negated in self._terms if not negated]
|
||||
|
||||
def matches(self, text: str, timeout: float | None = None, *, deadline: float | None = None) -> bool:
|
||||
"""True if `text` satisfies the query. Raises QueryTimeout when the time limit is exceeded."""
|
||||
if deadline is None and timeout is not None:
|
||||
deadline = time.monotonic() + timeout
|
||||
try:
|
||||
return self._root.evaluate(text, deadline)
|
||||
except TimeoutError as e:
|
||||
raise QueryTimeout(str(e) or "Search time limit exceeded") from None
|
||||
|
||||
def spans(self, text: str, timeout: float | None = None, *, deadline: float | None = None) -> list[tuple[int, int]]:
|
||||
"""Sorted, merged (start, end) ranges of all positive term matches in `text`."""
|
||||
if deadline is None and timeout is not None:
|
||||
deadline = time.monotonic() + timeout
|
||||
found = []
|
||||
try:
|
||||
for term in self.positive_terms:
|
||||
for m in term.pattern.finditer(text, timeout=_remaining(deadline)):
|
||||
if m.end() > m.start():
|
||||
found.append((m.start(), m.end()))
|
||||
except TimeoutError as e:
|
||||
raise QueryTimeout(str(e) or "Search time limit exceeded") from None
|
||||
return _merge(found)
|
||||
|
||||
def __repr__(self):
|
||||
return f"Query({self.text!r})"
|
||||
|
||||
|
||||
def _merge(spans: list[tuple[int, int]]) -> list[tuple[int, int]]:
|
||||
merged: list[list[int]] = []
|
||||
for start, end in sorted(spans):
|
||||
if merged and start <= merged[-1][1]:
|
||||
merged[-1][1] = max(merged[-1][1], end)
|
||||
else:
|
||||
merged.append([start, end])
|
||||
return [(s, e) for s, e in merged]
|
||||
|
||||
|
||||
def compile_query(text: str, *, max_length: int = 500, max_terms: int = 50) -> Query:
|
||||
"""Parse and compile a query string. Raises QuerySyntaxError on invalid input."""
|
||||
if len(text) > max_length:
|
||||
raise QuerySyntaxError(f"Query longer than {max_length} characters", max_length, "too_long")
|
||||
parser = _Parser(_tokenize(text), len(text))
|
||||
root = parser.parse()
|
||||
if len(parser.terms) > max_terms:
|
||||
raise QuerySyntaxError(f"More than {max_terms} search terms", 0, "too_many_terms")
|
||||
return Query(text, root, parser.terms)
|
||||
93
src/mdgrep/snippets.py
Normal file
93
src/mdgrep/snippets.py
Normal file
|
|
@ -0,0 +1,93 @@
|
|||
"""Turn a text plus match spans into display segments, shortened around the hits if needed."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
__all__ = ["segments", "ELLIPSIS"]
|
||||
|
||||
ELLIPSIS = " … "
|
||||
|
||||
|
||||
def segments(text: str, spans: list[tuple[int, int]], *, max_chars: int | None = None,
|
||||
context: int = 120) -> list[dict]:
|
||||
"""Split `text` into [{"t": str, "hit": bool}, ...].
|
||||
|
||||
If `max_chars` is set and the text is longer, only windows of `context` characters around
|
||||
the hits are kept (joined by ELLIPSIS segments). Without hits the beginning of the text is shown.
|
||||
"""
|
||||
if max_chars is None or len(text) <= max_chars:
|
||||
return _split(text, spans, 0, len(text))
|
||||
|
||||
windows = _windows(text, spans, context, max_chars)
|
||||
out: list[dict] = []
|
||||
for i, (start, end) in enumerate(windows):
|
||||
if start > 0 or i > 0:
|
||||
out.append({"t": ELLIPSIS, "hit": False})
|
||||
out.extend(_split(text, spans, start, end))
|
||||
if windows and windows[-1][1] < len(text):
|
||||
out.append({"t": ELLIPSIS, "hit": False})
|
||||
return out
|
||||
|
||||
|
||||
def _windows(text, spans, context, max_chars):
|
||||
if not spans:
|
||||
return [(0, _snap_end(text, max_chars))]
|
||||
windows: list[list[int]] = []
|
||||
budget = max_chars
|
||||
for start, end in spans:
|
||||
w_start = _snap_start(text, max(0, start - context))
|
||||
w_end = _snap_end(text, min(len(text), end + context))
|
||||
if windows and w_start <= windows[-1][1]:
|
||||
grow = w_end - windows[-1][1]
|
||||
if grow > 0:
|
||||
if grow > budget:
|
||||
break
|
||||
windows[-1][1] = w_end
|
||||
budget -= grow
|
||||
continue
|
||||
size = w_end - w_start
|
||||
if size > budget:
|
||||
if not windows: # always show the first hit
|
||||
windows.append([w_start, w_end])
|
||||
break
|
||||
windows.append([w_start, w_end])
|
||||
budget -= size
|
||||
return [(s, e) for s, e in windows]
|
||||
|
||||
|
||||
def _snap_start(text, pos):
|
||||
"""Move a window start back to the beginning of a word."""
|
||||
if pos <= 0:
|
||||
return 0
|
||||
limit = max(0, pos - 20)
|
||||
i = pos
|
||||
while i > limit and not text[i - 1].isspace():
|
||||
i -= 1
|
||||
return i if i > limit else pos
|
||||
|
||||
|
||||
def _snap_end(text, pos):
|
||||
"""Move a window end forward to the end of a word."""
|
||||
n = len(text)
|
||||
if pos >= n:
|
||||
return n
|
||||
limit = min(n, pos + 20)
|
||||
i = pos
|
||||
while i < limit and not text[i].isspace():
|
||||
i += 1
|
||||
return i
|
||||
|
||||
|
||||
def _split(text, spans, start, end):
|
||||
out = []
|
||||
pos = start
|
||||
for s, e in spans:
|
||||
if e <= start or s >= end:
|
||||
continue
|
||||
s, e = max(s, start), min(e, end)
|
||||
if s > pos:
|
||||
out.append({"t": text[pos:s], "hit": False})
|
||||
out.append({"t": text[s:e], "hit": True})
|
||||
pos = e
|
||||
if pos < end:
|
||||
out.append({"t": text[pos:end], "hit": False})
|
||||
return out
|
||||
79
src/mdgrep/static/mdgrep-widget.css
Normal file
79
src/mdgrep/static/mdgrep-widget.css
Normal file
|
|
@ -0,0 +1,79 @@
|
|||
/* mdgrep search widget v0.1.0 – MIT License
|
||||
*
|
||||
* Theming: set these custom properties on :root or any ancestor
|
||||
* --mdgrep-text --mdgrep-muted --mdgrep-surface --mdgrep-border
|
||||
* --mdgrep-accent --mdgrep-mark-bg --mdgrep-mark-text --mdgrep-error --mdgrep-radius
|
||||
*/
|
||||
|
||||
.mdgrep { color: var(--mdgrep-text, #1a1d2e); }
|
||||
.mdgrep [hidden] { display: none !important; }
|
||||
|
||||
.mdgrep-form { display: flex; flex-wrap: wrap; gap: 0.5rem; }
|
||||
|
||||
.mdgrep-input,
|
||||
.mdgrep-filter {
|
||||
font: inherit;
|
||||
color: inherit;
|
||||
background: var(--mdgrep-surface, #fff);
|
||||
border: 1px solid var(--mdgrep-border, #d5d8e3);
|
||||
border-radius: var(--mdgrep-radius, 0.5rem);
|
||||
padding: 0.5rem 0.75rem;
|
||||
}
|
||||
.mdgrep-input { flex: 1 1 16rem; min-width: 0; }
|
||||
.mdgrep-filter { flex: 0 0 auto; cursor: pointer; }
|
||||
.mdgrep-input:focus,
|
||||
.mdgrep-filter:focus { outline: 2px solid var(--mdgrep-accent, #4a7cff); outline-offset: 1px; }
|
||||
|
||||
.mdgrep-help { margin: 0.75rem 0 0; font-size: 0.875em; color: var(--mdgrep-muted, #6b7394); }
|
||||
.mdgrep-help summary { cursor: pointer; width: max-content; }
|
||||
.mdgrep-help table { border-collapse: collapse; margin: 0.5rem 0; background: none; border: none; }
|
||||
.mdgrep-help td { border: none; padding: 0.15rem 1rem 0.15rem 0; vertical-align: top; }
|
||||
.mdgrep-help code { white-space: nowrap; }
|
||||
|
||||
.mdgrep-status { margin: 0.75rem 0; min-height: 1.4em; font-size: 0.875em; color: var(--mdgrep-muted, #6b7394); }
|
||||
.mdgrep-status-error { color: var(--mdgrep-error, #c0392b); }
|
||||
.mdgrep-status-warn { color: var(--mdgrep-error, #c0392b); }
|
||||
.mdgrep-busy .mdgrep-results { opacity: 0.6; }
|
||||
|
||||
.mdgrep-results { list-style: none; margin: 0; padding: 0; display: flex; flex-direction: column; gap: 0.75rem; }
|
||||
|
||||
.mdgrep-card {
|
||||
margin: 0;
|
||||
padding: 0.75rem 1rem;
|
||||
background: var(--mdgrep-surface, #fff);
|
||||
border: 1px solid var(--mdgrep-border, #d5d8e3);
|
||||
border-radius: var(--mdgrep-radius, 0.5rem);
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
.mdgrep-card:hover { border-color: var(--mdgrep-accent, #4a7cff); }
|
||||
|
||||
.mdgrep-card-head { display: flex; flex-wrap: wrap; align-items: center; gap: 0.5rem; margin-bottom: 0.35rem;
|
||||
font-size: 0.8em; color: var(--mdgrep-muted, #6b7394); }
|
||||
.mdgrep-badge { padding: 0 0.45rem; border: 1px solid var(--mdgrep-border, #d5d8e3);
|
||||
border-radius: 999px; text-transform: uppercase; letter-spacing: 0.03em; }
|
||||
.mdgrep-date { font-style: italic; }
|
||||
|
||||
.mdgrep-title { display: block; font-weight: 700; margin-bottom: 0.3rem; text-decoration: none; }
|
||||
.mdgrep-snippet { display: block; white-space: pre-line; color: inherit; text-decoration: none; font-size: 0.925em; }
|
||||
a.mdgrep-snippet:visited { color: inherit; }
|
||||
.mdgrep-card a:hover { text-decoration: none; }
|
||||
|
||||
.mdgrep mark { background: var(--mdgrep-mark-bg, #ffe58a); color: var(--mdgrep-mark-text, inherit);
|
||||
border-radius: 2px; padding: 0 1px; }
|
||||
|
||||
.mdgrep-also { margin-top: 0.5rem; font-size: 0.8em; color: var(--mdgrep-muted, #6b7394); }
|
||||
.mdgrep-also summary { cursor: pointer; width: max-content; }
|
||||
.mdgrep-also ul { margin: 0.35rem 0 0; padding-left: 1.25rem; }
|
||||
|
||||
.mdgrep-more {
|
||||
display: block;
|
||||
margin: 1rem auto 0;
|
||||
font: inherit;
|
||||
color: inherit;
|
||||
background: var(--mdgrep-surface, #fff);
|
||||
border: 1px solid var(--mdgrep-border, #d5d8e3);
|
||||
border-radius: var(--mdgrep-radius, 0.5rem);
|
||||
padding: 0.4rem 1.25rem;
|
||||
cursor: pointer;
|
||||
}
|
||||
.mdgrep-more:hover { border-color: var(--mdgrep-accent, #4a7cff); }
|
||||
399
src/mdgrep/static/mdgrep-widget.js
Normal file
399
src/mdgrep/static/mdgrep-widget.js
Normal file
|
|
@ -0,0 +1,399 @@
|
|||
/*! mdgrep search widget v0.1.0 – MIT License
|
||||
*
|
||||
* Usage:
|
||||
* <link rel="stylesheet" href="mdgrep-widget.css">
|
||||
* <div id="search"></div>
|
||||
* <script src="mdgrep-widget.js"></script>
|
||||
* <script>
|
||||
* MdGrep.mount(document.getElementById('search'), {
|
||||
* endpoint: 'api/search',
|
||||
* labels: 'de',
|
||||
* filters: [{ name: 'lang', label: 'Sprache',
|
||||
* options: [{ value: '', label: 'Alle' }, { value: 'de', label: 'Deutsch' }] }],
|
||||
* });
|
||||
* </script>
|
||||
*
|
||||
* Or without script: <div data-mdgrep data-endpoint="api/search" data-labels="de"></div>
|
||||
* (filters as JSON in data-filters).
|
||||
*
|
||||
* Expects the JSON contract of mdgrep's Web API (SearchResult.to_dict()).
|
||||
* All result text is inserted with textContent – no HTML from the server is interpreted.
|
||||
*/
|
||||
(function (global) {
|
||||
'use strict';
|
||||
|
||||
var LABELS = {
|
||||
en: {
|
||||
placeholder: 'Search… e.g. (rust OR go) AND NOT crypto',
|
||||
inputLabel: 'Search query',
|
||||
help: 'Search syntax',
|
||||
helpRows: [
|
||||
['word', 'part of a word, upper/lower case ignored'],
|
||||
['"exact phrase"', 'words in this order'],
|
||||
['/regex/', 'regular expression, e.g. /llms?\\b/'],
|
||||
['a b a AND b', 'both terms (also &)'],
|
||||
['a OR b', 'at least one term (also |)'],
|
||||
['NOT a -a !a', 'term must not occur'],
|
||||
['( … )', 'grouping, e.g. (a OR b) AND c'],
|
||||
],
|
||||
minChars: function (n) { return 'Enter at least ' + n + ' characters.'; },
|
||||
searching: 'Searching…',
|
||||
none: 'No matches.',
|
||||
count: function (groups, blocks, ms) {
|
||||
return groups + (groups === 1 ? ' result' : ' results') +
|
||||
(blocks !== groups ? ' (' + blocks + ' matches)' : '') + ' · ' + Math.round(ms) + ' ms';
|
||||
},
|
||||
truncated: 'Time limit reached – results are incomplete.',
|
||||
more: 'Load more',
|
||||
alsoIn: function (n) { return 'also in ' + n + (n === 1 ? ' other place' : ' other places'); },
|
||||
errorAt: function (msg, pos) { return msg + ' (position ' + (pos + 1) + ')'; },
|
||||
rateLimited: 'Too many requests – please wait a moment.',
|
||||
failed: 'Search failed. Please try again later.',
|
||||
errors: {
|
||||
empty_query: 'Empty query',
|
||||
empty_phrase: 'Empty phrase',
|
||||
unterminated_phrase: 'Closing quote missing',
|
||||
unterminated_regex: 'Closing slash of the regular expression missing',
|
||||
invalid_regex: 'Invalid regular expression',
|
||||
missing_operand: 'Search term expected',
|
||||
unexpected_paren: 'Unexpected closing parenthesis',
|
||||
unexpected_token: 'Unexpected operator',
|
||||
unclosed_paren: 'Closing parenthesis missing',
|
||||
empty_group: 'Empty parentheses',
|
||||
too_long: 'Query too long',
|
||||
too_many_terms: 'Too many search terms',
|
||||
},
|
||||
},
|
||||
de: {
|
||||
placeholder: 'Suchen… z. B. (rust ODER go) UND NICHT crypto',
|
||||
inputLabel: 'Suchbegriff',
|
||||
help: 'Suchsyntax',
|
||||
helpRows: [
|
||||
['wort', 'Wortbestandteil, Groß-/Kleinschreibung egal'],
|
||||
['"genaue phrase"', 'Wörter in dieser Reihenfolge'],
|
||||
['/regex/', 'regulärer Ausdruck, z. B. /llms?\\b/'],
|
||||
['a b a UND b', 'beide Begriffe (auch AND, &)'],
|
||||
['a ODER b', 'mindestens ein Begriff (auch OR, |)'],
|
||||
['NICHT a -a !a', 'Begriff darf nicht vorkommen (auch NOT)'],
|
||||
['( … )', 'Klammern, z. B. (a ODER b) UND c'],
|
||||
],
|
||||
minChars: function (n) { return 'Bitte mindestens ' + n + ' Zeichen eingeben.'; },
|
||||
searching: 'Suche läuft…',
|
||||
none: 'Keine Treffer.',
|
||||
count: function (groups, blocks, ms) {
|
||||
return groups + (groups === 1 ? ' Ergebnis' : ' Ergebnisse') +
|
||||
(blocks !== groups ? ' (' + blocks + ' Fundstellen)' : '') + ' · ' + Math.round(ms) + ' ms';
|
||||
},
|
||||
truncated: 'Zeitlimit erreicht – die Ergebnisse sind unvollständig.',
|
||||
more: 'Mehr laden',
|
||||
alsoIn: function (n) { return 'auch in ' + n + (n === 1 ? ' weiterem Bericht' : ' weiteren Berichten'); },
|
||||
errorAt: function (msg, pos) { return msg + ' (Position ' + (pos + 1) + ')'; },
|
||||
rateLimited: 'Zu viele Anfragen – bitte einen Moment warten.',
|
||||
failed: 'Die Suche ist fehlgeschlagen. Bitte später erneut versuchen.',
|
||||
errors: {
|
||||
empty_query: 'Leere Suchanfrage',
|
||||
empty_phrase: 'Leere Phrase',
|
||||
unterminated_phrase: 'Schließendes Anführungszeichen fehlt',
|
||||
unterminated_regex: 'Schließender Schrägstrich des regulären Ausdrucks fehlt',
|
||||
invalid_regex: 'Ungültiger regulärer Ausdruck',
|
||||
missing_operand: 'Suchbegriff erwartet',
|
||||
unexpected_paren: 'Unerwartete schließende Klammer',
|
||||
unexpected_token: 'Unerwarteter Operator',
|
||||
unclosed_paren: 'Schließende Klammer fehlt',
|
||||
empty_group: 'Leere Klammern',
|
||||
too_long: 'Suchanfrage zu lang',
|
||||
too_many_terms: 'Zu viele Suchbegriffe',
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
var DEFAULTS = {
|
||||
endpoint: 'search',
|
||||
filters: [], // [{name, label, options: [{value, label}], value}]
|
||||
labels: 'en', // 'en' | 'de' | object overriding single labels
|
||||
live: true,
|
||||
debounce: 300,
|
||||
minChars: 2,
|
||||
limit: 30,
|
||||
syncUrl: true, // keep q and filters in the page URL (shareable, back button)
|
||||
autofocus: true,
|
||||
help: true,
|
||||
hrefField: 'href', // block.meta[hrefField] is the link target
|
||||
href: null, // function(block) → URL, overrides hrefField
|
||||
dateField: null, // block.meta[dateField] shown in the card header
|
||||
badgeFields: [], // block.meta keys shown as badges
|
||||
formatBadge: null, // function(field, value, block) → text
|
||||
alsoText: null, // function(block) → link text in the "also in" list
|
||||
};
|
||||
|
||||
function el(tag, className, text) {
|
||||
var node = document.createElement(tag);
|
||||
if (className) node.className = className;
|
||||
if (text != null) node.textContent = text;
|
||||
return node;
|
||||
}
|
||||
|
||||
function appendSegments(parent, segments) {
|
||||
(segments || []).forEach(function (s) {
|
||||
parent.appendChild(s.hit ? el('mark', null, s.t) : document.createTextNode(s.t));
|
||||
});
|
||||
}
|
||||
|
||||
function resolveLabels(labels) {
|
||||
if (typeof labels === 'string') return LABELS[labels] || LABELS.en;
|
||||
var base = LABELS[(labels && labels.lang) || 'en'] || LABELS.en;
|
||||
var out = {};
|
||||
Object.keys(base).forEach(function (k) { out[k] = base[k]; });
|
||||
Object.keys(labels || {}).forEach(function (k) { out[k] = labels[k]; });
|
||||
return out;
|
||||
}
|
||||
|
||||
function mount(root, options) {
|
||||
var o = {};
|
||||
Object.keys(DEFAULTS).forEach(function (k) { o[k] = DEFAULTS[k]; });
|
||||
Object.keys(options || {}).forEach(function (k) { o[k] = options[k]; });
|
||||
var L = resolveLabels(o.labels);
|
||||
|
||||
// ── DOM ──────────────────────────────────────────────────────────────
|
||||
root.classList.add('mdgrep');
|
||||
root.textContent = '';
|
||||
var form = el('form', 'mdgrep-form');
|
||||
form.setAttribute('role', 'search');
|
||||
var input = el('input', 'mdgrep-input');
|
||||
input.type = 'search';
|
||||
input.name = 'q';
|
||||
input.placeholder = L.placeholder;
|
||||
input.setAttribute('aria-label', L.inputLabel);
|
||||
input.autocomplete = 'off';
|
||||
input.spellcheck = false;
|
||||
form.appendChild(input);
|
||||
|
||||
var selects = {};
|
||||
o.filters.forEach(function (f) {
|
||||
var sel = el('select', 'mdgrep-filter');
|
||||
sel.name = f.name;
|
||||
sel.setAttribute('aria-label', f.label || f.name);
|
||||
(f.options || []).forEach(function (opt) {
|
||||
var option = el('option', null, opt.label);
|
||||
option.value = opt.value;
|
||||
sel.appendChild(option);
|
||||
});
|
||||
if (f.value != null) sel.value = f.value;
|
||||
selects[f.name] = sel;
|
||||
form.appendChild(sel);
|
||||
});
|
||||
root.appendChild(form);
|
||||
|
||||
if (o.help) {
|
||||
var help = el('details', 'mdgrep-help');
|
||||
help.appendChild(el('summary', null, L.help));
|
||||
var table = el('table');
|
||||
L.helpRows.forEach(function (row) {
|
||||
var tr = el('tr');
|
||||
var code = el('td');
|
||||
code.appendChild(el('code', null, row[0]));
|
||||
tr.appendChild(code);
|
||||
tr.appendChild(el('td', null, row[1]));
|
||||
table.appendChild(tr);
|
||||
});
|
||||
help.appendChild(table);
|
||||
root.appendChild(help);
|
||||
}
|
||||
|
||||
var status = el('div', 'mdgrep-status');
|
||||
status.setAttribute('aria-live', 'polite');
|
||||
root.appendChild(status);
|
||||
var list = el('ol', 'mdgrep-results');
|
||||
root.appendChild(list);
|
||||
var more = el('button', 'mdgrep-more', L.more);
|
||||
more.type = 'button';
|
||||
more.hidden = true;
|
||||
root.appendChild(more);
|
||||
|
||||
// ── State from URL ───────────────────────────────────────────────────
|
||||
if (o.syncUrl) {
|
||||
var params = new URLSearchParams(global.location.search);
|
||||
if (params.has('q')) input.value = params.get('q');
|
||||
Object.keys(selects).forEach(function (name) {
|
||||
if (params.has(name)) selects[name].value = params.get(name);
|
||||
});
|
||||
}
|
||||
|
||||
var controller = null;
|
||||
var timer = null;
|
||||
var nextOffset = null;
|
||||
|
||||
function filterValues() {
|
||||
var out = {};
|
||||
Object.keys(selects).forEach(function (name) { out[name] = selects[name].value; });
|
||||
return out;
|
||||
}
|
||||
|
||||
function updateUrl(q) {
|
||||
if (!o.syncUrl) return;
|
||||
var url = new URL(global.location.href);
|
||||
if (q) url.searchParams.set('q', q); else url.searchParams.delete('q');
|
||||
var values = filterValues();
|
||||
Object.keys(values).forEach(function (name) {
|
||||
if (values[name]) url.searchParams.set(name, values[name]); else url.searchParams.delete(name);
|
||||
});
|
||||
global.history.replaceState(null, '', url);
|
||||
}
|
||||
|
||||
function setStatus(text, kind) {
|
||||
status.textContent = text || '';
|
||||
status.className = 'mdgrep-status' + (kind ? ' mdgrep-status-' + kind : '');
|
||||
}
|
||||
|
||||
function hrefOf(block) {
|
||||
if (o.href) return o.href(block);
|
||||
var meta = block.meta || {};
|
||||
if (meta[o.hrefField]) return meta[o.hrefField];
|
||||
return block.anchor ? '#' + block.anchor : null;
|
||||
}
|
||||
|
||||
function link(block, className) {
|
||||
var href = hrefOf(block);
|
||||
var a = el(href ? 'a' : 'div', className);
|
||||
if (href) a.href = href;
|
||||
return a;
|
||||
}
|
||||
|
||||
function renderGroup(r) {
|
||||
var b = r.block;
|
||||
var meta = b.meta || {};
|
||||
var li = el('li', 'mdgrep-card');
|
||||
|
||||
var head = el('div', 'mdgrep-card-head');
|
||||
o.badgeFields.forEach(function (field) {
|
||||
if (meta[field] == null || meta[field] === '') return;
|
||||
var text = o.formatBadge ? o.formatBadge(field, meta[field], b) : String(meta[field]);
|
||||
if (text) head.appendChild(el('span', 'mdgrep-badge mdgrep-badge-' + field, text));
|
||||
});
|
||||
if (o.dateField && meta[o.dateField]) head.appendChild(el('span', 'mdgrep-date', meta[o.dateField]));
|
||||
if (head.childNodes.length) li.appendChild(head);
|
||||
|
||||
if (b.title) {
|
||||
var title = link(b, 'mdgrep-title');
|
||||
appendSegments(title, b.title_segments);
|
||||
li.appendChild(title);
|
||||
}
|
||||
var snippet = link(b, 'mdgrep-snippet');
|
||||
appendSegments(snippet, b.segments);
|
||||
li.appendChild(snippet);
|
||||
|
||||
if (r.also && r.also.length) {
|
||||
var details = el('details', 'mdgrep-also');
|
||||
details.appendChild(el('summary', null, L.alsoIn(r.also.length)));
|
||||
var ul = el('ul');
|
||||
r.also.forEach(function (other) {
|
||||
var item = el('li');
|
||||
var a = link(other, null);
|
||||
var m = other.meta || {};
|
||||
a.textContent = o.alsoText ? o.alsoText(other) : (o.dateField && m[o.dateField]) || other.title || other.id;
|
||||
item.appendChild(a);
|
||||
ul.appendChild(item);
|
||||
});
|
||||
details.appendChild(ul);
|
||||
li.appendChild(details);
|
||||
}
|
||||
return li;
|
||||
}
|
||||
|
||||
function search(append) {
|
||||
var q = input.value.trim();
|
||||
if (controller) controller.abort();
|
||||
clearTimeout(timer);
|
||||
updateUrl(q);
|
||||
if (!append) { list.textContent = ''; more.hidden = true; nextOffset = null; }
|
||||
if (q.length < o.minChars) {
|
||||
setStatus(q ? L.minChars(o.minChars) : '');
|
||||
return Promise.resolve(null);
|
||||
}
|
||||
|
||||
var url = new URL(o.endpoint, global.location.href);
|
||||
url.searchParams.set('q', q);
|
||||
var values = filterValues();
|
||||
Object.keys(values).forEach(function (name) { if (values[name]) url.searchParams.set(name, values[name]); });
|
||||
url.searchParams.set('limit', o.limit);
|
||||
if (append && nextOffset) url.searchParams.set('offset', nextOffset);
|
||||
|
||||
controller = new AbortController();
|
||||
var current = controller;
|
||||
if (!append) setStatus(L.searching, 'busy');
|
||||
root.classList.add('mdgrep-busy');
|
||||
|
||||
return fetch(url, { signal: current.signal, headers: { Accept: 'application/json' } })
|
||||
.then(function (resp) {
|
||||
if (resp.status === 429) throw { userMessage: L.rateLimited };
|
||||
return resp.json().then(function (data) {
|
||||
if (!resp.ok) {
|
||||
var err = data && data.error;
|
||||
if (!err) throw { userMessage: L.failed };
|
||||
throw { userMessage: L.errorAt(L.errors[err.code] || err.message, err.position), position: err.position };
|
||||
}
|
||||
return data;
|
||||
}, function () { throw { userMessage: L.failed }; });
|
||||
})
|
||||
.then(function (data) {
|
||||
if (current !== controller) return data;
|
||||
data.results.forEach(function (r) { list.appendChild(renderGroup(r)); });
|
||||
nextOffset = data.next_offset;
|
||||
more.hidden = nextOffset == null;
|
||||
if (!data.total_groups) setStatus(L.none);
|
||||
else setStatus(L.count(data.total_groups, data.total_blocks, data.elapsed_ms) +
|
||||
(data.truncated ? ' · ' + L.truncated : ''), data.truncated ? 'warn' : null);
|
||||
root.dispatchEvent(new CustomEvent('mdgrep:results', { detail: data }));
|
||||
return data;
|
||||
})
|
||||
.catch(function (e) {
|
||||
if (e && e.name === 'AbortError') return null;
|
||||
setStatus((e && e.userMessage) || L.failed, 'error');
|
||||
return null;
|
||||
})
|
||||
.then(function (data) {
|
||||
if (current === controller) root.classList.remove('mdgrep-busy');
|
||||
return data;
|
||||
});
|
||||
}
|
||||
|
||||
input.addEventListener('input', function () {
|
||||
if (!o.live) return;
|
||||
clearTimeout(timer);
|
||||
timer = setTimeout(function () { search(false); }, o.debounce);
|
||||
});
|
||||
form.addEventListener('submit', function (e) { e.preventDefault(); search(false); });
|
||||
Object.keys(selects).forEach(function (name) {
|
||||
selects[name].addEventListener('change', function () { search(false); });
|
||||
});
|
||||
more.addEventListener('click', function () { search(true); });
|
||||
|
||||
if (o.autofocus) input.focus();
|
||||
if (input.value.trim()) search(false);
|
||||
|
||||
return {
|
||||
search: function () { return search(false); },
|
||||
setQuery: function (q) { input.value = q; return search(false); },
|
||||
input: input,
|
||||
};
|
||||
}
|
||||
|
||||
function autoMount() {
|
||||
var nodes = document.querySelectorAll('[data-mdgrep]:not(.mdgrep)');
|
||||
Array.prototype.forEach.call(nodes, function (node) {
|
||||
var d = node.dataset;
|
||||
mount(node, {
|
||||
endpoint: d.endpoint || DEFAULTS.endpoint,
|
||||
labels: d.labels || DEFAULTS.labels,
|
||||
filters: d.filters ? JSON.parse(d.filters) : [],
|
||||
dateField: d.dateField || null,
|
||||
badgeFields: d.badgeFields ? d.badgeFields.split(',') : [],
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
global.MdGrep = { mount: mount, labels: LABELS, version: '0.1.0' };
|
||||
|
||||
if (document.readyState === 'loading') document.addEventListener('DOMContentLoaded', autoMount);
|
||||
else autoMount();
|
||||
})(window);
|
||||
161
src/mdgrep/web.py
Normal file
161
src/mdgrep/web.py
Normal file
|
|
@ -0,0 +1,161 @@
|
|||
"""Flask integration: a JSON search endpoint plus the widget's static files.
|
||||
|
||||
from mdgrep import SearchEngine, load_markdown
|
||||
from mdgrep.web import create_app
|
||||
|
||||
engine = SearchEngine(lambda: load_markdown("docs", split="list"))
|
||||
app = create_app(engine, filters=["lang"]) # GET /search?q=...&lang=...
|
||||
|
||||
Or mount into an existing app:
|
||||
|
||||
app.register_blueprint(create_blueprint(engine), url_prefix="/api")
|
||||
|
||||
Requires the optional dependency: pip install "mdgrep[web]".
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Hashable, Iterable
|
||||
|
||||
from flask import Blueprint, Flask, jsonify, request, send_from_directory
|
||||
|
||||
from .engine import Block, SearchEngine
|
||||
from .markdown import SPLIT_MODES, load_markdown
|
||||
from .query import QuerySyntaxError
|
||||
|
||||
__all__ = ["create_blueprint", "create_app", "STATIC_DIR"]
|
||||
|
||||
STATIC_DIR = Path(__file__).parent / "static"
|
||||
|
||||
|
||||
def create_blueprint(
|
||||
engine: SearchEngine,
|
||||
*,
|
||||
filters: Iterable[str] = (),
|
||||
group_by: Callable[[Block], Hashable | None] | None = None,
|
||||
sort_key: Callable[[Block], Any] | None = None,
|
||||
reverse: bool = False,
|
||||
default_limit: int = 30,
|
||||
max_limit: int = 100,
|
||||
time_budget: float = 1.5,
|
||||
snippet_chars: int | None = 700,
|
||||
context: int = 120,
|
||||
max_query_length: int = 300,
|
||||
hide_meta: Iterable[str] = ("markdown",),
|
||||
name: str = "mdgrep",
|
||||
) -> Blueprint:
|
||||
"""Blueprint with `GET /search` and the widget assets under `/mdgrep-static/`.
|
||||
|
||||
Query parameters: q, offset, limit and one parameter per name in `filters`
|
||||
(empty or "all" = no restriction, comma-separated = several values).
|
||||
Errors: HTTP 400 with {"error": {"code", "message", "position"}}.
|
||||
"""
|
||||
bp = Blueprint(name, __name__, static_folder=str(STATIC_DIR), static_url_path="/mdgrep-static")
|
||||
filter_names = list(filters)
|
||||
hidden = set(hide_meta)
|
||||
|
||||
@bp.get("/search")
|
||||
def search():
|
||||
q = request.args.get("q", "").strip()
|
||||
offset = _int_arg("offset", 0, 0, 10**9)
|
||||
limit = _int_arg("limit", default_limit, 1, max_limit)
|
||||
if not q:
|
||||
return jsonify(_empty(offset, limit))
|
||||
if len(q) > max_query_length:
|
||||
return _error("too_long", f"Query longer than {max_query_length} characters", max_query_length)
|
||||
|
||||
selected = {}
|
||||
for fname in filter_names:
|
||||
value = request.args.get(fname, "").strip()
|
||||
if value and value != "all":
|
||||
selected[fname] = value.split(",") if "," in value else value
|
||||
|
||||
try:
|
||||
result = engine.search(q, filters=selected, group_by=group_by, sort_key=sort_key,
|
||||
reverse=reverse, offset=offset, limit=limit,
|
||||
time_budget=time_budget, snippet_chars=snippet_chars,
|
||||
context=context)
|
||||
except QuerySyntaxError as e:
|
||||
return _error(e.code, e.message, e.position)
|
||||
|
||||
data = result.to_dict()
|
||||
if hidden:
|
||||
for r in data["results"]:
|
||||
for b in [r["block"], *r["also"]]:
|
||||
for key in hidden:
|
||||
b["meta"].pop(key, None)
|
||||
return jsonify(data)
|
||||
|
||||
return bp
|
||||
|
||||
|
||||
def create_app(engine: SearchEngine, *, url_prefix: str = "", static_dir: str | Path | None = None,
|
||||
**blueprint_options) -> Flask:
|
||||
"""Standalone Flask app. With `static_dir`, files from that directory are served at `/`
|
||||
(handy for local testing of a static site together with its search API)."""
|
||||
app = Flask(__name__, static_folder=None)
|
||||
app.json.ensure_ascii = False
|
||||
app.register_blueprint(create_blueprint(engine, **blueprint_options), url_prefix=url_prefix or None)
|
||||
|
||||
if static_dir is not None:
|
||||
root = Path(static_dir).resolve()
|
||||
|
||||
@app.get("/", defaults={"path": "index.html"})
|
||||
@app.get("/<path:path>")
|
||||
def static_files(path):
|
||||
if (root / path).is_dir():
|
||||
path = f"{path.rstrip('/')}/index.html"
|
||||
return send_from_directory(root, path)
|
||||
|
||||
return app
|
||||
|
||||
|
||||
def _int_arg(name, default, lo, hi):
|
||||
try:
|
||||
value = int(request.args.get(name, default))
|
||||
except (TypeError, ValueError):
|
||||
value = default
|
||||
return max(lo, min(hi, value))
|
||||
|
||||
|
||||
def _empty(offset, limit):
|
||||
return {"query": "", "total_groups": 0, "total_blocks": 0, "offset": offset, "limit": limit,
|
||||
"next_offset": None, "truncated": False, "elapsed_ms": 0, "results": []}
|
||||
|
||||
|
||||
def _error(code, message, position):
|
||||
return jsonify({"error": {"code": code, "message": message, "position": position}}), 400
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
"""Development server: python -m mdgrep.web --dir docs [--static site] [--port 8000]"""
|
||||
p = argparse.ArgumentParser(prog="python -m mdgrep.web", description="mdgrep development server")
|
||||
p.add_argument("--dir", required=True, help="directory with Markdown files")
|
||||
p.add_argument("--split", choices=SPLIT_MODES, default="paragraph")
|
||||
p.add_argument("--static", help="serve this directory at / (API then lives under --api-prefix)")
|
||||
p.add_argument("--api-prefix", default="/api")
|
||||
p.add_argument("--filters", default="", help="comma-separated meta keys usable as filters")
|
||||
p.add_argument("--host", default="127.0.0.1")
|
||||
p.add_argument("--port", type=int, default=8000)
|
||||
args = p.parse_args(argv)
|
||||
|
||||
md_dir = Path(args.dir)
|
||||
engine = SearchEngine(lambda: load_markdown(md_dir, split=args.split, relative_to=md_dir),
|
||||
reload_check=lambda: _fingerprint(md_dir), reload_interval=5)
|
||||
filters = [f for f in args.filters.split(",") if f]
|
||||
app = create_app(engine, url_prefix=args.api_prefix if args.static else "",
|
||||
static_dir=args.static, filters=filters)
|
||||
print(f"mdgrep: {len(engine.blocks)} blocks from {md_dir}")
|
||||
app.run(host=args.host, port=args.port, debug=False)
|
||||
|
||||
|
||||
def _fingerprint(directory: Path):
|
||||
stats = [p.stat() for p in directory.rglob("*.md")]
|
||||
return len(stats), max((s.st_mtime for s in stats), default=0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
60
tests/test_cli.py
Normal file
60
tests/test_cli.py
Normal file
|
|
@ -0,0 +1,60 @@
|
|||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
from mdgrep.cli import main
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def docs(tmp_path):
|
||||
(tmp_path / "a.md").write_text("# A\n\nRust is fast.\n\nGo is simple.\n", encoding="utf-8")
|
||||
(tmp_path / "b.md").write_text("- rust crypto wallet\n- python typing\n", encoding="utf-8")
|
||||
return tmp_path
|
||||
|
||||
|
||||
def run(capsys, *args):
|
||||
code = main([*args, "--color", "never"])
|
||||
out, err = capsys.readouterr()
|
||||
return code, out, err
|
||||
|
||||
|
||||
def test_match_output_and_exit_code(capsys, docs):
|
||||
code, out, _ = run(capsys, "rust -crypto", str(docs))
|
||||
assert code == 0
|
||||
assert f"{docs / 'a.md'}:3" in out
|
||||
assert "Rust is fast." in out
|
||||
assert "crypto" not in out
|
||||
|
||||
|
||||
def test_no_match_exit_1(capsys, docs):
|
||||
code, out, _ = run(capsys, "haskell", str(docs))
|
||||
assert code == 1 and out == ""
|
||||
|
||||
|
||||
def test_syntax_error_exit_2_with_caret(capsys, docs):
|
||||
code, _, err = run(capsys, "(rust", str(docs))
|
||||
assert code == 2
|
||||
assert "Missing closing parenthesis" in err and "^" in err
|
||||
|
||||
|
||||
def test_count_and_files(capsys, docs):
|
||||
assert run(capsys, "rust OR python", str(docs))[0] == 0
|
||||
_, out, _ = run(capsys, "rust OR python", str(docs), "--split", "list", "-c")
|
||||
assert out.strip() == "3"
|
||||
_, out, _ = run(capsys, "rust", str(docs), "-l")
|
||||
assert out.split() == [str(docs / "a.md"), str(docs / "b.md")]
|
||||
|
||||
|
||||
def test_json_output(capsys, docs):
|
||||
code, out, _ = run(capsys, "/fa.t/", str(docs), "--json")
|
||||
data = json.loads(out)
|
||||
assert code == 0 and data["total_blocks"] == 1
|
||||
block = data["results"][0]["block"]
|
||||
assert {"t": "fast", "hit": True} in block["segments"]
|
||||
assert "markdown" not in block["meta"]
|
||||
|
||||
|
||||
def test_color_highlight(capsys, docs):
|
||||
main(["rust", str(docs / "a.md"), "--color", "always"])
|
||||
out = capsys.readouterr().out
|
||||
assert "\033[1;31mRust\033[0m" in out
|
||||
116
tests/test_engine.py
Normal file
116
tests/test_engine.py
Normal file
|
|
@ -0,0 +1,116 @@
|
|||
import pytest
|
||||
|
||||
from mdgrep import Block, QuerySyntaxError, SearchEngine
|
||||
|
||||
|
||||
def blk(i, text, **meta):
|
||||
return Block(id=str(i), text=text, title=f"T{i} {text.split()[0]}", meta=meta)
|
||||
|
||||
|
||||
BLOCKS = [
|
||||
blk(1, "rust compiler news", lang="en", date="2026-01-01", url="u1"),
|
||||
blk(2, "rust compiler news again", lang="en", date="2026-02-01", url="u1"),
|
||||
blk(3, "Rust Übersetzer", lang="de", date="2026-01-15", url="u1"),
|
||||
blk(4, "go generics", lang="en", date="2026-03-01", url="u2"),
|
||||
blk(5, "python typing and rust bindings", lang="en", date="2025-12-01", url="u3"),
|
||||
]
|
||||
|
||||
|
||||
def ids(result):
|
||||
return [g.block.id for g in result.groups]
|
||||
|
||||
|
||||
def test_basic_search_original_order():
|
||||
res = SearchEngine(BLOCKS).search("rust")
|
||||
assert ids(res) == ["1", "2", "3", "5"]
|
||||
assert res.total_blocks == 4 and res.total_groups == 4
|
||||
|
||||
|
||||
def test_filters_single_and_multiple_values():
|
||||
eng = SearchEngine(BLOCKS)
|
||||
assert ids(eng.search("rust", filters={"lang": "de"})) == ["3"]
|
||||
assert ids(eng.search("rust", filters={"lang": ["de", "en"]})) == ["1", "2", "3", "5"]
|
||||
assert ids(eng.search("rust", filters={"lang": ""})) == ["1", "2", "3", "5"]
|
||||
|
||||
|
||||
def test_predicate():
|
||||
res = SearchEngine(BLOCKS).search("rust", predicate=lambda b: b.id != "1")
|
||||
assert ids(res) == ["2", "3", "5"]
|
||||
|
||||
|
||||
def test_sort_newest_first():
|
||||
res = SearchEngine(BLOCKS).search("rust", sort_key=lambda b: b.meta["date"], reverse=True)
|
||||
assert ids(res) == ["2", "3", "1", "5"]
|
||||
|
||||
|
||||
def test_grouping_uses_first_block_after_sorting():
|
||||
res = SearchEngine(BLOCKS).search(
|
||||
"rust", sort_key=lambda b: b.meta["date"], reverse=True,
|
||||
group_by=lambda b: (b.meta["lang"], b.meta["url"]))
|
||||
assert ids(res) == ["2", "3", "5"]
|
||||
assert [b.id for b in res.groups[0].also] == ["1"]
|
||||
assert res.total_groups == 3 and res.total_blocks == 4
|
||||
|
||||
|
||||
def test_title_is_searched():
|
||||
eng = SearchEngine([Block("t", "body text", title="Special Title")])
|
||||
res = eng.search("special")
|
||||
assert ids(res) == ["t"]
|
||||
assert res.groups[0].title_segments == [{"t": "Special", "hit": True}, {"t": " Title", "hit": False}]
|
||||
assert res.groups[0].hits == 0 # hits counts snippet matches
|
||||
|
||||
|
||||
def test_paging():
|
||||
eng = SearchEngine(BLOCKS)
|
||||
first = eng.search("rust", limit=2)
|
||||
assert ids(first) == ["1", "2"] and first.next_offset == 2
|
||||
second = eng.search("rust", limit=2, offset=first.next_offset)
|
||||
assert ids(second) == ["3", "5"] and second.next_offset is None
|
||||
|
||||
|
||||
def test_segments_and_title_segments():
|
||||
res = SearchEngine(BLOCKS).search("compiler")
|
||||
d = res.to_dict()["results"][0]["block"]
|
||||
assert {"t": "compiler", "hit": True} in d["segments"]
|
||||
assert d["hits"] == 1
|
||||
assert d["title_segments"] == [{"t": "T1 rust", "hit": False}]
|
||||
|
||||
|
||||
def test_to_dict_contract():
|
||||
d = SearchEngine(BLOCKS).search("go", limit=10).to_dict()
|
||||
assert set(d) == {"query", "total_groups", "total_blocks", "offset", "limit",
|
||||
"next_offset", "truncated", "elapsed_ms", "results"}
|
||||
r = d["results"][0]
|
||||
assert set(r) == {"block", "also"}
|
||||
assert r["block"]["meta"] == {"lang": "en", "date": "2026-03-01", "url": "u2"}
|
||||
|
||||
|
||||
def test_syntax_error_propagates():
|
||||
with pytest.raises(QuerySyntaxError):
|
||||
SearchEngine(BLOCKS).search("(rust")
|
||||
|
||||
|
||||
def test_time_budget_truncates():
|
||||
blocks = [Block(id=str(i), text="a" * 30 + "!") for i in range(50)]
|
||||
res = SearchEngine(blocks).search("/(a|aa)+$/", time_budget=0.3)
|
||||
assert res.truncated
|
||||
assert res.elapsed_ms < 3000
|
||||
|
||||
|
||||
def test_reload_on_token_change():
|
||||
state = {"token": 1, "blocks": [Block("a", "first")]}
|
||||
eng = SearchEngine(lambda: list(state["blocks"]), reload_check=lambda: state["token"],
|
||||
reload_interval=0)
|
||||
assert ids(eng.search("first")) == ["a"]
|
||||
state["blocks"] = [Block("b", "second")]
|
||||
assert ids(eng.search("second")) == [] # token unchanged → no reload
|
||||
state["token"] = 2
|
||||
assert ids(eng.search("second")) == ["b"]
|
||||
|
||||
|
||||
def test_reload_interval_respected():
|
||||
calls = []
|
||||
eng = SearchEngine(lambda: [], reload_check=lambda: calls.append(1) or 1, reload_interval=3600)
|
||||
eng.search("x")
|
||||
eng.search("x")
|
||||
assert len(calls) == 1 # only the initial check
|
||||
119
tests/test_markdown.py
Normal file
119
tests/test_markdown.py
Normal file
|
|
@ -0,0 +1,119 @@
|
|||
import pytest
|
||||
|
||||
from mdgrep.markdown import load_markdown, markdown_to_text, slugify, split_markdown
|
||||
|
||||
DOC = """# Report
|
||||
## Edition
|
||||
|
||||
1. **[Rust 2.0 released](https://example.com/rust)** (100 points by alice)
|
||||
|
||||
The Rust team announced a *major* release with `async` traits.
|
||||
|
||||
2. **[Go generics](https://example.com/go)** (50 points by bob)
|
||||
|
||||
Generics in Go are now __fast__.
|
||||
|
||||
## Insights
|
||||
|
||||
1. **Memory safety wins**
|
||||
|
||||
- **Trend:** Rust adoption grows.
|
||||
- **Why:** fewer bugs.
|
||||
|
||||
2. **Typed languages**
|
||||
lazy continuation line
|
||||
|
||||
A closing paragraph
|
||||
spanning two lines.
|
||||
|
||||
---
|
||||
|
||||
*Footer text*
|
||||
|
||||
```
|
||||
# not a heading
|
||||
- not an item
|
||||
```
|
||||
"""
|
||||
|
||||
|
||||
def kinds(blocks):
|
||||
return [(b.meta["kind"], b.meta["section"], b.line) for b in blocks]
|
||||
|
||||
|
||||
def test_split_list():
|
||||
blocks = split_markdown(DOC, source="doc.md", split="list")
|
||||
assert kinds(blocks) == [
|
||||
("item", "Edition", 4),
|
||||
("item", "Edition", 8),
|
||||
("item", "Insights", 14),
|
||||
("item", "Insights", 19),
|
||||
("paragraph", "Insights", 22),
|
||||
("paragraph", "Insights", 27),
|
||||
("paragraph", "Insights", 29),
|
||||
]
|
||||
first = blocks[0]
|
||||
assert first.text == ("Rust 2.0 released (100 points by alice)\n\n"
|
||||
"The Rust team announced a major release with async traits.")
|
||||
assert first.title == ""
|
||||
assert first.id == "doc.md:4" and first.source == "doc.md"
|
||||
assert first.meta["markdown"].startswith("1. **[Rust 2.0 released]")
|
||||
assert "Trend: Rust adoption grows." in blocks[2].text
|
||||
assert "lazy continuation line" in blocks[3].text
|
||||
assert "not a heading" in blocks[6].text
|
||||
|
||||
|
||||
def test_split_heading():
|
||||
blocks = split_markdown(DOC, split="heading")
|
||||
assert [(b.title, b.anchor) for b in blocks] == [
|
||||
("Report", "report"), ("Edition", "edition"), ("Insights", "insights")]
|
||||
assert "Go generics" in blocks[1].text
|
||||
assert not blocks[1].text.startswith("Edition")
|
||||
|
||||
|
||||
def test_split_paragraph():
|
||||
blocks = split_markdown(DOC, split="paragraph")
|
||||
texts = [b.text.split("\n")[0] for b in blocks]
|
||||
assert "Rust 2.0 released (100 points by alice)" in texts
|
||||
assert "The Rust team announced a major release with async traits." in texts
|
||||
assert "A closing paragraph" in texts
|
||||
fence = [b for b in blocks if "not a heading" in b.text]
|
||||
assert len(fence) == 1 and "not an item" in fence[0].text
|
||||
|
||||
|
||||
def test_split_file():
|
||||
blocks = split_markdown(DOC, source="x.md", split="file")
|
||||
assert len(blocks) == 1 and blocks[0].title == "Report" and blocks[0].line == 1
|
||||
|
||||
|
||||
def test_invalid_split_mode():
|
||||
with pytest.raises(ValueError):
|
||||
split_markdown("x", split="bogus")
|
||||
|
||||
|
||||
def test_markdown_to_text():
|
||||
md = ("- **Bold** and *italic* and __b2__ and `code` and ~~gone~~\n"
|
||||
" [link](https://x.y/a_(b))  <https://auto.link>\n"
|
||||
"> quote with snake_case_name")
|
||||
assert markdown_to_text(md) == (
|
||||
"Bold and italic and b2 and code and gone\n"
|
||||
"link alt https://auto.link\n"
|
||||
"quote with snake_case_name")
|
||||
|
||||
|
||||
def test_slugify():
|
||||
assert slugify("AI/ML Insights & Trends") == "aiml-insights--trends"
|
||||
assert slugify("Über **uns**") == "über-uns"
|
||||
|
||||
|
||||
def test_load_markdown_directory_and_meta(tmp_path):
|
||||
(tmp_path / "sub").mkdir()
|
||||
(tmp_path / "a.md").write_text("first para\n\nsecond para\n", encoding="utf-8")
|
||||
(tmp_path / "sub" / "b.md").write_text("- item one\n- item two\n", encoding="utf-8")
|
||||
(tmp_path / "ignore.txt").write_text("nope", encoding="utf-8")
|
||||
blocks = load_markdown(tmp_path, split="list", meta=lambda p: {"file": p.name})
|
||||
assert [(b.meta["file"], b.text) for b in blocks] == [
|
||||
("a.md", "first para"), ("a.md", "second para"),
|
||||
("b.md", "item one"), ("b.md", "item two")]
|
||||
rel = load_markdown(tmp_path, relative_to=tmp_path)
|
||||
assert [b.id for b in rel] == ["a.md:1", "a.md:3", "sub/b.md:1"]
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue