feat: mdgrep 0.1.0 – boolean full-text search for Markdown
- query language: substrings, phrases, /regex/, AND/OR/NOT (+ UND/ODER/NICHT), parentheses; case-insensitive; ReDoS protection via regex timeouts; syntax errors with stable code and position - SearchEngine over in-memory blocks: filters, predicate, grouping, sorting, paging, time budget, automatic reload, highlighted snippets - Markdown loader with split modes file/heading/list/paragraph - grep-like CLI (mdgrep), JSON output, grep exit codes - Flask blueprint/app with JSON API and widget assets, dev server - framework-free JS widget: live search, URL state, filters, paging, syntax help, en/de labels, CSS custom properties for theming Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dschlueter
commit
9c7b7e81f9
1 changed files with 2642 additions and 0 deletions
220
README.md
Normal file
220
README.md
Normal file
|
|
@ -0,0 +1,220 @@
|
|||
# mdgrep
|
||||
|
||||
Boolean full-text search over Markdown (and any other text) – as a **Python library**, a
|
||||
**grep-like command line tool**, a **Flask JSON API** and a **framework-free JavaScript search widget**.
|
||||
|
||||
- substrings, `"phrases"` and `/regular expressions/`, always case-insensitive (Unicode-aware)
|
||||
- `AND` / `OR` / `NOT` with parentheses – German keywords `UND` / `ODER` / `NICHT` work too
|
||||
- results as "boxes" (blocks) with highlighted context, grouping, sorting, filters and paging
|
||||
- protection against catastrophic regular expressions (time limit per search)
|
||||
- no search index: blocks live in memory, a few MB of text are searched in milliseconds
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install "mdgrep @ git+https://kitux.de/forgejo/dschlueter/mdgrep.git@v0.1.0" # library + CLI
|
||||
pip install "mdgrep[web] @ git+https://kitux.de/forgejo/dschlueter/mdgrep.git@v0.1.0" # + Flask API
|
||||
```
|
||||
|
||||
Requires Python ≥ 3.10. Dependencies: [`regex`](https://pypi.org/project/regex/) (and `flask` for `[web]`).
|
||||
|
||||
## Query syntax
|
||||
|
||||
| Query | Meaning |
|
||||
|---|---|
|
||||
| `rust` | substring, case-insensitive – also matches *Rust*, *trusted*, *Rustacean* |
|
||||
| `"open source"` | phrase; any whitespace (incl. line breaks) between the words |
|
||||
| `/llms?\b/` | regular expression (`\/` for a literal slash), case-insensitive |
|
||||
| `a b` · `a AND b` · `a UND b` · `a & b` | both |
|
||||
| `a OR b` · `a ODER b` · `a \| b` | at least one |
|
||||
| `NOT a` · `NICHT a` · `-a` · `!a` | must not occur |
|
||||
| `( … )` | grouping |
|
||||
|
||||
Precedence: `NOT` > `AND` > `OR`, so `a OR b AND c` means `a OR (b AND c)`.
|
||||
Keywords are only recognised in upper case (`and` is a normal search word).
|
||||
Whole words: use a regex, e.g. `/\bgo\b/`.
|
||||
|
||||
Examples: `(rust OR go) AND NOT crypto` · `"machine learning" -/pytorch|tensorflow/` · `llm (privacy OR datenschutz)`
|
||||
|
||||
Invalid queries raise `QuerySyntaxError` with a stable `code`, a `message` and the 0-based `position`:
|
||||
`empty_query`, `empty_phrase`, `unterminated_phrase`, `unterminated_regex`, `invalid_regex`,
|
||||
`missing_operand`, `unexpected_paren`, `unexpected_token`, `unclosed_paren`, `empty_group`,
|
||||
`too_long`, `too_many_terms`.
|
||||
|
||||
## Command line
|
||||
|
||||
```bash
|
||||
mdgrep '(rust OR go) AND NOT crypto' notes/ # search all *.md below notes/
|
||||
mdgrep '/llms?\b/' docs/*.md --split list # one result per list item
|
||||
mdgrep 'todo' . -l # file names only
|
||||
mdgrep 'release' . -c # number of matching blocks
|
||||
mdgrep '"open source"' . --json # JSON (same format as the Web API)
|
||||
python -m mdgrep --help
|
||||
```
|
||||
|
||||
`--split` chooses the unit that is searched and shown: `paragraph` (default), `list`
|
||||
(top-level list items, other text as paragraphs), `heading` (sections) or `file`.
|
||||
Exit codes as with grep: `0` match, `1` no match, `2` error (syntax error, time limit).
|
||||
|
||||
## Python API
|
||||
|
||||
```python
|
||||
from mdgrep import compile_query, load_markdown, SearchEngine, Block
|
||||
|
||||
# 1. Just the query language – on any string
|
||||
q = compile_query('(rust OR go) AND NOT /crypto(currency)?/')
|
||||
q.matches("Rust 2.0 is out") # True
|
||||
q.spans("Rust and Go") # [(0, 4), (9, 11)] – for highlighting
|
||||
|
||||
# 2. Markdown files → blocks → search
|
||||
blocks = load_markdown("docs/", split="list", relative_to="docs/",
|
||||
meta=lambda path: {"lang": path.stem.rsplit("_", 1)[-1]})
|
||||
engine = SearchEngine(blocks)
|
||||
result = engine.search("rust", filters={"lang": "de"}, limit=20)
|
||||
for group in result.groups:
|
||||
print(group.block.source, group.block.line, group.segments)
|
||||
result.to_dict() # JSON-ready, see "JSON format"
|
||||
|
||||
# 3. Your own data – any text source
|
||||
engine = SearchEngine([
|
||||
Block(id="42", title="Ticket 42", text="Login fails with SSO", meta={"status": "open"}),
|
||||
Block(id="43", title="Ticket 43", text="Dark mode toggle broken", meta={"status": "closed"}),
|
||||
])
|
||||
engine.search("login OR sso", filters={"status": "open"})
|
||||
```
|
||||
|
||||
### `Block`
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| `id` | unique id |
|
||||
| `text` | searched and shown as snippet |
|
||||
| `title` | searched and shown as heading (don't repeat it in `text`) |
|
||||
| `anchor` | optional fragment id, used by the widget as link target if no `href` is given |
|
||||
| `source`, `line` | origin (file, 1-based line) |
|
||||
| `meta` | free dict, passed through unchanged (filters work on it) |
|
||||
|
||||
`load_markdown()` adds `meta["section"]` (nearest heading), `meta["kind"]`
|
||||
(`file`/`section`/`item`/`paragraph`) and `meta["markdown"]` (raw source).
|
||||
|
||||
### `SearchEngine(source, *, reload_check=None, reload_interval=60)`
|
||||
|
||||
`source` is a list of blocks or a callable returning them. With a callable plus `reload_check`
|
||||
(returns any comparable token, e.g. newest file mtime) blocks are reloaded automatically when
|
||||
the token changes – checked at most every `reload_interval` seconds:
|
||||
|
||||
```python
|
||||
engine = SearchEngine(lambda: load_markdown("docs/"),
|
||||
reload_check=lambda: max(p.stat().st_mtime for p in Path("docs").rglob("*.md")))
|
||||
```
|
||||
|
||||
### `engine.search(query, **options)`
|
||||
|
||||
| Option | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `filters` | `None` | `{meta_key: value or [values]}`, all keys must match; empty values are ignored |
|
||||
| `predicate` | `None` | extra `callable(block) -> bool` |
|
||||
| `group_by` | `None` | `callable(block) -> key or None`; one result per key, the others in `also` |
|
||||
| `sort_key`, `reverse` | original order | sort matching blocks (e.g. newest first) |
|
||||
| `offset`, `limit` | `0`, `30` | paging, counted in groups (`limit=None` = all) |
|
||||
| `time_budget` | `1.5` | seconds for the whole search; on overrun `truncated=True` |
|
||||
| `snippet_chars`, `context` | `700`, `120` | longer texts are cut to windows around the hits |
|
||||
|
||||
## Web API (Flask)
|
||||
|
||||
```python
|
||||
from mdgrep import SearchEngine, load_markdown
|
||||
from mdgrep.web import create_app, create_blueprint
|
||||
|
||||
engine = SearchEngine(lambda: load_markdown("docs", split="list", relative_to="docs"))
|
||||
app = create_app(engine, filters=["lang"]) # standalone: GET /search
|
||||
# or inside an existing app:
|
||||
app.register_blueprint(create_blueprint(engine, filters=["lang"]), url_prefix="/api")
|
||||
```
|
||||
|
||||
`GET /search?q=…&offset=0&limit=30&<filter>=value` – for each name in `filters` an equally
|
||||
named parameter (`""`/`all` = no restriction, `a,b` = several values). The blueprint also serves the
|
||||
widget assets under `/mdgrep-static/`. `create_blueprint()` accepts the search options above plus
|
||||
`default_limit`, `max_limit`, `max_query_length` and `hide_meta` (meta keys removed from responses,
|
||||
default `("markdown",)`).
|
||||
|
||||
Development server (serves a static site and the API together):
|
||||
|
||||
```bash
|
||||
python -m mdgrep.web --dir examples/docs --split list --static examples/site --port 8000
|
||||
# → http://127.0.0.1:8000/ (API: /api/search)
|
||||
```
|
||||
|
||||
For production run the app with a WSGI server (e.g. `gunicorn -w 2 module:app`) behind a reverse
|
||||
proxy with rate limiting.
|
||||
|
||||
### JSON format
|
||||
|
||||
```json
|
||||
{
|
||||
"query": "rust", "total_groups": 12, "total_blocks": 15,
|
||||
"offset": 0, "limit": 30, "next_offset": null, "truncated": false, "elapsed_ms": 3.2,
|
||||
"results": [{
|
||||
"block": {
|
||||
"id": "a.md:4", "title": "…", "anchor": "", "source": "a.md", "line": 4, "meta": {},
|
||||
"title_segments": [{"t": "…", "hit": false}],
|
||||
"segments": [{"t": "The ", "hit": false}, {"t": "Rust", "hit": true}, {"t": " team …", "hit": false}],
|
||||
"hits": 1
|
||||
},
|
||||
"also": [{"id": "b.md:9", "title": "…", "anchor": "", "source": "b.md", "line": 9, "meta": {}}]
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
Errors: HTTP 400 `{"error": {"code": "unclosed_paren", "message": "…", "position": 0}}`.
|
||||
|
||||
## JavaScript widget
|
||||
|
||||
Framework-free, live search with debounce, cancels outdated requests, keeps the query in the URL
|
||||
(shareable, back button), filters, "load more", collapsible syntax help, error messages with
|
||||
position, English and German labels. All result text is inserted via `textContent` (no XSS).
|
||||
|
||||
```html
|
||||
<link rel="stylesheet" href="/api/mdgrep-static/mdgrep-widget.css">
|
||||
<div id="search"></div>
|
||||
<script src="/api/mdgrep-static/mdgrep-widget.js"></script>
|
||||
<script>
|
||||
MdGrep.mount(document.getElementById('search'), {
|
||||
endpoint: '/api/search',
|
||||
labels: 'de', // 'en' | 'de' | {…overrides}
|
||||
filters: [{ name: 'lang', label: 'Sprache',
|
||||
options: [{ value: '', label: 'Alle' }, { value: 'de', label: 'Deutsch' }] }],
|
||||
badgeFields: ['lang'], // meta keys shown as badges
|
||||
dateField: 'date', // meta key shown as date
|
||||
hrefField: 'href', // meta key with the link target (else #anchor)
|
||||
});
|
||||
</script>
|
||||
```
|
||||
|
||||
Without JavaScript code: `<div data-mdgrep data-endpoint="/api/search" data-labels="de"></div>`.
|
||||
|
||||
Other options: `live`, `debounce` (300 ms), `minChars` (2), `limit` (30), `syncUrl`, `autofocus`,
|
||||
`help`, `href(block)`, `formatBadge(field, value, block)`, `alsoText(block)`.
|
||||
The widget fires a `mdgrep:results` event with the response on its root element.
|
||||
|
||||
For static sites copy the assets instead of serving them from Flask:
|
||||
|
||||
```python
|
||||
import mdgrep
|
||||
mdgrep.export_static("site/static") # writes mdgrep-widget.js / .css
|
||||
```
|
||||
|
||||
Theming via CSS custom properties (defaults are light):
|
||||
`--mdgrep-text`, `--mdgrep-muted`, `--mdgrep-surface`, `--mdgrep-border`, `--mdgrep-accent`,
|
||||
`--mdgrep-mark-bg`, `--mdgrep-mark-text`, `--mdgrep-error`, `--mdgrep-radius`.
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
python -m venv .venv && .venv/bin/pip install -e ".[dev]"
|
||||
.venv/bin/pytest
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
Loading…
Reference in a new issue