mdgrep

Boolean full-text search over Markdown (and any other text) – as a Python library, a grep-like command line tool, a Flask JSON API and a framework-free JavaScript search widget.

• substrings, "phrases" and /regular expressions/, always case-insensitive (Unicode-aware)
• AND / OR / NOT with parentheses – German keywords UND / ODER / NICHT work too
• results as "boxes" (blocks) with highlighted context, grouping, sorting, filters and paging
• protection against catastrophic regular expressions (time limit per search)
• no search index: blocks live in memory, a few MB of text are searched in milliseconds

Installation

pip install "mdgrep @ git+https://kitux.de/forgejo/dschlueter/mdgrep.git@v0.1.1"        # library + CLI
pip install "mdgrep[web] @ git+https://kitux.de/forgejo/dschlueter/mdgrep.git@v0.1.1"   # + Flask API

Requires Python ≥ 3.10. Dependencies: regex (and flask for [web]).

Query syntax

Query	Meaning
rust	substring, case-insensitive – also matches Rust, trusted, Rustacean
"open source"	phrase; any whitespace (incl. line breaks) between the words
/llms?\b/	regular expression (\/ for a literal slash), case-insensitive
a b · a AND b · a UND b · a & b	both
a OR b · a ODER b · a | b	at least one
NOT a · NICHT a · -a · !a	must not occur
( … )	grouping

Precedence: NOT > AND > OR, so a OR b AND c means a OR (b AND c). Keywords are only recognised in upper case (and is a normal search word). Whole words: use a regex, e.g. /\bgo\b/.

Examples: (rust OR go) AND NOT crypto · "machine learning" -/pytorch|tensorflow/ · llm (privacy OR datenschutz)

Invalid queries raise QuerySyntaxError with a stable code, a message and the 0-based position: empty_query, empty_phrase, unterminated_phrase, unterminated_regex, invalid_regex, missing_operand, unexpected_paren, unexpected_token, unclosed_paren, empty_group, too_long, too_many_terms.

Command line

mdgrep '(rust OR go) AND NOT crypto' notes/            # search all *.md below notes/
mdgrep '/llms?\b/' docs/*.md --split list             # one result per list item
mdgrep 'todo' . -l                                     # file names only
mdgrep 'release' . -c                                  # number of matching blocks
mdgrep '"open source"' . --json                        # JSON (same format as the Web API)
python -m mdgrep --help

--split chooses the unit that is searched and shown: paragraph (default), list (top-level list items, other text as paragraphs), heading (sections) or file. Exit codes as with grep: 0 match, 1 no match, 2 error (syntax error, time limit).

Python API

from mdgrep import compile_query, load_markdown, SearchEngine, Block

# 1. Just the query language – on any string
q = compile_query('(rust OR go) AND NOT /crypto(currency)?/')
q.matches("Rust 2.0 is out")        # True
q.spans("Rust and Go")              # [(0, 4), (9, 11)]  – for highlighting

# 2. Markdown files → blocks → search
blocks = load_markdown("docs/", split="list", relative_to="docs/",
                       meta=lambda path: {"lang": path.stem.rsplit("_", 1)[-1]})
engine = SearchEngine(blocks)
result = engine.search("rust", filters={"lang": "de"}, limit=20)
for group in result.groups:
    print(group.block.source, group.block.line, group.segments)
result.to_dict()                     # JSON-ready, see "JSON format"

# 3. Your own data – any text source
engine = SearchEngine([
    Block(id="42", title="Ticket 42", text="Login fails with SSO", meta={"status": "open"}),
    Block(id="43", title="Ticket 43", text="Dark mode toggle broken", meta={"status": "closed"}),
])
engine.search("login OR sso", filters={"status": "open"})

Block

Field	Meaning
id	unique id
text	searched and shown as snippet
title	searched and shown as heading (don't repeat it in text)
anchor	optional fragment id, used by the widget as link target if no href is given
source, line	origin (file, 1-based line)
meta	free dict, passed through unchanged (filters work on it)

load_markdown() adds meta["section"] (nearest heading), meta["kind"] (file/section/item/paragraph) and meta["markdown"] (raw source).

dialect (for split="list") makes blocks line up with the HTML of your Markdown renderer: "commonmark" (default) or "python-markdown" (a list needs a blank line before it; nesting needs 4 spaces of indentation). Important if you link results to element ids in rendered HTML.

SearchEngine(source, *, reload_check=None, reload_interval=60)

source is a list of blocks or a callable returning them. With a callable plus reload_check (returns any comparable token, e.g. newest file mtime) blocks are reloaded automatically when the token changes – checked at most every reload_interval seconds:

engine = SearchEngine(lambda: load_markdown("docs/"),
                      reload_check=lambda: max(p.stat().st_mtime for p in Path("docs").rglob("*.md")))

engine.search(query, **options)

Option	Default	Meaning
filters	None	{meta_key: value or [values]}, all keys must match; empty values are ignored
predicate	None	extra callable(block) -> bool
group_by	None	callable(block) -> key or None; one result per key, the others in also
sort_key, reverse	original order	sort matching blocks (e.g. newest first)
offset, limit	0, 30	paging, counted in groups (limit=None = all)
time_budget	1.5	seconds for the whole search; on overrun truncated=True
snippet_chars, context	700, 120	longer texts are cut to windows around the hits

Web API (Flask)

from mdgrep import SearchEngine, load_markdown
from mdgrep.web import create_app, create_blueprint

engine = SearchEngine(lambda: load_markdown("docs", split="list", relative_to="docs"))
app = create_app(engine, filters=["lang"])                    # standalone: GET /search
# or inside an existing app:
app.register_blueprint(create_blueprint(engine, filters=["lang"]), url_prefix="/api")

GET /search?q=…&offset=0&limit=30&<filter>=value – for each name in filters an equally named parameter (""/all = no restriction, a,b = several values). The blueprint also serves the widget assets under /mdgrep-static/. create_blueprint() accepts the search options above plus default_limit, max_limit, max_query_length and hide_meta (meta keys removed from responses, default ("markdown",)).

Development server (serves a static site and the API together):

python -m mdgrep.web --dir examples/docs --split list --static examples/site --port 8000
# → http://127.0.0.1:8000/  (API: /api/search)

For production run the app with a WSGI server (e.g. gunicorn -w 2 module:app) behind a reverse proxy with rate limiting.

JSON format

{
  "query": "rust", "total_groups": 12, "total_blocks": 15,
  "offset": 0, "limit": 30, "next_offset": null, "truncated": false, "elapsed_ms": 3.2,
  "results": [{
    "block": {
      "id": "a.md:4", "title": "…", "anchor": "", "source": "a.md", "line": 4, "meta": {},
      "title_segments": [{"t": "…", "hit": false}],
      "segments": [{"t": "The ", "hit": false}, {"t": "Rust", "hit": true}, {"t": " team …", "hit": false}],
      "hits": 1
    },
    "also": [{"id": "b.md:9", "title": "…", "anchor": "", "source": "b.md", "line": 9, "meta": {}}]
  }]
}

Errors: HTTP 400 {"error": {"code": "unclosed_paren", "message": "…", "position": 0}}.

JavaScript widget

Framework-free, live search with debounce, cancels outdated requests, keeps the query in the URL (shareable, back button), filters, "load more", collapsible syntax help, error messages with position, English and German labels. All result text is inserted via textContent (no XSS).

<link rel="stylesheet" href="/api/mdgrep-static/mdgrep-widget.css">
<div id="search"></div>
<script src="/api/mdgrep-static/mdgrep-widget.js"></script>
<script>
  MdGrep.mount(document.getElementById('search'), {
    endpoint: '/api/search',
    labels: 'de',                         // 'en' | 'de' | {…overrides}
    filters: [{ name: 'lang', label: 'Sprache',
                options: [{ value: '', label: 'Alle' }, { value: 'de', label: 'Deutsch' }] }],
    badgeFields: ['lang'],                // meta keys shown as badges
    dateField: 'date',                    // meta key shown as date
    hrefField: 'href',                    // meta key with the link target (else #anchor)
  });
</script>

Without JavaScript code: <div data-mdgrep data-endpoint="/api/search" data-labels="de"></div>.

Other options: live, debounce (300 ms), minChars (2), limit (30), syncUrl, autofocus, help, href(block), formatBadge(field, value, block), alsoText(block). The widget fires a mdgrep:results event with the response on its root element.

For static sites copy the assets instead of serving them from Flask:

import mdgrep
mdgrep.export_static("site/static")     # writes mdgrep-widget.js / .css

Theming via CSS custom properties (defaults are light): --mdgrep-text, --mdgrep-muted, --mdgrep-surface, --mdgrep-border, --mdgrep-accent, --mdgrep-mark-bg, --mdgrep-mark-text, --mdgrep-error, --mdgrep-radius.

Development

python -m venv .venv && .venv/bin/pip install -e ".[dev]"
.venv/bin/pytest

License

MIT