mdgrep/README.md

# mdgrep

Boolean full-text search over Markdown (and any other text) – as a **Python library**, a
**grep-like command line tool**, a **Flask JSON API** and a **framework-free JavaScript search widget**.

- substrings, `"phrases"` and `/regular expressions/`, always case-insensitive (Unicode-aware)
- `AND` / `OR` / `NOT` with parentheses – German keywords `UND` / `ODER` / `NICHT` work too
- results as "boxes" (blocks) with highlighted context, grouping, sorting, filters and paging
- protection against catastrophic regular expressions (time limit per search)
- no search index: blocks live in memory, a few MB of text are searched in milliseconds

## Documentation

| Document | Content |
|---|---|
| [BEDIENUNGSANLEITUNG.md](BEDIENUNGSANLEITUNG.md) | User manual (German): installation, syntax, CLI, API, widget, operation, troubleshooting |
| [FUNKTIONSWEISE.md](FUNKTIONSWEISE.md) | How it works (German): architecture, data flow, design decisions, limits |
| [CHANGELOG.md](CHANGELOG.md) | Changes per version and what to do when updating |
| [LICENSE](LICENSE) | MIT |

## Installation

```bash
pip install "mdgrep @ git+https://kitux.de/forgejo/dschlueter/mdgrep.git@v0.1.2"        # library + CLI
pip install "mdgrep[web] @ git+https://kitux.de/forgejo/dschlueter/mdgrep.git@v0.1.2"   # + Flask API
```

Requires Python ≥ 3.10. Dependencies: [`regex`](https://pypi.org/project/regex/) (and `flask` for `[web]`).

## Query syntax

| Query | Meaning |
|---|---|
| `rust` | substring, case-insensitive – also matches *Rust*, *trusted*, *Rustacean* |
| `"open source"` | phrase; any whitespace (incl. line breaks) between the words |
| `/llms?\b/` | regular expression (`\/` for a literal slash), case-insensitive |
| `a b` · `a AND b` · `a UND b` · `a & b` | both |
| `a OR b` · `a ODER b` · `a \| b` | at least one |
| `NOT a` · `NICHT a` · `-a` · `!a` | must not occur |
| `( … )` | grouping |

Precedence: `NOT` > `AND` > `OR`, so `a OR b AND c` means `a OR (b AND c)`.
Keywords are only recognised in upper case (`and` is a normal search word).
Whole words: use a regex, e.g. `/\bgo\b/`.

Examples: `(rust OR go) AND NOT crypto` · `"machine learning" -/pytorch|tensorflow/` · `llm (privacy OR datenschutz)`

Invalid queries raise `QuerySyntaxError` with a stable `code`, a `message` and the 0-based `position`:
`empty_query`, `empty_phrase`, `unterminated_phrase`, `unterminated_regex`, `invalid_regex`,
`missing_operand`, `unexpected_paren`, `unexpected_token`, `unclosed_paren`, `empty_group`,
`too_long`, `too_many_terms`.

## Command line

```bash
mdgrep '(rust OR go) AND NOT crypto' notes/            # search all *.md below notes/
mdgrep '/llms?\b/' docs/*.md --split list             # one result per list item
mdgrep 'todo' . -l                                     # file names only
mdgrep 'release' . -c                                  # number of matching blocks
mdgrep '"open source"' . --json                        # JSON (same format as the Web API)
python -m mdgrep --help
```

`--split` chooses the unit that is searched and shown: `paragraph` (default), `list`
(top-level list items, other text as paragraphs), `heading` (sections) or `file`.
Exit codes as with grep: `0` match, `1` no match, `2` error (syntax error, time limit).

## Python API

```python
from mdgrep import compile_query, load_markdown, SearchEngine, Block

# 1. Just the query language – on any string
q = compile_query('(rust OR go) AND NOT /crypto(currency)?/')
q.matches("Rust 2.0 is out")        # True
q.spans("Rust and Go")              # [(0, 4), (9, 11)]  – for highlighting

# 2. Markdown files → blocks → search
blocks = load_markdown("docs/", split="list", relative_to="docs/",
                       meta=lambda path: {"lang": path.stem.rsplit("_", 1)[-1]})
engine = SearchEngine(blocks)
result = engine.search("rust", filters={"lang": "de"}, limit=20)
for group in result.groups:
    print(group.block.source, group.block.line, group.segments)
result.to_dict()                     # JSON-ready, see "JSON format"

# 3. Your own data – any text source
engine = SearchEngine([
    Block(id="42", title="Ticket 42", text="Login fails with SSO", meta={"status": "open"}),
    Block(id="43", title="Ticket 43", text="Dark mode toggle broken", meta={"status": "closed"}),
])
engine.search("login OR sso", filters={"status": "open"})
```

### `Block`

| Field | Meaning |
|---|---|
| `id` | unique id |
| `text` | searched and shown as snippet |
| `title` | searched and shown as heading (don't repeat it in `text`) |
| `anchor` | optional fragment id, used by the widget as link target if no `href` is given |
| `source`, `line` | origin (file, 1-based line) |
| `meta` | free dict, passed through unchanged (filters work on it) |

`load_markdown()` adds `meta["section"]` (nearest heading), `meta["kind"]`
(`file`/`section`/`item`/`paragraph`) and `meta["markdown"]` (raw source).

`dialect` (for `split="list"`) makes blocks line up with the HTML of your Markdown renderer:
`"commonmark"` (default) or `"python-markdown"` (a list needs a blank line before it; nesting
needs 4 spaces of indentation). Important if you link results to element ids in rendered HTML.

### `SearchEngine(source, *, reload_check=None, reload_interval=60)`

`source` is a list of blocks or a callable returning them. With a callable plus `reload_check`
(returns any comparable token, e.g. newest file mtime) blocks are reloaded automatically when
the token changes – checked at most every `reload_interval` seconds:

```python
engine = SearchEngine(lambda: load_markdown("docs/"),
                      reload_check=lambda: max(p.stat().st_mtime for p in Path("docs").rglob("*.md")))
```

### `engine.search(query, **options)`

| Option | Default | Meaning |
|---|---|---|
| `filters` | `None` | `{meta_key: value or [values]}`, all keys must match; empty values are ignored |
| `predicate` | `None` | extra `callable(block) -> bool` |
| `group_by` | `None` | `callable(block) -> key or None`; one result per key, the others in `also` |
| `sort_key`, `reverse` | original order | sort matching blocks (e.g. newest first) |
| `offset`, `limit` | `0`, `30` | paging, counted in groups (`limit=None` = all) |
| `time_budget` | `1.5` | seconds for the whole search; on overrun `truncated=True` |
| `snippet_chars`, `context` | `700`, `120` | longer texts are cut to windows around the hits |

## Web API (Flask)

```python
from mdgrep import SearchEngine, load_markdown
from mdgrep.web import create_app, create_blueprint

engine = SearchEngine(lambda: load_markdown("docs", split="list", relative_to="docs"))
app = create_app(engine, filters=["lang"])                    # standalone: GET /search
# or inside an existing app:
app.register_blueprint(create_blueprint(engine, filters=["lang"]), url_prefix="/api")
```

`GET /search?q=…&offset=0&limit=30&<filter>=value` – for each name in `filters` an equally
named parameter (`""`/`all` = no restriction, `a,b` = several values). The blueprint also serves the
widget assets under `/mdgrep-static/`. `create_blueprint()` accepts the search options above plus
`default_limit`, `max_limit`, `max_query_length` and `hide_meta` (meta keys removed from responses,
default `("markdown",)`).

Development server (serves a static site and the API together):

```bash
python -m mdgrep.web --dir examples/docs --split list --static examples/site --port 8000
# → http://127.0.0.1:8000/  (API: /api/search)
```

For production run the app with a WSGI server (e.g. `gunicorn -w 2 module:app`) behind a reverse
proxy with rate limiting.

### JSON format

```json
{
  "query": "rust", "total_groups": 12, "total_blocks": 15,
  "offset": 0, "limit": 30, "next_offset": null, "truncated": false, "elapsed_ms": 3.2,
  "results": [{
    "block": {
      "id": "a.md:4", "title": "…", "anchor": "", "source": "a.md", "line": 4, "meta": {},
      "title_segments": [{"t": "…", "hit": false}],
      "segments": [{"t": "The ", "hit": false}, {"t": "Rust", "hit": true}, {"t": " team …", "hit": false}],
      "hits": 1
    },
    "also": [{"id": "b.md:9", "title": "…", "anchor": "", "source": "b.md", "line": 9, "meta": {}}]
  }]
}
```

Errors: HTTP 400 `{"error": {"code": "unclosed_paren", "message": "…", "position": 0}}`.

## JavaScript widget

Framework-free, live search with debounce, cancels outdated requests, keeps the query in the URL
(shareable, back button), filters, "load more", collapsible syntax help, error messages with
position, English and German labels. All result text is inserted via `textContent` (no XSS).

```html
<link rel="stylesheet" href="/api/mdgrep-static/mdgrep-widget.css">
<div id="search"></div>
<script src="/api/mdgrep-static/mdgrep-widget.js"></script>
<script>
  MdGrep.mount(document.getElementById('search'), {
    endpoint: '/api/search',
    labels: 'de',                         // 'en' | 'de' | {…overrides}
    filters: [{ name: 'lang', label: 'Sprache',
                options: [{ value: '', label: 'Alle' }, { value: 'de', label: 'Deutsch' }] }],
    badgeFields: ['lang'],                // meta keys shown as badges
    dateField: 'date',                    // meta key shown as date
    hrefField: 'href',                    // meta key with the link target (else #anchor)
  });
</script>
```

Without JavaScript code: `<div data-mdgrep data-endpoint="/api/search" data-labels="de"></div>`.

Other options: `live`, `debounce` (300 ms), `minChars` (2), `limit` (30), `syncUrl`, `autofocus`,
`help`, `href(block)`, `formatBadge(field, value, block)`, `alsoText(block)`.
The widget fires a `mdgrep:results` event with the response on its root element.

For static sites copy the assets instead of serving them from Flask:

```python
import mdgrep
mdgrep.export_static("site/static")     # writes mdgrep-widget.js / .css
```

Theming via CSS custom properties (defaults are light):
`--mdgrep-text`, `--mdgrep-muted`, `--mdgrep-surface`, `--mdgrep-border`, `--mdgrep-accent`,
`--mdgrep-mark-bg`, `--mdgrep-mark-text`, `--mdgrep-error`, `--mdgrep-radius`.

## Development

```bash
python -m venv .venv && .venv/bin/pip install -e ".[dev]"
.venv/bin/pytest
```

## License

MIT