passageway-connector-markdown 0.3.0__tar.gz

passageway_connector_markdown-0.3.0/.gitignore

@@ -0,0 +1,11 @@
+docs/superpowers/
+.superpowers/
+.env
+.DS_Store
+.venv/
+__pycache__/
+.pytest_cache/
+.models/
+dist/
+site/
+.playwright-mcp/

passageway_connector_markdown-0.3.0/PKG-INFO

@@ -0,0 +1,132 @@
+Metadata-Version: 2.5
+Name: passageway-connector-markdown
+Version: 0.3.0
+Summary: passageway connector for a folder of markdown files with front matter
+License-Expression: PolyForm-Noncommercial-1.0.0
+Requires-Python: >=3.12
+Requires-Dist: passageway-sdk~=0.3.0
+Requires-Dist: pyyaml>=6.0
+Description-Content-Type: text/markdown
+
+# passageway-connector-markdown
+
+A [passageway](https://github.com/getpassageway/passageway#readme) connector that turns a folder of markdown files into searchable items: a docs folder, an Obsidian vault, a Jekyll or Hugo site, or the articles of the frozen Wikipedia corpus.
+
+## What it does
+
+- **Each markdown file is one item.** Its id is the path relative to the folder (or an `id` in its front matter) and its version is a hash of the file, so an edit re-indexes just that note.
+- **Front matter gives the metadata:** title, address, date, tags, id and weight (see the table below).
+- **Headings become sections.** Every heading level starts a section whose heading is the trail of its parents, such as `Setup › Linux`, so a hit says where in the note it is.
+- **Folders become the breadcrumb**, and **links between notes become related items:** `[text](other.md)` and `[[Other]]`.
+- **Files that cannot be indexed are skipped, not fatal.** A file that is not UTF-8 text, holds binary data or is too big is skipped with a warning on stderr, and the rest of the folder is indexed.
+
+## Install
+
+Add the connector to the environment that runs passageway (see [Install](https://github.com/getpassageway/passageway/blob/main/docs/install.md)):
+
+```bash
+uv tool install \
+  --with 'passageway-connector-markdown @ git+https://github.com/getpassageway/passageway#subdirectory=connectors/markdown' \
+  git+https://github.com/getpassageway/passageway
+```
+
+## An example
+
+A note `ops/rotating-certificates.md`:
+
+```markdown
+---
+title: Rotating certificates
+tags: [ops, security]
+url: https://wiki.example.org/ops/certs
+---
+# Rotating certificates
+
+Rotate the signing certificate every year.
+
+## Steps
+
+1. Create the new certificate.
+2. Follow [the deploy guide](deploy.md).
+```
+
+becomes this item (shortened):
+
+```json
+{
+  "id": "ops/rotating-certificates.md",
+  "title": "Rotating certificates",
+  "url": "https://wiki.example.org/ops/certs",
+  "path": ["ops"],
+  "labels": ["ops", "security"],
+  "links": ["ops/deploy.md"],
+  "sections": [
+    {
+      "heading": null,
+      "text": "Rotate the signing certificate every year."
+    },
+    {
+      "heading": "Steps",
+      "text": "1. Create the new certificate.\n2. Follow [the deploy guide](deploy.md)."
+    }
+  ]
+}
+```
+
+## Configure
+
+In `passageway.toml` (use the interpreter that has passageway installed, see [Install](https://github.com/getpassageway/passageway/blob/main/docs/install.md)):
+
+```toml
+[[connector]]
+name = "notes"
+command = ["/path/to/python", "-m", "passageway_connector_markdown"]
+env = { FOLDER = "/Users/you/notes", EXCLUDE = "drafts/**|*.draft.md" }
+```
+
+| Setting | Default | Meaning |
+|---|---|---|
+| `FOLDER` | required | The folder to index, including its subfolders |
+| `INCLUDE` | `*.md\|*.markdown` | Glob patterns for the files to index, separated by `\|`. `*` stays within a folder, `**` crosses folders, and a pattern without `/` matches the file name at any depth. Replaces the default |
+| `EXCLUDE` | | Glob patterns for files to leave out, the same syntax |
+| `MAX_FILE_BYTES` | `1048576` | Files larger than this are skipped with a warning. The core accepts at most 4 MiB per item line and the output escapes non-ASCII text, which can triple its size, so keep this at or below 1 MiB unless your notes are plain ASCII |
+
+Always skipped: hidden files and folders (names starting with `.`, such as `.git` and `.obsidian`) and `node_modules`. Symlinked folders are not followed, and a symlinked file counts only if it points inside `FOLDER`.
+
+## Front matter
+
+A file may start with a YAML block between `---` lines. These keys are read; everything else is ignored.
+
+| Key | Used for |
+|---|---|
+| `title` | The title. Without it: the first `# heading`, then the file name |
+| `url` | The address shown with a hit. Without it: the `file://` address of the file |
+| `updated` | The date shown with a hit. Without it: the file's modification time |
+| `tags` | Labels. A list, or a comma-separated string |
+| `id` | A stable id that survives renames and moves. Without it: the relative path. Two files with the same id fail the call and the message names both |
+| `weight` | A multiplier for ranking, from 0 to 2 (default 1) |
+
+All values are read as text, so `no` stays `no` and `2026-10-28` stays a date string. Front matter that is not valid YAML is reported on stderr and the file is indexed without it.
+
+## Behavior
+
+- **Sections:** every heading level (`#` to `######`) starts a section whose heading is the trail of its parents. Text before the first heading is its own section. The first `# heading` is the title (when the front matter has no title, or the same one) and is not repeated; a different `# heading` stays a section heading. A `#` inside a fenced code block is not a heading.
+- **Links:** `[text](other.md#part)` and `[[Other]]` or `[[Other|text]]` become related items when they point at a note in the folder. Wiki links match the file name without case; a name shared by two files links to neither. Links in code are ignored, and the text itself is not changed.
+- **Version:** a hash of the whole file and of the notes it links to, so any change to the file, front matter included, and a link that starts to resolve (its target was created) re-indexes the note.
+- **Bad files:** a file that is not UTF-8 text (a BOM is removed), contains binary data or is over `MAX_FILE_BYTES` is skipped with a warning on stderr, not an error.
+- **`changed`:** notes modified or moved in since the given time, from the later of each file's modification and change time, so edits appear within a minute, also for files copied in with their old dates.
+- **Cost:** every call reads the whole folder, because ids and links need all the notes. A skipped file is reported on every call.
+
+## Not supported
+
+- Headings underlined with `===` or `---` (setext); only `#` headings split. A `#` line inside an HTML comment still counts as a heading.
+- Glob character classes (`[ab]`) and braces (`{a,b}`); `*.md` is case sensitive, so `NOTE.MD` is not found without `INCLUDE=*.MD`. A pattern with a `/` is anchored at `FOLDER`, so `drafts/**` matches only the top-level `drafts`; use `**/drafts/**` for any depth.
+- Reference-style links (`[text][ref]`) and brackets inside link text.
+- File names in a different Unicode form than the link (`Café` typed with a combining accent).
+
+## Test
+
+```bash
+uv sync                                  # installs this connector too
+uv run pytest connectors/markdown        # offline, on temporary folders
+```

passageway_connector_markdown-0.3.0/README.md

@@ -0,0 +1,122 @@
+# passageway-connector-markdown
+
+A [passageway](https://github.com/getpassageway/passageway#readme) connector that turns a folder of markdown files into searchable items: a docs folder, an Obsidian vault, a Jekyll or Hugo site, or the articles of the frozen Wikipedia corpus.
+
+## What it does
+
+- **Each markdown file is one item.** Its id is the path relative to the folder (or an `id` in its front matter) and its version is a hash of the file, so an edit re-indexes just that note.
+- **Front matter gives the metadata:** title, address, date, tags, id and weight (see the table below).
+- **Headings become sections.** Every heading level starts a section whose heading is the trail of its parents, such as `Setup › Linux`, so a hit says where in the note it is.
+- **Folders become the breadcrumb**, and **links between notes become related items:** `[text](other.md)` and `[[Other]]`.
+- **Files that cannot be indexed are skipped, not fatal.** A file that is not UTF-8 text, holds binary data or is too big is skipped with a warning on stderr, and the rest of the folder is indexed.
+
+## Install
+
+Add the connector to the environment that runs passageway (see [Install](https://github.com/getpassageway/passageway/blob/main/docs/install.md)):
+
+```bash
+uv tool install \
+  --with 'passageway-connector-markdown @ git+https://github.com/getpassageway/passageway#subdirectory=connectors/markdown' \
+  git+https://github.com/getpassageway/passageway
+```
+
+## An example
+
+A note `ops/rotating-certificates.md`:
+
+```markdown
+---
+title: Rotating certificates
+tags: [ops, security]
+url: https://wiki.example.org/ops/certs
+---
+# Rotating certificates
+
+Rotate the signing certificate every year.
+
+## Steps
+
+1. Create the new certificate.
+2. Follow [the deploy guide](deploy.md).
+```
+
+becomes this item (shortened):
+
+```json
+{
+  "id": "ops/rotating-certificates.md",
+  "title": "Rotating certificates",
+  "url": "https://wiki.example.org/ops/certs",
+  "path": ["ops"],
+  "labels": ["ops", "security"],
+  "links": ["ops/deploy.md"],
+  "sections": [
+    {
+      "heading": null,
+      "text": "Rotate the signing certificate every year."
+    },
+    {
+      "heading": "Steps",
+      "text": "1. Create the new certificate.\n2. Follow [the deploy guide](deploy.md)."
+    }
+  ]
+}
+```
+
+## Configure
+
+In `passageway.toml` (use the interpreter that has passageway installed, see [Install](https://github.com/getpassageway/passageway/blob/main/docs/install.md)):
+
+```toml
+[[connector]]
+name = "notes"
+command = ["/path/to/python", "-m", "passageway_connector_markdown"]
+env = { FOLDER = "/Users/you/notes", EXCLUDE = "drafts/**|*.draft.md" }
+```
+
+| Setting | Default | Meaning |
+|---|---|---|
+| `FOLDER` | required | The folder to index, including its subfolders |
+| `INCLUDE` | `*.md\|*.markdown` | Glob patterns for the files to index, separated by `\|`. `*` stays within a folder, `**` crosses folders, and a pattern without `/` matches the file name at any depth. Replaces the default |
+| `EXCLUDE` | | Glob patterns for files to leave out, the same syntax |
+| `MAX_FILE_BYTES` | `1048576` | Files larger than this are skipped with a warning. The core accepts at most 4 MiB per item line and the output escapes non-ASCII text, which can triple its size, so keep this at or below 1 MiB unless your notes are plain ASCII |
+
+Always skipped: hidden files and folders (names starting with `.`, such as `.git` and `.obsidian`) and `node_modules`. Symlinked folders are not followed, and a symlinked file counts only if it points inside `FOLDER`.
+
+## Front matter
+
+A file may start with a YAML block between `---` lines. These keys are read; everything else is ignored.
+
+| Key | Used for |
+|---|---|
+| `title` | The title. Without it: the first `# heading`, then the file name |
+| `url` | The address shown with a hit. Without it: the `file://` address of the file |
+| `updated` | The date shown with a hit. Without it: the file's modification time |
+| `tags` | Labels. A list, or a comma-separated string |
+| `id` | A stable id that survives renames and moves. Without it: the relative path. Two files with the same id fail the call and the message names both |
+| `weight` | A multiplier for ranking, from 0 to 2 (default 1) |
+
+All values are read as text, so `no` stays `no` and `2026-10-28` stays a date string. Front matter that is not valid YAML is reported on stderr and the file is indexed without it.
+
+## Behavior
+
+- **Sections:** every heading level (`#` to `######`) starts a section whose heading is the trail of its parents. Text before the first heading is its own section. The first `# heading` is the title (when the front matter has no title, or the same one) and is not repeated; a different `# heading` stays a section heading. A `#` inside a fenced code block is not a heading.
+- **Links:** `[text](other.md#part)` and `[[Other]]` or `[[Other|text]]` become related items when they point at a note in the folder. Wiki links match the file name without case; a name shared by two files links to neither. Links in code are ignored, and the text itself is not changed.
+- **Version:** a hash of the whole file and of the notes it links to, so any change to the file, front matter included, and a link that starts to resolve (its target was created) re-indexes the note.
+- **Bad files:** a file that is not UTF-8 text (a BOM is removed), contains binary data or is over `MAX_FILE_BYTES` is skipped with a warning on stderr, not an error.
+- **`changed`:** notes modified or moved in since the given time, from the later of each file's modification and change time, so edits appear within a minute, also for files copied in with their old dates.
+- **Cost:** every call reads the whole folder, because ids and links need all the notes. A skipped file is reported on every call.
+
+## Not supported
+
+- Headings underlined with `===` or `---` (setext); only `#` headings split. A `#` line inside an HTML comment still counts as a heading.
+- Glob character classes (`[ab]`) and braces (`{a,b}`); `*.md` is case sensitive, so `NOTE.MD` is not found without `INCLUDE=*.MD`. A pattern with a `/` is anchored at `FOLDER`, so `drafts/**` matches only the top-level `drafts`; use `**/drafts/**` for any depth.
+- Reference-style links (`[text][ref]`) and brackets inside link text.
+- File names in a different Unicode form than the link (`Café` typed with a combining accent).
+
+## Test
+
+```bash
+uv sync                                  # installs this connector too
+uv run pytest connectors/markdown        # offline, on temporary folders
+```

passageway_connector_markdown-0.3.0/passageway_connector_markdown/__init__.py

@@ -0,0 +1 @@
+

passageway_connector_markdown-0.3.0/passageway_connector_markdown/__main__.py

@@ -0,0 +1,3 @@
+from passageway_connector_markdown.connector import main
+
+main()

passageway_connector_markdown-0.3.0/passageway_connector_markdown/connector.py

@@ -0,0 +1,244 @@
+"""passageway connector for a folder of markdown files with front matter."""
+
+import hashlib
+import os
+import posixpath
+import re
+from collections.abc import Iterator, Mapping
+from dataclasses import dataclass, replace
+from datetime import UTC, datetime
+from pathlib import Path, PurePosixPath
+from urllib.parse import unquote
+
+import yaml
+from passageway_sdk import Connector, SettingsError, SourceError, first_h1, split_markdown, strip_code
+from passageway_sdk.settings import names, whole
+
+DEFAULT_INCLUDE = ("*.md", "*.markdown")
+SKIP_DIRS = frozenset({"node_modules"})
+FRONT_MATTER = re.compile(r"\A---[ \t]*\r?\n(?:(.*?)\r?\n)?(?:---|\.\.\.)[ \t]*(?:\r?\n|\Z)", re.S)
+# [[wiki link]] or [text](target); no brackets or line breaks inside, which also keeps the matching linear
+LINK = re.compile(r"\[\[([^\[\]\n]+)\]\]|\[[^\[\]\n]*\]\(([^()\n]*)\)")
+SCHEME = re.compile(r"[a-zA-Z][a-zA-Z0-9+.-]*:")
+
+
+def _glob(pattern: str) -> re.Pattern:
+    """A glob over the path relative to the folder. `**` crosses folders; a pattern without `/` matches the file name anywhere."""
+    out, i = "", 0
+    while i < len(pattern):
+        if pattern.startswith("**/", i):
+            out, i = out + "(?:.*/)?", i + 3
+        elif pattern.startswith("**", i):
+            out, i = out + ".*", i + 2
+        elif pattern[i] == "*":
+            out, i = out + "[^/]*", i + 1
+        elif pattern[i] == "?":
+            out, i = out + "[^/]", i + 1
+        else:
+            out, i = out + re.escape(pattern[i]), i + 1
+    return re.compile(("" if "/" in pattern else "(?:.*/)?") + out + r"\Z")
+
+
+@dataclass(frozen=True)
+class Note:
+    rel: str  # the path relative to the folder, with `/`
+    path: Path
+    id: str
+    meta: dict
+    body: str
+    version: str  # the content and the links it resolves to: a link that starts to resolve changes it
+    mtime: float
+    changed: float  # the later of modification and change time: a note moved in keeps its old mtime
+    links: tuple[str, ...] = ()
+
+
+class Markdown(Connector):
+    name = "markdown connector"
+    expected_errors = (OSError,)
+
+    def __init__(self, env: Mapping[str, str]):
+        folder = env.get("FOLDER", "").strip()
+        if not folder or not Path(folder).is_dir():
+            raise SettingsError(f"FOLDER must be an existing folder, got {folder!r}")
+        self.root = Path(folder).resolve()
+        self.include = [_glob(p) for p in names(env, "INCLUDE") or DEFAULT_INCLUDE]
+        self.exclude = [_glob(p) for p in names(env, "EXCLUDE")]
+        self.max_bytes = whole(env, "MAX_FILE_BYTES", 1024 * 1024)
+
+    # -- the protocol ------------------------------------------------------------------------
+
+    def versions(self) -> dict[str, str]:
+        return {note.id: note.version for note in self._notes().values()}
+
+    def fetch(self, ids: list[str]) -> Iterator[dict]:
+        notes = self._notes()
+        for item_id in ids:
+            if item_id in notes:
+                yield self._item(notes[item_id])
+
+    def changed(self, since: datetime) -> Iterator[dict]:
+        # ponytail: every call reads the whole folder (ids and links need it); a cache if a vault ever has tens of thousands of notes
+        for note in self._notes().values():
+            if note.changed >= since.timestamp():
+                yield self._item(note)
+
+    # -- the folder --------------------------------------------------------------------------
+
+    def _wanted(self, rel: str) -> bool:
+        return any(g.match(rel) for g in self.include) and not any(g.match(rel) for g in self.exclude)
+
+    def _paths(self) -> Iterator[tuple[str, Path]]:
+        def unreadable(e: OSError) -> None:
+            self.warn(f"skipped {Path(e.filename).relative_to(self.root).as_posix()}/: {e.strerror}")
+
+        for dirpath, dirnames, filenames in os.walk(self.root, onerror=unreadable):  # does not follow symlinked folders
+            dirnames[:] = sorted(d for d in dirnames if not d.startswith(".") and d not in SKIP_DIRS
+                                 and not os.path.islink(os.path.join(dirpath, d)))
+            for name in sorted(filenames):
+                path = Path(dirpath, name)
+                rel = path.relative_to(self.root).as_posix()
+                if name.startswith(".") or not self._wanted(rel) or not path.is_file():
+                    continue
+                if path.is_symlink() and not path.resolve().is_relative_to(self.root):
+                    continue  # a link out of the folder would index files that are not in it
+                yield rel, path
+
+    def _notes(self) -> dict[str, Note]:
+        notes: dict[str, Note] = {}
+        for rel, path in self._paths():
+            note = self._read(rel, path)
+            if note is None:
+                continue
+            if note.id in notes:
+                raise SourceError(f"two notes have the id {note.id!r}: {notes[note.id].rel} and {rel}")
+            notes[note.id] = note
+        finder = _Links(notes)
+        for item_id, note in notes.items():
+            found = tuple(finder.of(note))
+            digest = hashlib.sha256((note.version + "\n".join(found)).encode()).hexdigest()[:16]
+            notes[item_id] = replace(note, links=found, version=digest)
+        return notes
+
+    def _read(self, rel: str, path: Path) -> Note | None:
+        try:
+            stat = path.stat()
+            if stat.st_size > self.max_bytes:
+                self.warn(f"skipped {rel}: larger than MAX_FILE_BYTES ({self.max_bytes})")
+                return None
+            raw = path.read_bytes()
+        except OSError as e:  # gone since the listing, or unreadable
+            self.warn(f"skipped {rel}: {e.strerror or e}")
+            return None
+        try:
+            if b"\0" in raw:
+                raise UnicodeDecodeError("utf-8", b"", 0, 1, "binary data")
+            text = raw.decode("utf-8-sig")  # also removes a BOM
+        except UnicodeDecodeError:
+            self.warn(f"skipped {rel}: not UTF-8 text")
+            return None
+        meta, body = self._front_matter(rel, text)
+        return Note(rel, path, _text(meta, "id") or rel, meta, body, hashlib.sha256(raw).hexdigest(), stat.st_mtime,
+                    max(stat.st_mtime, stat.st_ctime))
+
+    def _front_matter(self, rel: str, text: str) -> tuple[dict, str]:
+        match = FRONT_MATTER.match(text)
+        if not match:
+            return {}, text
+        body = text[match.end():]
+        try:
+            # BaseLoader keeps every scalar a string, so `no` stays "no" and a date stays "2026-10-28"; it builds no objects
+            meta = yaml.load(match.group(1) or "", Loader=yaml.BaseLoader)
+        except (yaml.YAMLError, RecursionError):  # RecursionError: a few thousand nested brackets
+            self.warn(f"{rel}: front matter is not valid YAML, indexed without it")
+            return {}, body
+        if meta is not None and not isinstance(meta, dict):
+            self.warn(f"{rel}: front matter is not a mapping of keys, indexed without it")
+            return {}, body
+        return meta or {}, body
+
+    # -- the item ----------------------------------------------------------------------------
+
+    def _item(self, note: Note) -> dict:
+        meta = note.meta
+        title = _text(meta, "title")
+        h1 = first_h1(note.body)
+        is_title = h1 is not None and (not title or h1.casefold() == title.casefold())  # then it is not repeated as a heading
+        item = {
+            "id": note.id,
+            "title": title or h1 or PurePosixPath(note.rel).stem,
+            "url": _text(meta, "url") or note.path.as_uri(),
+            "version": note.version,
+            "updated": _text(meta, "updated") or datetime.fromtimestamp(note.mtime, UTC).strftime("%Y-%m-%dT%H:%M:%SZ"),
+            "sections": split_markdown(note.body, skip_first_h1=is_title),
+        }
+        if folders := list(PurePosixPath(note.rel).parent.parts):
+            item["path"] = folders
+        if labels := _tags(meta.get("tags")):
+            item["labels"] = labels
+        if note.links:
+            item["links"] = list(note.links)
+        weight = _text(meta, "weight")
+        if weight is not None:
+            try:
+                value = float(weight)
+                if not 0 <= value <= 2:
+                    raise ValueError
+                item["weight"] = value
+            except ValueError:
+                self.warn(f"{note.rel}: weight must be a number from 0 to 2, got {weight!r}; ignored")
+        return item
+
+
+def _text(meta: dict, key: str) -> str | None:
+    value = meta.get(key)
+    return value.strip() or None if isinstance(value, str) else None
+
+
+def _tags(value) -> list[str]:
+    if isinstance(value, str):
+        value = value.split(",")
+    return [t.strip() for t in value if isinstance(t, str) and t.strip()] if isinstance(value, list) else []
+
+
+class _Links:
+    """Finds the notes a note links to: by path for [text](file.md), by file name for [[Name]]."""
+
+    def __init__(self, notes: dict[str, Note]):
+        self.by_path = {n.rel: n.id for n in notes.values()}
+        stems: dict[str, list[str]] = {}
+        for n in notes.values():
+            stems.setdefault(PurePosixPath(n.rel).stem.casefold(), []).append(n.id)
+        self.by_stem = {stem: ids[0] for stem, ids in stems.items() if len(ids) == 1}  # an ambiguous name links to nothing
+
+    def of(self, note: Note) -> list[str]:
+        found: list[str] = []
+        for wiki, target in LINK.findall(strip_code(note.body)):
+            item_id = self._wiki(wiki) if wiki else self._path(note, target)
+            if item_id and item_id != note.id and item_id not in found:
+                found.append(item_id)
+        return found
+
+    def _wiki(self, wiki: str) -> str | None:
+        target = re.split(r"\\?\|", wiki)[0].split("#")[0].strip()  # `\|` is how a table cell escapes the pipe
+        target = target[:-3] if target.lower().endswith(".md") else target
+        for candidate in (f"{target}.md", f"{target}.markdown"):  # [[folder/Name]] names the folder when the name is ambiguous
+            if candidate in self.by_path:
+                return self.by_path[candidate]
+        return self.by_stem.get(PurePosixPath(target).name.casefold())
+
+    def _path(self, note: Note, target: str) -> str | None:
+        target = target.strip()
+        if target.startswith("<") and ">" in target:  # <a target with spaces> and an optional title after it
+            target = target[1:target.index(">")]
+        else:
+            target = target.split()[0] if target else ""
+        target = unquote(target.split("#")[0].split("?")[0])
+        if not target or SCHEME.match(target):
+            return None
+        joined = target.lstrip("/") if target.startswith("/") else posixpath.join(posixpath.dirname(note.rel), target)
+        resolved = posixpath.normpath(joined)
+        return None if resolved.startswith("..") else self.by_path.get(resolved)
+
+
+def main() -> None:
+    Markdown.main()

passageway_connector_markdown-0.3.0/pyproject.toml

@@ -0,0 +1,22 @@
+[build-system]
+requires = ["hatchling==1.33.0"]
+build-backend = "hatchling.build"
+
+[project]
+name = "passageway-connector-markdown"
+version = "0.3.0"
+description = "passageway connector for a folder of markdown files with front matter"
+readme = "README.md"
+requires-python = ">=3.12"
+license = "PolyForm-Noncommercial-1.0.0"
+dependencies = ["passageway-sdk~=0.3.0", "pyyaml>=6.0"]
+
+[tool.hatch.build.targets.wheel]
+packages = ["passageway_connector_markdown"]
+
+[dependency-groups]
+dev = ["jsonschema>=4.23", "pytest>=8.3", "passageway"]
+
+[tool.uv.sources]
+passageway = { workspace = true }
+passageway-sdk = { workspace = true }

passageway_connector_markdown-0.3.0/tests/conftest.py

@@ -0,0 +1,40 @@
+import io
+import json
+
+import pytest
+
+from passageway_connector_markdown.connector import Markdown
+
+
+class Folder:
+    """A folder of notes and a way to run the connector on it in-process."""
+
+    def __init__(self, root):
+        self.root = root
+
+    def write(self, name, text, *, binary=False):
+        path = self.root / name
+        path.parent.mkdir(parents=True, exist_ok=True)
+        path.write_bytes(text) if binary else path.write_text(text, encoding="utf-8")
+        return path
+
+    def call(self, mode, stdin="", *, extra=(), **settings):
+        out, err = io.StringIO(), io.StringIO()
+        env = {"FOLDER": str(self.root), **settings}
+        code = Markdown.run(["markdown", mode, *extra], env, io.StringIO(stdin), out, err)
+        return code, [json.loads(line) for line in out.getvalue().splitlines()], err.getvalue()
+
+    def fetch(self, *ids, **settings):
+        code, records, err = self.call("fetch", "".join(json.dumps({"id": i}) + "\n" for i in ids), **settings)
+        assert code == 0, err
+        return {r["id"]: r for r in records}
+
+    def ids(self, **settings):
+        code, records, err = self.call("list", **settings)
+        assert code == 0, err
+        return sorted(r["id"] for r in records)
+
+
+@pytest.fixture
+def folder(tmp_path):
+    return Folder(tmp_path)

passageway_connector_markdown-0.3.0/tests/test_markdown_connector.py

@@ -0,0 +1,341 @@
+import os
+import time
+from datetime import datetime, timezone
+
+import pytest
+
+from passageway.protocol import parse_item
+
+
+# ---- which files ----------------------------------------------------------------------------------
+
+def test_markdown_files_are_found_in_subfolders_and_other_files_are_not(folder):
+    folder.write("a.md", "A")
+    folder.write("sub/b.markdown", "B")
+    folder.write("c.txt", "C")
+    folder.write("sub/deeper/d.md", "D")
+    assert folder.ids() == ["a.md", "sub/b.markdown", "sub/deeper/d.md"]
+
+
+def test_hidden_folders_and_files_and_node_modules_are_always_skipped(folder):
+    folder.write("keep.md", "K")
+    folder.write(".git/notes.md", "x")
+    folder.write(".obsidian/plugins/readme.md", "x")
+    folder.write(".hidden.md", "x")
+    folder.write("node_modules/pkg/readme.md", "x")
+    folder.write("sub/.trash/old.md", "x")
+    assert folder.ids() == ["keep.md"]
+
+
+def test_a_symlinked_folder_is_not_followed_and_a_symlink_loop_ends(folder):
+    folder.write("real/a.md", "A")
+    os.symlink(folder.root / "real", folder.root / "alias")
+    os.symlink(folder.root, folder.root / "real" / "loop")
+    assert folder.ids() == ["real/a.md"]
+
+
+def test_a_symlinked_file_counts_only_if_it_points_inside_the_folder(folder, tmp_path_factory):
+    outside = tmp_path_factory.mktemp("outside") / "secret.md"
+    outside.write_text("secret")
+    folder.write("inside.md", "I")
+    os.symlink(folder.root / "inside.md", folder.root / "link-in.md")
+    os.symlink(outside, folder.root / "link-out.md")
+    assert folder.ids() == ["inside.md", "link-in.md"]
+
+
+def test_include_replaces_the_default_and_exclude_removes(folder):
+    for name in ("a.md", "docs/b.md", "docs/c.md", "docs/c.draft.md", "other/d.md", "e.txt"):
+        folder.write(name, "x")
+    assert folder.ids(INCLUDE="docs/**/*.md") == ["docs/b.md", "docs/c.draft.md", "docs/c.md"]
+    assert folder.ids(INCLUDE="docs/*.md|a.md") == ["a.md", "docs/b.md", "docs/c.draft.md", "docs/c.md"]
+    assert folder.ids(EXCLUDE="*.draft.md|other/**") == ["a.md", "docs/b.md", "docs/c.md"]  # no slash: any depth
+    assert folder.ids(INCLUDE="*.txt") == ["e.txt"]
+
+
+# ---- the item --------------------------------------------------------------------------------------
+
+def test_the_item_is_valid_and_has_the_path_as_id_and_a_content_hash_as_version(folder):
+    folder.write("guides/setup.md", "# Setup guide\n\nInstall it.\n")
+    item = folder.fetch("guides/setup.md")["guides/setup.md"]
+    parsed = parse_item(item)  # the core's own parser accepts it
+    assert parsed.id == "guides/setup.md" and parsed.title == "Setup guide"
+    assert parsed.path == ("guides",) and parsed.url.startswith("file://") and parsed.url.endswith("/guides/setup.md")
+    assert [(s.heading, s.text) for s in parsed.sections] == [(None, "Install it.")]
+    folder.write("guides/setup.md", "# Setup guide\n\nInstall it, then run it.\n")
+    assert folder.fetch("guides/setup.md")["guides/setup.md"]["version"] != item["version"]
+
+
+def test_the_version_ignores_nothing_a_change_to_the_front_matter_changes_it(folder):
+    folder.write("a.md", "---\ntags: [x]\n---\nBody")
+    before = folder.fetch("a.md")["a.md"]["version"]
+    folder.write("a.md", "---\ntags: [y]\n---\nBody")
+    assert folder.fetch("a.md")["a.md"]["version"] != before
+
+
+def test_the_title_is_the_front_matter_title_else_the_first_heading_else_the_file_name(folder):
+    folder.write("a.md", "---\ntitle: From front matter\n---\n# Heading\nbody")
+    folder.write("b.md", "intro\n\n# From heading\nbody")
+    folder.write("c-note.md", "just text")
+    got = folder.fetch("a.md", "b.md", "c-note.md")
+    assert [got[i]["title"] for i in ("a.md", "b.md", "c-note.md")] == ["From front matter", "From heading", "c-note"]
+
+
+def test_front_matter_gives_url_updated_tags_weight_and_id(folder):
+    folder.write("a.md", "---\nurl: https://example.org/a\nupdated: 2026-10-05\ntags: [ops, security]\n"
+                         "weight: 1.5\nid: stable-a\n---\nBody")
+    item = folder.fetch("stable-a")["stable-a"]
+    assert (item["url"], item["updated"], item["labels"], item["weight"]) == (
+        "https://example.org/a", "2026-10-05", ["ops", "security"], 1.5)
+
+
+def test_tags_may_be_a_comma_separated_string_and_scalars_stay_text(folder):
+    folder.write("a.md", "---\ntags: ops, security\n---\nBody")
+    folder.write("b.md", "---\ntags:\n  - no\n  - 2026-10-28\n---\nBody")  # YAML 1.1 would make these False and a date
+    got = folder.fetch("a.md", "b.md")
+    assert got["a.md"]["labels"] == ["ops", "security"] and got["b.md"]["labels"] == ["no", "2026-10-28"]
+
+
+def test_updated_defaults_to_the_modification_time_in_utc(folder):
+    path = folder.write("a.md", "Body")
+    os.utime(path, (1_700_000_000, 1_700_000_000))
+    assert folder.fetch("a.md")["a.md"]["updated"] == "2023-11-14T22:13:20Z"
+
+
+def test_the_front_matter_is_not_indexed_as_text(folder):
+    folder.write("a.md", "---\ntitle: T\nsecret_key: zebra-key\n---\nBody text")
+    sections = folder.fetch("a.md")["a.md"]["sections"]
+    assert [s["text"] for s in sections] == ["Body text"]
+
+
+def test_broken_front_matter_warns_and_the_note_is_still_indexed(folder):
+    folder.write("a.md", "---\ntitle: [unclosed\n---\nBody text")
+    code, records, err = folder.call("fetch", '{"id": "a.md"}\n')
+    assert code == 0 and records[0]["title"] == "a" and records[0]["sections"][0]["text"] == "Body text"
+    assert "a.md" in err and "front matter" in err
+
+
+def test_a_front_matter_that_is_not_a_mapping_is_ignored_with_a_warning(folder):
+    folder.write("a.md", "---\n- just\n- a list\n---\nBody")
+    code, records, err = folder.call("fetch", '{"id": "a.md"}\n')
+    assert code == 0 and records[0]["sections"][0]["text"] == "Body" and "front matter" in err
+
+
+def test_an_unclosed_fence_means_no_front_matter(folder):
+    folder.write("a.md", "---\nnot front matter, a rule above text\n")
+    assert folder.fetch("a.md")["a.md"]["sections"][0]["text"].startswith("---")
+
+
+def test_a_bad_weight_is_ignored_with_a_warning(folder):
+    folder.write("a.md", "---\nweight: heavy\n---\nBody")
+    code, records, err = folder.call("fetch", '{"id": "a.md"}\n')
+    assert code == 0 and "weight" not in records[0] and "weight" in err
+
+
+def test_sections_follow_the_headings_and_the_first_h1_is_the_title(folder):
+    folder.write("a.md", "# Guide\nIntro.\n\n## Setup\nInstall.\n\n### Linux\nApt.\n")
+    item = folder.fetch("a.md")["a.md"]
+    assert item["title"] == "Guide"
+    assert [(s["heading"], s["text"]) for s in item["sections"]] == [
+        (None, "Intro."), ("Setup", "Install."), ("Setup › Linux", "Apt.")]
+
+
+# ---- ids ------------------------------------------------------------------------------------------
+
+def test_two_notes_with_the_same_id_fail_the_call_naming_both(folder):
+    folder.write("a.md", "---\nid: same\n---\nA")
+    folder.write("sub/b.md", "---\nid: same\n---\nB")
+    code, records, err = folder.call("list")
+    assert code == 1 and records == [] and "same" in err and "a.md" in err and "sub/b.md" in err
+
+
+# ---- links ----------------------------------------------------------------------------------------
+
+def test_links_to_other_notes_become_ids_of_existing_files(folder):
+    folder.write("guides/setup.md", "Setup")
+    folder.write("guides/usage.md", "Usage")
+    folder.write("index.md", "See [setup](guides/setup.md#linux), [usage](guides/usage.md?x=1), "
+                             "[web](https://example.org/x.md), [gone](missing.md), [anchor](#top), "
+                             "and [encoded](guides/set%75p.md).")
+    assert folder.fetch("index.md")["index.md"]["links"] == ["guides/setup.md", "guides/usage.md"]
+
+
+def test_relative_links_resolve_from_the_notes_own_folder(folder):
+    folder.write("a/one.md", "[two](../b/two.md) and [three](./three.md)")
+    folder.write("b/two.md", "Two")
+    folder.write("a/three.md", "Three")
+    assert folder.fetch("a/one.md")["a/one.md"]["links"] == ["b/two.md", "a/three.md"]
+
+
+def test_wiki_links_match_the_file_name_ignoring_case_and_skip_ambiguous_names(folder):
+    folder.write("Setup.md", "S")
+    folder.write("x/Dup.md", "D1")
+    folder.write("y/dup.md", "D2")
+    folder.write("n.md", "[[setup]] [[Setup|the guide]] [[Dup]] [[Nothing]] [[Setup#Linux]]")
+    assert folder.fetch("n.md")["n.md"]["links"] == ["Setup.md"]
+
+
+def test_links_use_front_matter_ids_and_ignore_code(folder):
+    folder.write("target.md", "---\nid: stable-target\n---\nT")
+    folder.write("other.md", "O")
+    folder.write("n.md", "[t](target.md)\n```\n[o](other.md)\n```\nand `[o](other.md)` inline")
+    assert folder.fetch("n.md")["n.md"]["links"] == ["stable-target"]
+
+
+def test_a_note_does_not_link_to_itself(folder):
+    folder.write("a.md", "[me](a.md) [[a]]")
+    assert "links" not in folder.fetch("a.md")["a.md"]
+
+
+# ---- bad files --------------------------------------------------------------------------------------
+
+def test_a_bom_is_removed(folder):
+    folder.write("a.md", b"\xef\xbb\xbf# Title\n\nBody", binary=True)
+    item = folder.fetch("a.md")["a.md"]
+    assert item["title"] == "Title" and item["sections"][0]["text"] == "Body"
+
+
+def test_files_that_are_not_text_are_skipped_with_a_warning(folder):
+    folder.write("ok.md", "fine")
+    folder.write("latin.md", "caf\xe9".encode("latin-1"), binary=True)
+    folder.write("binary.md", b"abc\x00def", binary=True)
+    code, records, err = folder.call("list")
+    assert code == 0 and [r["id"] for r in records] == ["ok.md"]
+    assert "latin.md" in err and "binary.md" in err
+
+
+def test_a_file_over_the_limit_is_skipped_with_a_warning(folder):
+    folder.write("small.md", "x" * 100)
+    folder.write("big.md", "x" * 101)
+    code, records, err = folder.call("list", MAX_FILE_BYTES="100")
+    assert code == 0 and [r["id"] for r in records] == ["small.md"] and "big.md" in err
+
+
+def test_a_file_that_vanishes_between_list_and_fetch_is_skipped(folder):
+    path = folder.write("a.md", "A")
+    folder.write("b.md", "B")
+    assert folder.ids() == ["a.md", "b.md"]
+    path.unlink()
+    assert list(folder.fetch("a.md", "b.md")) == ["b.md"]
+
+
+def test_non_ascii_text_is_written_ascii_safe(folder):
+    folder.write("a.md", "# Köln\n\nGröße", )
+    code, records, _ = folder.call("fetch", '{"id": "a.md"}\n')
+    assert records[0]["title"] == "Köln" and records[0]["sections"][0]["text"] == "Größe"
+
+
+# ---- check and changed ------------------------------------------------------------------------------
+
+def test_check_counts_the_notes(folder):
+    folder.write("a.md", "A")
+    folder.write("b.md", "B")
+    assert folder.call("check")[:2] == (0, [{"ok": True, "count": 2, "protocol": "1"}])
+
+
+def test_changed_returns_the_notes_modified_or_moved_in_since(folder):
+    soon = int(time.time()) + 3600  # whole seconds: a fraction can be stored a few nanoseconds short
+    old = folder.write("old.md", "Old")
+    new = folder.write("new.md", "New")
+    os.utime(old, (1_700_000_000, 1_700_000_000))  # an old mtime, but its ctime is now: it was just put here
+    os.utime(new, (soon, soon))
+    now = datetime.fromtimestamp(time.time() - 60, timezone.utc).isoformat()
+    assert sorted(r["id"] for r in folder.call("changed", extra=["--since", now])[1]) == ["new.md", "old.md"]
+    later = datetime.fromtimestamp(time.time() + 60, timezone.utc).isoformat()  # after the move, before the mtime of new
+    assert [r["id"] for r in folder.call("changed", extra=["--since", later])[1]] == ["new.md"]
+    exact = datetime.fromtimestamp(soon, timezone.utc).isoformat()
+    assert [r["id"] for r in folder.call("changed", extra=["--since", exact])[1]] == ["new.md"]  # inclusive
+
+
+# ---- settings ---------------------------------------------------------------------------------------
+
+@pytest.mark.parametrize("settings,message", [
+    ({"FOLDER": ""}, "FOLDER"),
+    ({"FOLDER": "/no/such/folder"}, "FOLDER"),
+    ({"MAX_FILE_BYTES": "lots"}, "MAX_FILE_BYTES"),
+    ({"MAX_FILE_BYTES": "0"}, "MAX_FILE_BYTES"),
+])
+def test_bad_settings_exit_2_with_a_message_and_no_output(folder, settings, message):
+    import io
+
+    from passageway_connector_markdown.connector import Markdown
+    out, err = io.StringIO(), io.StringIO()
+    env = {"FOLDER": str(folder.root), **settings}
+    assert Markdown.run(["markdown", "list"], env, io.StringIO(), out, err) == 2
+    assert out.getvalue() == "" and message in err.getvalue()
+
+
+# ---- review findings ------------------------------------------------------------------------------
+
+def test_deeply_nested_front_matter_warns_and_does_not_fail_the_source(folder):
+    folder.write("deep.md", "---\nx: " + "[" * 5000 + "]" * 5000 + "\n---\nBody")
+    folder.write("good.md", "Good")
+    code, records, err = folder.call("list")
+    assert code == 0 and sorted(r["id"] for r in records) == ["deep.md", "good.md"] and "deep.md" in err
+
+
+@pytest.mark.parametrize("text", ["[" * 300_000, "[[" * 150_000, "[a](" * 75_000])
+def test_a_pathological_line_does_not_make_link_finding_slow(folder, text):
+    folder.write("a.md", text)
+    started = time.monotonic()
+    assert folder.fetch("a.md")["a.md"]["id"] == "a.md"
+    assert time.monotonic() - started < 5
+
+
+def test_the_version_changes_when_a_link_starts_to_resolve(folder):
+    folder.write("a.md", "see [[New Idea]] and [x](other.md)")
+    before = folder.call("list")[1][0]["version"]
+    folder.write("New Idea.md", "N")
+    folder.write("other.md", "O")
+    after = {r["id"]: r["version"] for r in folder.call("list")[1]}
+    assert after["a.md"] != before
+    assert folder.fetch("a.md")["a.md"]["links"] == ["New Idea.md", "other.md"]
+
+
+def test_an_h1_that_is_not_the_title_stays_a_heading(folder):
+    folder.write("a.md", "---\ntitle: Front\n---\n# Intro\ntext\n## Setup\nx")
+    item = folder.fetch("a.md")["a.md"]
+    assert item["title"] == "Front"
+    assert [(s["heading"], s["text"]) for s in item["sections"]] == [("Intro", "text"), ("Intro › Setup", "x")]
+
+
+@pytest.mark.skipif(os.geteuid() == 0, reason="root can read anything")
+def test_an_unreadable_folder_is_reported_not_silently_dropped(folder):
+    folder.write("top.md", "T")
+    folder.write("locked/a.md", "A")
+    (folder.root / "locked").chmod(0)
+    try:
+        code, records, err = folder.call("list")
+    finally:
+        (folder.root / "locked").chmod(0o755)
+    assert code == 0 and [r["id"] for r in records] == ["top.md"] and "locked" in err
+
+
+def test_a_wiki_link_with_a_folder_picks_the_right_one_of_two_same_named_notes(folder):
+    folder.write("a/Note.md", "A")
+    folder.write("b/Note.md", "B")
+    folder.write("n.md", "[[a/Note]] and [[b/Note|the other]] and [[Note]]")
+    assert folder.fetch("n.md")["n.md"]["links"] == ["a/Note.md", "b/Note.md"]
+
+
+def test_a_link_target_in_angle_brackets_may_contain_spaces(folder):
+    folder.write("my note.md", "M")
+    folder.write("n.md", "[x](<my note.md>) and [y](<my note.md> \"title\")")
+    assert folder.fetch("n.md")["n.md"]["links"] == ["my note.md"]
+
+
+def test_an_escaped_pipe_in_a_wiki_link_inside_a_table(folder):
+    folder.write("Setup.md", "S")
+    folder.write("n.md", "| a |\n|---|\n| [[Setup\\|the guide]] |")
+    assert folder.fetch("n.md")["n.md"]["links"] == ["Setup.md"]
+
+
+def test_the_yaml_document_end_marker_closes_the_front_matter(folder):
+    folder.write("a.md", "---\ntitle: T\n...\nBody")
+    item = folder.fetch("a.md")["a.md"]
+    assert item["title"] == "T" and item["sections"][0]["text"] == "Body"
+
+
+def test_an_h1_equal_to_the_front_matter_title_is_not_repeated_in_the_headings(folder):
+    folder.write("a.md", "---\ntitle: Rotating Certificates\n---\n# rotating certificates\nIntro.\n\n## Steps\nx")
+    item = folder.fetch("a.md")["a.md"]
+    assert [(s["heading"], s["text"]) for s in item["sections"]] == [(None, "Intro."), ("Steps", "x")]

passageway_connector_markdown-0.3.0/tests/test_markdown_contract.py

@@ -0,0 +1,80 @@
+import json
+import os
+import re
+import subprocess
+import sys
+from pathlib import Path
+
+import jsonschema
+
+from passageway.protocol import Connector
+from passageway.testing.contract import check_connector
+
+README = Path(__file__).resolve().parents[1] / "README.md"
+SCHEMA = Path(__file__).resolve().parents[3] / "docs" / "reference" / "connector-protocol.schema.json"
+
+
+def notes(root):
+    (root / "guides").mkdir()
+    (root / "a.md").write_text("---\ntitle: Alpha\n---\n# Alpha\nAbout lighthouses.\n\n## Care\nPaint them. See [b](guides/b.md).")
+    (root / "guides" / "b.md").write_text("# Beta\nAbout harbours.")
+
+
+def command():
+    return [sys.executable, "-m", "passageway_connector_markdown"]
+
+
+def test_the_connector_passes_the_shared_contract_as_a_real_subprocess(tmp_path):
+    notes(tmp_path)
+    connector = Connector("markdown", command(), env={"FOLDER": str(tmp_path)})
+    check_connector(connector)
+    items = {i.id: i for i in connector.fetch(["a.md", "guides/b.md"])}
+    assert items["a.md"].links == ("guides/b.md",) and items["a.md"].title == "Alpha"
+
+
+def test_non_ascii_survives_a_subprocess_whose_stdout_is_ascii(tmp_path):
+    (tmp_path / "a.md").write_text("# Köln Süd\n\nGröße und Maße.", encoding="utf-8")
+    connector = Connector("markdown", command(), env={"FOLDER": str(tmp_path), "PYTHONIOENCODING": "ascii"})
+    item = connector.fetch(["a.md"])[0]
+    assert item.title == "Köln Süd" and item.sections[0].text == "Größe und Maße."
+
+
+def test_the_output_validates_against_the_published_json_schema(tmp_path):
+    notes(tmp_path)
+    schema = json.loads(SCHEMA.read_text())
+
+    def valid(name, record):
+        return jsonschema.Draft202012Validator({"$ref": f"#/$defs/{name}", "$defs": schema["$defs"]}).is_valid(record)
+
+    def lines(mode, stdin=""):
+        out = subprocess.run(command() + [mode], env={**os.environ, "FOLDER": str(tmp_path)}, input=stdin,
+                             capture_output=True, text=True, check=True).stdout
+        return [json.loads(line) for line in out.splitlines()]
+
+    assert lines("check") and all(valid("check", r) for r in lines("check"))
+    assert lines("list") and all(valid("listRecord", r) for r in lines("list"))
+    items = lines("fetch", '{"id": "a.md"}\n{"id": "guides/b.md"}\n')
+    assert len(items) == 2 and all(valid("item", r) for r in items)
+
+
+def test_every_setting_is_in_the_readme():
+    code = (Path(__file__).resolve().parents[1] / "passageway_connector_markdown" / "connector.py").read_text()
+    names = sorted(set(re.findall(r'(?:env\.get|names|whole)\(\s*(?:env,\s*)?"([A-Z_]+)"', code)))
+    assert len(names) >= 4, names  # the pattern must actually find the settings
+    readme = README.read_text()
+    for name in names:
+        assert f"`{name}`" in readme, f"{name} is read by the markdown connector but missing from its README"
+
+
+def test_the_example_in_the_readme_is_what_the_connector_really_produces(tmp_path):
+    readme = README.read_text()
+    note = re.search(r"```markdown\n(.*?)```", readme, re.S).group(1)
+    shown = json.loads(re.search(r"```json\n(.*?)```", readme, re.S).group(1))
+    (tmp_path / "ops").mkdir()
+    (tmp_path / "ops" / "rotating-certificates.md").write_text(note)
+    (tmp_path / "ops" / "deploy.md").write_text("# Deploy\nHow to deploy.")
+    out = subprocess.run(command() + ["fetch"], env={**os.environ, "FOLDER": str(tmp_path)},
+                         input='{"id": "ops/rotating-certificates.md"}\n', capture_output=True, text=True, check=True).stdout
+    real = json.loads(out)
+    for key, value in shown.items():
+        assert real[key] == value, f"the README shows {key} = {value!r} but the connector prints {real[key]!r}"