passageway-connector-mkdocs 0.4.0__tar.gz

passageway_connector_mkdocs-0.4.0/.gitignore

@@ -0,0 +1,11 @@
+docs/superpowers/
+.superpowers/
+.env
+.DS_Store
+.venv/
+__pycache__/
+.pytest_cache/
+.models/
+dist/
+site/
+.playwright-mcp/

passageway_connector_mkdocs-0.4.0/PKG-INFO

@@ -0,0 +1,80 @@
+Metadata-Version: 2.5
+Name: passageway-connector-mkdocs
+Version: 0.4.0
+Summary: passageway connector for websites built with MkDocs, read from their search index
+License-Expression: PolyForm-Noncommercial-1.0.0
+Requires-Python: >=3.12
+Requires-Dist: passageway-sdk[html,web]~=0.4.0
+Description-Content-Type: text/markdown
+
+# passageway-connector-mkdocs
+
+A [passageway](https://github.com/getpassageway/passageway#readme) connector that indexes a website built with [MkDocs](https://www.mkdocs.org/) or [Material for MkDocs](https://squidfunk.github.io/mkdocs-material/): someone else's documentation, when you have the site but not its repository. It reads the site's search index, the file its search box uses, so it needs no crawling, and every hit opens the page at its heading.
+
+## What it does
+
+- **Each section of the search index is one item.** A built MkDocs site publishes `search/search_index.json`, with an entry for every page and every heading in it. Each entry with text becomes an item whose address is the page at that heading (`https://docs.example.org/guide/setup/#linux`), so a hit lands on the passage, not at the top of a long page. The text before a page's first heading is an item of its own at the page's address.
+- **The page title is the item's title, the heading is its section,** and the folders of the address are its breadcrumb (`path`), so a `path:` filter works.
+- **The id is the address, the version a hash of the text,** so an edit re-indexes only the sections it touched.
+- **A site address means its search index.** `https://docs.example.org` is read as `https://docs.example.org/search/search_index.json`; a site under a path (`https://user.github.io/project`) works the same, and the index's own address can be given too.
+- **Both search plugins are understood.** Material for MkDocs writes HTML into the index, which becomes text with code blocks and lists kept. MkDocs' own search plugin writes plain text and repeats the whole page in the page's entry; only the part before the first heading is kept there, so a passage is not indexed twice.
+- **Any failure fails the whole listing.** A site without a search index, an index that is not one, a server that is down: the sync stops with a message that names the address, so a half-read site never deletes what was indexed before.
+
+When the site publishes an `llms-full.txt` (the `mkdocs-llmstxt` plugin writes one), the [llms.txt connector](https://getpassageway.com/connectors/llmstxt/) reads the markdown itself; when you have the repository, the [Markdown connector](https://getpassageway.com/connectors/markdown/) does. This connector is for the site as it is published.
+
+## Install
+
+Add the connector to the environment that runs passageway (see [Install](https://github.com/getpassageway/passageway/blob/main/docs/install.md)):
+
+```bash
+uv tool install --python 3.14 "passageway[mkdocs]"
+```
+
+## An example
+
+A site `https://docs.example.org` whose `search/search_index.json` is:
+
+```json
+{"docs": [
+  {"location": "guide/setup/", "title": "Setup", "text": "<p>How to set it up.</p>"},
+  {"location": "guide/setup/#linux", "title": "Linux", "text": "<p>Use the <code>apt</code> package.</p>"}
+]}
+```
+
+gives, for the second entry, the item:
+
+```json
+{
+  "id": "https://docs.example.org/guide/setup/#linux",
+  "title": "Setup",
+  "url": "https://docs.example.org/guide/setup/#linux",
+  "path": ["guide", "setup"],
+  "sections": [
+    {
+      "heading": "Linux",
+      "text": "Use the apt package."
+    }
+  ]
+}
+```
+
+and is configured as:
+
+```toml
+[[connector]]
+name = "vendor-docs"
+command = ["/path/to/python", "-m", "passageway_connector_mkdocs"]
+env = { URLS = "https://docs.example.org|https://user.github.io/project", WEIGHTS = "changelog=0.5" }
+```
+
+## Settings
+
+All settings are strings in the connector's `env`.
+
+| Setting | Meaning | Default |
+|---|---|---|
+| `URLS` | Required. Site addresses, or the addresses of their `search_index.json`, separated by `\|`. | |
+| `MAX_BYTES` | The largest search index that is read, in bytes. A larger one fails the sync. | `50000000` |
+| `WEIGHTS` | Lowers or raises the weight of the sections whose path starts with a prefix: `changelog=0.5,blog/=0.3` (comma-separated `prefix=weight`, the weight above 0, the first match wins). Release notes repeat the vocabulary of how-to pages and push them down the results; a weight below 1 ranks them lower. The path is the page's address without the site's own. | all sections 1.0 |
+
+Requests carry the user agent `passageway-connector-mkdocs`, follow redirects and verify HTTPS against the operating system's trust store.

passageway_connector_mkdocs-0.4.0/README.md

@@ -0,0 +1,71 @@
+# passageway-connector-mkdocs
+
+A [passageway](https://github.com/getpassageway/passageway#readme) connector that indexes a website built with [MkDocs](https://www.mkdocs.org/) or [Material for MkDocs](https://squidfunk.github.io/mkdocs-material/): someone else's documentation, when you have the site but not its repository. It reads the site's search index, the file its search box uses, so it needs no crawling, and every hit opens the page at its heading.
+
+## What it does
+
+- **Each section of the search index is one item.** A built MkDocs site publishes `search/search_index.json`, with an entry for every page and every heading in it. Each entry with text becomes an item whose address is the page at that heading (`https://docs.example.org/guide/setup/#linux`), so a hit lands on the passage, not at the top of a long page. The text before a page's first heading is an item of its own at the page's address.
+- **The page title is the item's title, the heading is its section,** and the folders of the address are its breadcrumb (`path`), so a `path:` filter works.
+- **The id is the address, the version a hash of the text,** so an edit re-indexes only the sections it touched.
+- **A site address means its search index.** `https://docs.example.org` is read as `https://docs.example.org/search/search_index.json`; a site under a path (`https://user.github.io/project`) works the same, and the index's own address can be given too.
+- **Both search plugins are understood.** Material for MkDocs writes HTML into the index, which becomes text with code blocks and lists kept. MkDocs' own search plugin writes plain text and repeats the whole page in the page's entry; only the part before the first heading is kept there, so a passage is not indexed twice.
+- **Any failure fails the whole listing.** A site without a search index, an index that is not one, a server that is down: the sync stops with a message that names the address, so a half-read site never deletes what was indexed before.
+
+When the site publishes an `llms-full.txt` (the `mkdocs-llmstxt` plugin writes one), the [llms.txt connector](https://getpassageway.com/connectors/llmstxt/) reads the markdown itself; when you have the repository, the [Markdown connector](https://getpassageway.com/connectors/markdown/) does. This connector is for the site as it is published.
+
+## Install
+
+Add the connector to the environment that runs passageway (see [Install](https://github.com/getpassageway/passageway/blob/main/docs/install.md)):
+
+```bash
+uv tool install --python 3.14 "passageway[mkdocs]"
+```
+
+## An example
+
+A site `https://docs.example.org` whose `search/search_index.json` is:
+
+```json
+{"docs": [
+  {"location": "guide/setup/", "title": "Setup", "text": "<p>How to set it up.</p>"},
+  {"location": "guide/setup/#linux", "title": "Linux", "text": "<p>Use the <code>apt</code> package.</p>"}
+]}
+```
+
+gives, for the second entry, the item:
+
+```json
+{
+  "id": "https://docs.example.org/guide/setup/#linux",
+  "title": "Setup",
+  "url": "https://docs.example.org/guide/setup/#linux",
+  "path": ["guide", "setup"],
+  "sections": [
+    {
+      "heading": "Linux",
+      "text": "Use the apt package."
+    }
+  ]
+}
+```
+
+and is configured as:
+
+```toml
+[[connector]]
+name = "vendor-docs"
+command = ["/path/to/python", "-m", "passageway_connector_mkdocs"]
+env = { URLS = "https://docs.example.org|https://user.github.io/project", WEIGHTS = "changelog=0.5" }
+```
+
+## Settings
+
+All settings are strings in the connector's `env`.
+
+| Setting | Meaning | Default |
+|---|---|---|
+| `URLS` | Required. Site addresses, or the addresses of their `search_index.json`, separated by `\|`. | |
+| `MAX_BYTES` | The largest search index that is read, in bytes. A larger one fails the sync. | `50000000` |
+| `WEIGHTS` | Lowers or raises the weight of the sections whose path starts with a prefix: `changelog=0.5,blog/=0.3` (comma-separated `prefix=weight`, the weight above 0, the first match wins). Release notes repeat the vocabulary of how-to pages and push them down the results; a weight below 1 ranks them lower. The path is the page's address without the site's own. | all sections 1.0 |
+
+Requests carry the user agent `passageway-connector-mkdocs`, follow redirects and verify HTTPS against the operating system's trust store.

passageway_connector_mkdocs-0.4.0/passageway_connector_mkdocs/__main__.py

@@ -0,0 +1,3 @@
+from passageway_connector_mkdocs.connector import main
+
+main()

passageway_connector_mkdocs-0.4.0/passageway_connector_mkdocs/connector.py

@@ -0,0 +1,146 @@
+"""passageway connector for websites built with MkDocs: one item per section of the site's search index."""
+
+import hashlib
+import json
+import time
+from collections.abc import Callable, Iterator, Mapping
+from dataclasses import dataclass
+from urllib.parse import urldefrag, urlparse
+
+import httpx
+from passageway_sdk import Connector, SettingsError, SourceError
+from passageway_sdk.html import clean, to_markdown
+from passageway_sdk.settings import names, weight_of, weights, whole
+from passageway_sdk.web import TooLarge, Web
+
+INDEX = "search/search_index.json"
+
+
+@dataclass(frozen=True)
+class Section:
+    id: str  # the address: the site, the page, and the heading's anchor if it has one
+    page_title: str
+    heading: str | None  # None for the text before the page's first heading
+    text: str
+    where: str  # the page's location below the site, for WEIGHTS and the breadcrumb
+
+
+def _base(url: str) -> str:
+    """The site an address names: the folder of a search index, else the address itself, ending in `/`."""
+    url = urldefrag(url)[0]
+    if url.endswith(INDEX):
+        return url.removesuffix(INDEX)
+    return url if url.endswith("/") else url + "/"
+
+
+def _intro(page_text: str, first: str | None) -> str:
+    """The page entry of MkDocs' own search plugin holds the whole page; keep what comes before its first section."""
+    if first:
+        at = page_text.find(first)
+        if at > 0:
+            return page_text[:at]
+    return page_text
+
+
+def sections(base: str, docs: list[dict]) -> list[Section]:
+    """The sections of a search index, in its order; entries without text are left out, a repeated address counts once."""
+    pages: dict[str, str] = {}  # page location -> page title
+    firsts: dict[str, tuple[str, str]] = {}  # page location -> (title, text) of its first section with text
+    texts = {d["location"]: to_markdown(d["text"]) for d in docs}
+    for d in docs:
+        page, _, anchor = d["location"].partition("#")
+        if not anchor:
+            pages.setdefault(page, clean(d["title"]))
+        elif texts[d["location"]]:
+            firsts.setdefault(page, (clean(d["title"]), texts[d["location"]]))
+    out, seen = [], set()
+    for d in docs:
+        location = d["location"]
+        if location in seen:
+            continue
+        seen.add(location)
+        page, _, anchor = location.partition("#")
+        text = texts[location]
+        if not anchor:
+            # the first section of this page, with its heading, starts where the intro ends
+            first_title, first = firsts.get(page, (None, None))
+            text = _intro(text, first)
+            if first_title and text.rstrip().endswith(first_title):
+                text = text.rstrip()[: -len(first_title)]
+            text = text.strip()
+        if text:
+            title = pages.get(page) or clean(d["title"]) or urlparse(base).netloc
+            out.append(Section(base + location, title, clean(d["title"]) if anchor else None, text, page))
+    return out
+
+
+class MkDocs(Connector):
+    name = "mkdocs connector"
+
+    def __init__(self, env: Mapping[str, str], *, transport: httpx.BaseTransport | None = None,
+                 sleep: Callable[[float], None] = time.sleep):
+        urls = names(env, "URLS")
+        if not urls:
+            raise SettingsError("URLS must name at least one site")
+        for url in urls:
+            if urlparse(url).scheme not in ("http", "https") or not urlparse(url).netloc:
+                raise SettingsError(f"URLS must be http(s) addresses, got {url!r}")
+        self.sites = list(dict.fromkeys(_base(u) for u in urls))
+        self.max_bytes = whole(env, "MAX_BYTES", 50_000_000)
+        self.weights = weights(env, "WEIGHTS")
+        self._web = Web("passageway-connector-mkdocs", max_bytes=self.max_bytes, transport=transport, sleep=sleep)
+
+    # -- the protocol ------------------------------------------------------------------------
+
+    def versions(self) -> dict[str, str]:
+        return {s.id: self._version(s) for site in self.sites for s in self._sections(site)}
+
+    def fetch(self, ids: list[str]) -> Iterator[dict]:
+        wanted = set(ids)
+        for site in self.sites:
+            if any(i.startswith(site) for i in wanted):
+                for section in self._sections(site):
+                    if section.id in wanted:
+                        yield self._item(section)
+
+    # -- the sites ---------------------------------------------------------------------------
+
+    def _sections(self, site: str) -> list[Section]:
+        url = site + INDEX
+        try:
+            text = self._web.get(url, required=True).text
+        except TooLarge:
+            raise SourceError(f"{url}: larger than MAX_BYTES ({self.max_bytes})") from None
+        try:
+            docs = json.loads(text)["docs"]
+            docs = [{"location": str(d["location"]), "title": str(d.get("title") or ""), "text": str(d.get("text") or "")}
+                    for d in docs]
+        except (ValueError, KeyError, TypeError):
+            raise SourceError(f"{url}: not an MkDocs search index") from None
+        return sections(site, docs)
+
+    def _weight(self, section: Section) -> float:
+        return weight_of(self.weights, section.where)
+
+    def _version(self, section: Section) -> str:
+        weight = self._weight(section)  # a changed weight is a changed item, so the index follows it
+        body = f"{section.page_title}\0{section.heading}\0{section.text}" + (f"\0{weight}" if weight != 1.0 else "")
+        return hashlib.sha256(body.encode()).hexdigest()[:16]
+
+    def _item(self, section: Section) -> dict:
+        item = {
+            "id": section.id,
+            "title": section.page_title,
+            "url": section.id,
+            "version": self._version(section),
+            "sections": [{"heading": section.heading, "text": section.text}],
+        }
+        if folders := [p for p in section.where.split("/") if p]:
+            item["path"] = folders
+        if (weight := self._weight(section)) != 1.0:
+            item["weight"] = weight
+        return item
+
+
+def main() -> None:
+    MkDocs.main()

passageway_connector_mkdocs-0.4.0/pyproject.toml

@@ -0,0 +1,22 @@
+[build-system]
+requires = ["hatchling==1.33.0"]
+build-backend = "hatchling.build"
+
+[project]
+name = "passageway-connector-mkdocs"
+version = "0.4.0"
+description = "passageway connector for websites built with MkDocs, read from their search index"
+readme = "README.md"
+requires-python = ">=3.12"
+license = "PolyForm-Noncommercial-1.0.0"
+dependencies = ["passageway-sdk[web,html]~=0.4.0"]
+
+[tool.hatch.build.targets.wheel]
+packages = ["passageway_connector_mkdocs"]
+
+[dependency-groups]
+dev = ["jsonschema>=4.23", "pytest>=8.3", "passageway"]
+
+[tool.uv.sources]
+passageway = { workspace = true }
+passageway-sdk = { workspace = true }

passageway_connector_mkdocs-0.4.0/tests/conftest.py

@@ -0,0 +1,61 @@
+import io
+import json
+
+import httpx
+import pytest
+
+from passageway_connector_mkdocs.connector import MkDocs
+
+SITE = "https://docs.example"
+
+# the shape Material for MkDocs writes: a page entry (its text before the first heading) and one entry per section, HTML in text
+INDEX = {
+    "config": {"lang": ["en"], "separator": "[\\s\\-]+"},
+    "docs": [
+        {"location": "", "title": "Home", "text": "<p>Welcome to the docs.</p>"},
+        {"location": "guide/setup/", "title": "Setup", "text": "<p>How to set it up.</p>"},
+        {"location": "guide/setup/#linux", "title": "Linux",
+         "text": "<p>Use the <code>apt</code> package.</p><pre><code>apt install example\n</code></pre>"},
+        {"location": "guide/setup/#windows", "title": "Windows", "text": ""},
+        {"location": "changelog/", "title": "Changelog", "text": "<p>Version 2 fixes the installer.</p>"},
+    ],
+}
+
+
+class Sites:
+    """A fake web: url -> JSON body; and a way to run the connector against it in-process."""
+
+    def __init__(self):
+        self.index = INDEX
+        self.pages = {f"{SITE}/search/search_index.json": INDEX}
+        self.requests = []
+        self.status = 200
+
+    def handler(self, request: httpx.Request) -> httpx.Response:
+        self.requests.append(str(request.url))
+        body = self.pages.get(str(request.url))
+        if body is None:
+            return httpx.Response(404)
+        return httpx.Response(self.status, text=body if isinstance(body, str) else json.dumps(body))
+
+    def call(self, mode, stdin="", **settings):
+        out, err = io.StringIO(), io.StringIO()
+        env = {"URLS": SITE, **settings}
+        code = MkDocs.run(["mkdocs", mode], env, io.StringIO(stdin), out, err, transport=httpx.MockTransport(self.handler),
+                          sleep=lambda s: None)
+        return code, [json.loads(line) for line in out.getvalue().splitlines()], err.getvalue()
+
+    def versions(self, **settings):
+        code, records, err = self.call("list", **settings)
+        assert code == 0, err
+        return {r["id"]: r["version"] for r in records}
+
+    def fetch(self, *ids, **settings):
+        code, records, err = self.call("fetch", "".join(json.dumps({"id": i}) + "\n" for i in ids), **settings)
+        assert code == 0, err
+        return {r["id"]: r for r in records}
+
+
+@pytest.fixture
+def sites():
+    return Sites()

passageway_connector_mkdocs-0.4.0/tests/test_mkdocs_connector.py

@@ -0,0 +1,79 @@
+import json
+
+SITE = "https://docs.example"  # the site of the fixture
+
+
+def test_every_section_with_text_is_an_item_whose_address_opens_at_its_heading(sites):
+    assert set(sites.versions()) == {f"{SITE}/", f"{SITE}/guide/setup/", f"{SITE}/guide/setup/#linux", f"{SITE}/changelog/"}
+
+
+def test_a_section_carries_its_page_title_and_heading_and_the_folders_as_breadcrumb(sites):
+    item = sites.fetch(f"{SITE}/guide/setup/#linux")[f"{SITE}/guide/setup/#linux"]
+    assert item["title"] == "Setup"
+    assert item["url"] == f"{SITE}/guide/setup/#linux"
+    assert item["path"] == ["guide", "setup"]
+    assert item["sections"] == [{"heading": "Linux", "text": "Use the apt package.\n\n```\napt install example\n```"}]
+
+
+def test_the_text_before_the_first_heading_is_the_page_item(sites):
+    item = sites.fetch(f"{SITE}/guide/setup/")[f"{SITE}/guide/setup/"]
+    assert item["title"] == "Setup" and item["sections"] == [{"heading": None, "text": "How to set it up."}]
+
+
+def test_plain_mkdocs_repeats_the_whole_page_in_its_page_entry_and_only_the_intro_is_kept(sites):
+    # mkdocs' own search plugin: tag-stripped text, and the page entry holds the whole page, headings included
+    sites.pages[f"{SITE}/search/search_index.json"] = {"docs": [
+        {"location": "setup/", "title": "Setup", "text": "How to set it up. Linux Use the package. Windows Use the installer."},
+        {"location": "setup/#linux", "title": "Linux", "text": "Use the package."},
+        {"location": "setup/#windows", "title": "Windows", "text": "Use the installer."},
+    ]}
+    items = sites.fetch(f"{SITE}/setup/", f"{SITE}/setup/#windows")
+    assert items[f"{SITE}/setup/"]["sections"] == [{"heading": None, "text": "How to set it up."}]
+    assert items[f"{SITE}/setup/#windows"]["sections"] == [{"heading": "Windows", "text": "Use the installer."}]
+
+
+def test_a_site_under_a_path_and_an_index_address_both_work(sites):
+    sites.pages["https://user.example/project/search/search_index.json"] = sites.index
+    assert f"https://user.example/project/guide/setup/#linux" in sites.versions(URLS="https://user.example/project")
+    assert f"https://user.example/project/changelog/" in sites.versions(URLS="https://user.example/project/search/search_index.json")
+
+
+def test_an_edit_changes_only_that_sections_version(sites):
+    before = sites.versions()
+    index = json.loads(json.dumps(sites.index))
+    index["docs"][2]["text"] = "<p>Use the <code>dnf</code> package.</p>"
+    sites.pages[f"{SITE}/search/search_index.json"] = index
+    after = sites.versions()
+    assert [i for i in before if before[i] != after[i]] == [f"{SITE}/guide/setup/#linux"]
+
+
+def test_a_weight_per_path_prefix_lowers_a_part_of_a_site_and_changes_its_version(sites):
+    before = sites.versions()
+    items = sites.fetch(f"{SITE}/changelog/", f"{SITE}/guide/setup/", WEIGHTS="changelog=0.5")
+    assert items[f"{SITE}/changelog/"]["weight"] == 0.5 and "weight" not in items[f"{SITE}/guide/setup/"]
+    assert sites.versions(WEIGHTS="changelog=0.5")[f"{SITE}/changelog/"] != before[f"{SITE}/changelog/"]
+
+
+def test_a_site_without_a_search_index_fails_the_listing_and_names_it(sites):
+    code, records, err = sites.call("list", URLS="https://other.example")
+    assert code == 1 and records == [] and "https://other.example/search/search_index.json" in err
+
+
+def test_a_server_that_is_down_fails_the_listing(sites):
+    sites.status = 503
+    assert sites.call("list")[0] == 1
+
+
+def test_an_index_that_is_not_an_mkdocs_index_fails_the_listing(sites):
+    sites.pages[f"{SITE}/search/search_index.json"] = "<html>not json</html>"
+    code, _, err = sites.call("list")
+    assert code == 1 and "search index" in err
+
+
+def test_fetch_skips_ids_that_are_gone_or_belong_to_no_site(sites):
+    assert sites.fetch(f"{SITE}/gone/", "https://elsewhere.example/x/", "not an address") == {}
+
+
+def test_settings_are_checked(sites):
+    for bad in ({"URLS": ""}, {"URLS": "ftp://docs.example"}, {"WEIGHTS": "changelog"}, {"MAX_BYTES": "0"}):
+        assert sites.call("list", **bad)[0] == 2, bad

passageway_connector_mkdocs-0.4.0/tests/test_mkdocs_contract.py

@@ -0,0 +1,96 @@
+import functools
+import http.server
+import json
+import os
+import re
+import subprocess
+import sys
+import threading
+from pathlib import Path
+
+import jsonschema
+import pytest
+
+from passageway.protocol import Connector
+from passageway.testing.contract import check_connector
+
+README = Path(__file__).resolve().parents[1] / "README.md"
+SCHEMA = Path(__file__).resolve().parents[3] / "docs" / "reference" / "connector-protocol.schema.json"
+INDEX = {"docs": [
+    {"location": "", "title": "Home", "text": "<p>Welcome.</p>"},
+    {"location": "setup/", "title": "Setup", "text": "<p>How to set it up.</p>"},
+    {"location": "setup/#linux", "title": "Linux", "text": "<p>Use the package manager.</p>"},
+]}
+
+
+class Site(str):
+    def __new__(cls, url, root):
+        site = super().__new__(cls, url)
+        site.root = root
+        return site
+
+
+@pytest.fixture
+def site(tmp_path):
+    """A real web server on localhost serving a built site's search index, so the connector runs as a real subprocess."""
+    (tmp_path / "search").mkdir()
+    (tmp_path / "search" / "search_index.json").write_text(json.dumps(INDEX))
+    handler = functools.partial(http.server.SimpleHTTPRequestHandler, directory=str(tmp_path))
+    handler.log_message = lambda *args: None
+    server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), handler)
+    threading.Thread(target=server.serve_forever, daemon=True).start()
+    yield Site(f"http://127.0.0.1:{server.server_port}", tmp_path)
+    server.shutdown()
+
+
+def command():
+    return [sys.executable, "-m", "passageway_connector_mkdocs"]
+
+
+def test_the_connector_passes_the_shared_contract_as_a_real_subprocess(site):
+    connector = Connector("mkdocs", command(), env={"URLS": site})
+    check_connector(connector)
+    items = {i.id: i for i in connector.fetch([f"{site}/setup/#linux"])}
+    assert items[f"{site}/setup/#linux"].title == "Setup"
+
+
+def test_the_output_validates_against_the_published_json_schema(site):
+    schema = json.loads(SCHEMA.read_text())
+
+    def valid(name, record):
+        return jsonschema.Draft202012Validator({"$ref": f"#/$defs/{name}", "$defs": schema["$defs"]}).is_valid(record)
+
+    def lines(mode, stdin=""):
+        out = subprocess.run(command() + [mode], env={**os.environ, "URLS": site}, input=stdin,
+                             capture_output=True, text=True, check=True).stdout
+        return [json.loads(line) for line in out.splitlines()]
+
+    assert lines("check") and all(valid("check", r) for r in lines("check"))
+    assert lines("list") and all(valid("listRecord", r) for r in lines("list"))
+    items = lines("fetch", f'{{"id": "{site}/setup/#linux"}}\n')
+    assert len(items) == 1 and valid("item", items[0])
+
+
+def test_every_setting_is_in_the_readme():
+    code = (Path(__file__).resolve().parents[1] / "passageway_connector_mkdocs" / "connector.py").read_text()
+    names = sorted(set(re.findall(r'(?:env\.get|names|whole|flag|seconds|weights)\(\s*(?:env,\s*)?"([A-Z_]+)"', code)))
+    assert len(names) >= 3, names  # the pattern must actually find the settings
+    readme = README.read_text()
+    for name in names:
+        assert f"`{name}`" in readme, f"{name} is read by the mkdocs connector but missing from its README"
+
+
+def test_the_example_in_the_readme_is_what_the_connector_really_produces(site):
+    readme = README.read_text()
+    index = json.loads(re.search(r"```json\n(.*?)```", readme, re.S).group(1))
+    shown = json.loads(re.findall(r"```json\n(.*?)```", readme, re.S)[1])
+    (site.root / "search" / "search_index.json").write_text(json.dumps(index))
+    location = re.search(r'"id": "https://docs\.example\.org/(.*?)"', readme).group(1)
+    out = subprocess.run(command() + ["fetch"], env={**os.environ, "URLS": site}, input=f'{{"id": "{site}/{location}"}}\n',
+                         capture_output=True, text=True, check=True).stdout
+    real = json.loads(out)
+    for key, value in shown.items():
+        if key in ("id", "url"):
+            value = value.replace("https://docs.example.org", site)
+        if key != "version":
+            assert real[key] == value, f"the README shows {key} = {value!r} but the connector prints {real[key]!r}"